novox/hq
9
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Design 42 zfs-owns-zfs: answer review round 1 against ADR 0679 version 11
mesh/delivery merged: source.main moved to novox/hq@b9cd0bc492ac
Details
mesh/check being carried to 224348ff4d69 on b9cd0bc492ac | from 9ebf6e58f0d7

Browse Source 
Cites the record's final clauses, drops the catalog check and every
write of the package's files, and writes the lead's answers as design
lines: a run that never waits on a scrub, waits ended by events,
ZFS snapshots kept in the ledger before any removal, hand acts as unit
instances, subjects per kind, the follower's findings, the readers of
the new manifest words and one node-engine integration test.
This commit is contained in:
jschoubben
2026-11-12 23:20:51 +01:00
parent f26c8afc20
commit 224348ff4d
3 changed files with 338 additions and 268 deletions
Download Patch File Download Diff File Expand all files Collapse all files
00-META/glossary.md
+1 -1
View File
@@ -1023,7 +1023,7 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283, 0564, 0668.
- **ZFS snapshot** — a read-only point in time of one ZFS dataset on a machine, kept inside that dataset's pool, so
it guards against a mistake and never against the pool's loss; taken by a person with `zfs_snapshot`, or by the zfs
module's scheduled run, which names its own `mesh-<kind>-<time>`, marks them and removes only those past their keep
count (ADR 0679 §7; design 42 zfs-owns-zfs §18, §19). Never bare, beside a ledger's snapshot and the stream snapshot
count (ADR 0679 §8; design 42 zfs-owns-zfs §21, §22). Never bare, beside a ledger's snapshot and the stream snapshot
(see [Homonyms](#homonyms)); not a backup or a restore point, which are kept elsewhere.
*Not:* ~~pool snapshot~~, ~~dataset snapshot~~
- **disk readings** — the size of each data item a node's modules declare, measured by its `measure` and kept
03-DESIGN/01-to-be/42-milestones/zfs-owns-zfs.md
+277 -226
View File
@@ -3,270 +3,321 @@ layer: to-be
design: 42
milestone: zfs-owns-zfs
updated: 2026-11-12
code: [mesh-catalog modules/zfs, mesh-catalog checks, mesh-host internal/apply]
code: [mesh-sdk, mesh-delivery, mesh-host internal/apply, mesh-catalog modules/zfs]
decisions:
- "ADR 0679"
- "ADR 0051"
- "ADR 0117"
- "ADR 0118"
- "ADR 0269"
- "ADR 0303"
- "ADR 0348"
- "ADR 0363"
- "ADR 0508"
- "ADR 0564"
- "ADR 0593"
- "ADR 0595"
- "ADR 0609"
- "ADR 0614"
---
# Design 42, milestone zfs-owns-zfs — the zfs module owns a machine's ZFS, and changes nothing that is not set
# Design 42, milestone zfs-owns-zfs — the zfs module owns a node's ZFS, and changes nothing that is not set
**What it builds.** On a machine that holds a ZFS pool (today the home server), the zfs module owns ZFS's packages,
files, units and, for one pool, its properties, cache and log devices, its datasets' properties and a schedule of
ZFS snapshots; each is changed only once its config value is set (ADR 0679). The purpose and the whole shape are the
main file's, [*The zfs module*](../42-the-machines-modules-in-order.md#the-zfs-module). This part holds every line
the milestone builds, cited as "design 42 zfs-owns-zfs §n".
**What it builds.** On a node that holds a ZFS pool (today the home server), the zfs module owns ZFS's packages, its
kernel module's options file, its host id, its own folder for ZFS's event daemon (ZED), its units and, for one pool,
its cache and log devices, its properties, its datasets' properties and a schedule of ZFS snapshots; each is changed
only once its config value is set and not empty (ADR 0679). The purpose and the whole shape are the main file's,
[*The zfs module*](../42-the-machines-modules-in-order.md#the-zfs-module). This part holds every line the milestone
builds, cited as "design 42 zfs-owns-zfs §n".
**Where.** Built under issue 1979. Three repositories, one delivery group with one check: `mesh-host` (the
node-engine: §1, §2, §3, its three subtasks of issue 1979), mesh-catalog's checks (§4) and mesh-catalog's
`modules/zfs` (§5 to §26). **Ordering edge:** the zfs module's new version needs the node-engine of §1, §2 and §3
on its machine (a node-engine capability edge the delivery plan derives, §5); §4 has no edge, it gates the catalog's
own check. Nothing waits on the node-engine's cron-to-calendar fix: the scheduled run's schedule is written in a form
today's node-engine already turns into a valid calendar (§18).
**Where.** Built under issue 1979. Four repositories, one delivery group with one check: mesh-sdk, mesh-delivery
(the manifest validator) and `mesh-host` (the node-engine) for §1 to §5, issue 1979's three node-engine subtasks
and the readers of their words; mesh-catalog's `modules/zfs` for §6 to §31. **Ordering edges** (ADR 0679 §16): the
readers in §4's order, then the zfs module's new version only on a node whose node-engine states the three
capabilities of §4 (a reader's capability the delivery plan derives, ADR 0614 §4).
## The prerequisites
## The prerequisites (ADR 0679 §16)
1. **The node-engine's apply of a module stops at its first failed resource.** When one resource of a module fails,
the node-engine applies none of that module's later resources in the declared order; the report names the
failed resource with its error and each one left unapplied, and the module's phase is `failed`. Resources already
applied stay as applied. Other modules on the node go on as today. Today the apply goes on, and stopped the
distribution's scrub timer after the module's own timer failed (ADR 0679 §14). Checked by a unit test: a module of
three resources whose second fails leaves the third untouched and reports it as left.
2. **The node-engine declares a resource only while a config value holds** (lead's answer, 2026-11-12, to this
part's design question: build it in the node-engine, design-it-as-it-should-be; ADR 0679 §3, §8). A resource of
any type may carry the manifest field `if-config`: a list of conditions, each `<key>` (the module's config value is
set and is neither `false`, `off`, `none` nor an empty list) or `<key>=<value>` (the value equals it; for a list
value, the list holds that entry). The resource is declared while any condition holds, and not at all while none
does. A resource that leaves the declaration this way is removed as any resource that leaves it: a file written
over gets its original back (ADR 0118), a unit gets its found state back (ADR 0269). The controller's catalog check
refuses an `if-config` key the module does not declare under `config`. Checked by the node-engine's table test (a
value unset, `off`, set; a list with and without the entry; a value going from set to unset removes the resource
and puts the original back) and the catalog check's own test.
3. **A template writes one line per entry** (lead's answer, 2026-11-12, as §2; ADR 0679 §4). A template file may
hold `${config-lines:<key>:<prefix>}`: for a list value one line per entry, `<prefix><entry>`, in the list's order;
1. **The node-engine's apply of a module stops at its first failed resource** (ADR 0679 §16 (a)). When one resource
of a module fails, the node-engine applies none of that module's later resources in the declared order; its apply
report names the failed resource with its error and each one left unapplied, and the module's phase is `failed`.
Resources already applied stay as applied; other modules on the node go on as today. Today the apply goes on, and
stopped the distribution's scrub timer after the module's own timer failed. Checked by a unit test (a module of
three resources whose second fails leaves the third untouched and reports it as left) and by §5.
2. **A resource is declared only while a config value holds** (ADR 0679 §16 (b), §3). A resource of any type may
carry the manifest field `if-config`: a list of conditions, each `<key>` (the module's config value is set and is
neither `false`, `off`, `none` nor an empty list) or `<key>=<value>` (the value equals it; for a list value, the
list holds that entry). The resource is declared while any condition holds, and not at all while none does. A
resource that leaves the declaration this way is removed as any resource that leaves it: a file the node-engine
wrote whole is deleted, a unit gets its found state back (ADR 0269 §2). The manifest validator refuses an
`if-config` key the module does not declare under `config`. Checked by the node-engine's table test (a value unset,
empty, `off`, set; a list with and without the entry; a value going from set to unset removes the resource) and
the validator's own test.
3. **A template writes one line per entry** (ADR 0679 §16 (c), §4). A template file may hold
`${config-lines:<key>:<prefix>}`: for a list value one line per entry, `<prefix><entry>`, in the list's order;
for a set scalar value one line `<prefix><value>`; nothing for an unset value or an empty list. With a fourth part,
`${config-lines:<key>:<prefix>:mib-bytes}`, a number given in MiB is written in bytes. The prefix is literal text
with no `}`, `$` or newline, refused by the catalog check; an entry holding a newline is refused before the file is
written, and the file is left as it was. Checked by the node-engine's table test (a list of two, an empty list, an
unset and a set scalar, `mib-bytes`, a newline in an entry) and the catalog check's own test.
4. **The catalog refuses a second writer of one file.** mesh-catalog's checks refuse a catalog in which two
modules' manifests declare a `file` resource at one path, naming both modules and the path (ADR 0679 §14; ADR 0051
§9, rival ownership). A file only read is not a writer. Checked by the check's own test over two manifests that
declare one path (refused) and two that declare different paths (passed); it runs in the catalog's check on
every pull request.
5. **The order on the machine.** The zfs module's new version declares its own timer resource (the scheduled run of
§18) before the resources that stop the distribution's scrub timers, so by §1 those are stopped only once the
module's own timer is active (ADR 0679 §14). The delivery plan places the node-engine of §1, §2 and §3 on the home server before the
module (the capability edge above). Checked by the module's manifest test, which pins the resource order, and by
acceptance test 3 of the main file.
with no `}`, `$` or newline, refused by the manifest validator; an entry holding a newline is refused before the
file is written, and the file is left as it was. Checked by the node-engine's table test (a list of two, an empty
list, an unset and a set scalar, `mib-bytes`, a newline in an entry) and the validator's own test.
4. **Every reader of the new words, in order** (design-it-as-it-should-be; ADR 0614 §4). mesh-sdk first: the
manifest types gain `if-config` on every resource and the template form `config-lines`, and the capability names
`if-config`, `config-lines` and `apply-stops-at-failure`. Then the manifest validator in mesh-delivery's indexer:
it reads both words and refuses them as §2 and §3 say. Then the node-engine: it builds §1 to §3 and states the
three capabilities. A manifest using `if-config` or `config-lines` reaches a node only through a node-engine that
states both; the delivery plan derives the edge from the stated capability, never from a fingerprint. Checked by
each repository's unit tests and by the delivery plan naming the edge.
5. **One integration test of the node-engine** (how-work-is-done: a change to how modules run carries one). It
starts the node-engine whole against the real bus with a module of a resource under `if-config`, a template with
`config-lines`, and a resource that fails: the conditioned resource appears when its value is set and goes when it
is unset, the template's lines follow the list, and the failing resource stops the module's apply with the later
resource left. It is the only integration test this milestone adds.
6. **The order on the node** (ADR 0679 §16, §9). The zfs module declares its scheduled run's resource (§20) before
the resources that stop the distribution's scrub timers, so by §1 those are stopped only once the run's timer is
active. The run's `schedule` is the list form `0,15,30,45 * * * *`, which today's node-engine writes as
`OnCalendar=*-*-* *:0,15,30,45:00`, so nothing waits on another fix. Checked by the module's manifest test (the
resource order and the schedule string) and by acceptance test 3 of the main file.
## What the module owns
6. **The packages.** The manifest declares `zfs-utils` and `zfs-dkms` present (ADR 0679 §1; ADR 0303 §2: the
node-engine installs a missing one, upgrades none, removes none). `zfs-utils-debug` is not declared and is left as
found. Fate: the README's "It installs nothing" goes; it says the two packages and why the debug package is not
the module's. Read by `pacman_packages` as declared by `zfs`. Checked by the manifest test.
7. **One pool, renamed `pool`.** The config value `scrub-pool` becomes `pool` (ADR 0679 §1): type string, trusted,
required, same description but for the rename. It is moved by mesh-config's rename window (mesh-config answers both
names meanwhile; its seat verb `renames` says the window), and every template and the module's code read `pool`.
One pool per assignment; the manifest has no per-pool form and none is added. The module reads, never writes,
`/etc/zfs/zpool.cache` and `/etc/zfs/zfs-list.cache`. Checked by the manifest test (no `scrub-pool` left) and the
rename's live read in `zfs_config`.
8. **What is not its own.** `sharenfs` and `sharesmb` are on no list of §14; a setting naming them is refused
(§9), and `zfs_config` never reports either as drift (ADR 0679 §2). The pool's files are the operator's (ADR
0051). Checked by §25's refused-forms test.
7. **The packages** (ADR 0679 §1; ADR 0303 §2). The manifest declares `zfs-utils` and `zfs-dkms` present: the
node-engine installs a missing one, upgrades none, removes none. `zfs-utils-debug` is not declared and is left as
found. Fate: the README's "It installs nothing" goes; it says the two packages and why the debug package is not the
module's. Read by `pacman_packages` as declared by `zfs`. Checked by the manifest test.
8. **One pool, renamed `pool`** (ADR 0679 §1; ADR 0609 §4). The config value `scrub-pool` becomes `pool` with
`renamed-from: scrub-pool`: type string, trusted, required. mesh-config moves the value under the new name and
answers both names during the rename window (design 32, *A config value renamed*); every template and the module's
code read `pool`. **Rollback:** a rolled-back version leaves pool and dataset properties and devices as the last
apply set them (ADR 0679 §15); once the rename window has closed, going back to the version with `scrub-pool` and
`zfs-mesh-scrub` is a new delivery of that version, with its own rename back, never a rollback. Checked by the
manifest test (no `scrub-pool` but in `renamed-from`) and `zfs_config` showing `pool` on the home server.
9. **The module's config file** (ADR 0679 §9; ADR 0348 §5). `/etc/zfs-tools/config.json`, root's, mode 0644, written
by the node-engine from a template holding every config value of the module, lists and objects rendered as ADR 0348
§5 says; it replaces `/etc/zfs-tools/scrub.conf`, which the node-engine removes as a resource that left. The apply
unit (§18) has `restart-on` this file: its change is the path by which mesh-config's `config-changed` for the
module's values reaches the apply. The scheduled run (§20) reads it at each run. `/var/lib/zfs-scrub` stays and is
read by the run as today. Checked by the manifest test.
10. **What it never writes** (ADR 0679 §1, §2). The package's own `/etc/default/zfs` and `/etc/zfs/zed.d`, with its
`zed.rc` and its links, are left as the package ships them, and nothing of them is given back on leaving; ZFS's own
`/etc/zfs/zpool.cache` and `/etc/zfs/zfs-list.cache` are read only. `zfs_config` shows each, and a `.pacnew`
beside `/etc/default/zfs` or the package's `zed.rc` is a warning (§29). A dataset's `sharenfs` is the seat
`node-nfs-server`'s, `sharesmb` the SMB module's (issue 613): neither is on a list of §16, a value naming either
is refused (§11), and `zfs_config` never reports either as differing. The pool's files are the operator's (ADR
0051). Checked by the call-site test (§19) and §31.
## The config values and their forms
9. **Every new value is "not managed" until set, and has a fixed form** (ADR 0679 §3, §4). Each value below has no
default in the manifest; absent means not managed: the module writes nothing, converges nothing and reports no
drift for it. A value of another form is refused by the module's own parse, before any file is written or command
run, emitting `config.refused` (§21). Trusted values (every list and object below, and `hostid`) are set on the
operator's Approve (ADR 0348 §4, §8). The module never reads a value from the machine into config.
- `arc-max`, `arc-min` (number): whole MiB, `arc-min` ≤ `arc-max`, `arc-max` at least 64 and below the machine's
memory read from `/proc/meminfo`. ZFS's read cache in memory (the ARC) is bounded by them.
- `module-options` (list, trusted): entries `name=digits`, the name from the module's fixed list `l2arc_write_max`,
`l2arc_write_boost`, `l2arc_headroom`, `l2arc_noprefetch`, `l2arc_rebuild_enabled`, `zfs_txg_timeout`,
`zfs_dirty_data_max`, `zfs_scan_vdev_limit`, `zfs_scrub_min_time_ms`, `zfs_resilver_min_time_ms`,
`zfs_arc_sys_free`; `zfs_arc_max` and `zfs_arc_min` are refused here, as `arc-max` and `arc-min` hold them. Reason
for the list: the ARC, L2ARC, transaction group and scrub parameters ADR 0679 §4 names, and nothing else.
- `zed` (string): `on` or `off`. `zed-handlers` (list, trusted): from `all-syslog` (the journal handler),
`resilver_finish-start-scrub` (scrub after resilver) and `history_event-zfs-list-cacher` (keeps the list cache
the mount reads); no notify handler is on the list (ADR 0679 §10).
- `trim-schedule` (string): `none`, `weekly` or `monthly`; `trim-day` and `trim-time` with `scrub-day`'s and
`scrub-time`'s forms.
- `cache-devices`, `log-devices` (lists, trusted): names under `/dev/disk/by-id`, letters, digits, `-`, `_`, `.`
and `:` only. `log-mirror` (boolean): when true, `log-devices` holds exactly two names, added as one mirror;
otherwise each is a log device of its own. Reason: the record names log devices by stable name; a mirror is the
form the pool's log has today.
- `pool-properties` (object, trusted): `autotrim`, `autoexpand`, `autoreplace` (`on`, `off`, `default`) and
`failmode` (`wait`, `continue`, `panic`, `default`).
- `dataset-properties` (object, trusted): a dataset of the pool (the pool's name or one beginning `<pool>/`) to
properties of §14's list with their forms.
- `snapshots` (object, trusted): `datasets` (list of datasets of the pool; absent means every filesystem of the
pool), and keep counts `hourly` (0 to 48), `daily` (0 to 62), `weekly` (0 to 104), `monthly` (0 to 120); absent
or all zero means none.
- `hostid` (string, trusted): `generate`, written once (§13).
Checked by §25's refused-forms tests, one table per value.
11. **Not managed until set and not empty; each of a fixed form** (ADR 0679 §3, §4). No new value has a default in
the manifest. An absent value and an empty list both mean not managed: the module writes nothing, converges
nothing and reports no difference for it, so nothing is ever removed by a blank value. A value of another form is
refused by the module's own parse before any file is written or command run, emitting `config.refused` (§25).
Trusted values (every list and object below, `log-mirror` and `hostid`) are set only on the operator's Approve (ADR
0348 §4, §8). The module never reads a value from the node into config.
- `arc-max`, `arc-min` (number): whole MiB, `arc-min` ≤ `arc-max`, `arc-max` at least 64 and below the node's
memory read from `/proc/meminfo`. ZFS's read cache in memory (the ARC) is bounded by them.
- `module-options` (list, trusted): entries `name=digits`, the name from the module's fixed list `l2arc_write_max`,
`l2arc_write_boost`, `l2arc_headroom`, `l2arc_noprefetch`, `l2arc_rebuild_enabled`, `zfs_txg_timeout`,
`zfs_dirty_data_max`, `zfs_scan_vdev_limit`, `zfs_scrub_min_time_ms`, `zfs_resilver_min_time_ms`,
`zfs_arc_sys_free`; `zfs_arc_max` and `zfs_arc_min` are refused here, as `arc-max` and `arc-min` hold them. Reason:
the ARC, L2ARC, transaction group and scrub parameters ADR 0679 §4 names, and nothing else.
- `hostid` (string, trusted): eight hexadecimal digits, not all zero (§15).
- `zed` (string): `on` or `off`. `zed-handlers` (list, trusted): from `all-syslog` (the journal handler) and
`resilver_finish-start-scrub` (scrub after resilver), the two ADR 0679 §12 names; no notify handler.
- `trim-schedule` (string): `none`, `weekly` or `monthly`; `trim-day` and `trim-time` with `scrub-day`'s and
`scrub-time`'s forms.
- `cache-devices`, `log-devices` (lists, trusted): names under `/dev/disk/by-id`, letters, digits, `-`, `_`, `.`
and `:` only. `log-mirror` (boolean, trusted): when true, `log-devices` holds exactly two names, added as one
mirror; otherwise each is a log device of its own. Reason: the log's form today is a mirror.
- `pool-properties` (object, trusted): `autotrim`, `autoexpand`, `autoreplace` (`on`, `off`, `default`) and
`failmode` (`wait`, `continue`, `panic`, `default`).
- `dataset-properties` (object, trusted): a dataset of the pool (the pool's name or one beginning `<pool>/`) to
properties of §16's list with their forms.
- `snapshots` (object, trusted): `none`, or keep counts `hourly`, `daily`, `weekly` and `monthly`, each a whole
number from 0 to 168, a kind with 0 not taken; for every filesystem of the pool. The bound 168 (a week of hours)
is the design's, so the ledger's cap (§26) stays bounded.
Checked by §31's refused-forms tests, one table per value, the empty list among them.
## Files, units and the event daemon
10. **The node-engine places the module's files** (ADR 0679 §1, §4, §8), by §2 and §3.
`/etc/modprobe.d/zfs.conf` carries `if-config: [arc-max, arc-min, module-options]` and holds, by §3, a header
comment, then `options zfs zfs_arc_max=` and `options zfs zfs_arc_min=` with `arc-max` and `arc-min` in bytes
(`mib-bytes`), then `options zfs ` with each `module-options` entry; with none of the three set the file is not
placed. The module's own `zed.rc` is placed in its folder (§12), not over the package's: it holds the package's
variables but every `ZED_NOTIFY_*` and mail line, and `ZED_SCRUB_AFTER_RESILVER=1`, which only the handler
`resilver_finish-start-scrub` reads, so it acts only where that handler is placed. The package's
`/etc/zfs/zed.d/zed.rc` and `/etc/default/zfs` are left as the package wrote them: no value of ADR 0679 changes
either, so writing them would only rewrite what ADR 0679 §3 leaves alone; `zfs_config` shows both as the
package's, and a `.pacnew` beside either as a warning. Every line is from a parsed value, never from free text, and
a test refuses a newline, a quote, `$`, a backtick and a semicolon in every field (§25). Checked by the manifest
test and §25.
11. **The units.** The node-engine declares `zfs-import-cache.service`, `zfs-mount.service`, `zfs.target` and
`zfs-import.target` running and enabled at boot, as they are on the home server today, so declaring them changes
nothing; `zfs-import-scan.service`, `zfs-share.service` and `zfs-volume-wait.service` are declared with no state
(ADR 0117), owned and left as found. `zfs-zed.service` is declared twice by §2: running and enabled with
`if-config: [zed=on]`, stopped and disabled with `if-config: [zed=off]`, and so not declared, and left as found,
while `zed` is not managed. The distribution's `zfs-trim-weekly@<pool>.timer` and `zfs-trim-monthly@<pool>.timer`
are stopped and disabled with `if-config: [trim-schedule]` (the module's run trims, §18). The distribution's scrub
timers are stopped and disabled after the module's own (§5). Read by `<node>/node-service-manager.status`.
Checked by the manifest test (each unit, its state and its condition).
12. **ZED's handlers without links** (ADR 0679 §11). The module's folder `/etc/zfs/mesh-zed.d` (root's, 0755), its
`zed.rc` (§10) and a drop-in for `zfs-zed.service` whose `ExecStart` adds `-d /etc/zfs/mesh-zed.d` carry
`if-config: [zed=on]`. Each handler is a file of its own in that folder, a copy of the package's script by its
content kept in the module's files archive, mode 0755, with `if-config: [zed-handlers=<its name>]`, so only the
handlers set are placed. The package's links in `/etc/zfs/zed.d` stay as the package left them and are not read
while the drop-in holds. The mesh creates no symlinks. The node-engine restarts `zfs-zed.service` on a change of
the drop-in, a handler or `zed.rc` (`restart-on`). Checked by the manifest test (no link resource, each handler's
condition) and a live `<node>/node-files.list` of the folder.
13. **The machine's ZFS id.** `/etc/hostid` is written only when `hostid` is `generate` and the file is absent, by the apply
of §16 with `zgenhostid`; never rewritten. Before the Approve, the value's description and `zfs_config` say that
the pool must then be imported once by hand after the next reboot to check it imports under the new id; the
module never reboots. Emits `config.applied` with target `file`. Checked by a unit test (absent file: one
`zgenhostid`; present file: none).
12. **The options file** (ADR 0679 §1, §4). `/etc/modprobe.d/zfs.conf`, root's, 0644, carries `if-config: [arc-max,
arc-min, module-options]` (§2) and holds a header comment, then by §3 `options zfs zfs_arc_max=` and `options zfs
zfs_arc_min=` with the values in bytes (`mib-bytes`), then `options zfs ` with each `module-options` entry. With
none of the three set it is not placed. Checked by the manifest test and §31's template test.
13. **The units** (ADR 0679 §1, §15; ADR 0117). Declared running and enabled at boot: `zfs-import-cache.service`,
`zfs-mount.service` and `zfs.target`, as they run on the home server today, so declaring them changes nothing.
Declared with no state, owned and left as found: `zfs-import-scan.service`, `zfs-share.service`,
`zfs-volume-wait.service` and `zfs-import.target` (reason: ADR 0679 §15 keeps only the first three running when the
module leaves, so no other ZFS unit is one the mesh would stop then). `zfs-zed.service`: running and enabled with
`if-config: [zed=on]`, stopped and disabled with `if-config: [zed=off]`, left as found while `zed` is not managed.
The distribution's `zfs-trim-weekly@<pool>.timer` and `zfs-trim-monthly@<pool>.timer` are stopped and disabled
with `if-config: [trim-schedule]` (the run trims, §20); its scrub timers are stopped and disabled after the run's
timer (§6). Read by `<node>/node-service-manager.status`. Checked by the manifest test (each unit, its state and
its condition).
14. **ZED runs from the module's own folder, without links** (ADR 0679 §13, §12). The folder `/etc/zfs/mesh-zed.d`
(root's, 0755), its `zed.rc` and a drop-in for `zfs-zed.service` whose `ExecStart` adds `-d /etc/zfs/mesh-zed.d`
carry `if-config: [zed=on]`. The module's `zed.rc` holds the package's variables but every `ZED_NOTIFY_*` and mail
line, and `ZED_SCRUB_AFTER_RESILVER=1`, which only the scrub-after-resilver handler reads. Each handler is a file
of its own in the folder, a copy of the package's script by its content kept in the module's files archive, mode
0755, with `if-config: [zed-handlers=<its name>]`, so only the handlers set are placed. The package's
`/etc/zfs/zed.d` is not read while the drop-in holds. The mesh creates no symlinks. The node-engine restarts
`zfs-zed.service` on a change of the drop-in, a handler or `zed.rc` (`restart-on`). Checked by the manifest test (no
link resource, each handler's condition).
15. **The node's ZFS id** (ADR 0679 §1, §3, §4). `/etc/hostid` holds four bytes, which no template writes, so the apply
(§18) writes it with ZFS's own writer, `zgenhostid -f <value>`, only while `hostid` is set, and only when the pool's
label (`zdb -C <pool>`, its `hostid`) is 0 or equals the value: then the next boot imports without `zpool import
-f`. Otherwise it writes nothing and emits `config.refused` naming why ("the pool's import would need -f"). Before
the Approve, the value's description and `zfs_config` say the change needs a check of the pool's import at the next
boot; the module never reboots. After the reboot, `zfs_pools` shows the pool imported. Emits `config.applied` with
target `file`. Checked by §31 (unset: no `zgenhostid`; a label of another id: refused; 0 or equal: one run).
## The pool
14. **Pool and dataset properties** (ADR 0679 §5). For each set pool property the apply runs
`zpool set <property>=<value> <pool>`, and for `default` `zpool set <property>=<ZFS's default> <pool>`. For each
set dataset property it runs `zfs set <property>=<value> <dataset>`, and for `default`
`zfs inherit <property> <dataset>`, so its source stays default. The dataset list is `compression` (`off`, `lz4`,
`zstd`, `zstd-1` to `zstd-19`, `gzip-1` to `gzip-9`, `zle`), `atime` and `relatime` (`on`, `off`), `recordsize`
(a power of two from `512` to `16M`, with `K` or `M`), `primarycache` and `secondarycache` (`all`, `metadata`,
`none`), `sync` (`standard`, `always`, `disabled`), `logbias` (`latency`, `throughput`), `quota` and
`reservation` (`none` or digits with `K`, `M`, `G` or `T`). A property already at its value runs nothing. Each
change is read back with `zpool get -Hp` or `zfs get -Hp` and emitted (§21). Checked by §25's convergence tests.
15. **Cache and log devices** (ADR 0679 §5). From `zpool status -P -L <pool>` the apply knows each device's kernel
path, and from the link under `/dev/disk/by-id` each declared name's kernel path. Then, in this order:
16. **Pool and dataset properties** (ADR 0679 §6). For each set pool property the apply runs `zpool set
<property>=<value> <pool>`, and for `default` `zpool set <property>=<ZFS's default> <pool>`. For each set dataset
property it runs `zfs set <property>=<value> <dataset>`, and for `default` `zfs inherit <property> <dataset>`, so
its source stays default. The dataset list is `compression` (`off`, `lz4`, `zstd`, `zstd-1` to `zstd-19`,
`gzip-1` to `gzip-9`, `zle`), `atime` and `relatime` (`on`, `off`), `recordsize` (a power of two from `512` to
`16M`, with `K` or `M`), `primarycache` and `secondarycache` (`all`, `metadata`, `none`), `sync` (`standard`,
`always`, `disabled`), `logbias` (`latency`, `throughput`), `quota` and `reservation` (`none` or digits with `K`,
`M`, `G` or `T`). A property already at its value runs nothing. Each change is read back with `zpool get -Hp` or
`zfs get -Hp` and emitted (§25). Checked by §31's convergence tests.
17. **Cache and log devices** (ADR 0679 §5). From `zpool status -P -L <pool>` the apply knows each device's kernel
path, and from the link under `/dev/disk/by-id` each declared name's. Then, in this order:
- an entry of the pool whose kernel path is a declared device's, but named by its kernel name, is removed
(`zpool remove <pool> <entry>`) and added again by its stable name;
- a declared device not in the pool is added with `zpool add <pool> cache|log [mirror] <names>`, never with
`-f`, and only after its checks: a device with a partition table or a file-system signature (`wipefs --no-act`)
is refused, but for a `zfs_member` signature, which is judged by its label (`zdb -l`): a label of this pool is
allowed, another pool's refused; a device that is a member of any imported pool is refused;
- a cache entry not declared is removed;
- a log device not declared, or one to rename, is removed only while the pool is ONLINE with no read, write or
checksum errors and no scrub or resilver runs; otherwise it is a wait, `config.waiting` {key, waits-on, since}.
- a declared device not in the pool is added with `zpool add <pool> cache|log [mirror] <names>`, never `-f`, and
only after its checks: a device with a partition table or a file-system signature (`wipefs --no-act <device>`)
is refused, but for a `zfs_member` signature, which is judged by its label (`zdb -l <device>`): this pool's
label is allowed, another pool's refused; a device that is a member of any imported pool is refused;
- only while `cache-devices` is set and not empty, a cache entry it does not name is removed;
- only while `log-devices` is set and not empty, a log device it does not name, one to rename, or the log when
`log-mirror` changes (a removal and a re-add, said as such) is removed only while the pool is ONLINE with no
read, write or checksum errors and no scrub or resilver runs; otherwise it is a wait, `config.waiting` {key,
waits-on, since}.
A declared device that is not present is a wait. Each change is read back from `zpool status -P` and emitted.
Checked by §25's convergence tests and acceptance test 2.
16. **Who applies, and when** (ADR 0679 §8). A `process` resource `zfs-config-apply`, `run: ./zfs-tools apply`, as
root, oneshot, ordered after `zfs-import.target` and enabled at boot; restarted by the node-engine on a change of
the module's config file (`restart-on`), which is how a set value reaches it from mesh-config (ADR 0348 §5, §6);
started by hand through `zfs_config_apply` (§22). It holds an exclusive lock on `/run/zfs-tools/apply.lock`; a
second start waits for it. It runs nothing while a scrub or resilver runs on the pool or before the pool is
imported: it emits `config.waiting` and the next start applies. It never runs on a timer. While the bus or the
control-node is down it applies the values its config file last held. It writes `arc-max` and `arc-min` live to
`/sys/module/zfs/parameters/zfs_arc_max` and `zfs_arc_min` as well as to §10's file and reads them back; each
`module-options` entry whose live value differs is a warning naming it, "takes effect at the next module load",
never success and never unhealthy.
17. **Never converged, always asked** (ADR 0679 §6). No setting adds, replaces, attaches, detaches or removes a data
device, a spare, a special or a dedup device; none upgrades the pool, destroys a dataset, rolls back, renames or
clears. The call-site test (`TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`, renamed for what it now holds)
allows `zpool add` only with `cache` or `log`, `zpool remove` only of a cache or log entry, `zpool set`, `zfs set`
and `zfs inherit` only for the lists of §9 and §14, `zfs destroy` only as §19 builds it, `zpool trim` only from
the scheduled run, and refuses `upgrade`, `replace`, `attach`, `detach`, `clear`, `rollback`, `rename`, `-f`, `-r`,
`-R` and `-d` everywhere.
Checked by §31's convergence tests (the empty list among them) and acceptance test 2.
18. **The apply: who, when, and what wakes a wait** (ADR 0679 §9). A `process` resource `zfs-config-apply`, `run:
./zfs-tools apply`, as root, oneshot, ordered after `zfs-import.target`, enabled at boot, with `restart-on` the
config file (§9); it converges §15 to §17 and writes the ARC values live. It holds an exclusive lock on
`/run/zfs-tools/apply.lock`; a second start waits for it. It converges nothing while a scrub or resilver runs on
the pool or before the pool is imported: it emits `config.waiting` with what it waits on and since when, and ends.
It never runs on a timer. **A wait is ended by an event:** when the event follower (§27) hears a scrub or a
resilver finish (ZFS's `scrub_finish` and `resilver_finish` events) or a declared device appear (a udev `add` of a
block device whose link under `/dev/disk/by-id` is declared, read with `udevadm monitor --udev
--subsystem-match=block`), it starts the apply unit through `<node>/node-service-manager.start`. While the bus or
the control-node is down, the apply converges from the config file it last got. It writes `arc-max` and `arc-min`
to `/sys/module/zfs/parameters/zfs_arc_max` and `zfs_arc_min` as well as to §12's file, and reads both back; each
`module-options` entry whose live value differs is a warning naming it, "takes effect at the next load of the
kernel module", never success and never unhealthy.
19. **The call-site test** (ADR 0679 §7). Every command the bundle runs is built by one of its argument functions, and
the test (`TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`, renamed `TestTheBundleRunsOnlyTheFormsOfADR0679`)
allows only: `zpool add <pool> cache|log`, `zpool remove` of a cache or log device, `zpool set`, `zfs set` and `zfs
inherit` of a listed property, `zfs snapshot`, `zfs set mesh:schedule=<kind>` on a ZFS snapshot named `mesh-`,
`zfs destroy -- <dataset>@<snapshot>`, `zpool scrub`, `zpool scrub -p` and `zpool scrub -s` of the pool, `zpool trim`
of the pool; and the reads `zpool status`, `zpool list`, `zpool get`, `zpool iostat`, `zpool events`, `zfs list`,
`zfs get` and `zfs holds`. It refuses every other `zpool` or `zfs` form and every `-f`, `-r`, `-R` and `-d`, so
data vdevs, spares, special and dedup devices, `upgrade`, a dataset's destroy, `rollback`, `rename` and `clear` are
never run (each an act of its own, asked of the operator). The same test pins the module's other root forms:
`wipefs --no-act <device>`, `zdb -l <device>`, `zdb -C <pool>`, `zgenhostid -f <value>`, `udevadm monitor --udev
--subsystem-match=block` and `systemctl show`. Fate: the README's list of allowed forms is replaced by this one.
## ZFS snapshots on a schedule
## The scheduled run and ZFS snapshots
18. **The scheduled run** (ADR 0679 §7, §14). The existing `process` resource `zfs-mesh-scrub` (`./zfs-tools
scrub-run`) becomes `zfs-mesh-run` (`./zfs-tools run`), root as today, and its `schedule` changes from
`*/15 * * * *` to the list form `0,15,30,45 * * * *`. Reason: today's node-engine writes a step `*/15` as
`*:*/15`, which the service manager refuses, so the timer failed; it writes the list form as
`*-*-* *:0,15,30,45:00`, which is valid, so the module's timer is active without the cron-to-calendar fix (lead's
answer, 2026-11-12). Checked by the module's manifest test (the schedule string) and by `systemd-analyze calendar`
of the generated timer's `OnCalendar` on the home server passing with a next run within 15 minutes (acceptance
test 3). It and does three things each time, in
order: the scrub when due (as today), a trim when `trim-schedule` is due (`zpool trim <pool>`, by `trim-day` and
`trim-time` as the scrub reads its own), and the ZFS snapshots when `snapshots` is set. Reason: one schedule
resource the node-engine already runs, and no second timer. For the snapshots: at the first run of each hour it
takes `hourly`, at the first run after 00:00 `daily`, on Monday `weekly`, on the first of the month `monthly`
(only kinds whose keep count is above 0), one `zfs snapshot -- <dataset>@mesh-<kind>-<UTC time YYYYMMDDTHHMMZ>`
per dataset, never recursive, and sets on each `zfs set mesh:schedule=<kind>`. No ZFS snapshot is taken while the
pool's capacity (`zpool list -Hp -o capacity`) is above 80 %: a warning says so, and `snapshot.skipped`
{pool, capacity} is emitted.
19. **Removed only by their own record.** After taking, per dataset and kind, the run removes the oldest past its keep
count, only when it carries `mesh:schedule=<kind>`, is in the ledger `snapshots` (§21), has no hold
(`zfs holds`) and no clone (`clones` empty): `zfs destroy -- <dataset>@<snapshot>`, never `-r`, `-R` or `-d`. A
ZFS snapshot taken by hand is never removed. Each removal emits `snapshot.removed` {snapshot, kind, space freed}
and records the ledger's `removed` operation.
20. **A person's ZFS snapshot stays apart.** `zfs_snapshot` refuses a name beginning `mesh-`, and emits
`snapshot.taken` with `by: person`; the run emits it with `by: schedule`. A waiter on a person's ZFS snapshot waits on
`snapshot.taken` with that snapshot's name, which no scheduled one carries. Fate: the README's "No tool of the
mesh destroys a snapshot" becomes §19's rule.
20. **The scheduled run** (ADR 0679 §9). The `process` resource `zfs-mesh-scrub` (`./zfs-tools scrub-run`) becomes
`zfs-mesh-run` (`./zfs-tools run`), root as today, with §6's schedule. Each run takes the apply's lock; when the lock
is held it does nothing and emits `run.skipped` {reason: `lock-held`}. It does nothing while a scrub or resilver
runs or before the pool is imported, emitting `run.skipped` with that reason. Otherwise, in order: it starts the
scrub when due and returns at once, never waiting on it (today's wait with `zpool wait -t scrub` goes; the scrub's
end is heard by the event follower, §27); starts a trim when `trim-schedule` is due (`zpool trim <pool>`, by
`trim-day` and `trim-time` in the node's time zone, as the scrub reads its own); and takes and removes the ZFS
snapshots (§21, §22) when `snapshots` is set. Checked by §31 (lock held: nothing run; scrub due: started and the
run ends).
21. **Taking scheduled ZFS snapshots** (ADR 0679 §8). Each kind whose keep count is above 0 is due at the first run of
the hour (hourly), the first run after midnight (daily), the first run on Monday (weekly) and the first run on the
first of the month (monthly), each read in the node's time zone against the ledger's last entry of that kind (§26).
For every filesystem of the pool it runs `zfs snapshot -- <dataset>@mesh-<kind>-<YYYYMMDDTHHMMZ>`, the time in UTC,
never recursive, then `zfs set mesh:schedule=<kind>` on it, and records the ledger's `taken` operation. The run
takes no ZFS snapshot it cannot record in the same run: with the bus down or the ledger full it takes none and
emits `snapshot.skipped.schedule` {pool, capacity, reason}. No ZFS snapshot is taken while the pool is more than 80
% full (`zpool list -Hp -o capacity`): a warning says so, and `snapshot.skipped.schedule` is emitted with reason
`full`.
22. **Removing them, only by their own record** (ADR 0679 §8; the operator, 2026-11-12: "we may absolutely not lose any
data"). After taking, the run reads the ledger in the same run (no read, no removal) and, per dataset and kind,
removes the oldest past its keep count, all of them in that run. Before any destroy it checks the name against
`<a dataset of the pool>@mesh-<kind>-<YYYYMMDDTHHMMZ>`, that it carries `mesh:schedule=<kind>`, is in the ledger,
has no hold (`zfs holds`) and no clone (`clones` empty); anything else is refused, emitted and left. Then `zfs
destroy -- <dataset>@<snapshot>`, the ledger's `removed` operation, and `snapshot.removed.schedule` {snapshot, kind,
space freed in bytes}. When `snapshots` goes back to `none`, the existing scheduled ZFS snapshots are kept, and a
warning names them; that change removes nothing. A ZFS snapshot taken by hand is never removed.
23. **A person's ZFS snapshot stays apart** (ADR 0679 §8, §10). `zfs_snapshot` refuses a name beginning `mesh-` and
emits `snapshot.taken.person`; the run emits `snapshot.taken.schedule`. A waiter on a person's ZFS snapshot waits
on `snapshot.taken.person`, which no scheduled one is said on. Fate: the README's "No tool of the mesh destroys a
snapshot" becomes §22's rule.
24. **Hand acts run as the run does** (ADR 0679 §14). `zfs_snapshots_run_now`, `zfs_snapshots_prune_now` and
`zfs_config_apply` each start a oneshot instance of the unit `zfs-mesh-run@<act>.service`, the act from the fixed
set `snapshots`, `prune` and `apply`, through `<node>/node-service-manager.start`, with `why`; the instance runs
under the same lock and guards as §18 and §20, and the tool answers from the state and events afterwards. Their
events carry `.person` (`snapshot.taken.person`, `snapshot.removed.person`, `snapshot.skipped.person`).
`zfs_snapshots_pause` writes the key `paused` {by, why, at} in the state `schedule`, which the run reads and then
takes and removes nothing; `zfs_snapshots_resume` deletes it.
## What it says, keeps and shows
21. **Events and kept data** (ADR 0679 §9; ADR 0363 §1; ADR 0595). The module declares a stream of its own, local
name `zfs`, kept 30 days, carrying `config.applied` {target (`pool`, `dataset`, `file` or `unit`), key, before,
after, read-back}, `config.refused` {key, value, reason}, `config.waiting` {key, waits-on, since},
`snapshot.taken` {snapshot, by}, `snapshot.removed`, `snapshot.skipped`, and today's `scrub.requested`; today's
module-level `emits` move to it. Its kept data, in `requires` by template: a state `applied`, one key per config
value, `{before, after, at, outcome}`, written by the apply and deleted by it when that value returns to not
managed (after its last `config.applied`); a ledger `snapshots`, one instance per ZFS snapshot the run took,
operations `taken` and `removed`, kept by the ledger's template. The apply process and the run declare the bus
account (`bus`) that writes them; the tools read them.
22. **Tools** (ADR 0679 §12). `zfs_config`: every owned value with its setting, what runs on the machine, its state
(`not managed`, `as set`, `waiting` with on what and since, `refused` with why, `takes effect at the next module
load`), and a `.pacnew` beside an owned file as a warning. `zfs_config_apply`: starts §16's unit through
`<node>/node-service-manager.start`, with `why`, and answers from the state `applied` afterwards.
`zfs_snapshots_schedule`: the schedule, its next run, what it would remove next and the ledger.
`zfs_snapshots_run_now` and `zfs_snapshots_prune_now`: a run's snapshot or removal step by hand, with `why`.
`zfs_snapshots_pause` and `zfs_snapshots_resume`: a key `paused` in the state `applied`, read by the run, said in
the health while it holds. The existing read tools stay. Each is in `replaces` where it replaces a command.
23. **Health** (ADR 0679 §10). `zfs_health` is unhealthy for a `refused` or a failed command in the state `applied`,
naming the key and why; warnings: a wait (naming on what and since when), the last scheduled ZFS snapshot older
than its cadence, a failed run, `autoreplace` on or a spare in the pool while `zed` is not `on`, the pool above
80 % with snapshots set, a parameter awaiting a module load. Pool faults reach the operator once, never from a ZED
handler: the event follower (`zfs-events`) is an observer, writing a finding `<pool>.<vdev or pool>.<state>` in its
bucket `findings` for a pool or device that leaves ONLINE, saying each as an event, and removing it when the state
is ONLINE again; the condition keeper raises and clears the condition from it (ADR 0564 §2; design 47 G §383). Its
process declares the bus account that writes them; today it writes the journal only.
24. **When the module leaves** (ADR 0679 §13; ADR 0303 §2; ADR 0269). The packages stay. The import, mount and target
units stay enabled and running. `/etc/default/zfs` and `zed.rc` get back their found content (ADR 0118);
`zfs.conf`, the folder `/etc/zfs/mesh-zed.d` and the drop-in are removed, and before leaving `zfs_config` says that
the next module load uses ZFS's defaults. The distribution's scrub timer is given back as found (ADR 0269). Pool
and dataset properties and devices stay as the last apply set them, also on a rollback of the module's version.
25. **Events, each kind on a subject of its own** (ADR 0679 §10). `config.applied` {target (`pool`, `dataset`,
`file` or `unit`), key, before, after, read-back}, `config.refused` {key, value, reason}, `config.waiting` {key,
waits-on, since}, `snapshot.taken.person` and `.schedule` {snapshot, by}, `snapshot.removed.person` and
`.schedule` {snapshot, kind, space freed in bytes}, `snapshot.skipped.person` and `.schedule` {pool, capacity,
reason}, `run.skipped` {reason}, and today's `scrub.requested`. They go on an events stream asked for by template
in `requires` alone and created by the bus module (ADR 0595 §1, §2, §4), kept 30 days; today's module-level `emits`
move to it. No module consumes `snapshot.taken` or `scrub.requested` today (read in mesh-catalog's manifests on
main); a session waiting through `mesh_wait` on `snapshot.taken` hears `snapshot.taken.person` after.
26. **Kept data, by template in `requires`** (ADR 0679 §10; ADR 0595 §1). A state `applied`, one key per config value,
`{before, after, at, outcome}`, written by the apply and deleted by it when that value goes back to not managed,
after its last `config.applied`. A state `schedule`, the key `paused` alone (§24). A ledger `snapshots`, one
instance per ZFS snapshot the run took, operations `taken` and `removed`, the run removing the instance of one it
destroyed; its cap is the keep counts' sum times two instances per dataset. A state `findings` (§27). The apply,
the run and the event follower declare the bus account that writes them; the tools read them.
27. **The event follower's findings** (ADR 0679 §12; ADR 0564 §2; design 47 G §383 to §388). The event follower
(`zfs-events`, root) is an observer: in its state `findings` it keeps one key `pool.<pool>.<device>` per device
of the pool (and `pool.<pool>.pool` for the pool) not ONLINE, its value the state, deleted when the device is
ONLINE again or leaves the pool; it says each change on its findings stream as design 47 G §383 says, and at its
start it reads `zpool status -P` against its keys and states them (§384). The condition keeper raises and clears
the condition from those events, so a pool fault reaches the operator once and never from a ZED handler. It also
starts the apply on the events of §18. Its process declares the bus account; today it writes the journal only.
28. **Tools** (ADR 0679 §14). `zfs_config`: every owned value with its config value, what runs on the node, its state
(`not managed`, `as set`, `waiting` with on what and since, `refused` with why, `takes effect at the next load of
the kernel module`), and the files of §10. `zfs_config_apply` (§24). `zfs_snapshots_schedule`: the schedule, its
next run, what it would remove next and the ledger. `zfs_snapshots_run_now`, `zfs_snapshots_prune_now`,
`zfs_snapshots_pause` and `zfs_snapshots_resume` (§24). `zfs_module_events`: the module's stream, the last N events
by kind (named so because `zfs_events` already reads ZFS's own events). `zfs_snapshots_check`: compares the ledger
with what ZFS holds and names each entry whose ZFS snapshot is gone and each `mesh-` snapshot not in the ledger,
changing nothing. `zfs_findings`: the follower's findings. The existing read tools stay. Each is in `replaces`
where it replaces a command.
29. **Health** (ADR 0679 §11). `zfs_health` is unhealthy for a `refused` or a failed command in the state `applied`,
naming the key and why. Warnings: a wait, naming on what and since when; a `.pacnew` beside `/etc/default/zfs` or
the package's `zed.rc`; the last scheduled ZFS snapshot older than its cadence; a failed run; `autoreplace` on or a
spare in the pool while `zed` is not `on`; the pool above 80 % with `snapshots` set; scheduled ZFS snapshots kept
after `snapshots` went to `none`; a parameter awaiting a load of the kernel module; `paused` set.
30. **When the module leaves** (ADR 0679 §15; ADR 0303 §2; ADR 0269 §2). The packages stay. `zfs-import-cache`,
`zfs-mount` and `zfs.target` are never stopped or disabled, and the module says so before it leaves (ADR 0679 §15
changes ADR 0269 §10 for them). Nothing of the package's files is given back, since the module never wrote them.
`zfs.conf`, `/etc/zfs/mesh-zed.d` and the drop-in are removed, and before leaving `zfs_config` says that the next
load of the kernel module uses ZFS's defaults. The distribution's scrub and trim timers are put back as the
module's declaration found them, started and enabled again where it stopped or disabled them. Pool and dataset
properties and devices stay as the last apply set them, also on a rollback (§8).
## How it is checked
25. **Unit tests**, table-driven, one test per area, within the module's 500: no settings gives no `zpool` or `zfs`
command that writes; each convergence plan from parsed `zpool status` and `zfs get` output (an add, a kernel name
to its stable name, a removal, a log removal refused while not healthy, a wait during a scrub); every refused form
(a newline, quote, `$`, backtick or semicolon in each field; a parameter not on the list; a property not on the
lists; a `sharenfs` setting; `-f`; a device with a signature or another pool's label); the call-site test of §17;
pruning never removes a `mesh-` ZFS snapshot without the property or not in the ledger, nor one with a hold or
clone; `zfs_snapshot` refuses a `mesh-` name; a parameter awaiting a module load is a warning; each change emits
its event with before and after; a scheduled `snapshot.taken` does not match a person's waiter. The module has no
integration test; the live checks stand for it, said in the pull request.
26. **Live, on the home server**: the main file's acceptance tests 1 to 3, in that order.
31. **Unit tests**, table-driven, one test per area, within the module's 500: no values gives no `zpool` or `zfs`
write but the due scrub; each convergence plan from parsed `zpool status` and `zfs get` output (an add, a kernel
name to its stable name, a removal, an empty list removing nothing, a log removal refused while not healthy, a
`log-mirror` change, a wait during a scrub); every refused form (a newline, quote, `$`, backtick or semicolon in
each field; a parameter or property not on its list; a `sharenfs` value; `-f`; a device with a signature or another
pool's label; a hostid of the wrong form or a label of another id); the call-site test of §19; the lock (a second
apply waits, a run with the lock held does nothing); the apply started only by `config-changed`, never a timer;
one scheduled resource; both ARC writes read back; applying from the last config file with the bus down; the
three templates (config file, options file, `zed.rc`); pruning never removes a ZFS snapshot whose name, property
or ledger entry does not fit, nor one with a hold or clone, and removes every surplus in one run; no ZFS snapshot
taken without a ledger record; `snapshots` back to `none` removes nothing; `zfs_snapshot` refuses a `mesh-`
name; a scheduled `snapshot.taken.schedule` does not wake a person's waiter; a parameter awaiting a load is a
warning; the `.pacnew` warning; each change emits its event with before and after; one test per tool. The module
adds no integration test; §5 is the milestone's, and the main file's live tests stand for the module whole, said
in the pull request.
03-DESIGN/01-to-be/42-the-machines-modules-in-order.md
+60 -41
View File
@@ -33,8 +33,8 @@ decisions:
- "ADR 0051"
- "ADR 0269"
- "ADR 0348"
- "ADR 0363"
- "ADR 0595"
- "ADR 0593"
---
# 42. The machines' modules, in order
@@ -157,68 +157,87 @@ on the machine; it is placed only on a machine that holds a pool. Its section is
handles the rest"; "the zfs module should/must own all zfs-configuration files and options"; "we may absolutely not
lose any data".*
**What it is for.** On a machine that holds a ZFS pool, the zfs module owns ZFS: what was installed and set by hand
and seen by nothing becomes the module's, and the operator's choices about it are config values (ADR 0679 §1). It
holds no seat (see the open question of its milestone part). It keeps reading and acting by hand as today: the pools,
the datasets, a named ZFS snapshot, a scrub's start, pause and stop.
**What it is for.** On a node that holds a ZFS pool, the zfs module owns ZFS: what was installed and set by hand and
seen by nothing becomes the module's, and the operator's choices about it are config values (ADR 0679 §1). It holds
no seat; a seat for it is a later record's. It keeps reading and acting by hand as today: the pools, the datasets, a
named ZFS snapshot, a scrub's start, pause and stop.
**The whole shape.**
- **What it owns** (ADR 0679 §1): the packages `zfs-utils` and `zfs-dkms`; the files `/etc/default/zfs`,
`/etc/zfs/zed.d/zed.rc` and `/etc/modprobe.d/zfs.conf`; which handlers ZFS's event daemon (ZED) runs; the states of
ZFS's import, mount, share, volume and target units and of the distribution's scrub and trim timers; and, for the one
pool its config value `pool` names, that pool's properties, its cache and log devices and its datasets' properties
from a fixed list. It reads, and never writes, what ZFS writes itself (its pool cache file and list cache).
- **What is not its own** (ADR 0679 §2): a dataset's `sharenfs` is the NFS share module's (design 52), `sharesmb` the
SMB module's once there is one; the pool's files are the operator's (ADR 0051); the weekly upgrade and a kernel
held for DKMS are the package manager's (ADR 0508).
- **An unset value changes nothing** (ADR 0679 §3): every new config value is "not managed" until the operator sets
it through mesh-config, on the Approve for a trusted list or object value (ADR 0348 §4, §8). Placing the new version
on a machine runs no `zpool` or `zfs` command that writes and rewrites none of its files.
- **Who applies** (ADR 0679 §8): the node-engine places the packages, files and units as the module's resources;
the module's root process `zfs-tools apply` converges the pool's devices and properties at its start and when its
config changes, never on a timer; the module's scheduled run takes and removes the scheduled ZFS snapshots
(ADR 0679 §7).
- **What it says** (ADR 0679 §9, §10): every change, refusal and wait is an event on the module's own stream, and
the last change per value and every scheduled ZFS snapshot are kept on the bus in its own state and ledger, asked
for by template (ADR 0595). Its health names a refusal or a failed command as unhealthy and a wait as a warning.
Pool faults reach the operator once, from the module's event follower through the condition keeper.
- **What it never does** (ADR 0679 §6): it converges no data device, spare, special or dedup device, upgrades no
- **What it owns** (ADR 0679 §1): the packages `zfs-utils` and `zfs-dkms`; `/etc/modprobe.d/zfs.conf`, placed only
while an ARC or module option is set; `/etc/hostid`; its own folder for ZFS's event daemon (ZED),
`/etc/zfs/mesh-zed.d`, with its `zed.rc` and handlers, and a drop-in for `zfs-zed`, placed only while ZED is on;
the states of ZFS's import, mount, share, volume and target units and of the distribution's scrub and trim timers;
and, for the one pool its config value `pool` names, that pool's cache and log devices, its properties and its
datasets' properties from fixed lists. The node-engine's composer refuses a second module owning one of its paths
(ADR 0593 §7, ADR 0051 §9).
- **What it never writes** (ADR 0679 §1, §2): the package's own `/etc/default/zfs` and `/etc/zfs/zed.d` with its
`zed.rc`, and what ZFS writes itself; it reads and shows them. A dataset's `sharenfs` is the seat
`node-nfs-server`'s, `sharesmb` the SMB module's; the pool's files are the operator's (ADR 0051); the weekly
upgrade and a kernel held for DKMS are the package manager's (ADR 0508).
- **An unset or empty value changes nothing** (ADR 0679 §3): every new config value is "not managed" until the
operator sets it through mesh-config, on the Approve for a trusted value (ADR 0348 §4, §8). Placing the new version
on a node runs no `zpool` or `zfs` write but the scrub its schedule already runs, and places none of its new files.
- **Who applies** (ADR 0679 §9): the node-engine places the packages, files and units as the module's resources,
some only while a value holds; the module's root process `zfs-tools apply` converges the pool's devices and
properties at its start and on `config-changed`, never on a timer; the module's one scheduled run, under the same
lock, starts the scrub and the trim when due and takes and removes the scheduled ZFS snapshots (ADR 0679 §8).
- **What it says** (ADR 0679 §10, §11, §12): every change, refusal and wait is an event, each kind on a subject of
its own, on an events stream asked for by template; the last change per value, the schedule's pause and every
scheduled ZFS snapshot are kept on the bus in its own states and ledger (ADR 0595). Its health names a refusal or a
failed command as unhealthy and a wait as a warning. Pool faults reach the operator once, from the module's event
follower's findings through the condition keeper.
- **What it never does** (ADR 0679 §7): it converges no data device, spare, special or dedup device, upgrades no
pool, destroys no dataset, rolls back, renames and clears nothing; each is an act of its own, decided separately.
- **When it leaves** (ADR 0679 §13): ZFS keeps running, the packages stay, and the files are given back.
- **When it leaves** (ADR 0679 §15): ZFS keeps running (the import, mount and ZFS target are never stopped), the
packages stay, its own files are removed and the distribution's timers are put back as found.
- **Its prerequisites** (ADR 0679 §16): three node-engine capabilities, built first in the same milestone under issue
1979, with their readers in mesh-sdk and the manifest validator.
```mermaid
flowchart LR
operator[the operator] -->|sets a config value| config[mesh-config]
config -->|the module's config file| engine[the node-engine]
engine -->|packages, files, units| machine[ZFS on the machine]
engine -->|packages, files, units| node[ZFS on the node]
engine -->|restarts on its config file| apply[zfs-tools apply, root]
apply -->|zpool add, remove, set; zfs set, inherit| machine
sched[the scheduled run] -->|ZFS snapshots, their removal, scrub, trim| machine
apply -->|config.applied, refused, waiting| stream[(the module's stream)]
sched -->|snapshot.taken, snapshot.removed| stream
apply -->|zpool add, remove, set; zfs set, inherit| node
sched[the scheduled run] -->|scrub, trim, ZFS snapshots and their removal| node
follower[the event follower] -->|scrub ended, device appeared: start| apply
apply -->|config.applied, refused, waiting| stream[(events stream)]
sched -->|snapshot.taken.schedule, removed, skipped| stream
apply --> applied[(state applied)]
sched --> ledger[(ledger snapshots)]
follower[the event follower] -->|pool faults| keeper[the condition keeper]
follower -->|findings| keeper[the condition keeper]
tools[the module's tools] -->|read| applied
tools -->|read| ledger
```
**Acceptance tests** (ADR 0679, *How it is checked*):
**Acceptance tests** (ADR 0679, *How it is checked*), each with its level; every live test is read by a tool call:
1. Placing the new version on the home server changes nothing: `zfs_config` shows every value "not managed", and
no `zpool` or `zfs` command that writes ran (the module's journal and its stream show none).
2. Setting `cache-devices` puts both cache devices in the pool by their stable names; after a reboot that swaps the
drives' kernel names, both stay ONLINE (`zfs_pool_layout`).
3. The module's own scrub timer is active before the distribution's scrub timer is stopped, read from
`<node>/node-service-manager.status` of both.
4. The unit tests of the milestone part's line 25 pass, within the module's budget of 500 unit tests.
1. *Live.* Placing the new version on the home server changes nothing: `zfs_config` shows every value "not managed",
and `zpool history` read before and after shows no command but the scheduled scrub.
2. *Live.* Setting `cache-devices` puts both cache devices in the pool by their stable names; after a reboot that
swaps the drives' kernel names, both stay ONLINE (`zfs_pool_layout`).
3. *Live.* The node-engine's apply report shows the module's run timer active before the distribution's scrub timer
is stopped, and `systemd-analyze calendar` of the generated `OnCalendar` passes with a next run within 15 minutes.
4. *Live.* With `hostid` unset, `/etc/hostid` is not written (`<node>/node-reads.stat`); `zfs_config` shows it "not
managed".
5. *Live.* With `snapshots` set, the next run's ZFS snapshot is in `zfs_snapshots` with `mesh:schedule` and in the
ledger (`zfs_snapshots_schedule`); after its keep count passes, its removal is in both and in `zfs_module_events`.
6. *Live.* A value of a refused form makes `zfs_health` unhealthy naming the key; a declared device that is not
present makes it a warning naming what it waits on.
7. *Live.* With `zed` on, `<node>/node-files.list` of `/etc/zfs/mesh-zed.d` shows its `zed.rc` and only the handlers
set, and `<node>/node-service-manager.status` shows `zfs-zed` running with the drop-in; with `zed` off, stopped.
8. *Live, on a test placement.* After the module leaves, `zfs-import-cache`, `zfs-mount` and `zfs.target` are still
running (`<node>/node-service-manager.status`) and `zfs_pools` shows the pool imported.
9. *Unit and integration.* The milestone part's §31 unit tests pass within the module's budget of 500, and its §5
integration test passes in the node-engine.
## Milestones
| milestone | what it builds | part | state |
|---|---|---|---|
| zfs-owns-zfs | the zfs module's ownership of ZFS (ADR 0679), with its prerequisites in the node-engine and the catalog, under issue 1979 | [42-milestones/zfs-owns-zfs.md](42-milestones/zfs-owns-zfs.md) | designed |
| zfs-owns-zfs | the zfs module's ownership of ZFS (ADR 0679), with its three node-engine prerequisites and their readers, under issue 1979 | [42-milestones/zfs-owns-zfs.md](42-milestones/zfs-owns-zfs.md) | designed |
The modules of the tables above were built before designs were cut into milestone parts; their remaining lines move
into a part of their own at their next amendment (playbook 02, *The shape of a design*).