novox/hq
9
0
Fork 0
Code Issues Pull Requests 2 Releases Wiki Activity

Design 42 zfs-owns-zfs: answer review round 1 against ADR 0679 version 11
mesh/delivery merged: source.main moved to novox/hq@b9cd0bc492ac
Details
mesh/check being carried to 224348ff4d69 on b9cd0bc492ac | from 9ebf6e58f0d7

Browse Source 
Cites the record's final clauses, drops the catalog check and every
write of the package's files, and writes the lead's answers as design
lines: a run that never waits on a scrub, waits ended by events,
ZFS snapshots kept in the ledger before any removal, hand acts as unit
instances, subjects per kind, the follower's findings, the readers of
the new manifest words and one node-engine integration test.
This commit is contained in:
jschoubben
2026-11-12 23:20:51 +01:00
parent f26c8afc20
commit 224348ff4d
3 changed files with 338 additions and 268 deletions
Download Patch File Download Diff File Expand all files Collapse all files
00-META/glossary.md
+1 -1
View File
@@ -1023,7 +1023,7 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283, 0564, 0668.
- **ZFS snapshot** — a read-only point in time of one ZFS dataset on a machine, kept inside that dataset's pool, so - **ZFS snapshot** — a read-only point in time of one ZFS dataset on a machine, kept inside that dataset's pool, so
it guards against a mistake and never against the pool's loss; taken by a person with `zfs_snapshot`, or by the zfs it guards against a mistake and never against the pool's loss; taken by a person with `zfs_snapshot`, or by the zfs
module's scheduled run, which names its own `mesh-<kind>-<time>`, marks them and removes only those past their keep module's scheduled run, which names its own `mesh-<kind>-<time>`, marks them and removes only those past their keep
count (ADR 0679 §7; design 42 zfs-owns-zfs §18, §19). Never bare, beside a ledger's snapshot and the stream snapshot count (ADR 0679 §8; design 42 zfs-owns-zfs §21, §22). Never bare, beside a ledger's snapshot and the stream snapshot
(see [Homonyms](#homonyms)); not a backup or a restore point, which are kept elsewhere. (see [Homonyms](#homonyms)); not a backup or a restore point, which are kept elsewhere.
*Not:* ~~pool snapshot~~, ~~dataset snapshot~~ *Not:* ~~pool snapshot~~, ~~dataset snapshot~~
- **disk readings** — the size of each data item a node's modules declare, measured by its `measure` and kept - **disk readings** — the size of each data item a node's modules declare, measured by its `measure` and kept
03-DESIGN/01-to-be/42-milestones/zfs-owns-zfs.md
+277 -226
View File
@@ -3,270 +3,321 @@ layer: to-be
design: 42 design: 42
milestone: zfs-owns-zfs milestone: zfs-owns-zfs
updated: 2026-11-12 updated: 2026-11-12
code: [mesh-catalog modules/zfs, mesh-catalog checks, mesh-host internal/apply] code: [mesh-sdk, mesh-delivery, mesh-host internal/apply, mesh-catalog modules/zfs]
decisions: decisions:
- "ADR 0679" - "ADR 0679"
- "ADR 0051" - "ADR 0051"
- "ADR 0117" - "ADR 0117"
- "ADR 0118"
- "ADR 0269" - "ADR 0269"
- "ADR 0303" - "ADR 0303"
- "ADR 0348" - "ADR 0348"
- "ADR 0363"
- "ADR 0508" - "ADR 0508"
- "ADR 0564" - "ADR 0564"
- "ADR 0593"
- "ADR 0595" - "ADR 0595"
- "ADR 0609"
- "ADR 0614"
--- ---
# Design 42, milestone zfs-owns-zfs — the zfs module owns a machine's ZFS, and changes nothing that is not set # Design 42, milestone zfs-owns-zfs — the zfs module owns a node's ZFS, and changes nothing that is not set
**What it builds.** On a machine that holds a ZFS pool (today the home server), the zfs module owns ZFS's packages, **What it builds.** On a node that holds a ZFS pool (today the home server), the zfs module owns ZFS's packages, its
files, units and, for one pool, its properties, cache and log devices, its datasets' properties and a schedule of kernel module's options file, its host id, its own folder for ZFS's event daemon (ZED), its units and, for one pool,
ZFS snapshots; each is changed only once its config value is set (ADR 0679). The purpose and the whole shape are the its cache and log devices, its properties, its datasets' properties and a schedule of ZFS snapshots; each is changed
main file's, [*The zfs module*](../42-the-machines-modules-in-order.md#the-zfs-module). This part holds every line only once its config value is set and not empty (ADR 0679). The purpose and the whole shape are the main file's,
the milestone builds, cited as "design 42 zfs-owns-zfs §n". [*The zfs module*](../42-the-machines-modules-in-order.md#the-zfs-module). This part holds every line the milestone
builds, cited as "design 42 zfs-owns-zfs §n".
**Where.** Built under issue 1979. Three repositories, one delivery group with one check: `mesh-host` (the **Where.** Built under issue 1979. Four repositories, one delivery group with one check: mesh-sdk, mesh-delivery
node-engine: §1, §2, §3, its three subtasks of issue 1979), mesh-catalog's checks (§4) and mesh-catalog's (the manifest validator) and `mesh-host` (the node-engine) for §1 to §5, issue 1979's three node-engine subtasks
`modules/zfs` (§5 to §26). **Ordering edge:** the zfs module's new version needs the node-engine of §1, §2 and §3 and the readers of their words; mesh-catalog's `modules/zfs` for §6 to §31. **Ordering edges** (ADR 0679 §16): the
on its machine (a node-engine capability edge the delivery plan derives, §5); §4 has no edge, it gates the catalog's readers in §4's order, then the zfs module's new version only on a node whose node-engine states the three
own check. Nothing waits on the node-engine's cron-to-calendar fix: the scheduled run's schedule is written in a form capabilities of §4 (a reader's capability the delivery plan derives, ADR 0614 §4).
today's node-engine already turns into a valid calendar (§18).
## The prerequisites ## The prerequisites (ADR 0679 §16)
1. **The node-engine's apply of a module stops at its first failed resource.** When one resource of a module fails, 1. **The node-engine's apply of a module stops at its first failed resource** (ADR 0679 §16 (a)). When one resource
the node-engine applies none of that module's later resources in the declared order; the report names the of a module fails, the node-engine applies none of that module's later resources in the declared order; its apply
failed resource with its error and each one left unapplied, and the module's phase is `failed`. Resources already report names the failed resource with its error and each one left unapplied, and the module's phase is `failed`.
applied stay as applied. Other modules on the node go on as today. Today the apply goes on, and stopped the Resources already applied stay as applied; other modules on the node go on as today. Today the apply goes on, and
distribution's scrub timer after the module's own timer failed (ADR 0679 §14). Checked by a unit test: a module of stopped the distribution's scrub timer after the module's own timer failed. Checked by a unit test (a module of
three resources whose second fails leaves the third untouched and reports it as left. three resources whose second fails leaves the third untouched and reports it as left) and by §5.
2. **The node-engine declares a resource only while a config value holds** (lead's answer, 2026-11-12, to this 2. **A resource is declared only while a config value holds** (ADR 0679 §16 (b), §3). A resource of any type may
part's design question: build it in the node-engine, design-it-as-it-should-be; ADR 0679 §3, §8). A resource of carry the manifest field `if-config`: a list of conditions, each `<key>` (the module's config value is set and is
any type may carry the manifest field `if-config`: a list of conditions, each `<key>` (the module's config value is neither `false`, `off`, `none` nor an empty list) or `<key>=<value>` (the value equals it; for a list value, the
set and is neither `false`, `off`, `none` nor an empty list) or `<key>=<value>` (the value equals it; for a list list holds that entry). The resource is declared while any condition holds, and not at all while none does. A
value, the list holds that entry). The resource is declared while any condition holds, and not at all while none resource that leaves the declaration this way is removed as any resource that leaves it: a file the node-engine
does. A resource that leaves the declaration this way is removed as any resource that leaves it: a file written wrote whole is deleted, a unit gets its found state back (ADR 0269 §2). The manifest validator refuses an
over gets its original back (ADR 0118), a unit gets its found state back (ADR 0269). The controller's catalog check `if-config` key the module does not declare under `config`. Checked by the node-engine's table test (a value unset,
refuses an `if-config` key the module does not declare under `config`. Checked by the node-engine's table test (a empty, `off`, set; a list with and without the entry; a value going from set to unset removes the resource) and
value unset, `off`, set; a list with and without the entry; a value going from set to unset removes the resource the validator's own test.
and puts the original back) and the catalog check's own test. 3. **A template writes one line per entry** (ADR 0679 §16 (c), §4). A template file may hold
3. **A template writes one line per entry** (lead's answer, 2026-11-12, as §2; ADR 0679 §4). A template file may `${config-lines:<key>:<prefix>}`: for a list value one line per entry, `<prefix><entry>`, in the list's order;
hold `${config-lines:<key>:<prefix>}`: for a list value one line per entry, `<prefix><entry>`, in the list's order;
for a set scalar value one line `<prefix><value>`; nothing for an unset value or an empty list. With a fourth part, for a set scalar value one line `<prefix><value>`; nothing for an unset value or an empty list. With a fourth part,
`${config-lines:<key>:<prefix>:mib-bytes}`, a number given in MiB is written in bytes. The prefix is literal text `${config-lines:<key>:<prefix>:mib-bytes}`, a number given in MiB is written in bytes. The prefix is literal text
with no `}`, `$` or newline, refused by the catalog check; an entry holding a newline is refused before the file is with no `}`, `$` or newline, refused by the manifest validator; an entry holding a newline is refused before the
written, and the file is left as it was. Checked by the node-engine's table test (a list of two, an empty list, an file is written, and the file is left as it was. Checked by the node-engine's table test (a list of two, an empty
unset and a set scalar, `mib-bytes`, a newline in an entry) and the catalog check's own test. list, an unset and a set scalar, `mib-bytes`, a newline in an entry) and the validator's own test.
4. **The catalog refuses a second writer of one file.** mesh-catalog's checks refuse a catalog in which two 4. **Every reader of the new words, in order** (design-it-as-it-should-be; ADR 0614 §4). mesh-sdk first: the
modules' manifests declare a `file` resource at one path, naming both modules and the path (ADR 0679 §14; ADR 0051 manifest types gain `if-config` on every resource and the template form `config-lines`, and the capability names
§9, rival ownership). A file only read is not a writer. Checked by the check's own test over two manifests that `if-config`, `config-lines` and `apply-stops-at-failure`. Then the manifest validator in mesh-delivery's indexer:
declare one path (refused) and two that declare different paths (passed); it runs in the catalog's check on it reads both words and refuses them as §2 and §3 say. Then the node-engine: it builds §1 to §3 and states the
every pull request. three capabilities. A manifest using `if-config` or `config-lines` reaches a node only through a node-engine that
5. **The order on the machine.** The zfs module's new version declares its own timer resource (the scheduled run of states both; the delivery plan derives the edge from the stated capability, never from a fingerprint. Checked by
§18) before the resources that stop the distribution's scrub timers, so by §1 those are stopped only once the each repository's unit tests and by the delivery plan naming the edge.
module's own timer is active (ADR 0679 §14). The delivery plan places the node-engine of §1, §2 and §3 on the home server before the 5. **One integration test of the node-engine** (how-work-is-done: a change to how modules run carries one). It
module (the capability edge above). Checked by the module's manifest test, which pins the resource order, and by starts the node-engine whole against the real bus with a module of a resource under `if-config`, a template with
acceptance test 3 of the main file. `config-lines`, and a resource that fails: the conditioned resource appears when its value is set and goes when it
is unset, the template's lines follow the list, and the failing resource stops the module's apply with the later
resource left. It is the only integration test this milestone adds.
6. **The order on the node** (ADR 0679 §16, §9). The zfs module declares its scheduled run's resource (§20) before
the resources that stop the distribution's scrub timers, so by §1 those are stopped only once the run's timer is
active. The run's `schedule` is the list form `0,15,30,45 * * * *`, which today's node-engine writes as
`OnCalendar=*-*-* *:0,15,30,45:00`, so nothing waits on another fix. Checked by the module's manifest test (the
resource order and the schedule string) and by acceptance test 3 of the main file.
## What the module owns ## What the module owns
6. **The packages.** The manifest declares `zfs-utils` and `zfs-dkms` present (ADR 0679 §1; ADR 0303 §2: the 7. **The packages** (ADR 0679 §1; ADR 0303 §2). The manifest declares `zfs-utils` and `zfs-dkms` present: the
node-engine installs a missing one, upgrades none, removes none). `zfs-utils-debug` is not declared and is left as node-engine installs a missing one, upgrades none, removes none. `zfs-utils-debug` is not declared and is left as
found. Fate: the README's "It installs nothing" goes; it says the two packages and why the debug package is not found. Fate: the README's "It installs nothing" goes; it says the two packages and why the debug package is not the
the module's. Read by `pacman_packages` as declared by `zfs`. Checked by the manifest test. module's. Read by `pacman_packages` as declared by `zfs`. Checked by the manifest test.
7. **One pool, renamed `pool`.** The config value `scrub-pool` becomes `pool` (ADR 0679 §1): type string, trusted, 8. **One pool, renamed `pool`** (ADR 0679 §1; ADR 0609 §4). The config value `scrub-pool` becomes `pool` with
required, same description but for the rename. It is moved by mesh-config's rename window (mesh-config answers both `renamed-from: scrub-pool`: type string, trusted, required. mesh-config moves the value under the new name and
names meanwhile; its seat verb `renames` says the window), and every template and the module's code read `pool`. answers both names during the rename window (design 32, *A config value renamed*); every template and the module's
One pool per assignment; the manifest has no per-pool form and none is added. The module reads, never writes, code read `pool`. **Rollback:** a rolled-back version leaves pool and dataset properties and devices as the last
`/etc/zfs/zpool.cache` and `/etc/zfs/zfs-list.cache`. Checked by the manifest test (no `scrub-pool` left) and the apply set them (ADR 0679 §15); once the rename window has closed, going back to the version with `scrub-pool` and
rename's live read in `zfs_config`. `zfs-mesh-scrub` is a new delivery of that version, with its own rename back, never a rollback. Checked by the
8. **What is not its own.** `sharenfs` and `sharesmb` are on no list of §14; a setting naming them is refused manifest test (no `scrub-pool` but in `renamed-from`) and `zfs_config` showing `pool` on the home server.
(§9), and `zfs_config` never reports either as drift (ADR 0679 §2). The pool's files are the operator's (ADR 9. **The module's config file** (ADR 0679 §9; ADR 0348 §5). `/etc/zfs-tools/config.json`, root's, mode 0644, written
0051). Checked by §25's refused-forms test. by the node-engine from a template holding every config value of the module, lists and objects rendered as ADR 0348
§5 says; it replaces `/etc/zfs-tools/scrub.conf`, which the node-engine removes as a resource that left. The apply
unit (§18) has `restart-on` this file: its change is the path by which mesh-config's `config-changed` for the
module's values reaches the apply. The scheduled run (§20) reads it at each run. `/var/lib/zfs-scrub` stays and is
read by the run as today. Checked by the manifest test.
10. **What it never writes** (ADR 0679 §1, §2). The package's own `/etc/default/zfs` and `/etc/zfs/zed.d`, with its
`zed.rc` and its links, are left as the package ships them, and nothing of them is given back on leaving; ZFS's own
`/etc/zfs/zpool.cache` and `/etc/zfs/zfs-list.cache` are read only. `zfs_config` shows each, and a `.pacnew`
beside `/etc/default/zfs` or the package's `zed.rc` is a warning (§29). A dataset's `sharenfs` is the seat
`node-nfs-server`'s, `sharesmb` the SMB module's (issue 613): neither is on a list of §16, a value naming either
is refused (§11), and `zfs_config` never reports either as differing. The pool's files are the operator's (ADR
0051). Checked by the call-site test (§19) and §31.
## The config values and their forms ## The config values and their forms
9. **Every new value is "not managed" until set, and has a fixed form** (ADR 0679 §3, §4). Each value below has no 11. **Not managed until set and not empty; each of a fixed form** (ADR 0679 §3, §4). No new value has a default in
default in the manifest; absent means not managed: the module writes nothing, converges nothing and reports no the manifest. An absent value and an empty list both mean not managed: the module writes nothing, converges
drift for it. A value of another form is refused by the module's own parse, before any file is written or command nothing and reports no difference for it, so nothing is ever removed by a blank value. A value of another form is
run, emitting `config.refused` (§21). Trusted values (every list and object below, and `hostid`) are set on the refused by the module's own parse before any file is written or command run, emitting `config.refused` (§25).
operator's Approve (ADR 0348 §4, §8). The module never reads a value from the machine into config. Trusted values (every list and object below, `log-mirror` and `hostid`) are set only on the operator's Approve (ADR
- `arc-max`, `arc-min` (number): whole MiB, `arc-min` ≤ `arc-max`, `arc-max` at least 64 and below the machine's 0348 §4, §8). The module never reads a value from the node into config.
memory read from `/proc/meminfo`. ZFS's read cache in memory (the ARC) is bounded by them. - `arc-max`, `arc-min` (number): whole MiB, `arc-min` ≤ `arc-max`, `arc-max` at least 64 and below the node's
- `module-options` (list, trusted): entries `name=digits`, the name from the module's fixed list `l2arc_write_max`, memory read from `/proc/meminfo`. ZFS's read cache in memory (the ARC) is bounded by them.
`l2arc_write_boost`, `l2arc_headroom`, `l2arc_noprefetch`, `l2arc_rebuild_enabled`, `zfs_txg_timeout`, - `module-options` (list, trusted): entries `name=digits`, the name from the module's fixed list `l2arc_write_max`,
`zfs_dirty_data_max`, `zfs_scan_vdev_limit`, `zfs_scrub_min_time_ms`, `zfs_resilver_min_time_ms`, `l2arc_write_boost`, `l2arc_headroom`, `l2arc_noprefetch`, `l2arc_rebuild_enabled`, `zfs_txg_timeout`,
`zfs_arc_sys_free`; `zfs_arc_max` and `zfs_arc_min` are refused here, as `arc-max` and `arc-min` hold them. Reason `zfs_dirty_data_max`, `zfs_scan_vdev_limit`, `zfs_scrub_min_time_ms`, `zfs_resilver_min_time_ms`,
for the list: the ARC, L2ARC, transaction group and scrub parameters ADR 0679 §4 names, and nothing else. `zfs_arc_sys_free`; `zfs_arc_max` and `zfs_arc_min` are refused here, as `arc-max` and `arc-min` hold them. Reason:
- `zed` (string): `on` or `off`. `zed-handlers` (list, trusted): from `all-syslog` (the journal handler), the ARC, L2ARC, transaction group and scrub parameters ADR 0679 §4 names, and nothing else.
`resilver_finish-start-scrub` (scrub after resilver) and `history_event-zfs-list-cacher` (keeps the list cache - `hostid` (string, trusted): eight hexadecimal digits, not all zero (§15).
the mount reads); no notify handler is on the list (ADR 0679 §10). - `zed` (string): `on` or `off`. `zed-handlers` (list, trusted): from `all-syslog` (the journal handler) and
- `trim-schedule` (string): `none`, `weekly` or `monthly`; `trim-day` and `trim-time` with `scrub-day`'s and `resilver_finish-start-scrub` (scrub after resilver), the two ADR 0679 §12 names; no notify handler.
`scrub-time`'s forms. - `trim-schedule` (string): `none`, `weekly` or `monthly`; `trim-day` and `trim-time` with `scrub-day`'s and
- `cache-devices`, `log-devices` (lists, trusted): names under `/dev/disk/by-id`, letters, digits, `-`, `_`, `.` `scrub-time`'s forms.
and `:` only. `log-mirror` (boolean): when true, `log-devices` holds exactly two names, added as one mirror; - `cache-devices`, `log-devices` (lists, trusted): names under `/dev/disk/by-id`, letters, digits, `-`, `_`, `.`
otherwise each is a log device of its own. Reason: the record names log devices by stable name; a mirror is the and `:` only. `log-mirror` (boolean, trusted): when true, `log-devices` holds exactly two names, added as one
form the pool's log has today. mirror; otherwise each is a log device of its own. Reason: the log's form today is a mirror.
- `pool-properties` (object, trusted): `autotrim`, `autoexpand`, `autoreplace` (`on`, `off`, `default`) and - `pool-properties` (object, trusted): `autotrim`, `autoexpand`, `autoreplace` (`on`, `off`, `default`) and
`failmode` (`wait`, `continue`, `panic`, `default`). `failmode` (`wait`, `continue`, `panic`, `default`).
- `dataset-properties` (object, trusted): a dataset of the pool (the pool's name or one beginning `<pool>/`) to - `dataset-properties` (object, trusted): a dataset of the pool (the pool's name or one beginning `<pool>/`) to
properties of §14's list with their forms. properties of §16's list with their forms.
- `snapshots` (object, trusted): `datasets` (list of datasets of the pool; absent means every filesystem of the - `snapshots` (object, trusted): `none`, or keep counts `hourly`, `daily`, `weekly` and `monthly`, each a whole
pool), and keep counts `hourly` (0 to 48), `daily` (0 to 62), `weekly` (0 to 104), `monthly` (0 to 120); absent number from 0 to 168, a kind with 0 not taken; for every filesystem of the pool. The bound 168 (a week of hours)
or all zero means none. is the design's, so the ledger's cap (§26) stays bounded.
- `hostid` (string, trusted): `generate`, written once (§13). Checked by §31's refused-forms tests, one table per value, the empty list among them.
Checked by §25's refused-forms tests, one table per value.
## Files, units and the event daemon ## Files, units and the event daemon
10. **The node-engine places the module's files** (ADR 0679 §1, §4, §8), by §2 and §3. 12. **The options file** (ADR 0679 §1, §4). `/etc/modprobe.d/zfs.conf`, root's, 0644, carries `if-config: [arc-max,
`/etc/modprobe.d/zfs.conf` carries `if-config: [arc-max, arc-min, module-options]` and holds, by §3, a header arc-min, module-options]` (§2) and holds a header comment, then by §3 `options zfs zfs_arc_max=` and `options zfs
comment, then `options zfs zfs_arc_max=` and `options zfs zfs_arc_min=` with `arc-max` and `arc-min` in bytes zfs_arc_min=` with the values in bytes (`mib-bytes`), then `options zfs ` with each `module-options` entry. With
(`mib-bytes`), then `options zfs ` with each `module-options` entry; with none of the three set the file is not none of the three set it is not placed. Checked by the manifest test and §31's template test.
placed. The module's own `zed.rc` is placed in its folder (§12), not over the package's: it holds the package's 13. **The units** (ADR 0679 §1, §15; ADR 0117). Declared running and enabled at boot: `zfs-import-cache.service`,
variables but every `ZED_NOTIFY_*` and mail line, and `ZED_SCRUB_AFTER_RESILVER=1`, which only the handler `zfs-mount.service` and `zfs.target`, as they run on the home server today, so declaring them changes nothing.
`resilver_finish-start-scrub` reads, so it acts only where that handler is placed. The package's Declared with no state, owned and left as found: `zfs-import-scan.service`, `zfs-share.service`,
`/etc/zfs/zed.d/zed.rc` and `/etc/default/zfs` are left as the package wrote them: no value of ADR 0679 changes `zfs-volume-wait.service` and `zfs-import.target` (reason: ADR 0679 §15 keeps only the first three running when the
either, so writing them would only rewrite what ADR 0679 §3 leaves alone; `zfs_config` shows both as the module leaves, so no other ZFS unit is one the mesh would stop then). `zfs-zed.service`: running and enabled with
package's, and a `.pacnew` beside either as a warning. Every line is from a parsed value, never from free text, and `if-config: [zed=on]`, stopped and disabled with `if-config: [zed=off]`, left as found while `zed` is not managed.
a test refuses a newline, a quote, `$`, a backtick and a semicolon in every field (§25). Checked by the manifest The distribution's `zfs-trim-weekly@<pool>.timer` and `zfs-trim-monthly@<pool>.timer` are stopped and disabled
test and §25. with `if-config: [trim-schedule]` (the run trims, §20); its scrub timers are stopped and disabled after the run's
11. **The units.** The node-engine declares `zfs-import-cache.service`, `zfs-mount.service`, `zfs.target` and timer (§6). Read by `<node>/node-service-manager.status`. Checked by the manifest test (each unit, its state and
`zfs-import.target` running and enabled at boot, as they are on the home server today, so declaring them changes its condition).
nothing; `zfs-import-scan.service`, `zfs-share.service` and `zfs-volume-wait.service` are declared with no state 14. **ZED runs from the module's own folder, without links** (ADR 0679 §13, §12). The folder `/etc/zfs/mesh-zed.d`
(ADR 0117), owned and left as found. `zfs-zed.service` is declared twice by §2: running and enabled with (root's, 0755), its `zed.rc` and a drop-in for `zfs-zed.service` whose `ExecStart` adds `-d /etc/zfs/mesh-zed.d`
`if-config: [zed=on]`, stopped and disabled with `if-config: [zed=off]`, and so not declared, and left as found, carry `if-config: [zed=on]`. The module's `zed.rc` holds the package's variables but every `ZED_NOTIFY_*` and mail
while `zed` is not managed. The distribution's `zfs-trim-weekly@<pool>.timer` and `zfs-trim-monthly@<pool>.timer` line, and `ZED_SCRUB_AFTER_RESILVER=1`, which only the scrub-after-resilver handler reads. Each handler is a file
are stopped and disabled with `if-config: [trim-schedule]` (the module's run trims, §18). The distribution's scrub of its own in the folder, a copy of the package's script by its content kept in the module's files archive, mode
timers are stopped and disabled after the module's own (§5). Read by `<node>/node-service-manager.status`. 0755, with `if-config: [zed-handlers=<its name>]`, so only the handlers set are placed. The package's
Checked by the manifest test (each unit, its state and its condition). `/etc/zfs/zed.d` is not read while the drop-in holds. The mesh creates no symlinks. The node-engine restarts
12. **ZED's handlers without links** (ADR 0679 §11). The module's folder `/etc/zfs/mesh-zed.d` (root's, 0755), its `zfs-zed.service` on a change of the drop-in, a handler or `zed.rc` (`restart-on`). Checked by the manifest test (no
`zed.rc` (§10) and a drop-in for `zfs-zed.service` whose `ExecStart` adds `-d /etc/zfs/mesh-zed.d` carry link resource, each handler's condition).
`if-config: [zed=on]`. Each handler is a file of its own in that folder, a copy of the package's script by its 15. **The node's ZFS id** (ADR 0679 §1, §3, §4). `/etc/hostid` holds four bytes, which no template writes, so the apply
content kept in the module's files archive, mode 0755, with `if-config: [zed-handlers=<its name>]`, so only the (§18) writes it with ZFS's own writer, `zgenhostid -f <value>`, only while `hostid` is set, and only when the pool's
handlers set are placed. The package's links in `/etc/zfs/zed.d` stay as the package left them and are not read label (`zdb -C <pool>`, its `hostid`) is 0 or equals the value: then the next boot imports without `zpool import
while the drop-in holds. The mesh creates no symlinks. The node-engine restarts `zfs-zed.service` on a change of -f`. Otherwise it writes nothing and emits `config.refused` naming why ("the pool's import would need -f"). Before
the drop-in, a handler or `zed.rc` (`restart-on`). Checked by the manifest test (no link resource, each handler's the Approve, the value's description and `zfs_config` say the change needs a check of the pool's import at the next
condition) and a live `<node>/node-files.list` of the folder. boot; the module never reboots. After the reboot, `zfs_pools` shows the pool imported. Emits `config.applied` with
13. **The machine's ZFS id.** `/etc/hostid` is written only when `hostid` is `generate` and the file is absent, by the apply target `file`. Checked by §31 (unset: no `zgenhostid`; a label of another id: refused; 0 or equal: one run).
of §16 with `zgenhostid`; never rewritten. Before the Approve, the value's description and `zfs_config` say that
the pool must then be imported once by hand after the next reboot to check it imports under the new id; the
module never reboots. Emits `config.applied` with target `file`. Checked by a unit test (absent file: one
`zgenhostid`; present file: none).
## The pool ## The pool
14. **Pool and dataset properties** (ADR 0679 §5). For each set pool property the apply runs 16. **Pool and dataset properties** (ADR 0679 §6). For each set pool property the apply runs `zpool set
`zpool set <property>=<value> <pool>`, and for `default` `zpool set <property>=<ZFS's default> <pool>`. For each <property>=<value> <pool>`, and for `default` `zpool set <property>=<ZFS's default> <pool>`. For each set dataset
set dataset property it runs `zfs set <property>=<value> <dataset>`, and for `default` property it runs `zfs set <property>=<value> <dataset>`, and for `default` `zfs inherit <property> <dataset>`, so
`zfs inherit <property> <dataset>`, so its source stays default. The dataset list is `compression` (`off`, `lz4`, its source stays default. The dataset list is `compression` (`off`, `lz4`, `zstd`, `zstd-1` to `zstd-19`,
`zstd`, `zstd-1` to `zstd-19`, `gzip-1` to `gzip-9`, `zle`), `atime` and `relatime` (`on`, `off`), `recordsize` `gzip-1` to `gzip-9`, `zle`), `atime` and `relatime` (`on`, `off`), `recordsize` (a power of two from `512` to
(a power of two from `512` to `16M`, with `K` or `M`), `primarycache` and `secondarycache` (`all`, `metadata`, `16M`, with `K` or `M`), `primarycache` and `secondarycache` (`all`, `metadata`, `none`), `sync` (`standard`,
`none`), `sync` (`standard`, `always`, `disabled`), `logbias` (`latency`, `throughput`), `quota` and `always`, `disabled`), `logbias` (`latency`, `throughput`), `quota` and `reservation` (`none` or digits with `K`,
`reservation` (`none` or digits with `K`, `M`, `G` or `T`). A property already at its value runs nothing. Each `M`, `G` or `T`). A property already at its value runs nothing. Each change is read back with `zpool get -Hp` or
change is read back with `zpool get -Hp` or `zfs get -Hp` and emitted (§21). Checked by §25's convergence tests. `zfs get -Hp` and emitted (§25). Checked by §31's convergence tests.
15. **Cache and log devices** (ADR 0679 §5). From `zpool status -P -L <pool>` the apply knows each device's kernel 17. **Cache and log devices** (ADR 0679 §5). From `zpool status -P -L <pool>` the apply knows each device's kernel
path, and from the link under `/dev/disk/by-id` each declared name's kernel path. Then, in this order: path, and from the link under `/dev/disk/by-id` each declared name's. Then, in this order:
- an entry of the pool whose kernel path is a declared device's, but named by its kernel name, is removed - an entry of the pool whose kernel path is a declared device's, but named by its kernel name, is removed
(`zpool remove <pool> <entry>`) and added again by its stable name; (`zpool remove <pool> <entry>`) and added again by its stable name;
- a declared device not in the pool is added with `zpool add <pool> cache|log [mirror] <names>`, never with - a declared device not in the pool is added with `zpool add <pool> cache|log [mirror] <names>`, never `-f`, and
`-f`, and only after its checks: a device with a partition table or a file-system signature (`wipefs --no-act`) only after its checks: a device with a partition table or a file-system signature (`wipefs --no-act <device>`)
is refused, but for a `zfs_member` signature, which is judged by its label (`zdb -l`): a label of this pool is is refused, but for a `zfs_member` signature, which is judged by its label (`zdb -l <device>`): this pool's
allowed, another pool's refused; a device that is a member of any imported pool is refused; label is allowed, another pool's refused; a device that is a member of any imported pool is refused;
- a cache entry not declared is removed; - only while `cache-devices` is set and not empty, a cache entry it does not name is removed;
- a log device not declared, or one to rename, is removed only while the pool is ONLINE with no read, write or - only while `log-devices` is set and not empty, a log device it does not name, one to rename, or the log when
checksum errors and no scrub or resilver runs; otherwise it is a wait, `config.waiting` {key, waits-on, since}. `log-mirror` changes (a removal and a re-add, said as such) is removed only while the pool is ONLINE with no
read, write or checksum errors and no scrub or resilver runs; otherwise it is a wait, `config.waiting` {key,
waits-on, since}.
A declared device that is not present is a wait. Each change is read back from `zpool status -P` and emitted. A declared device that is not present is a wait. Each change is read back from `zpool status -P` and emitted.
Checked by §25's convergence tests and acceptance test 2. Checked by §31's convergence tests (the empty list among them) and acceptance test 2.
16. **Who applies, and when** (ADR 0679 §8). A `process` resource `zfs-config-apply`, `run: ./zfs-tools apply`, as 18. **The apply: who, when, and what wakes a wait** (ADR 0679 §9). A `process` resource `zfs-config-apply`, `run:
root, oneshot, ordered after `zfs-import.target` and enabled at boot; restarted by the node-engine on a change of ./zfs-tools apply`, as root, oneshot, ordered after `zfs-import.target`, enabled at boot, with `restart-on` the
the module's config file (`restart-on`), which is how a set value reaches it from mesh-config (ADR 0348 §5, §6); config file (§9); it converges §15 to §17 and writes the ARC values live. It holds an exclusive lock on
started by hand through `zfs_config_apply` (§22). It holds an exclusive lock on `/run/zfs-tools/apply.lock`; a `/run/zfs-tools/apply.lock`; a second start waits for it. It converges nothing while a scrub or resilver runs on
second start waits for it. It runs nothing while a scrub or resilver runs on the pool or before the pool is the pool or before the pool is imported: it emits `config.waiting` with what it waits on and since when, and ends.
imported: it emits `config.waiting` and the next start applies. It never runs on a timer. While the bus or the It never runs on a timer. **A wait is ended by an event:** when the event follower (§27) hears a scrub or a
control-node is down it applies the values its config file last held. It writes `arc-max` and `arc-min` live to resilver finish (ZFS's `scrub_finish` and `resilver_finish` events) or a declared device appear (a udev `add` of a
`/sys/module/zfs/parameters/zfs_arc_max` and `zfs_arc_min` as well as to §10's file and reads them back; each block device whose link under `/dev/disk/by-id` is declared, read with `udevadm monitor --udev
`module-options` entry whose live value differs is a warning naming it, "takes effect at the next module load", --subsystem-match=block`), it starts the apply unit through `<node>/node-service-manager.start`. While the bus or
never success and never unhealthy. the control-node is down, the apply converges from the config file it last got. It writes `arc-max` and `arc-min`
17. **Never converged, always asked** (ADR 0679 §6). No setting adds, replaces, attaches, detaches or removes a data to `/sys/module/zfs/parameters/zfs_arc_max` and `zfs_arc_min` as well as to §12's file, and reads both back; each
device, a spare, a special or a dedup device; none upgrades the pool, destroys a dataset, rolls back, renames or `module-options` entry whose live value differs is a warning naming it, "takes effect at the next load of the
clears. The call-site test (`TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`, renamed for what it now holds) kernel module", never success and never unhealthy.
allows `zpool add` only with `cache` or `log`, `zpool remove` only of a cache or log entry, `zpool set`, `zfs set` 19. **The call-site test** (ADR 0679 §7). Every command the bundle runs is built by one of its argument functions, and
and `zfs inherit` only for the lists of §9 and §14, `zfs destroy` only as §19 builds it, `zpool trim` only from the test (`TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`, renamed `TestTheBundleRunsOnlyTheFormsOfADR0679`)
the scheduled run, and refuses `upgrade`, `replace`, `attach`, `detach`, `clear`, `rollback`, `rename`, `-f`, `-r`, allows only: `zpool add <pool> cache|log`, `zpool remove` of a cache or log device, `zpool set`, `zfs set` and `zfs
`-R` and `-d` everywhere. inherit` of a listed property, `zfs snapshot`, `zfs set mesh:schedule=<kind>` on a ZFS snapshot named `mesh-`,
`zfs destroy -- <dataset>@<snapshot>`, `zpool scrub`, `zpool scrub -p` and `zpool scrub -s` of the pool, `zpool trim`
of the pool; and the reads `zpool status`, `zpool list`, `zpool get`, `zpool iostat`, `zpool events`, `zfs list`,
`zfs get` and `zfs holds`. It refuses every other `zpool` or `zfs` form and every `-f`, `-r`, `-R` and `-d`, so
data vdevs, spares, special and dedup devices, `upgrade`, a dataset's destroy, `rollback`, `rename` and `clear` are
never run (each an act of its own, asked of the operator). The same test pins the module's other root forms:
`wipefs --no-act <device>`, `zdb -l <device>`, `zdb -C <pool>`, `zgenhostid -f <value>`, `udevadm monitor --udev
--subsystem-match=block` and `systemctl show`. Fate: the README's list of allowed forms is replaced by this one.
## ZFS snapshots on a schedule ## The scheduled run and ZFS snapshots
18. **The scheduled run** (ADR 0679 §7, §14). The existing `process` resource `zfs-mesh-scrub` (`./zfs-tools 20. **The scheduled run** (ADR 0679 §9). The `process` resource `zfs-mesh-scrub` (`./zfs-tools scrub-run`) becomes
scrub-run`) becomes `zfs-mesh-run` (`./zfs-tools run`), root as today, and its `schedule` changes from `zfs-mesh-run` (`./zfs-tools run`), root as today, with §6's schedule. Each run takes the apply's lock; when the lock
`*/15 * * * *` to the list form `0,15,30,45 * * * *`. Reason: today's node-engine writes a step `*/15` as is held it does nothing and emits `run.skipped` {reason: `lock-held`}. It does nothing while a scrub or resilver
`*:*/15`, which the service manager refuses, so the timer failed; it writes the list form as runs or before the pool is imported, emitting `run.skipped` with that reason. Otherwise, in order: it starts the
`*-*-* *:0,15,30,45:00`, which is valid, so the module's timer is active without the cron-to-calendar fix (lead's scrub when due and returns at once, never waiting on it (today's wait with `zpool wait -t scrub` goes; the scrub's
answer, 2026-11-12). Checked by the module's manifest test (the schedule string) and by `systemd-analyze calendar` end is heard by the event follower, §27); starts a trim when `trim-schedule` is due (`zpool trim <pool>`, by
of the generated timer's `OnCalendar` on the home server passing with a next run within 15 minutes (acceptance `trim-day` and `trim-time` in the node's time zone, as the scrub reads its own); and takes and removes the ZFS
test 3). It and does three things each time, in snapshots (§21, §22) when `snapshots` is set. Checked by §31 (lock held: nothing run; scrub due: started and the
order: the scrub when due (as today), a trim when `trim-schedule` is due (`zpool trim <pool>`, by `trim-day` and run ends).
`trim-time` as the scrub reads its own), and the ZFS snapshots when `snapshots` is set. Reason: one schedule 21. **Taking scheduled ZFS snapshots** (ADR 0679 §8). Each kind whose keep count is above 0 is due at the first run of
resource the node-engine already runs, and no second timer. For the snapshots: at the first run of each hour it the hour (hourly), the first run after midnight (daily), the first run on Monday (weekly) and the first run on the
takes `hourly`, at the first run after 00:00 `daily`, on Monday `weekly`, on the first of the month `monthly` first of the month (monthly), each read in the node's time zone against the ledger's last entry of that kind (§26).
(only kinds whose keep count is above 0), one `zfs snapshot -- <dataset>@mesh-<kind>-<UTC time YYYYMMDDTHHMMZ>` For every filesystem of the pool it runs `zfs snapshot -- <dataset>@mesh-<kind>-<YYYYMMDDTHHMMZ>`, the time in UTC,
per dataset, never recursive, and sets on each `zfs set mesh:schedule=<kind>`. No ZFS snapshot is taken while the never recursive, then `zfs set mesh:schedule=<kind>` on it, and records the ledger's `taken` operation. The run
pool's capacity (`zpool list -Hp -o capacity`) is above 80 %: a warning says so, and `snapshot.skipped` takes no ZFS snapshot it cannot record in the same run: with the bus down or the ledger full it takes none and
{pool, capacity} is emitted. emits `snapshot.skipped.schedule` {pool, capacity, reason}. No ZFS snapshot is taken while the pool is more than 80
19. **Removed only by their own record.** After taking, per dataset and kind, the run removes the oldest past its keep % full (`zpool list -Hp -o capacity`): a warning says so, and `snapshot.skipped.schedule` is emitted with reason
count, only when it carries `mesh:schedule=<kind>`, is in the ledger `snapshots` (§21), has no hold `full`.
(`zfs holds`) and no clone (`clones` empty): `zfs destroy -- <dataset>@<snapshot>`, never `-r`, `-R` or `-d`. A 22. **Removing them, only by their own record** (ADR 0679 §8; the operator, 2026-11-12: "we may absolutely not lose any
ZFS snapshot taken by hand is never removed. Each removal emits `snapshot.removed` {snapshot, kind, space freed} data"). After taking, the run reads the ledger in the same run (no read, no removal) and, per dataset and kind,
and records the ledger's `removed` operation. removes the oldest past its keep count, all of them in that run. Before any destroy it checks the name against
20. **A person's ZFS snapshot stays apart.** `zfs_snapshot` refuses a name beginning `mesh-`, and emits `<a dataset of the pool>@mesh-<kind>-<YYYYMMDDTHHMMZ>`, that it carries `mesh:schedule=<kind>`, is in the ledger,
`snapshot.taken` with `by: person`; the run emits it with `by: schedule`. A waiter on a person's ZFS snapshot waits on has no hold (`zfs holds`) and no clone (`clones` empty); anything else is refused, emitted and left. Then `zfs
`snapshot.taken` with that snapshot's name, which no scheduled one carries. Fate: the README's "No tool of the destroy -- <dataset>@<snapshot>`, the ledger's `removed` operation, and `snapshot.removed.schedule` {snapshot, kind,
mesh destroys a snapshot" becomes §19's rule. space freed in bytes}. When `snapshots` goes back to `none`, the existing scheduled ZFS snapshots are kept, and a
warning names them; that change removes nothing. A ZFS snapshot taken by hand is never removed.
23. **A person's ZFS snapshot stays apart** (ADR 0679 §8, §10). `zfs_snapshot` refuses a name beginning `mesh-` and
emits `snapshot.taken.person`; the run emits `snapshot.taken.schedule`. A waiter on a person's ZFS snapshot waits
on `snapshot.taken.person`, which no scheduled one is said on. Fate: the README's "No tool of the mesh destroys a
snapshot" becomes §22's rule.
24. **Hand acts run as the run does** (ADR 0679 §14). `zfs_snapshots_run_now`, `zfs_snapshots_prune_now` and
`zfs_config_apply` each start a oneshot instance of the unit `zfs-mesh-run@<act>.service`, the act from the fixed
set `snapshots`, `prune` and `apply`, through `<node>/node-service-manager.start`, with `why`; the instance runs
under the same lock and guards as §18 and §20, and the tool answers from the state and events afterwards. Their
events carry `.person` (`snapshot.taken.person`, `snapshot.removed.person`, `snapshot.skipped.person`).
`zfs_snapshots_pause` writes the key `paused` {by, why, at} in the state `schedule`, which the run reads and then
takes and removes nothing; `zfs_snapshots_resume` deletes it.
## What it says, keeps and shows ## What it says, keeps and shows
21. **Events and kept data** (ADR 0679 §9; ADR 0363 §1; ADR 0595). The module declares a stream of its own, local 25. **Events, each kind on a subject of its own** (ADR 0679 §10). `config.applied` {target (`pool`, `dataset`,
name `zfs`, kept 30 days, carrying `config.applied` {target (`pool`, `dataset`, `file` or `unit`), key, before, `file` or `unit`), key, before, after, read-back}, `config.refused` {key, value, reason}, `config.waiting` {key,
after, read-back}, `config.refused` {key, value, reason}, `config.waiting` {key, waits-on, since}, waits-on, since}, `snapshot.taken.person` and `.schedule` {snapshot, by}, `snapshot.removed.person` and
`snapshot.taken` {snapshot, by}, `snapshot.removed`, `snapshot.skipped`, and today's `scrub.requested`; today's `.schedule` {snapshot, kind, space freed in bytes}, `snapshot.skipped.person` and `.schedule` {pool, capacity,
module-level `emits` move to it. Its kept data, in `requires` by template: a state `applied`, one key per config reason}, `run.skipped` {reason}, and today's `scrub.requested`. They go on an events stream asked for by template
value, `{before, after, at, outcome}`, written by the apply and deleted by it when that value returns to not in `requires` alone and created by the bus module (ADR 0595 §1, §2, §4), kept 30 days; today's module-level `emits`
managed (after its last `config.applied`); a ledger `snapshots`, one instance per ZFS snapshot the run took, move to it. No module consumes `snapshot.taken` or `scrub.requested` today (read in mesh-catalog's manifests on
operations `taken` and `removed`, kept by the ledger's template. The apply process and the run declare the bus main); a session waiting through `mesh_wait` on `snapshot.taken` hears `snapshot.taken.person` after.
account (`bus`) that writes them; the tools read them. 26. **Kept data, by template in `requires`** (ADR 0679 §10; ADR 0595 §1). A state `applied`, one key per config value,
22. **Tools** (ADR 0679 §12). `zfs_config`: every owned value with its setting, what runs on the machine, its state `{before, after, at, outcome}`, written by the apply and deleted by it when that value goes back to not managed,
(`not managed`, `as set`, `waiting` with on what and since, `refused` with why, `takes effect at the next module after its last `config.applied`. A state `schedule`, the key `paused` alone (§24). A ledger `snapshots`, one
load`), and a `.pacnew` beside an owned file as a warning. `zfs_config_apply`: starts §16's unit through instance per ZFS snapshot the run took, operations `taken` and `removed`, the run removing the instance of one it
`<node>/node-service-manager.start`, with `why`, and answers from the state `applied` afterwards. destroyed; its cap is the keep counts' sum times two instances per dataset. A state `findings` (§27). The apply,
`zfs_snapshots_schedule`: the schedule, its next run, what it would remove next and the ledger. the run and the event follower declare the bus account that writes them; the tools read them.
`zfs_snapshots_run_now` and `zfs_snapshots_prune_now`: a run's snapshot or removal step by hand, with `why`. 27. **The event follower's findings** (ADR 0679 §12; ADR 0564 §2; design 47 G §383 to §388). The event follower
`zfs_snapshots_pause` and `zfs_snapshots_resume`: a key `paused` in the state `applied`, read by the run, said in (`zfs-events`, root) is an observer: in its state `findings` it keeps one key `pool.<pool>.<device>` per device
the health while it holds. The existing read tools stay. Each is in `replaces` where it replaces a command. of the pool (and `pool.<pool>.pool` for the pool) not ONLINE, its value the state, deleted when the device is
23. **Health** (ADR 0679 §10). `zfs_health` is unhealthy for a `refused` or a failed command in the state `applied`, ONLINE again or leaves the pool; it says each change on its findings stream as design 47 G §383 says, and at its
naming the key and why; warnings: a wait (naming on what and since when), the last scheduled ZFS snapshot older start it reads `zpool status -P` against its keys and states them (§384). The condition keeper raises and clears
than its cadence, a failed run, `autoreplace` on or a spare in the pool while `zed` is not `on`, the pool above the condition from those events, so a pool fault reaches the operator once and never from a ZED handler. It also
80 % with snapshots set, a parameter awaiting a module load. Pool faults reach the operator once, never from a ZED starts the apply on the events of §18. Its process declares the bus account; today it writes the journal only.
handler: the event follower (`zfs-events`) is an observer, writing a finding `<pool>.<vdev or pool>.<state>` in its 28. **Tools** (ADR 0679 §14). `zfs_config`: every owned value with its config value, what runs on the node, its state
bucket `findings` for a pool or device that leaves ONLINE, saying each as an event, and removing it when the state (`not managed`, `as set`, `waiting` with on what and since, `refused` with why, `takes effect at the next load of
is ONLINE again; the condition keeper raises and clears the condition from it (ADR 0564 §2; design 47 G §383). Its the kernel module`), and the files of §10. `zfs_config_apply` (§24). `zfs_snapshots_schedule`: the schedule, its
process declares the bus account that writes them; today it writes the journal only. next run, what it would remove next and the ledger. `zfs_snapshots_run_now`, `zfs_snapshots_prune_now`,
24. **When the module leaves** (ADR 0679 §13; ADR 0303 §2; ADR 0269). The packages stay. The import, mount and target `zfs_snapshots_pause` and `zfs_snapshots_resume` (§24). `zfs_module_events`: the module's stream, the last N events
units stay enabled and running. `/etc/default/zfs` and `zed.rc` get back their found content (ADR 0118); by kind (named so because `zfs_events` already reads ZFS's own events). `zfs_snapshots_check`: compares the ledger
`zfs.conf`, the folder `/etc/zfs/mesh-zed.d` and the drop-in are removed, and before leaving `zfs_config` says that with what ZFS holds and names each entry whose ZFS snapshot is gone and each `mesh-` snapshot not in the ledger,
the next module load uses ZFS's defaults. The distribution's scrub timer is given back as found (ADR 0269). Pool changing nothing. `zfs_findings`: the follower's findings. The existing read tools stay. Each is in `replaces`
and dataset properties and devices stay as the last apply set them, also on a rollback of the module's version. where it replaces a command.
29. **Health** (ADR 0679 §11). `zfs_health` is unhealthy for a `refused` or a failed command in the state `applied`,
naming the key and why. Warnings: a wait, naming on what and since when; a `.pacnew` beside `/etc/default/zfs` or
the package's `zed.rc`; the last scheduled ZFS snapshot older than its cadence; a failed run; `autoreplace` on or a
spare in the pool while `zed` is not `on`; the pool above 80 % with `snapshots` set; scheduled ZFS snapshots kept
after `snapshots` went to `none`; a parameter awaiting a load of the kernel module; `paused` set.
30. **When the module leaves** (ADR 0679 §15; ADR 0303 §2; ADR 0269 §2). The packages stay. `zfs-import-cache`,
`zfs-mount` and `zfs.target` are never stopped or disabled, and the module says so before it leaves (ADR 0679 §15
changes ADR 0269 §10 for them). Nothing of the package's files is given back, since the module never wrote them.
`zfs.conf`, `/etc/zfs/mesh-zed.d` and the drop-in are removed, and before leaving `zfs_config` says that the next
load of the kernel module uses ZFS's defaults. The distribution's scrub and trim timers are put back as the
module's declaration found them, started and enabled again where it stopped or disabled them. Pool and dataset
properties and devices stay as the last apply set them, also on a rollback (§8).
## How it is checked ## How it is checked
25. **Unit tests**, table-driven, one test per area, within the module's 500: no settings gives no `zpool` or `zfs` 31. **Unit tests**, table-driven, one test per area, within the module's 500: no values gives no `zpool` or `zfs`
command that writes; each convergence plan from parsed `zpool status` and `zfs get` output (an add, a kernel name write but the due scrub; each convergence plan from parsed `zpool status` and `zfs get` output (an add, a kernel
to its stable name, a removal, a log removal refused while not healthy, a wait during a scrub); every refused form name to its stable name, a removal, an empty list removing nothing, a log removal refused while not healthy, a
(a newline, quote, `$`, backtick or semicolon in each field; a parameter not on the list; a property not on the `log-mirror` change, a wait during a scrub); every refused form (a newline, quote, `$`, backtick or semicolon in
lists; a `sharenfs` setting; `-f`; a device with a signature or another pool's label); the call-site test of §17; each field; a parameter or property not on its list; a `sharenfs` value; `-f`; a device with a signature or another
pruning never removes a `mesh-` ZFS snapshot without the property or not in the ledger, nor one with a hold or pool's label; a hostid of the wrong form or a label of another id); the call-site test of §19; the lock (a second
clone; `zfs_snapshot` refuses a `mesh-` name; a parameter awaiting a module load is a warning; each change emits apply waits, a run with the lock held does nothing); the apply started only by `config-changed`, never a timer;
its event with before and after; a scheduled `snapshot.taken` does not match a person's waiter. The module has no one scheduled resource; both ARC writes read back; applying from the last config file with the bus down; the
integration test; the live checks stand for it, said in the pull request. three templates (config file, options file, `zed.rc`); pruning never removes a ZFS snapshot whose name, property
26. **Live, on the home server**: the main file's acceptance tests 1 to 3, in that order. or ledger entry does not fit, nor one with a hold or clone, and removes every surplus in one run; no ZFS snapshot
taken without a ledger record; `snapshots` back to `none` removes nothing; `zfs_snapshot` refuses a `mesh-`
name; a scheduled `snapshot.taken.schedule` does not wake a person's waiter; a parameter awaiting a load is a
warning; the `.pacnew` warning; each change emits its event with before and after; one test per tool. The module
adds no integration test; §5 is the milestone's, and the main file's live tests stand for the module whole, said
in the pull request.
03-DESIGN/01-to-be/42-the-machines-modules-in-order.md
+60 -41
View File
@@ -33,8 +33,8 @@ decisions:
- "ADR 0051" - "ADR 0051"
- "ADR 0269" - "ADR 0269"
- "ADR 0348" - "ADR 0348"
- "ADR 0363"
- "ADR 0595" - "ADR 0595"
- "ADR 0593"
--- ---
# 42. The machines' modules, in order # 42. The machines' modules, in order
@@ -157,68 +157,87 @@ on the machine; it is placed only on a machine that holds a pool. Its section is
handles the rest"; "the zfs module should/must own all zfs-configuration files and options"; "we may absolutely not handles the rest"; "the zfs module should/must own all zfs-configuration files and options"; "we may absolutely not
lose any data".* lose any data".*
**What it is for.** On a machine that holds a ZFS pool, the zfs module owns ZFS: what was installed and set by hand **What it is for.** On a node that holds a ZFS pool, the zfs module owns ZFS: what was installed and set by hand and
and seen by nothing becomes the module's, and the operator's choices about it are config values (ADR 0679 §1). It seen by nothing becomes the module's, and the operator's choices about it are config values (ADR 0679 §1). It holds
holds no seat (see the open question of its milestone part). It keeps reading and acting by hand as today: the pools, no seat; a seat for it is a later record's. It keeps reading and acting by hand as today: the pools, the datasets, a
the datasets, a named ZFS snapshot, a scrub's start, pause and stop. named ZFS snapshot, a scrub's start, pause and stop.
**The whole shape.** **The whole shape.**
- **What it owns** (ADR 0679 §1): the packages `zfs-utils` and `zfs-dkms`; the files `/etc/default/zfs`, - **What it owns** (ADR 0679 §1): the packages `zfs-utils` and `zfs-dkms`; `/etc/modprobe.d/zfs.conf`, placed only
`/etc/zfs/zed.d/zed.rc` and `/etc/modprobe.d/zfs.conf`; which handlers ZFS's event daemon (ZED) runs; the states of while an ARC or module option is set; `/etc/hostid`; its own folder for ZFS's event daemon (ZED),
ZFS's import, mount, share, volume and target units and of the distribution's scrub and trim timers; and, for the one `/etc/zfs/mesh-zed.d`, with its `zed.rc` and handlers, and a drop-in for `zfs-zed`, placed only while ZED is on;
pool its config value `pool` names, that pool's properties, its cache and log devices and its datasets' properties the states of ZFS's import, mount, share, volume and target units and of the distribution's scrub and trim timers;
from a fixed list. It reads, and never writes, what ZFS writes itself (its pool cache file and list cache). and, for the one pool its config value `pool` names, that pool's cache and log devices, its properties and its
- **What is not its own** (ADR 0679 §2): a dataset's `sharenfs` is the NFS share module's (design 52), `sharesmb` the datasets' properties from fixed lists. The node-engine's composer refuses a second module owning one of its paths
SMB module's once there is one; the pool's files are the operator's (ADR 0051); the weekly upgrade and a kernel (ADR 0593 §7, ADR 0051 §9).
held for DKMS are the package manager's (ADR 0508). - **What it never writes** (ADR 0679 §1, §2): the package's own `/etc/default/zfs` and `/etc/zfs/zed.d` with its
- **An unset value changes nothing** (ADR 0679 §3): every new config value is "not managed" until the operator sets `zed.rc`, and what ZFS writes itself; it reads and shows them. A dataset's `sharenfs` is the seat
it through mesh-config, on the Approve for a trusted list or object value (ADR 0348 §4, §8). Placing the new version `node-nfs-server`'s, `sharesmb` the SMB module's; the pool's files are the operator's (ADR 0051); the weekly
on a machine runs no `zpool` or `zfs` command that writes and rewrites none of its files. upgrade and a kernel held for DKMS are the package manager's (ADR 0508).
- **Who applies** (ADR 0679 §8): the node-engine places the packages, files and units as the module's resources; - **An unset or empty value changes nothing** (ADR 0679 §3): every new config value is "not managed" until the
the module's root process `zfs-tools apply` converges the pool's devices and properties at its start and when its operator sets it through mesh-config, on the Approve for a trusted value (ADR 0348 §4, §8). Placing the new version
config changes, never on a timer; the module's scheduled run takes and removes the scheduled ZFS snapshots on a node runs no `zpool` or `zfs` write but the scrub its schedule already runs, and places none of its new files.
(ADR 0679 §7). - **Who applies** (ADR 0679 §9): the node-engine places the packages, files and units as the module's resources,
- **What it says** (ADR 0679 §9, §10): every change, refusal and wait is an event on the module's own stream, and some only while a value holds; the module's root process `zfs-tools apply` converges the pool's devices and
the last change per value and every scheduled ZFS snapshot are kept on the bus in its own state and ledger, asked properties at its start and on `config-changed`, never on a timer; the module's one scheduled run, under the same
for by template (ADR 0595). Its health names a refusal or a failed command as unhealthy and a wait as a warning. lock, starts the scrub and the trim when due and takes and removes the scheduled ZFS snapshots (ADR 0679 §8).
Pool faults reach the operator once, from the module's event follower through the condition keeper. - **What it says** (ADR 0679 §10, §11, §12): every change, refusal and wait is an event, each kind on a subject of
- **What it never does** (ADR 0679 §6): it converges no data device, spare, special or dedup device, upgrades no its own, on an events stream asked for by template; the last change per value, the schedule's pause and every
scheduled ZFS snapshot are kept on the bus in its own states and ledger (ADR 0595). Its health names a refusal or a
failed command as unhealthy and a wait as a warning. Pool faults reach the operator once, from the module's event
follower's findings through the condition keeper.
- **What it never does** (ADR 0679 §7): it converges no data device, spare, special or dedup device, upgrades no
pool, destroys no dataset, rolls back, renames and clears nothing; each is an act of its own, decided separately. pool, destroys no dataset, rolls back, renames and clears nothing; each is an act of its own, decided separately.
- **When it leaves** (ADR 0679 §13): ZFS keeps running, the packages stay, and the files are given back. - **When it leaves** (ADR 0679 §15): ZFS keeps running (the import, mount and ZFS target are never stopped), the
packages stay, its own files are removed and the distribution's timers are put back as found.
- **Its prerequisites** (ADR 0679 §16): three node-engine capabilities, built first in the same milestone under issue
1979, with their readers in mesh-sdk and the manifest validator.
```mermaid ```mermaid
flowchart LR flowchart LR
operator[the operator] -->|sets a config value| config[mesh-config] operator[the operator] -->|sets a config value| config[mesh-config]
config -->|the module's config file| engine[the node-engine] config -->|the module's config file| engine[the node-engine]
engine -->|packages, files, units| machine[ZFS on the machine] engine -->|packages, files, units| node[ZFS on the node]
engine -->|restarts on its config file| apply[zfs-tools apply, root] engine -->|restarts on its config file| apply[zfs-tools apply, root]
apply -->|zpool add, remove, set; zfs set, inherit| machine apply -->|zpool add, remove, set; zfs set, inherit| node
sched[the scheduled run] -->|ZFS snapshots, their removal, scrub, trim| machine sched[the scheduled run] -->|scrub, trim, ZFS snapshots and their removal| node
apply -->|config.applied, refused, waiting| stream[(the module's stream)] follower[the event follower] -->|scrub ended, device appeared: start| apply
sched -->|snapshot.taken, snapshot.removed| stream apply -->|config.applied, refused, waiting| stream[(events stream)]
sched -->|snapshot.taken.schedule, removed, skipped| stream
apply --> applied[(state applied)] apply --> applied[(state applied)]
sched --> ledger[(ledger snapshots)] sched --> ledger[(ledger snapshots)]
follower[the event follower] -->|pool faults| keeper[the condition keeper] follower -->|findings| keeper[the condition keeper]
tools[the module's tools] -->|read| applied tools[the module's tools] -->|read| applied
tools -->|read| ledger tools -->|read| ledger
``` ```
**Acceptance tests** (ADR 0679, *How it is checked*): **Acceptance tests** (ADR 0679, *How it is checked*), each with its level; every live test is read by a tool call:
1. Placing the new version on the home server changes nothing: `zfs_config` shows every value "not managed", and 1. *Live.* Placing the new version on the home server changes nothing: `zfs_config` shows every value "not managed",
no `zpool` or `zfs` command that writes ran (the module's journal and its stream show none). and `zpool history` read before and after shows no command but the scheduled scrub.
2. Setting `cache-devices` puts both cache devices in the pool by their stable names; after a reboot that swaps the 2. *Live.* Setting `cache-devices` puts both cache devices in the pool by their stable names; after a reboot that
drives' kernel names, both stay ONLINE (`zfs_pool_layout`). swaps the drives' kernel names, both stay ONLINE (`zfs_pool_layout`).
3. The module's own scrub timer is active before the distribution's scrub timer is stopped, read from 3. *Live.* The node-engine's apply report shows the module's run timer active before the distribution's scrub timer
`<node>/node-service-manager.status` of both. is stopped, and `systemd-analyze calendar` of the generated `OnCalendar` passes with a next run within 15 minutes.
4. The unit tests of the milestone part's line 25 pass, within the module's budget of 500 unit tests. 4. *Live.* With `hostid` unset, `/etc/hostid` is not written (`<node>/node-reads.stat`); `zfs_config` shows it "not
managed".
5. *Live.* With `snapshots` set, the next run's ZFS snapshot is in `zfs_snapshots` with `mesh:schedule` and in the
ledger (`zfs_snapshots_schedule`); after its keep count passes, its removal is in both and in `zfs_module_events`.
6. *Live.* A value of a refused form makes `zfs_health` unhealthy naming the key; a declared device that is not
present makes it a warning naming what it waits on.
7. *Live.* With `zed` on, `<node>/node-files.list` of `/etc/zfs/mesh-zed.d` shows its `zed.rc` and only the handlers
set, and `<node>/node-service-manager.status` shows `zfs-zed` running with the drop-in; with `zed` off, stopped.
8. *Live, on a test placement.* After the module leaves, `zfs-import-cache`, `zfs-mount` and `zfs.target` are still
running (`<node>/node-service-manager.status`) and `zfs_pools` shows the pool imported.
9. *Unit and integration.* The milestone part's §31 unit tests pass within the module's budget of 500, and its §5
integration test passes in the node-engine.
## Milestones ## Milestones
| milestone | what it builds | part | state | | milestone | what it builds | part | state |
|---|---|---|---| |---|---|---|---|
| zfs-owns-zfs | the zfs module's ownership of ZFS (ADR 0679), with its prerequisites in the node-engine and the catalog, under issue 1979 | [42-milestones/zfs-owns-zfs.md](42-milestones/zfs-owns-zfs.md) | designed | | zfs-owns-zfs | the zfs module's ownership of ZFS (ADR 0679), with its three node-engine prerequisites and their readers, under issue 1979 | [42-milestones/zfs-owns-zfs.md](42-milestones/zfs-owns-zfs.md) | designed |
The modules of the tables above were built before designs were cut into milestone parts; their remaining lines move The modules of the tables above were built before designs were cut into milestone parts; their remaining lines move
into a part of their own at their next amendment (playbook 02, *The shape of a design*). into a part of their own at their next amendment (playbook 02, *The shape of a design*).