novox/hq
9
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Design 42: the zfs module owns a machine's ZFS and changes nothing unset (ADR 0679)

Browse Source 
Adds row 12 and a section to the main file, and the milestone part
zfs-owns-zfs with numbered lines and its three prerequisites, so a
builder takes ownership of ZFS's packages, files, units and one pool's
devices, properties and scheduled ZFS snapshots without a choice of
its own. Leaves one design question open (a resource that follows a
value only once it is set). Adds ZFS snapshot to the glossary beside
the two snapshots it already names.
This commit is contained in:
jschoubben
2026-11-12 22:32:54 +01:00
parent 4eba1f451b
commit 6772904668
3 changed files with 352 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
00-META/glossary.md
+7
View File
@@ -1020,6 +1020,12 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283, 0564, 0668.
`recreate` of a protected stream, in place (design 47 G §681), and the backup holder only stores copies (design 47 G §554).
**aside** — live kept data a restore set aside, removed only by the hand tool `recovery_discard_aside`, which the operator and sessions call.
*Not:* ~~restorer~~, ~~restore account~~
- **ZFS snapshot** — a read-only point in time of one ZFS dataset on a machine, kept inside that dataset's pool, so
it guards against a mistake and never against the pool's loss; taken by a person with `zfs_snapshot`, or by the zfs
module's scheduled run, which names its own `mesh-<kind>-<time>`, marks them and removes only those past their keep
count (ADR 0679 §7; design 42 zfs-owns-zfs §17, §18). Never bare, beside a ledger's snapshot and the stream snapshot
(see [Homonyms](#homonyms)); not a backup or a restore point, which are kept elsewhere.
*Not:* ~~pool snapshot~~, ~~dataset snapshot~~
- **disk readings** — the size of each data item a node's modules declare, measured by its `measure` and kept
with its largest size in seven days by disk-load on that node, in its own state, one key
`<node>.<module>.<item>` written only when it changes; the holder of `node-backup`, mounts, nfs-server and the
@@ -1452,6 +1458,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
| | anything a module reads that someone may change | **config value** | Config |
| snapshot | the latest state of one ledger instance, from which a rebuild starts | **snapshot** of a ledger's state | Data |
| | the bus's own backup of each stream | **stream snapshot** | Data |
| | a read-only point in time of one ZFS dataset, inside its pool | **ZFS snapshot** | Data |
| template | a storage type with a configuration, asked for in `requires` | **template** of bus state | Data |
| | a file with placeholders a provisioner fills | **template file** | Config |
| option | a setting of a template a module may change | **option** of a template | Data |
03-DESIGN/01-to-be/42-milestones/zfs-owns-zfs.md
+261
View File
@@ -0,0 +1,261 @@
---
layer: to-be
design: 42
milestone: zfs-owns-zfs
updated: 2026-11-12
code: [mesh-catalog modules/zfs, mesh-catalog checks, mesh-host internal/apply]
decisions:
- "ADR 0679"
- "ADR 0051"
- "ADR 0117"
- "ADR 0118"
- "ADR 0269"
- "ADR 0303"
- "ADR 0348"
- "ADR 0363"
- "ADR 0508"
- "ADR 0564"
- "ADR 0595"
---
# Design 42, milestone zfs-owns-zfs — the zfs module owns a machine's ZFS, and changes nothing that is not set
**What it builds.** On a machine that holds a ZFS pool (today the home server), the zfs module owns ZFS's packages,
files, units and, for one pool, its properties, cache and log devices, its datasets' properties and a schedule of
ZFS snapshots; each is changed only once its config value is set (ADR 0679). The purpose and the whole shape are the
main file's, [*The zfs module*](../42-the-machines-modules-in-order.md#the-zfs-module). This part holds every line
the milestone builds, cited as "design 42 zfs-owns-zfs §n".
**Where.** Three repositories, one delivery group with one check: `mesh-host` (the node-engine: §1, §2),
mesh-catalog's checks (§3) and mesh-catalog's `modules/zfs` (§4 to §25). **Ordering edge:** the zfs module's new
version needs the node-engine of §1 and §2 on its machine (a node-engine capability edge the delivery plan derives,
§4); §3 has no edge, it gates the catalog's own check.
**Open before develop.** Lines §9, §10 and §11 carry an open design question (Q1 at the end of this part); their
subtasks wait for its answer. Nothing else in this part waits on it.
## The prerequisites
1. **The node-engine writes a valid calendar for a cron step.** The node-engine's `calendarFor` (the schedule of a
`process` resource turned into a timer's `OnCalendar`) writes a field `*/N` as `0/N` for the minute and the hour
and as `1/N` for the day of the month and the month; the day of the week is unchanged. Today `*/15 * * * *` becomes
`*-*-* *:*/15:00`, which the service manager refuses, so the zfs module's scrub timer fails (ADR 0679 §14).
Checked by a table test of `calendarFor` over `*/15 * * * *` (`*-*-* *:0/15:00`), `0 */6 * * *`
(`*-*-* 0/6:00:00`), `0 0 */2 * *` (`*-*-1/2 00:00:00`) and the existing cases unchanged.
2. **The node-engine's apply of a module stops at its first failed resource.** When one resource of a module fails,
the node-engine applies none of that module's later resources in the declared order; the report names the
failed resource with its error and each one left unapplied, and the module's phase is `failed`. Resources already
applied stay as applied. Other modules on the node go on as today. Today the apply goes on, and stopped the
distribution's scrub timer after the module's own timer failed (ADR 0679 §14). Checked by a unit test: a module of
three resources whose second fails leaves the third untouched and reports it as left.
3. **The catalog refuses a second writer of one file.** mesh-catalog's checks refuse a catalog in which two
modules' manifests declare a `file` resource at one path, naming both modules and the path (ADR 0679 §14; ADR 0051
§9, rival ownership). A file only read is not a writer. Checked by the check's own test over two manifests that
declare one path (refused) and two that declare different paths (passed); it runs in the catalog's check on
every pull request.
4. **The order on the machine.** The zfs module's new version declares its own timer resource (the scheduled run of
§17) before the resources that stop the distribution's scrub timers, so by §2 those are stopped only once the
module's own timer is active (ADR 0679 §14). The delivery plan places the node-engine of §1 and §2 on the home
server before the module (the capability edge above). Checked by the module's manifest test, which pins the
resource order, and by acceptance test 3 of the main file.
## What the module owns
5. **The packages.** The manifest declares `zfs-utils` and `zfs-dkms` present (ADR 0679 §1; ADR 0303 §2: the
node-engine installs a missing one, upgrades none, removes none). `zfs-utils-debug` is not declared and is left as
found. Fate: the README's "It installs nothing" goes; it says the two packages and why the debug package is not
the module's. Read by `pacman_packages` as declared by `zfs`. Checked by the manifest test.
6. **One pool, renamed `pool`.** The config value `scrub-pool` becomes `pool` (ADR 0679 §1): type string, trusted,
required, same description but for the rename. It is moved by mesh-config's rename window (mesh-config answers both
names meanwhile; its seat verb `renames` says the window), and every template and the module's code read `pool`.
One pool per assignment; the manifest has no per-pool form and none is added. The module reads, never writes,
`/etc/zfs/zpool.cache` and `/etc/zfs/zfs-list.cache`. Checked by the manifest test (no `scrub-pool` left) and the
rename's live read in `zfs_config`.
7. **What is not its own.** `sharenfs` and `sharesmb` are on no list of §13; a setting naming them is refused
(§8), and `zfs_config` never reports either as drift (ADR 0679 §2). The pool's files are the operator's (ADR
0051). Checked by §24's refused-forms test.
## The config values and their forms
8. **Every new value is "not managed" until set, and has a fixed form** (ADR 0679 §3, §4). Each value below has no
default in the manifest; absent means not managed: the module writes nothing, converges nothing and reports no
drift for it. A value of another form is refused by the module's own parse, before any file is written or command
run, emitting `config.refused` (§18). Trusted values (every list and object below, and `hostid`) are set on the
operator's Approve (ADR 0348 §4, §8). The module never reads a value from the machine into config.
- `arc-max`, `arc-min` (number): whole MiB, `arc-min` ≤ `arc-max`, `arc-max` at least 64 and below the machine's
memory read from `/proc/meminfo`. ZFS's read cache in memory (the ARC) is bounded by them.
- `module-options` (list, trusted): entries `name=digits`, the name from the module's fixed list `l2arc_write_max`,
`l2arc_write_boost`, `l2arc_headroom`, `l2arc_noprefetch`, `l2arc_rebuild_enabled`, `zfs_txg_timeout`,
`zfs_dirty_data_max`, `zfs_scan_vdev_limit`, `zfs_scrub_min_time_ms`, `zfs_resilver_min_time_ms`,
`zfs_arc_sys_free`; `zfs_arc_max` and `zfs_arc_min` are refused here, as `arc-max` and `arc-min` hold them. Reason
for the list: the ARC, L2ARC, transaction group and scrub parameters ADR 0679 §4 names, and nothing else.
- `zed` (string): `on` or `off`. `zed-handlers` (list, trusted): from `all-syslog` (the journal handler),
`resilver_finish-start-scrub` (scrub after resilver) and `history_event-zfs-list-cacher` (keeps the list cache
the mount reads); no notify handler is on the list (ADR 0679 §10).
- `trim-schedule` (string): `none`, `weekly` or `monthly`; `trim-day` and `trim-time` with `scrub-day`'s and
`scrub-time`'s forms.
- `cache-devices`, `log-devices` (lists, trusted): names under `/dev/disk/by-id`, letters, digits, `-`, `_`, `.`
and `:` only. `log-mirror` (boolean): when true, `log-devices` holds exactly two names, added as one mirror;
otherwise each is a log device of its own. Reason: the record names log devices by stable name; a mirror is the
form the pool's log has today.
- `pool-properties` (object, trusted): `autotrim`, `autoexpand`, `autoreplace` (`on`, `off`, `default`) and
`failmode` (`wait`, `continue`, `panic`, `default`).
- `dataset-properties` (object, trusted): a dataset of the pool (the pool's name or one beginning `<pool>/`) to
properties of §13's list with their forms.
- `snapshots` (object, trusted): `datasets` (list of datasets of the pool; absent means every filesystem of the
pool), and keep counts `hourly` (0 to 48), `daily` (0 to 62), `weekly` (0 to 104), `monthly` (0 to 120); absent
or all zero means none.
- `hostid` (string, trusted): `generate`, written once (§12).
Checked by §24's refused-forms tests, one table per value.
## Files, units and the event daemon
9. **The node-engine places the module's files** (ADR 0679 §1, §4, §8). `/etc/modprobe.d/zfs.conf` holds one
`options zfs` line per set value: `zfs_arc_max` and `zfs_arc_min` in bytes from `arc-max` and `arc-min`, then each
`module-options` entry; with nothing set it holds only its header comment, which sets nothing. `/etc/zfs/zed.d/zed.rc`
holds the package's own lines with `ZED_SCRUB_AFTER_RESILVER=1` when `resilver_finish-start-scrub` is among the
handlers, and no `ZED_NOTIFY_*` or mail line. `/etc/default/zfs` holds the package's own content; no value of ADR
0679 changes it, so it is the module's only to have one writer. Every line is generated from parsed values, never
from free text, and a test refuses a newline, a quote, `$`, a backtick and a semicolon in every field (§24). A file
the node-engine writes over keeps its original, put back when the module leaves (ADR 0118, §22). **Open: Q1** (how a
file is written only when its value is set, and how one line per list entry is rendered).
10. **The units.** The node-engine declares `zfs-import-cache.service`, `zfs-mount.service`, `zfs.target` and
`zfs-import.target` running and enabled at boot, as they are on the home server today, so declaring them changes
nothing; `zfs-import-scan.service`, `zfs-share.service` and `zfs-volume-wait.service` are declared with no state
(ADR 0117), owned and left as found; `zfs-zed.service` is running and enabled when `zed` is `on`, stopped and
disabled when `off`, and left as found when not managed; the distribution's `zfs-trim-weekly@<pool>.timer` and
`zfs-trim-monthly@<pool>.timer` are stopped and disabled once `trim-schedule` is set (the module's run trims,
§17), and the distribution's scrub timers stopped and disabled after the module's own (§4). Read by
`<node>/node-service-manager.status`. **Open: Q1** (a unit state that follows a value only once it is set).
11. **ZED's handlers without links** (ADR 0679 §11). With `zed` on, the module places each handler of `zed-handlers`
as a file in its own folder `/etc/zfs/mesh-zed.d` (a copy of the package's script by its content, mode 0755,
root's) and a drop-in for `zfs-zed.service` whose `ExecStart` adds `-d /etc/zfs/mesh-zed.d`; `zed.rc` is read
from there too, as a copy of §9's. The package's links in `/etc/zfs/zed.d` stay as the package left them and are
not read while the drop-in holds. The mesh creates no symlinks. The node-engine restarts `zfs-zed.service` on a
change of the drop-in, a handler or `zed.rc`. Checked by the manifest test (no link resource) and a live
`<node>/node-files.list` of the folder. **Open: Q1** (placed only when `zed` is on).
12. **The machine's ZFS id.** `/etc/hostid` is written only when `hostid` is `generate` and the file is absent, by the apply
of §15 with `zgenhostid`; never rewritten. Before the Approve, the value's description and `zfs_config` say that
the pool must then be imported once by hand after the next reboot to check it imports under the new id; the
module never reboots. Emits `config.applied` with target `file`. Checked by a unit test (absent file: one
`zgenhostid`; present file: none).
## The pool
13. **Pool and dataset properties** (ADR 0679 §5). For each set pool property the apply runs
`zpool set <property>=<value> <pool>`, and for `default` `zpool set <property>=<ZFS's default> <pool>`. For each
set dataset property it runs `zfs set <property>=<value> <dataset>`, and for `default`
`zfs inherit <property> <dataset>`, so its source stays default. The dataset list is `compression` (`off`, `lz4`,
`zstd`, `zstd-1` to `zstd-19`, `gzip-1` to `gzip-9`, `zle`), `atime` and `relatime` (`on`, `off`), `recordsize`
(a power of two from `512` to `16M`, with `K` or `M`), `primarycache` and `secondarycache` (`all`, `metadata`,
`none`), `sync` (`standard`, `always`, `disabled`), `logbias` (`latency`, `throughput`), `quota` and
`reservation` (`none` or digits with `K`, `M`, `G` or `T`). A property already at its value runs nothing. Each
change is read back with `zpool get -Hp` or `zfs get -Hp` and emitted (§18). Checked by §24's convergence tests.
14. **Cache and log devices** (ADR 0679 §5). From `zpool status -P -L <pool>` the apply knows each device's kernel
path, and from the link under `/dev/disk/by-id` each declared name's kernel path. Then, in this order:
- an entry of the pool whose kernel path is a declared device's, but named by its kernel name, is removed
(`zpool remove <pool> <entry>`) and added again by its stable name;
- a declared device not in the pool is added with `zpool add <pool> cache|log [mirror] <names>`, never with
`-f`, and only after its checks: a device with a partition table or a file-system signature (`wipefs --no-act`)
is refused, but for a `zfs_member` signature, which is judged by its label (`zdb -l`): a label of this pool is
allowed, another pool's refused; a device that is a member of any imported pool is refused;
- a cache entry not declared is removed;
- a log device not declared, or one to rename, is removed only while the pool is ONLINE with no read, write or
checksum errors and no scrub or resilver runs; otherwise it is a wait, `config.waiting` {key, waits-on, since}.
A declared device that is not present is a wait. Each change is read back from `zpool status -P` and emitted.
Checked by §24's convergence tests and acceptance test 2.
15. **Who applies, and when** (ADR 0679 §8). A `process` resource `zfs-config-apply`, `run: ./zfs-tools apply`, as
root, oneshot, ordered after `zfs-import.target` and enabled at boot; restarted by the node-engine on a change of
the module's config file (`restart-on`), which is how a set value reaches it from mesh-config (ADR 0348 §5, §6);
started by hand through `zfs_config_apply` (§21). It holds an exclusive lock on `/run/zfs-tools/apply.lock`; a
second start waits for it. It runs nothing while a scrub or resilver runs on the pool or before the pool is
imported: it emits `config.waiting` and the next start applies. It never runs on a timer. While the bus or the
control-node is down it applies the values its config file last held. It writes `arc-max` and `arc-min` live to
`/sys/module/zfs/parameters/zfs_arc_max` and `zfs_arc_min` as well as to §9's file and reads them back; each
`module-options` entry whose live value differs is a warning naming it, "takes effect at the next module load",
never success and never unhealthy.
16. **Never converged, always asked** (ADR 0679 §6). No setting adds, replaces, attaches, detaches or removes a data
device, a spare, a special or a dedup device; none upgrades the pool, destroys a dataset, rolls back, renames or
clears. The call-site test (`TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`, renamed for what it now holds)
allows `zpool add` only with `cache` or `log`, `zpool remove` only of a cache or log entry, `zpool set`, `zfs set`
and `zfs inherit` only for the lists of §8 and §13, `zfs destroy` only as §17 builds it, `zpool trim` only from
the scheduled run, and refuses `upgrade`, `replace`, `attach`, `detach`, `clear`, `rollback`, `rename`, `-f`, `-r`,
`-R` and `-d` everywhere.
## ZFS snapshots on a schedule
17. **The scheduled run** (ADR 0679 §7). The existing `process` resource `zfs-mesh-scrub` (`./zfs-tools scrub-run`,
every 15 minutes) becomes `zfs-mesh-run` (`./zfs-tools run`), root as today, and does three things each time, in
order: the scrub when due (as today), a trim when `trim-schedule` is due (`zpool trim <pool>`, by `trim-day` and
`trim-time` as the scrub reads its own), and the ZFS snapshots when `snapshots` is set. Reason: one schedule
resource the node-engine already runs, and no second timer. For the snapshots: at the first run of each hour it
takes `hourly`, at the first run after 00:00 `daily`, on Monday `weekly`, on the first of the month `monthly`
(only kinds whose keep count is above 0), one `zfs snapshot -- <dataset>@mesh-<kind>-<UTC time YYYYMMDDTHHMMZ>`
per dataset, never recursive, and sets on each `zfs set mesh:schedule=<kind>`. No ZFS snapshot is taken while the
pool's capacity (`zpool list -Hp -o capacity`) is above 80 %: a warning says so, and `snapshot.skipped`
{pool, capacity} is emitted.
18. **Removed only by their own record.** After taking, per dataset and kind, the run removes the oldest past its keep
count, only when it carries `mesh:schedule=<kind>`, is in the ledger `snapshots` (§20), has no hold
(`zfs holds`) and no clone (`clones` empty): `zfs destroy -- <dataset>@<snapshot>`, never `-r`, `-R` or `-d`. A
ZFS snapshot taken by hand is never removed. Each removal emits `snapshot.removed` {snapshot, kind, space freed}
and records the ledger's `removed` operation.
19. **A person's ZFS snapshot stays apart.** `zfs_snapshot` refuses a name beginning `mesh-`, and emits
`snapshot.taken` with `by: person`; the run emits it with `by: schedule`. A waiter on a person's ZFS snapshot waits on
`snapshot.taken` with that snapshot's name, which no scheduled one carries. Fate: the README's "No tool of the
mesh destroys a snapshot" becomes §18's rule.
## What it says, keeps and shows
20. **Events and kept data** (ADR 0679 §9; ADR 0363 §1; ADR 0595). The module declares a stream of its own, local
name `zfs`, kept 30 days, carrying `config.applied` {target (`pool`, `dataset`, `file` or `unit`), key, before,
after, read-back}, `config.refused` {key, value, reason}, `config.waiting` {key, waits-on, since},
`snapshot.taken` {snapshot, by}, `snapshot.removed`, `snapshot.skipped`, and today's `scrub.requested`; today's
module-level `emits` move to it. Its kept data, in `requires` by template: a state `applied`, one key per config
value, `{before, after, at, outcome}`, written by the apply and deleted by it when that value returns to not
managed (after its last `config.applied`); a ledger `snapshots`, one instance per ZFS snapshot the run took,
operations `taken` and `removed`, kept by the ledger's template. The apply process and the run declare the bus
account (`bus`) that writes them; the tools read them.
21. **Tools** (ADR 0679 §12). `zfs_config`: every owned value with its setting, what runs on the machine, its state
(`not managed`, `as set`, `waiting` with on what and since, `refused` with why, `takes effect at the next module
load`), and a `.pacnew` beside an owned file as a warning. `zfs_config_apply`: starts §15's unit through
`<node>/node-service-manager.start`, with `why`, and answers from the state `applied` afterwards.
`zfs_snapshots_schedule`: the schedule, its next run, what it would remove next and the ledger.
`zfs_snapshots_run_now` and `zfs_snapshots_prune_now`: a run's snapshot or removal step by hand, with `why`.
`zfs_snapshots_pause` and `zfs_snapshots_resume`: a key `paused` in the state `applied`, read by the run, said in
the health while it holds. The existing read tools stay. Each is in `replaces` where it replaces a command.
22. **Health** (ADR 0679 §10). `zfs_health` is unhealthy for a `refused` or a failed command in the state `applied`,
naming the key and why; warnings: a wait (naming on what and since when), the last scheduled ZFS snapshot older
than its cadence, a failed run, `autoreplace` on or a spare in the pool while `zed` is not `on`, the pool above
80 % with snapshots set, a parameter awaiting a module load. Pool faults reach the operator once, never from a ZED
handler: the event follower (`zfs-events`) is an observer, writing a finding `<pool>.<vdev or pool>.<state>` in its
bucket `findings` for a pool or device that leaves ONLINE, saying each as an event, and removing it when the state
is ONLINE again; the condition keeper raises and clears the condition from it (ADR 0564 §2; design 47 G §383). Its
process declares the bus account that writes them; today it writes the journal only.
23. **When the module leaves** (ADR 0679 §13; ADR 0303 §2; ADR 0269). The packages stay. The import, mount and target
units stay enabled and running. `/etc/default/zfs` and `zed.rc` get back their found content (ADR 0118);
`zfs.conf`, the folder `/etc/zfs/mesh-zed.d` and the drop-in are removed, and before leaving `zfs_config` says that
the next module load uses ZFS's defaults. The distribution's scrub timer is given back as found (ADR 0269). Pool
and dataset properties and devices stay as the last apply set them, also on a rollback of the module's version.
## How it is checked
24. **Unit tests**, table-driven, one test per area, within the module's 500: no settings gives no `zpool` or `zfs`
command that writes; each convergence plan from parsed `zpool status` and `zfs get` output (an add, a kernel name
to its stable name, a removal, a log removal refused while not healthy, a wait during a scrub); every refused form
(a newline, quote, `$`, backtick or semicolon in each field; a parameter not on the list; a property not on the
lists; a `sharenfs` setting; `-f`; a device with a signature or another pool's label); the call-site test of §16;
pruning never removes a `mesh-` ZFS snapshot without the property or not in the ledger, nor one with a hold or
clone; `zfs_snapshot` refuses a `mesh-` name; a parameter awaiting a module load is a warning; each change emits
its event with before and after; a scheduled `snapshot.taken` does not match a person's waiter. The module has no
integration test; the live checks stand for it, said in the pull request.
25. **Live, on the home server**: the main file's acceptance tests 1 to 3, in that order.
## Open design questions
- **Q1. A resource that follows a value only once it is set.** ADR 0679 §3 says an unset value writes nothing, §4
that the options file and `zed.rc` are generated line by line from the values, and §8 that the node-engine places
them. The manifest has no resource declared only when a config value is set (only `if-capability`), and a template
fills `${config:<key>}` but makes no line per list entry. §9 to §11 cannot be built as written until this is
answered. Recommendation: build both in the node-engine in this milestone, as a fourth prerequisite: a resource
field `if-config: <key>` (the resource is declared only while that value is set) and a template form that renders
one line per entry of a list value; the record then holds as written.
03-DESIGN/01-to-be/42-the-machines-modules-in-order.md
+84 -1
View File
@@ -2,7 +2,7 @@
layer: to-be
status: in-progress
code: [mesh-catalog, mesh-controller, mesh-host]
updated: 2026-10-22
updated: 2026-11-12
decisions:
- "ADR 0369"
- "ADR 0303"
@@ -29,6 +29,12 @@ decisions:
- "ADR 0507"
- "ADR 0508"
- "ADR 0510"
- "ADR 0679"
- "ADR 0051"
- "ADR 0269"
- "ADR 0348"
- "ADR 0363"
- "ADR 0595"
---
# 42. The machines' modules, in order
@@ -80,6 +86,7 @@ In order:
| 9 | `ssh-client` | everything under `~/.ssh` (research 027/03) | a predecessor's entries winning over the mesh's; stale keys |
| 10 | scripts | the operator's own scripts, shared and per role (research 027/03) | under no version control, copied by hand |
| 11 | `kernel` | kernel, microcode, boot entries, the fallback kernel and its boot entry, and the kernel's run-time settings per machine (transparent huge pages first) | two machines without microcode; huge pages left at the distribution's `always` on a machine whose slow disks they flood (issue 746); only one machine with a fallback kernel |
| 12 | `zfs` | on a machine that holds a ZFS pool (today the home server): ZFS's packages, its kernel module's options, its event daemon's handlers, its import, mount, share and trim units, and, for the one pool its config names, that pool's properties, its cache and log devices, its datasets' properties and a schedule of ZFS snapshots, each changed only once its config value is set (ADR 0679) | ZFS installed by hand and declared by no module; no bound on its read cache on a machine that also runs its apps; cache devices named by kernel names that swap between boots; no scheduled ZFS snapshots; the pool left with no scrub schedule by a failed timer |
`systemd`, `pacman` and `docker` hold the three seats that apply resources
(ADR 0207):
@@ -131,6 +138,10 @@ before it passed; the boot-entry script in stand-in trees of both loaders), an i
keepers of two made-up nodes on a real bus, and, after the delivery, the view answering every node and every
boot menu listing the fallback kernel.
`zfs` comes after `kernel` because its kernel module is built by DKMS against the kernels `kernel` and `pacman` put
on the machine; it is placed only on a machine that holds a pool. Its section is below,
[*The zfs module*](#the-zfs-module).
**Added 2026-10-04.** Two more modules for every machine:
- `power` holds `node-power` (ADR 0211). Other modules contribute code for
its moments (after boot, before sleep, after waking, before shutdown, on mains, on battery), and it
@@ -140,6 +151,78 @@ boot menu listing the fallback kernel.
waits for a reboot. A live restart in the middle of a full upgrade took down a workstation's
logins on the day this was written.
## The zfs module
*Added 2026-11-12, ADR 0679; the operator, 2026-11-12: "you could configure the disk by id and make sure the module
handles the rest"; "the zfs module should/must own all zfs-configuration files and options"; "we may absolutely not
lose any data".*
**What it is for.** On a machine that holds a ZFS pool, the zfs module owns ZFS: what was installed and set by hand
and seen by nothing becomes the module's, and the operator's choices about it are config values (ADR 0679 §1). It
holds no seat (see the open question of its milestone part). It keeps reading and acting by hand as today: the pools,
the datasets, a named ZFS snapshot, a scrub's start, pause and stop.
**The whole shape.**
- **What it owns** (ADR 0679 §1): the packages `zfs-utils` and `zfs-dkms`; the files `/etc/default/zfs`,
`/etc/zfs/zed.d/zed.rc` and `/etc/modprobe.d/zfs.conf`; which handlers ZFS's event daemon (ZED) runs; the states of
ZFS's import, mount, share, volume and target units and of the distribution's scrub and trim timers; and, for the one
pool its config value `pool` names, that pool's properties, its cache and log devices and its datasets' properties
from a fixed list. It reads, and never writes, what ZFS writes itself (its pool cache file and list cache).
- **What is not its own** (ADR 0679 §2): a dataset's `sharenfs` is the NFS share module's (design 52), `sharesmb` the
SMB module's once there is one; the pool's files are the operator's (ADR 0051); the weekly upgrade and a kernel
held for DKMS are the package manager's (ADR 0508).
- **An unset value changes nothing** (ADR 0679 §3): every new config value is "not managed" until the operator sets
it through mesh-config, on the Approve for a trusted list or object value (ADR 0348 §4, §8). Placing the new version
on a machine runs no `zpool` or `zfs` command that writes and rewrites none of its files.
- **Who applies** (ADR 0679 §8): the node-engine places the packages, files and units as the module's resources;
the module's root process `zfs-tools apply` converges the pool's devices and properties at its start and when its
config changes, never on a timer; the module's scheduled run takes and removes the scheduled ZFS snapshots
(ADR 0679 §7).
- **What it says** (ADR 0679 §9, §10): every change, refusal and wait is an event on the module's own stream, and
the last change per value and every scheduled ZFS snapshot are kept on the bus in its own state and ledger, asked
for by template (ADR 0595). Its health names a refusal or a failed command as unhealthy and a wait as a warning.
Pool faults reach the operator once, from the module's event follower through the condition keeper.
- **What it never does** (ADR 0679 §6): it converges no data device, spare, special or dedup device, upgrades no
pool, destroys no dataset, rolls back, renames and clears nothing; each is an act of its own, decided separately.
- **When it leaves** (ADR 0679 §13): ZFS keeps running, the packages stay, and the files are given back.
```mermaid
flowchart LR
operator[the operator] -->|sets a config value| config[mesh-config]
config -->|the module's config file| engine[the node-engine]
engine -->|packages, files, units| machine[ZFS on the machine]
engine -->|restarts on its config file| apply[zfs-tools apply, root]
apply -->|zpool add, remove, set; zfs set, inherit| machine
sched[the scheduled run] -->|ZFS snapshots, their removal, scrub, trim| machine
apply -->|config.applied, refused, waiting| stream[(the module's stream)]
sched -->|snapshot.taken, snapshot.removed| stream
apply --> applied[(state applied)]
sched --> ledger[(ledger snapshots)]
follower[the event follower] -->|pool faults| keeper[the condition keeper]
tools[the module's tools] -->|read| applied
tools -->|read| ledger
```
**Acceptance tests** (ADR 0679, *How it is checked*):
1. Placing the new version on the home server changes nothing: `zfs_config` shows every value "not managed", and
no `zpool` or `zfs` command that writes ran (the module's journal and its stream show none).
2. Setting `cache-devices` puts both cache devices in the pool by their stable names; after a reboot that swaps the
drives' kernel names, both stay ONLINE (`zfs_pool_layout`).
3. The module's own scrub timer is active before the distribution's scrub timer is stopped, read from
`<node>/node-service-manager.status` of both.
4. The unit tests of the milestone part's line 24 pass, within the module's budget of 500 unit tests.
## Milestones
| milestone | what it builds | part | state |
|---|---|---|---|
| zfs-owns-zfs | the zfs module's ownership of ZFS (ADR 0679), with its three prerequisites in the node-engine and the catalog | [42-milestones/zfs-owns-zfs.md](42-milestones/zfs-owns-zfs.md) | designed |
The modules of the tables above were built before designs were cut into milestone parts; their remaining lines move
into a part of their own at their next amendment (playbook 02, *The shape of a design*).
## Phase 2 — both workstations
In order: