|
|
|
@@ -0,0 +1,323 @@
|
|
|
|
|
---
|
|
|
|
|
layer: to-be
|
|
|
|
|
design: 42
|
|
|
|
|
milestone: zfs-owns-zfs
|
|
|
|
|
updated: 2026-11-12
|
|
|
|
|
code: [mesh-sdk, mesh-delivery, mesh-host internal/apply, mesh-catalog modules/zfs]
|
|
|
|
|
decisions:
|
|
|
|
|
- "ADR 0679"
|
|
|
|
|
- "ADR 0051"
|
|
|
|
|
- "ADR 0117"
|
|
|
|
|
- "ADR 0269"
|
|
|
|
|
- "ADR 0303"
|
|
|
|
|
- "ADR 0348"
|
|
|
|
|
- "ADR 0508"
|
|
|
|
|
- "ADR 0564"
|
|
|
|
|
- "ADR 0593"
|
|
|
|
|
- "ADR 0595"
|
|
|
|
|
- "ADR 0609"
|
|
|
|
|
- "ADR 0614"
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# Design 42, milestone zfs-owns-zfs — the zfs module owns a node's ZFS, and changes nothing that is not set
|
|
|
|
|
|
|
|
|
|
**What it builds.** On a node that holds a ZFS pool (today the home server), the zfs module owns ZFS's packages, its
|
|
|
|
|
kernel module's options file, its host id, its own folder for ZFS's event daemon (ZED), its units and, for one pool,
|
|
|
|
|
its cache and log devices, its properties, its datasets' properties and a schedule of ZFS snapshots; each is changed
|
|
|
|
|
only once its config value is set and not empty (ADR 0679). The purpose and the whole shape are the main file's,
|
|
|
|
|
[*The zfs module*](../42-the-machines-modules-in-order.md#the-zfs-module). This part holds every line the milestone
|
|
|
|
|
builds, cited as "design 42 zfs-owns-zfs §n".
|
|
|
|
|
|
|
|
|
|
**Where.** Built under issue 1979. Four repositories, one delivery group with one check: mesh-sdk, mesh-delivery
|
|
|
|
|
(the manifest validator) and `mesh-host` (the node-engine) for §1 to §5, issue 1979's three node-engine subtasks
|
|
|
|
|
and the readers of their words; mesh-catalog's `modules/zfs` for §6 to §31. **Ordering edges** (ADR 0679 §16): the
|
|
|
|
|
readers in §4's order, then the zfs module's new version only on a node whose node-engine states the three
|
|
|
|
|
capabilities of §4 (a reader's capability the delivery plan derives, ADR 0614 §4).
|
|
|
|
|
|
|
|
|
|
## The prerequisites (ADR 0679 §16)
|
|
|
|
|
|
|
|
|
|
1. **The node-engine's apply of a module stops at its first failed resource** (ADR 0679 §16 (a)). When one resource
|
|
|
|
|
of a module fails, the node-engine applies none of that module's later resources in the declared order; its apply
|
|
|
|
|
report names the failed resource with its error and each one left unapplied, and the module's phase is `failed`.
|
|
|
|
|
Resources already applied stay as applied; other modules on the node go on as today. Today the apply goes on, and
|
|
|
|
|
stopped the distribution's scrub timer after the module's own timer failed. Checked by a unit test (a module of
|
|
|
|
|
three resources whose second fails leaves the third untouched and reports it as left) and by §5.
|
|
|
|
|
2. **A resource is declared only while a config value holds** (ADR 0679 §16 (b), §3). A resource of any type may
|
|
|
|
|
carry the manifest field `if-config`: a list of conditions, each `<key>` (the module's config value is set and is
|
|
|
|
|
neither `false`, `off`, `none` nor an empty list) or `<key>=<value>` (the value equals it; for a list value, the
|
|
|
|
|
list holds that entry). The resource is declared while any condition holds, and not at all while none does. A
|
|
|
|
|
resource that leaves the declaration this way is removed as any resource that leaves it: a file the node-engine
|
|
|
|
|
wrote whole is deleted, a unit gets its found state back (ADR 0269 §2). The manifest validator refuses an
|
|
|
|
|
`if-config` key the module does not declare under `config`. Checked by the node-engine's table test (a value unset,
|
|
|
|
|
empty, `off`, set; a list with and without the entry; a value going from set to unset removes the resource) and
|
|
|
|
|
the validator's own test.
|
|
|
|
|
3. **A template writes one line per entry** (ADR 0679 §16 (c), §4). A template file may hold
|
|
|
|
|
`${config-lines:<key>:<prefix>}`: for a list value one line per entry, `<prefix><entry>`, in the list's order;
|
|
|
|
|
for a set scalar value one line `<prefix><value>`; nothing for an unset value or an empty list. With a fourth part,
|
|
|
|
|
`${config-lines:<key>:<prefix>:mib-bytes}`, a number given in MiB is written in bytes. The prefix is literal text
|
|
|
|
|
with no `}`, `$` or newline, refused by the manifest validator; an entry holding a newline is refused before the
|
|
|
|
|
file is written, and the file is left as it was. Checked by the node-engine's table test (a list of two, an empty
|
|
|
|
|
list, an unset and a set scalar, `mib-bytes`, a newline in an entry) and the validator's own test.
|
|
|
|
|
4. **Every reader of the new words, in order** (design-it-as-it-should-be; ADR 0614 §4). mesh-sdk first: the
|
|
|
|
|
manifest types gain `if-config` on every resource and the template form `config-lines`, and the capability names
|
|
|
|
|
`if-config`, `config-lines` and `apply-stops-at-failure`. Then the manifest validator in mesh-delivery's indexer:
|
|
|
|
|
it reads both words and refuses them as §2 and §3 say. Then the node-engine: it builds §1 to §3 and states the
|
|
|
|
|
three capabilities. A manifest using `if-config` or `config-lines` reaches a node only through a node-engine that
|
|
|
|
|
states both; the delivery plan derives the edge from the stated capability, never from a fingerprint. Checked by
|
|
|
|
|
each repository's unit tests and by the delivery plan naming the edge.
|
|
|
|
|
5. **One integration test of the node-engine** (how-work-is-done: a change to how modules run carries one). It
|
|
|
|
|
starts the node-engine whole against the real bus with a module of a resource under `if-config`, a template with
|
|
|
|
|
`config-lines`, and a resource that fails: the conditioned resource appears when its value is set and goes when it
|
|
|
|
|
is unset, the template's lines follow the list, and the failing resource stops the module's apply with the later
|
|
|
|
|
resource left. It is the only integration test this milestone adds.
|
|
|
|
|
6. **The order on the node** (ADR 0679 §16, §9). The zfs module declares its scheduled run's resource (§20) before
|
|
|
|
|
the resources that stop the distribution's scrub timers, so by §1 those are stopped only once the run's timer is
|
|
|
|
|
active. The run's `schedule` is the list form `0,15,30,45 * * * *`, which today's node-engine writes as
|
|
|
|
|
`OnCalendar=*-*-* *:0,15,30,45:00`, so nothing waits on another fix. Checked by the module's manifest test (the
|
|
|
|
|
resource order and the schedule string) and by acceptance test 3 of the main file.
|
|
|
|
|
|
|
|
|
|
## What the module owns
|
|
|
|
|
|
|
|
|
|
7. **The packages** (ADR 0679 §1; ADR 0303 §2). The manifest declares `zfs-utils` and `zfs-dkms` present: the
|
|
|
|
|
node-engine installs a missing one, upgrades none, removes none. `zfs-utils-debug` is not declared and is left as
|
|
|
|
|
found. Fate: the README's "It installs nothing" goes; it says the two packages and why the debug package is not the
|
|
|
|
|
module's. Read by `pacman_packages` as declared by `zfs`. Checked by the manifest test.
|
|
|
|
|
8. **One pool, renamed `pool`** (ADR 0679 §1; ADR 0609 §4). The config value `scrub-pool` becomes `pool` with
|
|
|
|
|
`renamed-from: scrub-pool`: type string, trusted, required. mesh-config moves the value under the new name and
|
|
|
|
|
answers both names during the rename window (design 32, *A config value renamed*); every template and the module's
|
|
|
|
|
code read `pool`. **Rollback:** a rolled-back version leaves pool and dataset properties and devices as the last
|
|
|
|
|
apply set them (ADR 0679 §15); once the rename window has closed, going back to the version with `scrub-pool` and
|
|
|
|
|
`zfs-mesh-scrub` is a new delivery of that version, with its own rename back, never a rollback. Checked by the
|
|
|
|
|
manifest test (no `scrub-pool` but in `renamed-from`) and `zfs_config` showing `pool` on the home server.
|
|
|
|
|
9. **The module's config file** (ADR 0679 §9; ADR 0348 §5). `/etc/zfs-tools/config.json`, root's, mode 0644, written
|
|
|
|
|
by the node-engine from a template holding every config value of the module, lists and objects rendered as ADR 0348
|
|
|
|
|
§5 says; it replaces `/etc/zfs-tools/scrub.conf`, which the node-engine removes as a resource that left. The apply
|
|
|
|
|
unit (§18) has `restart-on` this file: its change is the path by which mesh-config's `config-changed` for the
|
|
|
|
|
module's values reaches the apply. The scheduled run (§20) reads it at each run. `/var/lib/zfs-scrub` stays and is
|
|
|
|
|
read by the run as today. Checked by the manifest test.
|
|
|
|
|
10. **What it never writes** (ADR 0679 §1, §2). The package's own `/etc/default/zfs` and `/etc/zfs/zed.d`, with its
|
|
|
|
|
`zed.rc` and its links, are left as the package ships them, and nothing of them is given back on leaving; ZFS's own
|
|
|
|
|
`/etc/zfs/zpool.cache` and `/etc/zfs/zfs-list.cache` are read only. `zfs_config` shows each, and a `.pacnew`
|
|
|
|
|
beside `/etc/default/zfs` or the package's `zed.rc` is a warning (§29). A dataset's `sharenfs` is the seat
|
|
|
|
|
`node-nfs-server`'s, `sharesmb` the SMB module's (issue 613): neither is on a list of §16, a value naming either
|
|
|
|
|
is refused (§11), and `zfs_config` never reports either as differing. The pool's files are the operator's (ADR
|
|
|
|
|
0051). Checked by the call-site test (§19) and §31.
|
|
|
|
|
|
|
|
|
|
## The config values and their forms
|
|
|
|
|
|
|
|
|
|
11. **Not managed until set and not empty; each of a fixed form** (ADR 0679 §3, §4). No new value has a default in
|
|
|
|
|
the manifest. An absent value and an empty list both mean not managed: the module writes nothing, converges
|
|
|
|
|
nothing and reports no difference for it, so nothing is ever removed by a blank value. A value of another form is
|
|
|
|
|
refused by the module's own parse before any file is written or command run, emitting `config.refused` (§25).
|
|
|
|
|
Trusted values (every list and object below, `log-mirror` and `hostid`) are set only on the operator's Approve (ADR
|
|
|
|
|
0348 §4, §8). The module never reads a value from the node into config.
|
|
|
|
|
- `arc-max`, `arc-min` (number): whole MiB, `arc-min` ≤ `arc-max`, `arc-max` at least 64 and below the node's
|
|
|
|
|
memory read from `/proc/meminfo`. ZFS's read cache in memory (the ARC) is bounded by them.
|
|
|
|
|
- `module-options` (list, trusted): entries `name=digits`, the name from the module's fixed list `l2arc_write_max`,
|
|
|
|
|
`l2arc_write_boost`, `l2arc_headroom`, `l2arc_noprefetch`, `l2arc_rebuild_enabled`, `zfs_txg_timeout`,
|
|
|
|
|
`zfs_dirty_data_max`, `zfs_scan_vdev_limit`, `zfs_scrub_min_time_ms`, `zfs_resilver_min_time_ms`,
|
|
|
|
|
`zfs_arc_sys_free`; `zfs_arc_max` and `zfs_arc_min` are refused here, as `arc-max` and `arc-min` hold them. Reason:
|
|
|
|
|
the ARC, L2ARC, transaction group and scrub parameters ADR 0679 §4 names, and nothing else.
|
|
|
|
|
- `hostid` (string, trusted): eight hexadecimal digits, not all zero (§15).
|
|
|
|
|
- `zed` (string): `on` or `off`. `zed-handlers` (list, trusted): from `all-syslog` (the journal handler) and
|
|
|
|
|
`resilver_finish-start-scrub` (scrub after resilver), the two ADR 0679 §12 names; no notify handler.
|
|
|
|
|
- `trim-schedule` (string): `none`, `weekly` or `monthly`; `trim-day` and `trim-time` with `scrub-day`'s and
|
|
|
|
|
`scrub-time`'s forms.
|
|
|
|
|
- `cache-devices`, `log-devices` (lists, trusted): names under `/dev/disk/by-id`, letters, digits, `-`, `_`, `.`
|
|
|
|
|
and `:` only. `log-mirror` (boolean, trusted): when true, `log-devices` holds exactly two names, added as one
|
|
|
|
|
mirror; otherwise each is a log device of its own. Reason: the log's form today is a mirror.
|
|
|
|
|
- `pool-properties` (object, trusted): `autotrim`, `autoexpand`, `autoreplace` (`on`, `off`, `default`) and
|
|
|
|
|
`failmode` (`wait`, `continue`, `panic`, `default`).
|
|
|
|
|
- `dataset-properties` (object, trusted): a dataset of the pool (the pool's name or one beginning `<pool>/`) to
|
|
|
|
|
properties of §16's list with their forms.
|
|
|
|
|
- `snapshots` (object, trusted): `none`, or keep counts `hourly`, `daily`, `weekly` and `monthly`, each a whole
|
|
|
|
|
number from 0 to 168, a kind with 0 not taken; for every filesystem of the pool. The bound 168 (a week of hours)
|
|
|
|
|
is the design's, so the ledger's cap (§26) stays bounded.
|
|
|
|
|
Checked by §31's refused-forms tests, one table per value, the empty list among them.
|
|
|
|
|
|
|
|
|
|
## Files, units and the event daemon
|
|
|
|
|
|
|
|
|
|
12. **The options file** (ADR 0679 §1, §4). `/etc/modprobe.d/zfs.conf`, root's, 0644, carries `if-config: [arc-max,
|
|
|
|
|
arc-min, module-options]` (§2) and holds a header comment, then by §3 `options zfs zfs_arc_max=` and `options zfs
|
|
|
|
|
zfs_arc_min=` with the values in bytes (`mib-bytes`), then `options zfs ` with each `module-options` entry. With
|
|
|
|
|
none of the three set it is not placed. Checked by the manifest test and §31's template test.
|
|
|
|
|
13. **The units** (ADR 0679 §1, §15; ADR 0117). Declared running and enabled at boot: `zfs-import-cache.service`,
|
|
|
|
|
`zfs-mount.service` and `zfs.target`, as they run on the home server today, so declaring them changes nothing.
|
|
|
|
|
Declared with no state, owned and left as found: `zfs-import-scan.service`, `zfs-share.service`,
|
|
|
|
|
`zfs-volume-wait.service` and `zfs-import.target` (reason: ADR 0679 §15 keeps only the first three running when the
|
|
|
|
|
module leaves, so no other ZFS unit is one the mesh would stop then). `zfs-zed.service`: running and enabled with
|
|
|
|
|
`if-config: [zed=on]`, stopped and disabled with `if-config: [zed=off]`, left as found while `zed` is not managed.
|
|
|
|
|
The distribution's `zfs-trim-weekly@<pool>.timer` and `zfs-trim-monthly@<pool>.timer` are stopped and disabled
|
|
|
|
|
with `if-config: [trim-schedule]` (the run trims, §20); its scrub timers are stopped and disabled after the run's
|
|
|
|
|
timer (§6). Read by `<node>/node-service-manager.status`. Checked by the manifest test (each unit, its state and
|
|
|
|
|
its condition).
|
|
|
|
|
14. **ZED runs from the module's own folder, without links** (ADR 0679 §13, §12). The folder `/etc/zfs/mesh-zed.d`
|
|
|
|
|
(root's, 0755), its `zed.rc` and a drop-in for `zfs-zed.service` whose `ExecStart` adds `-d /etc/zfs/mesh-zed.d`
|
|
|
|
|
carry `if-config: [zed=on]`. The module's `zed.rc` holds the package's variables but every `ZED_NOTIFY_*` and mail
|
|
|
|
|
line, and `ZED_SCRUB_AFTER_RESILVER=1`, which only the scrub-after-resilver handler reads. Each handler is a file
|
|
|
|
|
of its own in the folder, a copy of the package's script by its content kept in the module's files archive, mode
|
|
|
|
|
0755, with `if-config: [zed-handlers=<its name>]`, so only the handlers set are placed. The package's
|
|
|
|
|
`/etc/zfs/zed.d` is not read while the drop-in holds. The mesh creates no symlinks. The node-engine restarts
|
|
|
|
|
`zfs-zed.service` on a change of the drop-in, a handler or `zed.rc` (`restart-on`). Checked by the manifest test (no
|
|
|
|
|
link resource, each handler's condition).
|
|
|
|
|
15. **The node's ZFS id** (ADR 0679 §1, §3, §4). `/etc/hostid` holds four bytes, which no template writes, so the apply
|
|
|
|
|
(§18) writes it with ZFS's own writer, `zgenhostid -f <value>`, only while `hostid` is set, and only when the pool's
|
|
|
|
|
label (`zdb -C <pool>`, its `hostid`) is 0 or equals the value: then the next boot imports without `zpool import
|
|
|
|
|
-f`. Otherwise it writes nothing and emits `config.refused` naming why ("the pool's import would need -f"). Before
|
|
|
|
|
the Approve, the value's description and `zfs_config` say the change needs a check of the pool's import at the next
|
|
|
|
|
boot; the module never reboots. After the reboot, `zfs_pools` shows the pool imported. Emits `config.applied` with
|
|
|
|
|
target `file`. Checked by §31 (unset: no `zgenhostid`; a label of another id: refused; 0 or equal: one run).
|
|
|
|
|
|
|
|
|
|
## The pool
|
|
|
|
|
|
|
|
|
|
16. **Pool and dataset properties** (ADR 0679 §6). For each set pool property the apply runs `zpool set
|
|
|
|
|
<property>=<value> <pool>`, and for `default` `zpool set <property>=<ZFS's default> <pool>`. For each set dataset
|
|
|
|
|
property it runs `zfs set <property>=<value> <dataset>`, and for `default` `zfs inherit <property> <dataset>`, so
|
|
|
|
|
its source stays default. The dataset list is `compression` (`off`, `lz4`, `zstd`, `zstd-1` to `zstd-19`,
|
|
|
|
|
`gzip-1` to `gzip-9`, `zle`), `atime` and `relatime` (`on`, `off`), `recordsize` (a power of two from `512` to
|
|
|
|
|
`16M`, with `K` or `M`), `primarycache` and `secondarycache` (`all`, `metadata`, `none`), `sync` (`standard`,
|
|
|
|
|
`always`, `disabled`), `logbias` (`latency`, `throughput`), `quota` and `reservation` (`none` or digits with `K`,
|
|
|
|
|
`M`, `G` or `T`). A property already at its value runs nothing. Each change is read back with `zpool get -Hp` or
|
|
|
|
|
`zfs get -Hp` and emitted (§25). Checked by §31's convergence tests.
|
|
|
|
|
17. **Cache and log devices** (ADR 0679 §5). From `zpool status -P -L <pool>` the apply knows each device's kernel
|
|
|
|
|
path, and from the link under `/dev/disk/by-id` each declared name's. Then, in this order:
|
|
|
|
|
- an entry of the pool whose kernel path is a declared device's, but named by its kernel name, is removed
|
|
|
|
|
(`zpool remove <pool> <entry>`) and added again by its stable name;
|
|
|
|
|
- a declared device not in the pool is added with `zpool add <pool> cache|log [mirror] <names>`, never `-f`, and
|
|
|
|
|
only after its checks: a device with a partition table or a file-system signature (`wipefs --no-act <device>`)
|
|
|
|
|
is refused, but for a `zfs_member` signature, which is judged by its label (`zdb -l <device>`): this pool's
|
|
|
|
|
label is allowed, another pool's refused; a device that is a member of any imported pool is refused;
|
|
|
|
|
- only while `cache-devices` is set and not empty, a cache entry it does not name is removed;
|
|
|
|
|
- only while `log-devices` is set and not empty, a log device it does not name, one to rename, or the log when
|
|
|
|
|
`log-mirror` changes (a removal and a re-add, said as such) is removed only while the pool is ONLINE with no
|
|
|
|
|
read, write or checksum errors and no scrub or resilver runs; otherwise it is a wait, `config.waiting` {key,
|
|
|
|
|
waits-on, since}.
|
|
|
|
|
A declared device that is not present is a wait. Each change is read back from `zpool status -P` and emitted.
|
|
|
|
|
Checked by §31's convergence tests (the empty list among them) and acceptance test 2.
|
|
|
|
|
18. **The apply: who, when, and what wakes a wait** (ADR 0679 §9). A `process` resource `zfs-config-apply`, `run:
|
|
|
|
|
./zfs-tools apply`, as root, oneshot, ordered after `zfs-import.target`, enabled at boot, with `restart-on` the
|
|
|
|
|
config file (§9); it converges §15 to §17 and writes the ARC values live. It holds an exclusive lock on
|
|
|
|
|
`/run/zfs-tools/apply.lock`; a second start waits for it. It converges nothing while a scrub or resilver runs on
|
|
|
|
|
the pool or before the pool is imported: it emits `config.waiting` with what it waits on and since when, and ends.
|
|
|
|
|
It never runs on a timer. **A wait is ended by an event:** when the event follower (§27) hears a scrub or a
|
|
|
|
|
resilver finish (ZFS's `scrub_finish` and `resilver_finish` events) or a declared device appear (a udev `add` of a
|
|
|
|
|
block device whose link under `/dev/disk/by-id` is declared, read with `udevadm monitor --udev
|
|
|
|
|
--subsystem-match=block`), it starts the apply unit through `<node>/node-service-manager.start`. While the bus or
|
|
|
|
|
the control-node is down, the apply converges from the config file it last got. It writes `arc-max` and `arc-min`
|
|
|
|
|
to `/sys/module/zfs/parameters/zfs_arc_max` and `zfs_arc_min` as well as to §12's file, and reads both back; each
|
|
|
|
|
`module-options` entry whose live value differs is a warning naming it, "takes effect at the next load of the
|
|
|
|
|
kernel module", never success and never unhealthy.
|
|
|
|
|
19. **The call-site test** (ADR 0679 §7). Every command the bundle runs is built by one of its argument functions, and
|
|
|
|
|
the test (`TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`, renamed `TestTheBundleRunsOnlyTheFormsOfADR0679`)
|
|
|
|
|
allows only: `zpool add <pool> cache|log`, `zpool remove` of a cache or log device, `zpool set`, `zfs set` and `zfs
|
|
|
|
|
inherit` of a listed property, `zfs snapshot`, `zfs set mesh:schedule=<kind>` on a ZFS snapshot named `mesh-`,
|
|
|
|
|
`zfs destroy -- <dataset>@<snapshot>`, `zpool scrub`, `zpool scrub -p` and `zpool scrub -s` of the pool, `zpool trim`
|
|
|
|
|
of the pool; and the reads `zpool status`, `zpool list`, `zpool get`, `zpool iostat`, `zpool events`, `zfs list`,
|
|
|
|
|
`zfs get` and `zfs holds`. It refuses every other `zpool` or `zfs` form and every `-f`, `-r`, `-R` and `-d`, so
|
|
|
|
|
data vdevs, spares, special and dedup devices, `upgrade`, a dataset's destroy, `rollback`, `rename` and `clear` are
|
|
|
|
|
never run (each an act of its own, asked of the operator). The same test pins the module's other root forms:
|
|
|
|
|
`wipefs --no-act <device>`, `zdb -l <device>`, `zdb -C <pool>`, `zgenhostid -f <value>`, `udevadm monitor --udev
|
|
|
|
|
--subsystem-match=block` and `systemctl show`. Fate: the README's list of allowed forms is replaced by this one.
|
|
|
|
|
|
|
|
|
|
## The scheduled run and ZFS snapshots
|
|
|
|
|
|
|
|
|
|
20. **The scheduled run** (ADR 0679 §9). The `process` resource `zfs-mesh-scrub` (`./zfs-tools scrub-run`) becomes
|
|
|
|
|
`zfs-mesh-run` (`./zfs-tools run`), root as today, with §6's schedule. Each run takes the apply's lock; when the lock
|
|
|
|
|
is held it does nothing and emits `run.skipped` {reason: `lock-held`}. It does nothing while a scrub or resilver
|
|
|
|
|
runs or before the pool is imported, emitting `run.skipped` with that reason. Otherwise, in order: it starts the
|
|
|
|
|
scrub when due and returns at once, never waiting on it (today's wait with `zpool wait -t scrub` goes; the scrub's
|
|
|
|
|
end is heard by the event follower, §27); starts a trim when `trim-schedule` is due (`zpool trim <pool>`, by
|
|
|
|
|
`trim-day` and `trim-time` in the node's time zone, as the scrub reads its own); and takes and removes the ZFS
|
|
|
|
|
snapshots (§21, §22) when `snapshots` is set. Checked by §31 (lock held: nothing run; scrub due: started and the
|
|
|
|
|
run ends).
|
|
|
|
|
21. **Taking scheduled ZFS snapshots** (ADR 0679 §8). Each kind whose keep count is above 0 is due at the first run of
|
|
|
|
|
the hour (hourly), the first run after midnight (daily), the first run on Monday (weekly) and the first run on the
|
|
|
|
|
first of the month (monthly), each read in the node's time zone against the ledger's last entry of that kind (§26).
|
|
|
|
|
For every filesystem of the pool it runs `zfs snapshot -- <dataset>@mesh-<kind>-<YYYYMMDDTHHMMZ>`, the time in UTC,
|
|
|
|
|
never recursive, then `zfs set mesh:schedule=<kind>` on it, and records the ledger's `taken` operation. The run
|
|
|
|
|
takes no ZFS snapshot it cannot record in the same run: with the bus down or the ledger full it takes none and
|
|
|
|
|
emits `snapshot.skipped.schedule` {pool, capacity, reason}. No ZFS snapshot is taken while the pool is more than 80
|
|
|
|
|
% full (`zpool list -Hp -o capacity`): a warning says so, and `snapshot.skipped.schedule` is emitted with reason
|
|
|
|
|
`full`.
|
|
|
|
|
22. **Removing them, only by their own record** (ADR 0679 §8; the operator, 2026-11-12: "we may absolutely not lose any
|
|
|
|
|
data"). After taking, the run reads the ledger in the same run (no read, no removal) and, per dataset and kind,
|
|
|
|
|
removes the oldest past its keep count, all of them in that run. Before any destroy it checks the name against
|
|
|
|
|
`<a dataset of the pool>@mesh-<kind>-<YYYYMMDDTHHMMZ>`, that it carries `mesh:schedule=<kind>`, is in the ledger,
|
|
|
|
|
has no hold (`zfs holds`) and no clone (`clones` empty); anything else is refused, emitted and left. Then `zfs
|
|
|
|
|
destroy -- <dataset>@<snapshot>`, the ledger's `removed` operation, and `snapshot.removed.schedule` {snapshot, kind,
|
|
|
|
|
space freed in bytes}. When `snapshots` goes back to `none`, the existing scheduled ZFS snapshots are kept, and a
|
|
|
|
|
warning names them; that change removes nothing. A ZFS snapshot taken by hand is never removed.
|
|
|
|
|
23. **A person's ZFS snapshot stays apart** (ADR 0679 §8, §10). `zfs_snapshot` refuses a name beginning `mesh-` and
|
|
|
|
|
emits `snapshot.taken.person`; the run emits `snapshot.taken.schedule`. A waiter on a person's ZFS snapshot waits
|
|
|
|
|
on `snapshot.taken.person`, which no scheduled one is said on. Fate: the README's "No tool of the mesh destroys a
|
|
|
|
|
snapshot" becomes §22's rule.
|
|
|
|
|
24. **Hand acts run as the run does** (ADR 0679 §14). `zfs_snapshots_run_now`, `zfs_snapshots_prune_now` and
|
|
|
|
|
`zfs_config_apply` each start a oneshot instance of the unit `zfs-mesh-run@<act>.service`, the act from the fixed
|
|
|
|
|
set `snapshots`, `prune` and `apply`, through `<node>/node-service-manager.start`, with `why`; the instance runs
|
|
|
|
|
under the same lock and guards as §18 and §20, and the tool answers from the state and events afterwards. Their
|
|
|
|
|
events carry `.person` (`snapshot.taken.person`, `snapshot.removed.person`, `snapshot.skipped.person`).
|
|
|
|
|
`zfs_snapshots_pause` writes the key `paused` {by, why, at} in the state `schedule`, which the run reads and then
|
|
|
|
|
takes and removes nothing; `zfs_snapshots_resume` deletes it.
|
|
|
|
|
|
|
|
|
|
## What it says, keeps and shows
|
|
|
|
|
|
|
|
|
|
25. **Events, each kind on a subject of its own** (ADR 0679 §10). `config.applied` {target (`pool`, `dataset`,
|
|
|
|
|
`file` or `unit`), key, before, after, read-back}, `config.refused` {key, value, reason}, `config.waiting` {key,
|
|
|
|
|
waits-on, since}, `snapshot.taken.person` and `.schedule` {snapshot, by}, `snapshot.removed.person` and
|
|
|
|
|
`.schedule` {snapshot, kind, space freed in bytes}, `snapshot.skipped.person` and `.schedule` {pool, capacity,
|
|
|
|
|
reason}, `run.skipped` {reason}, and today's `scrub.requested`. They go on an events stream asked for by template
|
|
|
|
|
in `requires` alone and created by the bus module (ADR 0595 §1, §2, §4), kept 30 days; today's module-level `emits`
|
|
|
|
|
move to it. No module consumes `snapshot.taken` or `scrub.requested` today (read in mesh-catalog's manifests on
|
|
|
|
|
main); a session waiting through `mesh_wait` on `snapshot.taken` hears `snapshot.taken.person` after.
|
|
|
|
|
26. **Kept data, by template in `requires`** (ADR 0679 §10; ADR 0595 §1). A state `applied`, one key per config value,
|
|
|
|
|
`{before, after, at, outcome}`, written by the apply and deleted by it when that value goes back to not managed,
|
|
|
|
|
after its last `config.applied`. A state `schedule`, the key `paused` alone (§24). A ledger `snapshots`, one
|
|
|
|
|
instance per ZFS snapshot the run took, operations `taken` and `removed`, the run removing the instance of one it
|
|
|
|
|
destroyed; its cap is the keep counts' sum times two instances per dataset. A state `findings` (§27). The apply,
|
|
|
|
|
the run and the event follower declare the bus account that writes them; the tools read them.
|
|
|
|
|
27. **The event follower's findings** (ADR 0679 §12; ADR 0564 §2; design 47 G §383 to §388). The event follower
|
|
|
|
|
(`zfs-events`, root) is an observer: in its state `findings` it keeps one key `pool.<pool>.<device>` per device
|
|
|
|
|
of the pool (and `pool.<pool>.pool` for the pool) not ONLINE, its value the state, deleted when the device is
|
|
|
|
|
ONLINE again or leaves the pool; it says each change on its findings stream as design 47 G §383 says, and at its
|
|
|
|
|
start it reads `zpool status -P` against its keys and states them (§384). The condition keeper raises and clears
|
|
|
|
|
the condition from those events, so a pool fault reaches the operator once and never from a ZED handler. It also
|
|
|
|
|
starts the apply on the events of §18. Its process declares the bus account; today it writes the journal only.
|
|
|
|
|
28. **Tools** (ADR 0679 §14). `zfs_config`: every owned value with its config value, what runs on the node, its state
|
|
|
|
|
(`not managed`, `as set`, `waiting` with on what and since, `refused` with why, `takes effect at the next load of
|
|
|
|
|
the kernel module`), and the files of §10. `zfs_config_apply` (§24). `zfs_snapshots_schedule`: the schedule, its
|
|
|
|
|
next run, what it would remove next and the ledger. `zfs_snapshots_run_now`, `zfs_snapshots_prune_now`,
|
|
|
|
|
`zfs_snapshots_pause` and `zfs_snapshots_resume` (§24). `zfs_module_events`: the module's stream, the last N events
|
|
|
|
|
by kind (named so because `zfs_events` already reads ZFS's own events). `zfs_snapshots_check`: compares the ledger
|
|
|
|
|
with what ZFS holds and names each entry whose ZFS snapshot is gone and each `mesh-` snapshot not in the ledger,
|
|
|
|
|
changing nothing. `zfs_findings`: the follower's findings. The existing read tools stay. Each is in `replaces`
|
|
|
|
|
where it replaces a command.
|
|
|
|
|
29. **Health** (ADR 0679 §11). `zfs_health` is unhealthy for a `refused` or a failed command in the state `applied`,
|
|
|
|
|
naming the key and why. Warnings: a wait, naming on what and since when; a `.pacnew` beside `/etc/default/zfs` or
|
|
|
|
|
the package's `zed.rc`; the last scheduled ZFS snapshot older than its cadence; a failed run; `autoreplace` on or a
|
|
|
|
|
spare in the pool while `zed` is not `on`; the pool above 80 % with `snapshots` set; scheduled ZFS snapshots kept
|
|
|
|
|
after `snapshots` went to `none`; a parameter awaiting a load of the kernel module; `paused` set.
|
|
|
|
|
30. **When the module leaves** (ADR 0679 §15; ADR 0303 §2; ADR 0269 §2). The packages stay. `zfs-import-cache`,
|
|
|
|
|
`zfs-mount` and `zfs.target` are never stopped or disabled, and the module says so before it leaves (ADR 0679 §15
|
|
|
|
|
changes ADR 0269 §10 for them). Nothing of the package's files is given back, since the module never wrote them.
|
|
|
|
|
`zfs.conf`, `/etc/zfs/mesh-zed.d` and the drop-in are removed, and before leaving `zfs_config` says that the next
|
|
|
|
|
load of the kernel module uses ZFS's defaults. The distribution's scrub and trim timers are put back as the
|
|
|
|
|
module's declaration found them, started and enabled again where it stopped or disabled them. Pool and dataset
|
|
|
|
|
properties and devices stay as the last apply set them, also on a rollback (§8).
|
|
|
|
|
|
|
|
|
|
## How it is checked
|
|
|
|
|
|
|
|
|
|
31. **Unit tests**, table-driven, one test per area, within the module's 500: no values gives no `zpool` or `zfs`
|
|
|
|
|
write but the due scrub; each convergence plan from parsed `zpool status` and `zfs get` output (an add, a kernel
|
|
|
|
|
name to its stable name, a removal, an empty list removing nothing, a log removal refused while not healthy, a
|
|
|
|
|
`log-mirror` change, a wait during a scrub); every refused form (a newline, quote, `$`, backtick or semicolon in
|
|
|
|
|
each field; a parameter or property not on its list; a `sharenfs` value; `-f`; a device with a signature or another
|
|
|
|
|
pool's label; a hostid of the wrong form or a label of another id); the call-site test of §19; the lock (a second
|
|
|
|
|
apply waits, a run with the lock held does nothing); the apply started only by `config-changed`, never a timer;
|
|
|
|
|
one scheduled resource; both ARC writes read back; applying from the last config file with the bus down; the
|
|
|
|
|
three templates (config file, options file, `zed.rc`); pruning never removes a ZFS snapshot whose name, property
|
|
|
|
|
or ledger entry does not fit, nor one with a hold or clone, and removes every surplus in one run; no ZFS snapshot
|
|
|
|
|
taken without a ledger record; `snapshots` back to `none` removes nothing; `zfs_snapshot` refuses a `mesh-`
|
|
|
|
|
name; a scheduled `snapshot.taken.schedule` does not wake a person's waiter; a parameter awaiting a load is a
|
|
|
|
|
warning; the `.pacnew` warning; each change emits its event with before and after; one test per tool. The module
|
|
|
|
|
adds no integration test; §5 is the milestone's, and the main file's live tests stand for the module whole, said
|
|
|
|
|
in the pull request.
|