mesh-catalog/modules/zfs/README.md

zfs

The mesh's view of ZFS on a machine, and what it owns of it: the packages, the pool's properties, cache and log devices, the node's ZFS id, the ARC, the event daemon, the scrub and trim schedule and scheduled ZFS snapshots. It changes only what its config values set: a value that is not set, or an empty list, is not managed, and nothing is ever removed by a blank value (ADR 0679). The pool's data is the operator's, which the mesh does not own (novox/hq ADR 0051).

What it installs

zfs-utils and zfs-dkms are declared present: the node-engine installs a missing one, upgrades none, removes none. zfs-utils-debug is not the module's and is left as found. The module never writes the package's own /etc/default/zfs or /etc/zfs/zed.d; a .pacnew beside /etc/default/zfs or the package's zed.rc is a warning.

What it does not do

It never destroys a dataset, rolls back, renames, clears, sends, receives, replaces, attaches or detaches, and it adds no data vdev, spare, special or dedup device. Each of those changes or removes the operator's data or the pool's layout and needs a decision of its own first. A dataset's sharenfs is the seat node-nfs-server's and sharesmb the SMB module's: a value naming either is refused.

This is checked by TestTheBundleRunsOnlyTheFormsOfADR0679. Every command the bundle runs is built by one of its argument functions, and each function is held to its program, its verb and its whole form:

• zpool: add <pool> cache|log (never -f), remove of a cache or log device, set of a listed pool property, scrub (with -p or -s only from a person's act), trim of the pool, and the reads status, list, get, iostat, events and history;
• zfs: snapshot, set mesh:schedule=<kind> on a snapshot named mesh-, destroy -- <dataset>@<snapshot>, set and inherit of a listed dataset property, and the reads list, get and holds;
• wipefs --no-act <device>, zdb -l <device> and zdb -C <pool>, zgenhostid -f <value> only with the value the parse validated, udevadm monitor --udev --subsystem-match=block, and systemctl show.

Any other zpool or zfs form, and every -f, -r, -R and -d, is refused; so is any call site that builds its own argument list.

On a machine without ZFS, every tool answers applicable: false with the reason, and the health check is unhealthy and says to unassign the module.

It holds no seat. A new node-* seat is the controller's to define (novox/hq ADR 0121, ADR 0246).

Config values

value	form	managed by
pool (was scrub-pool)	the pool's name, trusted, required	the scrub, trim, ZFS snapshots and the apply
scrub-cadence, scrub-day, scrub-time	as before, with their defaults	the scheduled run
arc-max, arc-min	whole MiB	written live and into /etc/modprobe.d/zfs.conf, read back
module-options	name=digits from a fixed list	the options file; a live parameter also live, others at the next load of the kernel module
hostid	eight hexadecimal digits	/etc/hostid, by zgenhostid -f, only when the pool's label holds 0 or this value
zed, zed-handlers	on or off; all-syslog, resilver_finish-start-scrub	the module's own ZED folder, without links
trim-schedule, trim-day, trim-time	none, weekly or monthly with a day and a time	the scheduled run
cache-devices, log-devices, log-mirror	names under /dev/disk/by-id	the apply, after its checks
pool-properties, dataset-properties	the listed properties	the apply, read back
snapshots	{"none": true} or keep counts hourly, daily, weekly, monthly, 0 to 168	the scheduled run

A value of another form is refused before anything is written or run (config.refused); one refused value refuses the whole config. A keep count of 0 is that kind's none: none is taken, and its existing scheduled ZFS snapshots are kept with a warning. A trim schedule of weekly or monthly without a trim-day or trim-time is refused. Changing hostid needs a check of the pool's import at the next boot; the module never reboots.

The apply, the run, the keeper and the follower

The module writes /etc/zfs-tools/config.json (all its values as one JSON object). The apply (zfs-tools apply, root, run-once, started again by the config file's change) converges the node under an exclusive lock, waits (config.waiting) while a scrub or resilver runs or before the pool is imported, and exits non-zero only when a command it ran failed. The event follower (zfs-tools events, root) starts the apply at its own start and when a scrub or resilver ends or a declared device appears, and keeps the findings of a pool or device not ONLINE in its state findings. The scheduled run (zfs-tools run, root, every quarter of an hour) takes the same lock, starts a due scrub or trim and returns without waiting, takes and removes the scheduled ZFS snapshots and acts on a person's hand acts. The three root processes hold no bus account: the keeper (zfs-tools keeper, the module's user zfs-keeper) holds the module's one, and serves a unix socket only root may open, with a closed list of calls: emit an event of the module, put, delete, get or list a key of the states applied, schedule and findings, and invoke zfs_snapshots_record, zfs_snapshots_schedule or the service manager's start of the module's two units. With the keeper not answering they act as with the bus down and warn.

Scheduled ZFS snapshots

Each kind with a keep count above 0 is due at the first run of the hour, day, Monday or month, against the ledger's last entry. For every filesystem of the pool the run takes zfs snapshot -- <dataset>@mesh-<kind>-<UTC stamp>, never recursive, sets mesh:schedule, and records taken in the ledger snapshots (one instance per dataset and kind, its state a map keyed by the ZFS snapshot's name) through the tool zfs_snapshots_record. It takes none it cannot record, none with the ledger unreadable or full, and none while the pool is more than 80 % full. It removes the oldest past a keep count only by their own record: name, mesh:schedule and ledger entry fit, no hold and no clone. A snapshot taken by hand (zfs_snapshot, which refuses a mesh- name, event snapshot.taken.person) is never removed.

Tools

The tools run as the operator account. What ZFS keeps for root goes through sudo -n with a fixed argument list, never a shell. A command that runs out of time is ended with everything it started.

tool	what
zfs_datasets, zfs_snapshots, zfs_pools, zfs_space, zfs_pool_layout, zfs_scrub, zfs_io, zfs_arc, zfs_events, zfs_properties, zfs_health	the reads (below)
zfs_snapshot	takes dataset@name, a person's act with why
zfs_scrub_start, zfs_scrub_pause, zfs_scrub_stop	a person's scrub acts
zfs_config	every value with its config value, what runs, how it stands, the package's files, and what the module says before it leaves
zfs_config_apply	converge now, as the apply does
zfs_snapshots_schedule	the keep counts, next moments, what the next run would remove, and the ledger
zfs_snapshots_run_now, zfs_snapshots_prune_now, zfs_snapshots_pause, zfs_snapshots_resume	a person's acts on the scheduled ZFS snapshots; the hand acts answer "started" at once and zfs_module_events reads the outcome
zfs_module_events	the module's own events, newest first (zfs_events reads ZFS's)
zfs_snapshots_check	the ledger against what ZFS holds; changes nothing
zfs_findings	the follower's findings
zfs_snapshots_record	the run's own record of a taken or removed ZFS snapshot

When the module leaves a node, the next load of the kernel module uses ZFS's defaults, zfs-import-cache, zfs-mount and zfs.target stay running, /etc/hostid stays, and pool and dataset properties and devices stay as the last apply set them.

The snapshot

zfs_snapshot {dataset, name, why} snapshots one dataset, never its children. There is no recursive, and passing one is refused. It refuses all of these before anything runs:

• a dataset that zfs list does not show on this machine;
• a name with anything but letters, digits, ., _ and -, or a whole name longer than 255 characters;
• a snapshot that already exists;
• a missing why, or a why longer than one line.

It runs sudo -n zfs snapshot -- <dataset>@<name>. It answers from zfs list afterwards, not from what it ran: the snapshot made, its creation time, and any other snapshot that came or went during the call. It also gives the dataset's free space and the pool's capacity. From then on, the snapshot holds every block of the dataset that is deleted or overwritten, so that space is not freed until the snapshot is destroyed by hand. The scheduled run destroys only the scheduled ZFS snapshots it took and recorded, by their own record.

The act is recorded as the event snapshot.taken.person, and ZFS keeps the command in the pool's own history.

The scrub schedule

The module owns the schedule of the pool its pool setting names (before this version scrub-pool, which mesh-config renames).

setting	default	what
pool	none: the operator's value	the pool to scrub
scrub-cadence	weekly	weekly, or monthly (the first given weekday of each month)
scrub-day	Mon	Mon, Tue, Wed, Thu, Fri, Sat or Sun, nothing else
scrub-time	00:00	HH:MM from 00:00 to 23:59, in the machine's time zone

The defaults are what the distribution's weekly timer did on the home server before the module, so taking it over does not move the schedule. The operator chose monthly for the home server.

The scheduled run. The node-engine runs the module's scheduled process zfs-mesh-run (zfs-tools run, as root) from its own timer zfs-mesh-run.timer, every quarter of an hour. The run reads /etc/zfs-tools/config.json. It finds the slot, which is the last scheduled time not after now. If the slot is not yet served, the run starts the scrub with zpool scrub -- <pool> and returns at once: it never waits for it, and the event follower hears the scrub end. If a scrub or resilver runs, or the pool is not imported, the run does nothing and says run.skipped.

It never pauses or stops a scrub. A resilver that is running defers the scrub to the next run after it. The served slot is kept in /var/lib/zfs-scrub/scrub-served.json, in a folder of root's alone (0700), apart from the account's state. The run writes it only into a real folder of its own and never through a link: a new file opened with O_NOFOLLOW and O_EXCL, then renamed over the old one. The tools run as the operator account and cannot read it, so they judge a missed scrub from status. Without that file, a scrub that status shows started in the slot serves it. So a scrub starts within 15 minutes of its time, and after the machine was off, within 15 minutes of its coming back.

No setting value reaches a unit file. The day, the time and the cadence go only into the configuration. The run checks them against fixed sets (the seven weekday names, and HH:MM). It refuses any other value and runs nothing; a value like *:* or Mon..Fri is impossible. The pool's name reaches the names of the two distribution timers. The controller fills it there and refuses anything but letters, digits, :, _, . and - (mesh-controller #171).

Exactly one schedule per pool. The distribution's zfs-scrub-weekly@<pool>.timer and zfs-scrub-monthly@<pool>.timer are held stopped and disabled, and the trim timers with them while trim-schedule is set. They come after the scheduled run in the declaration, so a run that cannot be installed stops the apply before the timer that scrubs the pool today is touched. On the home server the weekly one was enabled by hand, outside the mesh. A scrub already running is not stopped by this; only its timer is.

Unassigning the module gives the distribution's timer back running and enabled, as the node-engine found it, once mesh-host #66 (hq ADR 0269, proposed) is merged. Until then the node-engine never starts a unit on the way out, and unassigning would leave the pool with no scrub schedule. Do not unassign the module before #66 is merged.

One pool per assignment in this version. A manifest's resources are fixed, and a setting cannot make one timer per pool in a list. The home server has one pool.

The scrub acts

zfs_scrub_start, zfs_scrub_pause and zfs_scrub_stop take {pool, why}. They refuse a pool that zpool list does not show, and a missing why. They run sudo -n zpool scrub [-p|-s] -- <pool> and answer what status says afterwards. Each is recorded as the event scrub.requested.

A scrub reads every block and rewrites, from the pool's redundancy, only a block whose checksum is wrong. It changes no file's content. It adds read load for its whole length, which on the home server's pool was about 46 hours for the weekly scrub that ended 2026-10-06. These three are candidates for the approval mark (novox/hq research 039): a person's acts that never change data, and that cost hours of I/O.

Resilvers

ZFS starts a resilver by itself when a disk is replaced or reattached, so there is no schedule to own. The module watches and reports it:

• zfs_scrub shows a running resilver, pool-wide or a vdev's rebuild, with its progress and the time to go;
• the health check warns while one runs, and is unhealthy when one finished with errors;
• the event follower writes its start and finish to the journal.

Health

The module runs two long-running processes of its own binary, and the scheduled run:

• The watch (zfs-tools watch) runs as the operator account and carries the health check, zfs_health (novox/hq ADR 0240). Each minute it reads the pools and writes a line in its journal whenever their health changes.
• The event follower (zfs-tools events) runs as root and follows zpool events -f. It is described in the next section.
• The scheduled run (zfs-tools run) is not long-running and has no health check of its own; zfs_health says when it failed or did not start a due scrub.

zfs_health is unhealthy for a fault now:

• a pool not ONLINE (the answer gives status's reason and action), or a device not online;
• a device's error counts that rose since the last look. A rise stays unhealthy for 15 minutes, so the node-engine's two looks see it;
• data errors, or a resilver that finished with errors;
• a pool more than 90 % full;
• no pool imported, or no ZFS on the machine.

These are warnings, which leave it healthy:

• error counts from before ("N past checksum errors; clear them by hand once understood"). The first look after the bundle starts takes every count as past;
• a pool more than 80 % full;
• a resilver running;
• a last finished scrub older than the schedule allows (weekly 9 days, monthly 38 days), or 35 days for a pool without a schedule;
• a scheduled scrub that has not started an hour after its time;
• a scheduled run that failed, with no scrub finished since;
• a schedule whose timer is not active and enabled, or whose settings are not valid.

The error counts last seen are kept in /var/lib/zfs-tools/errors.json, so a restart of the bundle does not take every count as past again.

The event journal

ZFS keeps its events in a queue in the kernel, in memory, from the moment its module loads, so a reboot loses them. The follower reads the queue from its oldest event, then follows. So the pool's import, and whatever came while the follower was down, reach the journal too. It writes each error and change of state in one line, resilver and scrub starts and ends among them. It also writes ZFS's own "dropped N events" line when the queue overflowed before it was read.

A cursor of the boot and the last event read is kept in /run/zfs-events/cursor, which a reboot empties as it empties ZFS's queue. With it, a follower that restarts does not write an event twice. After five follows in a row that end within a minute, the follower exits non-zero, so the service manager counts the restarts and the unit's state shows the failure.

What survives a reboot is what reached the journal. That holds where journald keeps its journal on disk. On the home server it does: its journal is in /var/log/journal, and it holds entries from before the reboot of 2026-10-04 02:09, as read through the mesh on 2026-10-09. Events ZFS emits after the follower has stopped during a shutdown are lost with the queue.

Tests

go test ./... runs against OpenZFS 2.4.4 output in cmd/zfs-tools/testdata. Each file has the form that version's zpool_main.c and libzfs print, with documentation names in place of the real pools and disks. The tests cover:

• parsing every command's output, including the scan lines (human sizes, rebuilds per vdev, error scrubs, paused scrubs) and iostat -H without section lines;
• a timed-out command ended with its children;
• the snapshot's refusals, read-back, space statement and unknown outcome;
• the scrub acts, the schedule's slots, and the scheduled run (start and return, never pause, defer to a resilver, refuse an invalid setting);
• the config's parse, one table per value; each convergence plan (properties, devices, the node's ZFS id, the ARC); the lock; the scheduled ZFS snapshots' taking and removal; the hand acts; the keeper's closed list of calls and its socket; the tools; the health;
• the follower's cursor, dropped events, kill before wait, and exit after repeated failures;
• the health verdicts, past against new errors;
• a machine without ZFS;
• the manifest (values, files, units, order, bus state, keeper) against the tools the bundle serves and the call-site test.

The module adds no integration test (its test budget is 500 unit tests and 50 integration tests): the main design's live tests stand for the module whole.

Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

• /var/lib/zfs-tools: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• /run/zfs-events: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• /etc/zfs-tools: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• /var/lib/zfs-scrub: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• /run/zfs-tools: everything in it is the program's (the lock); the mesh moves nothing out.
• the keeper's folder: everything in it is the program's (its socket and its credential).
• /etc/zfs/mesh-zed.d: the module's own, while zed is on.