mesh-catalog/modules/zfs/README.md

zfs

The mesh's view of ZFS on a machine: every pool's health, layout, space, scrubs, I/O, cache and events, and every dataset with its space and snapshots. It owns the pool's scrub schedule, run by a scheduled process of its own. It has four acts: a named snapshot of one dataset, and a scrub's start, pause and stop. It is for the home server, whose pool holds the operator's media library, which the mesh does not own (novox/hq ADR 0051). So the module reads, and what it does only adds a snapshot or reads the pool through.

What it does not do

There is no destroy, rollback, set, send, receive, rename, clone, replace, attach, detach or clear. Each of those changes or removes the operator's data, or the pool's layout, and needs a decision of its own first. A resilver is ZFS's own: it starts one when a disk is replaced or comes back. The module watches and reports resilvers, and starts none.

This is checked by TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub. Every command the bundle runs is built by one of its argument functions, and each function is held to its program and verb:

• zfs: list, get and snapshot;
• zpool: list, status, iostat, events, history, wait -t scrub and scrub (with -p or -s only, and those only from the person's acts, never from the scheduled run);
• systemctl: show.

The test fails on any call site that builds its own argument list, through ask, AsRoot, Escalated, .Run(, exec.Command(, exec.CommandContext(, os.StartProcess or syscall.Exec.

It installs nothing. ZFS belongs to the machine, and its kernel module is built against that machine's kernel. On a machine without ZFS, every tool answers applicable: false with the reason, and the health check is unhealthy and says to unassign the module, because a module put where it does nothing must not look healthy.

It holds no seat. A new node-* seat is the controller's to define (novox/hq ADR 0121, ADR 0246).

Tools

The tools run as the operator account. What ZFS lets any account read runs as that account. What ZFS keeps for root goes through sudo -n with a fixed argument list, never a shell: events, history, the snapshot and the scrub acts. Sizes are whole bytes, read under -p, except where status prints human sizes even under -p (a scan's sizes). Each answer also says its findings in lines under said.

A command that runs out of time is ended with everything it started. It runs in a process group of its own. At the deadline the group gets SIGTERM, which sudo passes on to the command it runs as root, and after five seconds SIGKILL. An act that ran out of time is answered from what the machine shows afterwards, or as "outcome unknown".

tool	what	replaces
zfs_datasets	Every filesystem and volume: mountpoint, used, available, referenced, compression ratio, sharenfs, and the number of snapshots with the space they hold together	zfs list
zfs_snapshots	The snapshots of one dataset (optional), oldest first: creation time, used (the space only that snapshot holds) and referenced	zfs list -t snapshot
zfs_snapshot	Takes dataset@name (see below)	zfs snapshot
zfs_pools	Each pool's state with status's reason and action, space, capacity and fragmentation %, device error count, data errors and scans	zpool list, zpool status -x
zfs_space	Each pool's space, then every dataset, biggest first, with what uses its space, quotas and reservations, compression and the savings	zfs list -o space, zpool list -v
zfs_pool_layout	Vdevs and disks, with state, error counts and status's note; log, cache, special and dedup devices; spares	zpool status, zpool status -v
zfs_scrub	Each pool's scans: its scrub or error scrub, and its resilver (pool-wide, or each vdev's rebuild), with progress and time to go while one runs; the days since the last finished scrub; and the module's schedule with its next run. history: true also reads the pool's history	zpool history -i
zfs_scrub_start	Starts a scrub of pool, or resumes a paused one, with why	zpool scrub
zfs_scrub_pause	Pauses one, keeping its place, with why	zpool scrub -p
zfs_scrub_stop	Stops one, losing its place, with why	zpool scrub -s
zfs_io	A sample of seconds (default 3, from 1 to 10): operations and bytes per second per pool, vdev and disk	zpool iostat -v
zfs_arc	ARC size, target, minimum and maximum, hit ratio since boot, and L2ARC where present	arcstat, arc_summary
zfs_events	Recent events, newest first, as root. By default it shows only errors and changes of state, and all: true shows every event	zpool events -v
zfs_properties	Every property of one dataset with its source. The value of keylocation is withheld	zfs get all
zfs_health	The health check (below)

The snapshot

zfs_snapshot {dataset, name, why} snapshots one dataset, never its children. There is no recursive, and passing one is refused. It refuses all of these before anything runs:

• a dataset that zfs list does not show on this machine;
• a name with anything but letters, digits, ., _ and -, or a whole name longer than 255 characters;
• a snapshot that already exists;
• a missing why, or a why longer than one line.

It runs sudo -n zfs snapshot -- <dataset>@<name>. It answers from zfs list afterwards, not from what it ran: the snapshot made, its creation time, and any other snapshot that came or went during the call. It also gives the dataset's free space and the pool's capacity. From then on, the snapshot holds every block of the dataset that is deleted or overwritten, so that space is not freed until the snapshot is destroyed by hand. No tool of the mesh destroys a snapshot.

The act is recorded as the event snapshot.taken, and ZFS keeps the command in the pool's own history.

The scrub schedule

The module owns the schedule of the pool its scrub-pool setting names.

setting	default	what
scrub-pool	none: the operator's value	the pool to scrub
scrub-cadence	weekly	weekly, or monthly (the first given weekday of each month)
scrub-day	Mon	Mon, Tue, Wed, Thu, Fri, Sat or Sun, nothing else
scrub-time	00:00	HH:MM from 00:00 to 23:59, in the machine's time zone

The defaults are what the distribution's weekly timer did on the home server before the module, so taking it over does not move the schedule. The operator chose monthly for the home server.

The scheduled run. The node-engine runs the module's scheduled process zfs-mesh-scrub (zfs-tools scrub-run, as root) from its own timer zfs-mesh-scrub.timer, every quarter of an hour. The run reads the configuration the mesh writes from the settings (/etc/zfs-tools/scrub.conf). It finds the slot, which is the last scheduled time not after now. If the slot is not yet served, the run acts as upstream's zfs-scrub@.service does:

• if a scrub is running, it waits for it with zpool wait -t scrub -- <pool>;
• otherwise it starts one with zpool scrub -- <pool> and waits for it the same way;
• if the start is refused because a scrub has just begun, it waits for that one.

It never pauses or stops a scrub. The unit has no ExecStop, so stopping the unit ends only the wait. A resilver that is running defers the scrub to the next run after it. The served slot is kept in /var/lib/zfs-scrub/scrub-served.json, in a folder of root's alone (0700), apart from the account's state. The run writes it only into a real folder of its own and never through a link: a new file opened with O_NOFOLLOW and O_EXCL, then renamed over the old one. The tools run as the operator account and cannot read it, so they judge a missed scrub from status. Without that file, a scrub that status shows started in the slot serves it. So a scrub starts within 15 minutes of its time, and after the machine was off, within 15 minutes of its coming back.

No setting value reaches a unit file. The day, the time and the cadence go only into the configuration. The run checks them against fixed sets (the seven weekday names, and HH:MM). It refuses any other value and runs nothing; a value like *:* or Mon..Fri is impossible. The pool's name reaches the names of the two distribution timers. The controller fills it there and refuses anything but letters, digits, :, _, . and - (mesh-controller #171).

Exactly one schedule per pool. The distribution's zfs-scrub-weekly@<pool>.timer and zfs-scrub-monthly@<pool>.timer are held stopped and disabled. They come after the scheduled run in the declaration, so a run that cannot be installed stops the apply before the timer that scrubs the pool today is touched. On the home server the weekly one was enabled by hand, outside the mesh. A scrub already running is not stopped by this; only its timer is.

Unassigning the module gives the distribution's timer back running and enabled, as the node-engine found it, once mesh-host #66 (hq ADR 0269, proposed) is merged. Until then the node-engine never starts a unit on the way out, and unassigning would leave the pool with no scrub schedule. Do not unassign the module before #66 is merged.

One pool per assignment in this version. A manifest's resources are fixed, and a setting cannot make one timer per pool in a list. The home server has one pool.

The scrub acts

zfs_scrub_start, zfs_scrub_pause and zfs_scrub_stop take {pool, why}. They refuse a pool that zpool list does not show, and a missing why. They run sudo -n zpool scrub [-p|-s] -- <pool> and answer what status says afterwards. Each is recorded as the event scrub.requested.

A scrub reads every block and rewrites, from the pool's redundancy, only a block whose checksum is wrong. It changes no file's content. It adds read load for its whole length, which on the home server's pool was about 46 hours for the weekly scrub that ended 2026-10-06. These three are candidates for the approval mark (novox/hq research 039): a person's acts that never change data, and that cost hours of I/O.

Resilvers

ZFS starts a resilver by itself when a disk is replaced or reattached, so there is no schedule to own. The module watches and reports it:

• zfs_scrub shows a running resilver, pool-wide or a vdev's rebuild, with its progress and the time to go;
• the health check warns while one runs, and is unhealthy when one finished with errors;
• the event follower writes its start and finish to the journal.

Health

The module runs two long-running processes of its own binary, and the scheduled run:

• The watch (zfs-tools watch) runs as the operator account and carries the health check, zfs_health (novox/hq ADR 0240). Each minute it reads the pools and writes a line in its journal whenever their health changes.
• The event follower (zfs-tools events) runs as root and follows zpool events -f. It is described in the next section.
• The scheduled run (zfs-tools scrub-run) is not long-running and has no health check of its own; zfs_health says when it failed or did not start a due scrub.

zfs_health is unhealthy for a fault now:

• a pool not ONLINE (the answer gives status's reason and action), or a device not online;
• a device's error counts that rose since the last look. A rise stays unhealthy for 15 minutes, so the node-engine's two looks see it;
• data errors, or a resilver that finished with errors;
• a pool more than 90 % full;
• no pool imported, or no ZFS on the machine.

These are warnings, which leave it healthy:

• error counts from before ("N past checksum errors; clear them by hand once understood"). The first look after the bundle starts takes every count as past;
• a pool more than 80 % full;
• a resilver running;
• a last finished scrub older than the schedule allows (weekly 9 days, monthly 38 days), or 35 days for a pool without a schedule;
• a scheduled scrub that has not started an hour after its time;
• a scheduled run that failed, with no scrub finished since;
• a schedule whose timer is not active and enabled, or whose settings are not valid.

The error counts last seen are kept in /var/lib/zfs-tools/errors.json, so a restart of the bundle does not take every count as past again.

The event journal

ZFS keeps its events in a queue in the kernel, in memory, from the moment its module loads, so a reboot loses them. The follower reads the queue from its oldest event, then follows. So the pool's import, and whatever came while the follower was down, reach the journal too. It writes each error and change of state in one line, resilver and scrub starts and ends among them. It also writes ZFS's own "dropped N events" line when the queue overflowed before it was read.

A cursor of the boot and the last event read is kept in /run/zfs-events/cursor, which a reboot empties as it empties ZFS's queue. With it, a follower that restarts does not write an event twice. After five follows in a row that end within a minute, the follower exits non-zero, so the service manager counts the restarts and the unit's state shows the failure.

What survives a reboot is what reached the journal. That holds where journald keeps its journal on disk. On the home server it does: its journal is in /var/log/journal, and it holds entries from before the reboot of 2026-10-04 02:09, as read through the mesh on 2026-10-09. Events ZFS emits after the follower has stopped during a shutdown are lost with the queue.

Tests

go test ./... runs against OpenZFS 2.4.4 output in cmd/zfs-tools/testdata. Each file has the form that version's zpool_main.c and libzfs print, with documentation names in place of the real pools and disks. The tests cover:

• parsing every command's output, including the scan lines (human sizes, rebuilds per vdev, error scrubs, paused scrubs) and iostat -H without section lines;
• a timed-out command ended with its children;
• the snapshot's refusals, read-back, space statement and unknown outcome;
• the scrub acts, the schedule's slots, and the scheduled run (start, wait, never pause, defer to a resilver, refuse an invalid setting);
• the follower's cursor, dropped events, kill before wait, and exit after repeated failures;
• the health verdicts, past against new errors;
• a machine without ZFS;
• the manifest against the tools the bundle serves.