mesh-catalog/modules/zfs/README.md

zfs

The mesh's view of ZFS on a machine: every pool's health, layout, space, scrubs, I/O, cache and events, and every dataset with its space and snapshots. It owns the pool's scrub schedule. It has four acts: a named snapshot of one dataset, and a scrub's start, pause and stop. It is for the home server, whose pool holds the operator's media library, which the mesh does not own (novox/hq ADR 0051). So the module reads, and what it does only adds a snapshot or reads the pool through.

What it does not do

There is no destroy, rollback, set, send, receive, rename, clone, replace, attach, detach or clear. Each of those changes or removes the operator's data, or the pool's layout, and needs a decision of its own first. A resilver is ZFS's own: it starts one when a disk is replaced or comes back. The module watches and reports resilvers, and starts none.

This is checked by TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub. Every command the bundle runs is built by one of its argument functions, and each function is held to its program and verb:

• zfs: list, get and snapshot;
• zpool: list, status, iostat, events, history and scrub (with -p or -s only);
• systemctl: show.

The test fails on any call site that builds its own argument list, through ask, AsRoot, Escalated, .Run(, exec.Command(, exec.CommandContext(, os.StartProcess or syscall.Exec.

It installs nothing. ZFS belongs to the machine, and its kernel module is built against that machine's kernel. On a machine without ZFS, every tool answers applicable: false with the reason, and the health check is unhealthy and says to unassign the module, because a module put where it does nothing must not look healthy.

It holds no seat. A new node-* seat is the controller's to define (novox/hq ADR 0121, ADR 0246).

Tools

The tools run as the operator account. What ZFS lets any account read runs as that account. What ZFS keeps for root goes through sudo -n with a fixed argument list, never a shell: events, history, the snapshot and the scrub acts. Sizes are whole bytes, read under -p, except where status prints human sizes even under -p (a scan's sizes). Each answer also says its findings in lines under said.

A command that runs out of time is ended with everything it started. It runs in a process group of its own. At the deadline the group gets SIGTERM, which sudo passes on to the command it runs as root, and after five seconds SIGKILL. An act that ran out of time is answered from what the machine shows afterwards, or as "outcome unknown".

tool	what	replaces
zfs_datasets	Every filesystem and volume: mountpoint, used, available, referenced, compression ratio, sharenfs, and the number of snapshots with the space they hold together	zfs list
zfs_snapshots	The snapshots of one dataset (optional), oldest first: creation time, used (the space only that snapshot holds) and referenced	zfs list -t snapshot
zfs_snapshot	Takes dataset@name (see below)	zfs snapshot
zfs_pools	Each pool's state with status's reason and action, space, capacity and fragmentation %, device error count, data errors and scans	zpool list, zpool status -x
zfs_space	Each pool's space, then every dataset, biggest first, with what uses its space, quotas and reservations, compression and the savings	zfs list -o space, zpool list -v
zfs_pool_layout	Vdevs and disks, with state, error counts and status's note; log, cache, special and dedup devices; spares	zpool status, zpool status -v
zfs_scrub	Each pool's scans: its scrub or error scrub, and its resilver (pool-wide, or each vdev's rebuild), with progress and time to go while one runs; the days since the last finished scrub; and the module's schedule with its next run. history: true also reads the pool's history	zpool history -i
zfs_scrub_start	Starts a scrub of pool, or resumes a paused one, with why	zpool scrub
zfs_scrub_pause	Pauses one, keeping its place, with why	zpool scrub -p
zfs_scrub_stop	Stops one, losing its place, with why	zpool scrub -s
zfs_io	A sample of seconds (default 3, from 1 to 10): operations and bytes per second per pool, vdev and disk	zpool iostat -v
zfs_arc	ARC size, target, minimum and maximum, hit ratio since boot, and L2ARC where present	arcstat, arc_summary
zfs_events	Recent events, newest first, as root. By default it shows only errors and changes of state, and all: true shows every event	zpool events -v
zfs_properties	Every property of one dataset with its source. The value of keylocation is withheld	zfs get all
zfs_health	The health check (below)

The snapshot

zfs_snapshot {dataset, name, why} snapshots one dataset, never its children. There is no recursive, and passing one is refused. It refuses all of these before anything runs:

• a dataset that zfs list does not show on this machine;
• a name with anything but letters, digits, ., _ and -, or a whole name longer than 255 characters;
• a snapshot that already exists;
• a missing why, or a why longer than one line.

It runs sudo -n zfs snapshot -- <dataset>@<name>. It answers from zfs list afterwards, not from what it ran: the snapshot made, its creation time, and any other snapshot that came or went during the call. It also gives the dataset's free space and the pool's capacity. From then on, the snapshot holds every block of the dataset that is deleted or overwritten, so that space is not freed until the snapshot is destroyed by hand. No tool of the mesh destroys a snapshot.

The act is recorded as the event snapshot.taken, and ZFS keeps the command in the pool's own history.

The scrub schedule

The module owns the schedule of the pool its scrub-pool setting names.

setting	default	what
scrub-pool	none: the operator's value	the pool to scrub
scrub-cadence	weekly	weekly, or monthly (the first given weekday of each month)
scrub-day	Mon	Mon to Sun
scrub-time	00:00	HH:MM, in the machine's time zone

The defaults are what the distribution's weekly timer did on the home server before the module, so taking it over does not move the schedule.

The mesh writes two timer templates of the module's own, zfs-mesh-scrub-weekly@.timer and zfs-mesh-scrub-monthly@.timer, and the service they start, zfs-mesh-scrub@.service. The service runs /usr/bin/zpool scrub -w -- <pool> with no shell; stopping it while a scrub runs pauses the scrub, as the distribution's unit does. The mesh enables the one timer the cadence names, for the one pool.

Exactly one schedule per pool. In the same declaration, the distribution's own zfs-scrub-weekly@<pool>.timer and zfs-scrub-monthly@<pool>.timer are held stopped and disabled. On the home server the weekly one was enabled by hand, outside the mesh. A scrub already running is not stopped by this; only its timer is. The day and time cannot be expressed with the distribution's templates, which fire at "weekly" or "monthly" with a random delay, so the module's own timer replaces them.

Unassigning the module leaves the pool with no scrub schedule. The node-engine gives a unit back as it found it but never starts one on the way out: the distribution's timer stays stopped and disabled, and the module's own timer and units go. Re-enable a schedule by hand, or assign the module again.

One pool per assignment in this version. A manifest's resources are fixed, and a setting cannot make one timer per pool in a list. The home server has one pool. A second pool needs either a second set of resources or a controller feature that makes resources per item of a list.

The setting values reach the timer file as they are. The tools check them (zfs_scrub and the health say an invalid one), and the service manager refuses a calendar it cannot read. The values are the operator's own, so they are not escaped.

The scrub acts

zfs_scrub_start, zfs_scrub_pause and zfs_scrub_stop take {pool, why}. They refuse a pool that zpool list does not show, and a missing why. They run sudo -n zpool scrub [-p|-s] -- <pool> and answer what status says afterwards. Each is recorded as the event scrub.requested.

A scrub reads every block and rewrites, from the pool's redundancy, only a block whose checksum is wrong. It changes no file's content. It adds read load for its whole length, which on the home server's pool was about 46 hours for the weekly scrub that ended 2026-10-06. These three are candidates for the approval mark (novox/hq research 039): a person's acts that never change data, and that cost hours of I/O.

Resilvers

ZFS starts a resilver by itself when a disk is replaced or reattached, so there is no schedule to own. The module watches and reports it:

• zfs_scrub shows a running resilver, pool-wide or a vdev's rebuild, with its progress and the time to go;
• the health check warns while one runs, and is unhealthy when one finished with errors;
• the event follower writes its start and finish to the journal.

Health

The module runs two long-running processes of its own binary, and the timer:

• The watch (zfs-tools watch) runs as the operator account and carries the health check, zfs_health (novox/hq ADR 0240). Each minute it reads the pools and writes a line in its journal whenever their health changes.
• The event follower (zfs-tools events) runs as root and follows zpool events -f. It is described in the next section.
• The scrub timer is judged by its unit.

zfs_health is unhealthy for a fault now:

• a pool not ONLINE (the answer gives status's reason and action), or a device not online;
• a device's error counts that rose since the last look. A rise stays unhealthy for 15 minutes, so the node-engine's two looks see it;
• data errors, or a resilver that finished with errors;
• a pool more than 90 % full;
• no pool imported, or no ZFS on the machine.

These are warnings, which leave it healthy:

• error counts from before ("N past checksum errors; clear them by hand once understood"). The first look after the bundle starts takes every count as past;
• a pool more than 80 % full;
• a resilver running;
• a last finished scrub older than the schedule allows (weekly 9 days, monthly 38 days), or 35 days for a pool without a schedule;
• a scheduled scrub that failed, with no scrub finished since;
• a schedule whose timer is not active and enabled, or whose settings are not valid.

The event journal

ZFS keeps its events in a queue in the kernel, in memory, from the moment its module loads, so a reboot loses them. The follower reads the queue from its oldest event, then follows. So the pool's import, and whatever came while the follower was down, reach the journal too. It writes each error and change of state in one line, resilver and scrub starts and ends among them. It also writes ZFS's own "dropped N events" line when the queue overflowed before it was read.

A cursor of the boot and the last event read is kept in /run/zfs-events/cursor, which a reboot empties as it empties ZFS's queue. With it, a follower that restarts does not write an event twice. After five follows in a row that end within a minute, the follower exits non-zero, so the service manager counts the restarts and the unit's state shows the failure.

What survives a reboot is what reached the journal. That holds where journald keeps its journal on disk. On the home server it does: its journal is in /var/log/journal, and it holds entries from before the reboot of 2026-10-04 02:09, as read through the mesh on 2026-10-09. Events ZFS emits after the follower has stopped during a shutdown are lost with the queue.

Tests

go test ./... runs against OpenZFS 2.4.4 output in cmd/zfs-tools/testdata. Each file has the form that version's zpool_main.c and libzfs print, with documentation names in place of the real pools and disks. The tests cover:

• parsing every command's output, including the scan lines (human sizes, rebuilds per vdev, error scrubs, paused scrubs) and iostat -H without section lines;
• a timed-out command ended with its children;
• the snapshot's refusals, read-back, space statement and unknown outcome;
• the scrub acts and the schedule;
• the follower's cursor, dropped events, kill before wait, and exit after repeated failures;
• the health verdicts, past against new errors;
• a machine without ZFS;
• the manifest against the tools the bundle serves.