mesh-catalog/modules/zfs/README.md

# zfs

The mesh's view of ZFS on a machine: every pool's health, layout, space, scrubs, I/O, cache and events,
and every dataset with its space and snapshots. It owns the pool's scrub schedule, run by a scheduled process of its own. It has four acts: a named
snapshot of one dataset, and a scrub's start, pause and stop. It is for the home server, whose pool holds
the operator's media library, which the mesh does not own (novox/hq ADR 0051). So the module reads, and
what it does only adds a snapshot or reads the pool through.

## What it does not do

There is no destroy, rollback, set, send, receive, rename, clone, replace, attach, detach or clear. Each of
those changes or removes the operator's data, or the pool's layout, and needs a decision of its own first.
A resilver is ZFS's own: it starts one when a disk is replaced or comes back. The module watches and
reports resilvers, and starts none.

This is checked by `TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`. Every command the bundle runs is
built by one of its argument functions, and each function is held to its program and verb:

- `zfs`: `list`, `get` and `snapshot`;
- `zpool`: `list`, `status`, `iostat`, `events`, `history`, `wait -t scrub` and `scrub` (with `-p` or `-s`
  only, and those only from the person's acts, never from the scheduled run);
- `systemctl`: `show`.

The test fails on any call site that builds its own argument list, through `ask`, `AsRoot`, `Escalated`,
`.Run(`, `exec.Command(`, `exec.CommandContext(`, `os.StartProcess` or `syscall.Exec`.

It installs nothing. ZFS belongs to the machine, and its kernel module is built against that machine's
kernel. On a machine without ZFS, every tool answers `applicable: false` with the reason, and the health
check is unhealthy and says to unassign the module, because a module put where it does nothing must not
look healthy.

It holds no seat. A new `node-*` seat is the controller's to define (novox/hq ADR 0121, ADR 0246).

## Tools

The tools run as the operator account. What ZFS lets any account read runs as that account. What ZFS
keeps for root goes through `sudo -n` with a fixed argument list, never a shell: events, history, the
snapshot and the scrub acts. Sizes are whole bytes, read under `-p`, except where status prints human sizes
even under `-p` (a scan's sizes). Each answer also says its findings in lines under `said`.

A command that runs out of time is ended with everything it started. It runs in a process group of its
own. At the deadline the group gets SIGTERM, which sudo passes on to the command it runs as root, and
after five seconds SIGKILL. An act that ran out of time is answered from what the machine shows afterwards,
or as "outcome unknown".

| tool | what | replaces |
|---|---|---|
| `zfs_datasets` | Every filesystem and volume: mountpoint, used, available, referenced, compression ratio, sharenfs, and the number of snapshots with the space they hold together | `zfs list` |
| `zfs_snapshots` | The snapshots of one `dataset` (optional), oldest first: creation time, used (the space only that snapshot holds) and referenced | `zfs list -t snapshot` |
| `zfs_snapshot` | Takes `dataset@name` (see below) | `zfs snapshot` |
| `zfs_pools` | Each pool's state with status's reason and action, space, capacity and fragmentation %, device error count, data errors and scans | `zpool list`, `zpool status -x` |
| `zfs_space` | Each pool's space, then every dataset, biggest first, with what uses its space, quotas and reservations, compression and the savings | `zfs list -o space`, `zpool list -v` |
| `zfs_pool_layout` | Vdevs and disks, with state, error counts and status's note; log, cache, special and dedup devices; spares | `zpool status`, `zpool status -v` |
| `zfs_scrub` | Each pool's scans: its scrub or error scrub, and its resilver (pool-wide, or each vdev's rebuild), with progress and time to go while one runs; the days since the last finished scrub; and the module's schedule with its next run. `history: true` also reads the pool's history | `zpool history -i` |
| `zfs_scrub_start` | Starts a scrub of `pool`, or resumes a paused one, with `why` | `zpool scrub` |
| `zfs_scrub_pause` | Pauses one, keeping its place, with `why` | `zpool scrub -p` |
| `zfs_scrub_stop` | Stops one, losing its place, with `why` | `zpool scrub -s` |
| `zfs_io` | A sample of `seconds` (default 3, from 1 to 10): operations and bytes per second per pool, vdev and disk | `zpool iostat -v` |
| `zfs_arc` | ARC size, target, minimum and maximum, hit ratio since boot, and L2ARC where present | `arcstat`, `arc_summary` |
| `zfs_events` | Recent events, newest first, as root. By default it shows only errors and changes of state, and `all: true` shows every event | `zpool events -v` |
| `zfs_properties` | Every property of one `dataset` with its source. The value of `keylocation` is withheld | `zfs get all` |
| `zfs_health` | The health check (below) | |

## The snapshot

`zfs_snapshot {dataset, name, why}` snapshots one dataset, never its children. There is no `recursive`,
and passing one is refused. It refuses all of these before anything runs:

- a dataset that `zfs list` does not show on this machine;
- a name with anything but letters, digits, `.`, `_` and `-`, or a whole name longer than 255 characters;
- a snapshot that already exists;
- a missing `why`, or a `why` longer than one line.

It runs `sudo -n zfs snapshot -- <dataset>@<name>`. It answers from `zfs list` afterwards, not from what
it ran: the snapshot made, its creation time, and any other snapshot that came or went during the call. It
also gives the dataset's free space and the pool's capacity. **From then on, the snapshot holds every block
of the dataset that is deleted or overwritten, so that space is not freed until the snapshot is destroyed by
hand.** No tool of the mesh destroys a snapshot.

The act is recorded as the event `snapshot.taken`, and ZFS keeps the command in the pool's own history.

## The scrub schedule

The module owns the schedule of the pool its `scrub-pool` setting names.

| setting | default | what |
|---|---|---|
| `scrub-pool` | none: the operator's value | the pool to scrub |
| `scrub-cadence` | `weekly` | `weekly`, or `monthly` (the first given weekday of each month) |
| `scrub-day` | `Mon` | `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat` or `Sun`, nothing else |
| `scrub-time` | `00:00` | `HH:MM` from `00:00` to `23:59`, in the machine's time zone |

The defaults are what the distribution's weekly timer did on the home server before the module, so taking
it over does not move the schedule. The operator chose monthly for the home server.

**The scheduled run.** The node-engine runs the module's scheduled process `zfs-mesh-scrub` (`zfs-tools
scrub-run`, as root) from its own timer `zfs-mesh-scrub.timer`, every quarter of an hour. The run reads the
configuration the mesh writes from the settings (`/etc/zfs-tools/scrub.conf`). It finds the slot, which is
the last scheduled time not after now. If the slot is not yet served, the run acts as upstream's
`zfs-scrub@.service` does:

- if a scrub is running, it waits for it with `zpool wait -t scrub -- <pool>`;
- otherwise it starts one with `zpool scrub -- <pool>` and waits for it the same way;
- if the start is refused because a scrub has just begun, it waits for that one.

It never pauses or stops a scrub. The unit has no ExecStop, so stopping the unit ends only the wait. A
resilver that is running defers the scrub to the next run after it. The served slot is kept in
`/var/lib/zfs-scrub/scrub-served.json`, in a folder of root's alone (0700), apart from the account's state.
The run writes it only into a real folder of its own and never through a link: a new file opened with
`O_NOFOLLOW` and `O_EXCL`, then renamed over the old one. The tools run as the operator account and cannot
read it, so they judge a missed scrub from status. Without that file, a scrub that status shows started in the slot
serves it. So a scrub starts within 15 minutes of its time, and after the machine was off, within 15
minutes of its coming back.

**No setting value reaches a unit file.** The day, the time and the cadence go only into the
configuration. The run checks them against fixed sets (the seven weekday names, and `HH:MM`). It refuses
any other value and runs nothing; a value like `*:*` or `Mon..Fri` is impossible. The pool's name reaches
the names of the two distribution timers. The controller fills it there and refuses anything but letters,
digits, `:`, `_`, `.` and `-` (mesh-controller #171).

**Exactly one schedule per pool.** The distribution's `zfs-scrub-weekly@<pool>.timer` and
`zfs-scrub-monthly@<pool>.timer` are held stopped and disabled. They come after the scheduled run in the
declaration, so a run that cannot be installed stops the apply before the timer that scrubs the pool today
is touched. On the home server the weekly one was enabled by hand, outside the mesh. A scrub already
running is not stopped by this; only its timer is.

**Unassigning the module gives the distribution's timer back running and enabled**, as the node-engine
found it, once mesh-host #66 (hq ADR 0269, proposed) is merged. Until then the node-engine never starts a
unit on the way out, and unassigning would leave the pool with no scrub schedule. Do not unassign the
module before #66 is merged.

**One pool per assignment in this version.** A manifest's resources are fixed, and a setting cannot make
one timer per pool in a list. The home server has one pool.

### The scrub acts

`zfs_scrub_start`, `zfs_scrub_pause` and `zfs_scrub_stop` take `{pool, why}`. They refuse a pool that
`zpool list` does not show, and a missing `why`. They run `sudo -n zpool scrub [-p|-s] -- <pool>` and
answer what status says afterwards. Each is recorded as the event `scrub.requested`.

A scrub reads every block and rewrites, from the pool's redundancy, only a block whose checksum is wrong.
It changes no file's content. It adds read load for its whole length, which on the home server's pool was
about 46 hours for the weekly scrub that ended 2026-10-06. These three are **candidates for the approval
mark** (novox/hq research 039): a person's acts that never change data, and that cost hours of I/O.

## Resilvers

ZFS starts a resilver by itself when a disk is replaced or reattached, so there is no schedule to own. The
module watches and reports it:

- `zfs_scrub` shows a running resilver, pool-wide or a vdev's rebuild, with its progress and the time to go;
- the health check warns while one runs, and is unhealthy when one finished with errors;
- the event follower writes its start and finish to the journal.

## Health

The module runs two long-running processes of its own binary, and the scheduled run:

- **The watch** (`zfs-tools watch`) runs as the operator account and carries the health check,
  `zfs_health` (novox/hq ADR 0240). Each minute it reads the pools and writes a line in its journal whenever
  their health changes.
- **The event follower** (`zfs-tools events`) runs as root and follows `zpool events -f`. It is described
  in the next section.
- **The scheduled run** (`zfs-tools scrub-run`) is not long-running and has no health check of its own;
  `zfs_health` says when it failed or did not start a due scrub.

`zfs_health` is **unhealthy** for a fault now:

- a pool not ONLINE (the answer gives status's reason and action), or a device not online;
- a device's error counts that rose since the last look. A rise stays unhealthy for 15 minutes, so the
  node-engine's two looks see it;
- data errors, or a resilver that finished with errors;
- a pool more than 90 % full;
- no pool imported, or no ZFS on the machine.

These are **warnings**, which leave it healthy:

- error counts from before ("N past checksum errors; clear them by hand once understood"). The first look
  after the bundle starts takes every count as past;
- a pool more than 80 % full;
- a resilver running;
- a last finished scrub older than the schedule allows (weekly 9 days, monthly 38 days), or 35 days for a
  pool without a schedule;
- a scheduled scrub that has not started an hour after its time;
- a scheduled run that failed, with no scrub finished since;
- a schedule whose timer is not active and enabled, or whose settings are not valid.

The error counts last seen are kept in `/var/lib/zfs-tools/errors.json`, so a restart of the bundle does
not take every count as past again.

## The event journal

ZFS keeps its events in a queue in the kernel, in memory, from the moment its module loads, so a reboot
loses them. The follower reads the queue from its oldest event, then follows. So the pool's import, and
whatever came while the follower was down, reach the journal too. It writes each error and change of state
in one line, resilver and scrub starts and ends among them. It also writes ZFS's own "dropped N events" line
when the queue overflowed before it was read.

A cursor of the boot and the last event read is kept in `/run/zfs-events/cursor`, which a reboot empties as
it empties ZFS's queue. With it, a follower that restarts does not write an event twice. After five
follows in a row that end within a minute, the follower exits non-zero, so the service manager counts the
restarts and the unit's state shows the failure.

**What survives a reboot** is what reached the journal. That holds where journald keeps its journal on
disk. On the home server it does: its journal is in `/var/log/journal`, and it holds entries from before the
reboot of 2026-10-04 02:09, as read through the mesh on 2026-10-09. Events ZFS emits after the follower has
stopped during a shutdown are lost with the queue.

## Tests

`go test ./...` runs against OpenZFS 2.4.4 output in `cmd/zfs-tools/testdata`. Each file has the form that
version's `zpool_main.c` and `libzfs` print, with documentation names in place of the real pools and disks.
The tests cover:

- parsing every command's output, including the scan lines (human sizes, rebuilds per vdev, error scrubs,
  paused scrubs) and `iostat -H` without section lines;
- a timed-out command ended with its children;
- the snapshot's refusals, read-back, space statement and unknown outcome;
- the scrub acts, the schedule's slots, and the scheduled run (start, wait, never pause, defer to a
  resilver, refuse an invalid setting);
- the follower's cursor, dropped events, kill before wait, and exit after repeated failures;
- the health verdicts, past against new errors;
- a machine without ZFS;
- the manifest against the tools the bundle serves.