mesh-catalog/modules/zfs/README.md

# zfs

The mesh's view of ZFS on a machine, and what it owns of it: the packages, the pool's properties, cache and log devices,
the node's ZFS id, the ARC, the event daemon, the scrub and trim schedule and scheduled ZFS snapshots. It changes only what
its config values set: a value that is not set, or an empty list, is not managed, and nothing is ever removed by a blank
value (ADR 0679). The pool's data is the operator's, which the mesh does not own (novox/hq ADR 0051).

## What it installs

`zfs-utils` and `zfs-dkms` are declared present: the node-engine installs a missing one, upgrades none, removes none.
`zfs-utils-debug` is not the module's and is left as found. The module never writes the package's own `/etc/default/zfs`
or `/etc/zfs/zed.d`; a `.pacnew` beside `/etc/default/zfs` or the package's `zed.rc` is a warning.

## What it does not do

It never destroys a dataset, rolls back, renames, clears, sends, receives, replaces, attaches or detaches, and it adds
no data vdev, spare, special or dedup device. Each of those changes or removes the operator's data or the pool's layout
and needs a decision of its own first. A dataset's `sharenfs` is the seat `node-nfs-server`'s and `sharesmb` the SMB module's:
a value naming either is refused.

This is checked by `TestTheBundleRunsOnlyTheFormsOfADR0679`. Every command the bundle runs is built by one of its argument
functions, and each function is held to its program, its verb and its whole form:

- `zpool`: `add <pool> cache|log` (never `-f`), `remove` of a cache or log device, `set` of a listed pool property, `scrub`
  (with `-p` or `-s` only from a person's act), `trim` of the pool, and the reads `status`, `list`, `get`, `iostat`, `events`
  and `history`;
- `zfs`: `snapshot`, `set mesh:schedule=<kind>` on a snapshot named `mesh-`, `destroy -- <dataset>@<snapshot>`, `set` and
  `inherit` of a listed dataset property, and the reads `list`, `get` and `holds`;
- `wipefs --no-act <device>`, `zdb -l <device>` and `zdb -C <pool>`, `zgenhostid -f <value>` only with the value the parse
  validated, `udevadm monitor --udev --subsystem-match=block`, and `systemctl show`.

Any other `zpool` or `zfs` form, and every `-f`, `-r`, `-R` and `-d`, is refused; so is any call site that builds its own argument
list.

On a machine without ZFS, every tool answers `applicable: false` with the reason, and the health check is unhealthy and says
to unassign the module.

It holds no seat. A new `node-*` seat is the controller's to define (novox/hq ADR 0121, ADR 0246).

## Config values

| value | form | managed by |
|---|---|---|
| `pool` (was `scrub-pool`) | the pool's name, trusted, required | the scrub, trim, ZFS snapshots and the apply |
| `scrub-cadence`, `scrub-day`, `scrub-time` | as before, with their defaults | the scheduled run |
| `arc-max`, `arc-min` | whole MiB | written live and into `/etc/modprobe.d/zfs.conf`, read back |
| `module-options` | `name=digits` from a fixed list | the options file; a live parameter also live, others at the next load of the kernel module |
| `hostid` | eight hexadecimal digits | `/etc/hostid`, by `zgenhostid -f`, only when the pool's label holds 0 or this value |
| `zed`, `zed-handlers` | `on` or `off`; `all-syslog`, `resilver_finish-start-scrub` | the module's own ZED folder, without links |
| `trim-schedule`, `trim-day`, `trim-time` | `none`, `weekly` or `monthly` with a day and a time | the scheduled run |
| `cache-devices`, `log-devices`, `log-mirror` | names under `/dev/disk/by-id` | the apply, after its checks |
| `pool-properties`, `dataset-properties` | the listed properties | the apply, read back |
| `snapshots` | `{"none": true}` or keep counts `hourly`, `daily`, `weekly`, `monthly`, 0 to 168 | the scheduled run |

A value of another form is refused before anything is written or run (`config.refused`); one refused value refuses the whole
config. A keep count of 0 is that kind's none: none is taken, and its existing scheduled ZFS snapshots are kept with a warning.
A trim schedule of `weekly` or `monthly` without a `trim-day` or `trim-time` is refused. Changing `hostid` needs a check of the
pool's import at the next boot; the module never reboots.

## The apply, the run, the keeper and the follower

The module writes `/etc/zfs-tools/config.json` (all its values as one JSON object). The apply (`zfs-tools apply`, root,
run-once, started again by the config file's change) converges the node under an exclusive lock, waits (`config.waiting`) while a scrub
or resilver runs or before the pool is imported, and exits non-zero only when a command it ran failed. The event follower
(`zfs-tools events`, root) starts the apply at its own start and when a scrub or resilver ends or a declared device appears, and keeps the
findings of a pool or device not ONLINE in its state `findings`. The scheduled run (`zfs-tools run`, root, every quarter of an hour) takes
the same lock, starts a due scrub or trim and returns without waiting, takes and removes the scheduled ZFS snapshots and acts on a person's
hand acts. The three root processes hold no bus account: the keeper (`zfs-tools keeper`, the module's user `zfs-keeper`) holds the
module's one, and serves a unix socket only root may open, with a closed list of calls: emit an event of the module, put, delete, get or list a key of the states `applied`,
`schedule` and `findings`, and invoke `zfs_snapshots_record`, `zfs_snapshots_schedule` or the service manager's `start` of the module's two
units. With the keeper not answering they act as with the bus down and warn.

## Scheduled ZFS snapshots

Each kind with a keep count above 0 is due at the first run of the hour, day, Monday or month, against the ledger's last entry. For every
filesystem of the pool the run takes `zfs snapshot -- <dataset>@mesh-<kind>-<UTC stamp>`, never recursive, sets `mesh:schedule`, and records
`taken` in the ledger `snapshots` (one instance per dataset and kind, its state a map keyed by the ZFS snapshot's name) through the tool
`zfs_snapshots_record`. It takes none it cannot record, none with the ledger unreadable or full, and none while the pool is more than 80 % full.
It removes the oldest past a keep count only by their own record: name, `mesh:schedule` and ledger entry fit, no hold and no clone.
A snapshot taken by hand (`zfs_snapshot`, which refuses a `mesh-` name, event `snapshot.taken.person`) is never removed.

## Tools

The tools run as the operator account. What ZFS keeps for root goes through `sudo -n` with a fixed argument list, never a shell. A command that
runs out of time is ended with everything it started.

| tool | what |
|---|---|
| `zfs_datasets`, `zfs_snapshots`, `zfs_pools`, `zfs_space`, `zfs_pool_layout`, `zfs_scrub`, `zfs_io`, `zfs_arc`, `zfs_events`, `zfs_properties`, `zfs_health` | the reads (below) |
| `zfs_snapshot` | takes `dataset@name`, a person's act with why |
| `zfs_scrub_start`, `zfs_scrub_pause`, `zfs_scrub_stop` | a person's scrub acts |
| `zfs_config` | every value with its config value, what runs, how it stands, the package's files, and what the module says before it leaves |
| `zfs_config_apply` | converge now, as the apply does |
| `zfs_snapshots_schedule` | the keep counts, next moments, what the next run would remove, and the ledger |
| `zfs_snapshots_run_now`, `zfs_snapshots_prune_now`, `zfs_snapshots_pause`, `zfs_snapshots_resume` | a person's acts on the scheduled ZFS snapshots; the hand acts answer "started" at once and `zfs_module_events` reads the outcome |
| `zfs_module_events` | the module's own events, newest first (`zfs_events` reads ZFS's) |
| `zfs_snapshots_check` | the ledger against what ZFS holds; changes nothing |
| `zfs_findings` | the follower's findings |
| `zfs_snapshots_record` | the run's own record of a `taken` or `removed` ZFS snapshot |

When the module leaves a node, the next load of the kernel module uses ZFS's defaults, `zfs-import-cache`, `zfs-mount` and `zfs.target` stay
running, `/etc/hostid` stays, and pool and dataset properties and devices stay as the last apply set them.

## The snapshot

`zfs_snapshot {dataset, name, why}` snapshots one dataset, never its children. There is no `recursive`,
and passing one is refused. It refuses all of these before anything runs:

- a dataset that `zfs list` does not show on this machine;
- a name with anything but letters, digits, `.`, `_` and `-`, or a whole name longer than 255 characters;
- a snapshot that already exists;
- a missing `why`, or a `why` longer than one line.

It runs `sudo -n zfs snapshot -- <dataset>@<name>`. It answers from `zfs list` afterwards, not from what
it ran: the snapshot made, its creation time, and any other snapshot that came or went during the call. It
also gives the dataset's free space and the pool's capacity. **From then on, the snapshot holds every block
of the dataset that is deleted or overwritten, so that space is not freed until the snapshot is destroyed by
hand.** The scheduled run destroys only the scheduled ZFS snapshots it took and recorded, by their own record.

The act is recorded as the event `snapshot.taken.person`, and ZFS keeps the command in the pool's own history.

## The scrub schedule

The module owns the schedule of the pool its `pool` setting names (before this version `scrub-pool`, which mesh-config renames).

| setting | default | what |
|---|---|---|
| `pool` | none: the operator's value | the pool to scrub |
| `scrub-cadence` | `weekly` | `weekly`, or `monthly` (the first given weekday of each month) |
| `scrub-day` | `Mon` | `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat` or `Sun`, nothing else |
| `scrub-time` | `00:00` | `HH:MM` from `00:00` to `23:59`, in the machine's time zone |

The defaults are what the distribution's weekly timer did on the home server before the module, so taking
it over does not move the schedule. The operator chose monthly for the home server.

**The scheduled run.** The node-engine runs the module's scheduled process `zfs-mesh-run` (`zfs-tools run`, as root) from its
own timer `zfs-mesh-run.timer`, every quarter of an hour. The run reads `/etc/zfs-tools/config.json`. It finds the slot, which
is the last scheduled time not after now. If the slot is not yet served, the run starts the scrub with `zpool scrub -- <pool>`
and returns at once: it never waits for it, and the event follower hears the scrub end. If a scrub or resilver runs, or the pool
is not imported, the run does nothing and says `run.skipped`.

It never pauses or stops a scrub. A
resilver that is running defers the scrub to the next run after it. The served slot is kept in
`/var/lib/zfs-scrub/scrub-served.json`, in a folder of root's alone (0700), apart from the account's state.
The run writes it only into a real folder of its own and never through a link: a new file opened with
`O_NOFOLLOW` and `O_EXCL`, then renamed over the old one. The tools run as the operator account and cannot
read it, so they judge a missed scrub from status. Without that file, a scrub that status shows started in the slot
serves it. So a scrub starts within 15 minutes of its time, and after the machine was off, within 15
minutes of its coming back.

**No setting value reaches a unit file.** The day, the time and the cadence go only into the
configuration. The run checks them against fixed sets (the seven weekday names, and `HH:MM`). It refuses
any other value and runs nothing; a value like `*:*` or `Mon..Fri` is impossible. The pool's name reaches
the names of the two distribution timers. The controller fills it there and refuses anything but letters,
digits, `:`, `_`, `.` and `-` (mesh-controller #171).

**Exactly one schedule per pool.** The distribution's `zfs-scrub-weekly@<pool>.timer` and
`zfs-scrub-monthly@<pool>.timer` are held stopped and disabled, and the trim timers with them while `trim-schedule` is set. They come after the scheduled run in the
declaration, so a run that cannot be installed stops the apply before the timer that scrubs the pool today
is touched. On the home server the weekly one was enabled by hand, outside the mesh. A scrub already
running is not stopped by this; only its timer is.

**Unassigning the module gives the distribution's timer back running and enabled**, as the node-engine
found it, once mesh-host #66 (hq ADR 0269, proposed) is merged. Until then the node-engine never starts a
unit on the way out, and unassigning would leave the pool with no scrub schedule. Do not unassign the
module before #66 is merged.

**One pool per assignment in this version.** A manifest's resources are fixed, and a setting cannot make
one timer per pool in a list. The home server has one pool.

### The scrub acts

`zfs_scrub_start`, `zfs_scrub_pause` and `zfs_scrub_stop` take `{pool, why}`. They refuse a pool that
`zpool list` does not show, and a missing `why`. They run `sudo -n zpool scrub [-p|-s] -- <pool>` and
answer what status says afterwards. Each is recorded as the event `scrub.requested`.

A scrub reads every block and rewrites, from the pool's redundancy, only a block whose checksum is wrong.
It changes no file's content. It adds read load for its whole length, which on the home server's pool was
about 46 hours for the weekly scrub that ended 2026-10-06. These three are **candidates for the approval
mark** (novox/hq research 039): a person's acts that never change data, and that cost hours of I/O.

## Resilvers

ZFS starts a resilver by itself when a disk is replaced or reattached, so there is no schedule to own. The
module watches and reports it:

- `zfs_scrub` shows a running resilver, pool-wide or a vdev's rebuild, with its progress and the time to go;
- the health check warns while one runs, and is unhealthy when one finished with errors;
- the event follower writes its start and finish to the journal.

## Health

The module runs two long-running processes of its own binary, and the scheduled run:

- **The watch** (`zfs-tools watch`) runs as the operator account and carries the health check,
  `zfs_health` (novox/hq ADR 0240). Each minute it reads the pools and writes a line in its journal whenever
  their health changes.
- **The event follower** (`zfs-tools events`) runs as root and follows `zpool events -f`. It is described
  in the next section.
- **The scheduled run** (`zfs-tools run`) is not long-running and has no health check of its own;
  `zfs_health` says when it failed or did not start a due scrub.

`zfs_health` is **unhealthy** for a fault now:

- a pool not ONLINE (the answer gives status's reason and action), or a device not online;
- a device's error counts that rose since the last look. A rise stays unhealthy for 15 minutes, so the
  node-engine's two looks see it;
- data errors, or a resilver that finished with errors;
- a pool more than 90 % full;
- no pool imported, or no ZFS on the machine.

These are **warnings**, which leave it healthy:

- error counts from before ("N past checksum errors; clear them by hand once understood"). The first look
  after the bundle starts takes every count as past;
- a pool more than 80 % full;
- a resilver running;
- a last finished scrub older than the schedule allows (weekly 9 days, monthly 38 days), or 35 days for a
  pool without a schedule;
- a scheduled scrub that has not started an hour after its time;
- a scheduled run that failed, with no scrub finished since;
- a schedule whose timer is not active and enabled, or whose settings are not valid.

The error counts last seen are kept in `/var/lib/zfs-tools/errors.json`, so a restart of the bundle does
not take every count as past again.

## The event journal

ZFS keeps its events in a queue in the kernel, in memory, from the moment its module loads, so a reboot
loses them. The follower reads the queue from its oldest event, then follows. So the pool's import, and
whatever came while the follower was down, reach the journal too. It writes each error and change of state
in one line, resilver and scrub starts and ends among them. It also writes ZFS's own "dropped N events" line
when the queue overflowed before it was read.

A cursor of the boot and the last event read is kept in `/run/zfs-events/cursor`, which a reboot empties as
it empties ZFS's queue. With it, a follower that restarts does not write an event twice. After five
follows in a row that end within a minute, the follower exits non-zero, so the service manager counts the
restarts and the unit's state shows the failure.

**What survives a reboot** is what reached the journal. That holds where journald keeps its journal on
disk. On the home server it does: its journal is in `/var/log/journal`, and it holds entries from before the
reboot of 2026-10-04 02:09, as read through the mesh on 2026-10-09. Events ZFS emits after the follower has
stopped during a shutdown are lost with the queue.

## Tests

`go test ./...` runs against OpenZFS 2.4.4 output in `cmd/zfs-tools/testdata`. Each file has the form that
version's `zpool_main.c` and `libzfs` print, with documentation names in place of the real pools and disks.
The tests cover:

- parsing every command's output, including the scan lines (human sizes, rebuilds per vdev, error scrubs,
  paused scrubs) and `iostat -H` without section lines;
- a timed-out command ended with its children;
- the snapshot's refusals, read-back, space statement and unknown outcome;
- the scrub acts, the schedule's slots, and the scheduled run (start and return, never pause, defer to a
  resilver, refuse an invalid setting);
- the config's parse, one table per value; each convergence plan (properties, devices, the node's ZFS id, the ARC); the lock; the
  scheduled ZFS snapshots' taking and removal; the hand acts; the keeper's closed list of calls and its socket; the tools; the health;
- the follower's cursor, dropped events, kill before wait, and exit after repeated failures;
- the health verdicts, past against new errors;
- a machine without ZFS;
- the manifest (values, files, units, order, bus state, keeper) against the tools the bundle serves and the call-site test.

The module adds no integration test (its test budget is 500 unit tests and 50 integration tests): the main design's live tests stand for
the module whole.

## Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

- `/var/lib/zfs-tools`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/run/zfs-events`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/etc/zfs-tools`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/var/lib/zfs-scrub`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/run/zfs-tools`: everything in it is the program's (the lock); the mesh moves nothing out.
- the keeper's folder: everything in it is the program's (its socket and its credential).
- `/etc/zfs/mesh-zed.d`: the module's own, while `zed` is on.