novox/mesh-catalog
9
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Z19 zfs-readme-budget: the README says what the module owns and how it is checked, and the test budget is a test

Browse Source 
Builds: design 42 zfs-owns-zfs §31, §7, §19, §23
This commit is contained in:
jschoubben
2026-11-13 01:06:04 +01:00
parent 8927d68900
commit 7831f14a6e
2 changed files with 124 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
modules/zfs/README.md
+108 -64
View File
@@ -1,65 +1,105 @@
# zfs
The mesh's view of ZFS on a machine: every pool's health, layout, space, scrubs, I/O, cache and events,
and every dataset with its space and snapshots. It owns the pool's scrub schedule, run by a scheduled process of its own. It has four acts: a named
snapshot of one dataset, and a scrub's start, pause and stop. It is for the home server, whose pool holds
the operator's media library, which the mesh does not own (novox/hq ADR 0051). So the module reads, and
what it does only adds a snapshot or reads the pool through.
The mesh's view of ZFS on a machine, and what it owns of it: the packages, the pool's properties, cache and log devices,
the node's ZFS id, the ARC, the event daemon, the scrub and trim schedule and scheduled ZFS snapshots. It changes only what
its config values set: a value that is not set, or an empty list, is not managed, and nothing is ever removed by a blank
value (ADR 0679). The pool's data is the operator's, which the mesh does not own (novox/hq ADR 0051).
## What it installs
`zfs-utils` and `zfs-dkms` are declared present: the node-engine installs a missing one, upgrades none, removes none.
`zfs-utils-debug` is not the module's and is left as found. The module never writes the package's own `/etc/default/zfs`
or `/etc/zfs/zed.d`; a `.pacnew` beside `/etc/default/zfs` or the package's `zed.rc` is a warning.
## What it does not do
There is no destroy, rollback, set, send, receive, rename, clone, replace, attach, detach or clear. Each of
those changes or removes the operator's data, or the pool's layout, and needs a decision of its own first.
A resilver is ZFS's own: it starts one when a disk is replaced or comes back. The module watches and
reports resilvers, and starts none.
It never destroys a dataset, rolls back, renames, clears, sends, receives, replaces, attaches or detaches, and it adds
no data vdev, spare, special or dedup device. Each of those changes or removes the operator's data or the pool's layout
and needs a decision of its own first. A dataset's `sharenfs` is the seat `node-nfs-server`'s and `sharesmb` the SMB module's:
a value naming either is refused.
This is checked by `TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`. Every command the bundle runs is
built by one of its argument functions, and each function is held to its program and verb:
This is checked by `TestTheBundleRunsOnlyTheFormsOfADR0679`. Every command the bundle runs is built by one of its argument
functions, and each function is held to its program, its verb and its whole form:
- `zfs`: `list`, `get` and `snapshot`;
- `zpool`: `list`, `status`, `iostat`, `events`, `history`, `wait -t scrub` and `scrub` (with `-p` or `-s`
only, and those only from the person's acts, never from the scheduled run);
- `systemctl`: `show`.
- `zpool`: `add <pool> cache|log` (never `-f`), `remove` of a cache or log device, `set` of a listed pool property, `scrub`
(with `-p` or `-s` only from a person's act), `trim` of the pool, and the reads `status`, `list`, `get`, `iostat`, `events`
and `history`;
- `zfs`: `snapshot`, `set mesh:schedule=<kind>` on a snapshot named `mesh-`, `destroy -- <dataset>@<snapshot>`, `set` and
`inherit` of a listed dataset property, and the reads `list`, `get` and `holds`;
- `wipefs --no-act <device>`, `zdb -l <device>` and `zdb -C <pool>`, `zgenhostid -f <value>` only with the value the parse
validated, `udevadm monitor --udev --subsystem-match=block`, and `systemctl show`.
The test fails on any call site that builds its own argument list, through `ask`, `AsRoot`, `Escalated`,
`.Run(`, `exec.Command(`, `exec.CommandContext(`, `os.StartProcess` or `syscall.Exec`.
Any other `zpool` or `zfs` form, and every `-f`, `-r`, `-R` and `-d`, is refused; so is any call site that builds its own argument
list.
It installs nothing. ZFS belongs to the machine, and its kernel module is built against that machine's
kernel. On a machine without ZFS, every tool answers `applicable: false` with the reason, and the health
check is unhealthy and says to unassign the module, because a module put where it does nothing must not
look healthy.
On a machine without ZFS, every tool answers `applicable: false` with the reason, and the health check is unhealthy and says
to unassign the module.
It holds no seat. A new `node-*` seat is the controller's to define (novox/hq ADR 0121, ADR 0246).
## Config values
| value | form | managed by |
|---|---|---|
| `pool` (was `scrub-pool`) | the pool's name, trusted, required | the scrub, trim, ZFS snapshots and the apply |
| `scrub-cadence`, `scrub-day`, `scrub-time` | as before, with their defaults | the scheduled run |
| `arc-max`, `arc-min` | whole MiB | written live and into `/etc/modprobe.d/zfs.conf`, read back |
| `module-options` | `name=digits` from a fixed list | the options file; a live parameter also live, others at the next load of the kernel module |
| `hostid` | eight hexadecimal digits | `/etc/hostid`, by `zgenhostid -f`, only when the pool's label holds 0 or this value |
| `zed`, `zed-handlers` | `on` or `off`; `all-syslog`, `resilver_finish-start-scrub` | the module's own ZED folder, without links |
| `trim-schedule`, `trim-day`, `trim-time` | `none`, `weekly` or `monthly` with a day and a time | the scheduled run |
| `cache-devices`, `log-devices`, `log-mirror` | names under `/dev/disk/by-id` | the apply, after its checks |
| `pool-properties`, `dataset-properties` | the listed properties | the apply, read back |
| `snapshots` | `{"none": true}` or keep counts `hourly`, `daily`, `weekly`, `monthly`, 0 to 168 | the scheduled run |
A value of another form is refused before anything is written or run (`config.refused`); one refused value refuses the whole
config. A keep count of 0 is that kind's none: none is taken, and its existing scheduled ZFS snapshots are kept with a warning.
A trim schedule of `weekly` or `monthly` without a `trim-day` or `trim-time` is refused. Changing `hostid` needs a check of the
pool's import at the next boot; the module never reboots.
## The apply, the run, the keeper and the follower
The module writes `/etc/zfs-tools/config.json` (all its values as one JSON object). The apply (`zfs-tools apply`, root,
run-once, started again by the config file's change) converges the node under an exclusive lock, waits (`config.waiting`) while a scrub
or resilver runs or before the pool is imported, and exits non-zero only when a command it ran failed. The event follower
(`zfs-tools events`, root) starts the apply at its own start and when a scrub or resilver ends or a declared device appears, and keeps the
findings of a pool or device not ONLINE in its state `findings`. The scheduled run (`zfs-tools run`, root, every quarter of an hour) takes
the same lock, starts a due scrub or trim and returns without waiting, takes and removes the scheduled ZFS snapshots and acts on a person's
hand acts. The three root processes hold no bus account: the keeper (`zfs-tools keeper`, the module's user `zfs-keeper`) holds the
module's one, and serves a unix socket only root may open, with a closed list of calls: emit an event of the module, put, delete, get or list a key of the states `applied`,
`schedule` and `findings`, and invoke `zfs_snapshots_record`, `zfs_snapshots_schedule` or the service manager's `start` of the module's two
units. With the keeper not answering they act as with the bus down and warn.
## Scheduled ZFS snapshots
Each kind with a keep count above 0 is due at the first run of the hour, day, Monday or month, against the ledger's last entry. For every
filesystem of the pool the run takes `zfs snapshot -- <dataset>@mesh-<kind>-<UTC stamp>`, never recursive, sets `mesh:schedule`, and records
`taken` in the ledger `snapshots` (one instance per dataset and kind, its state a map keyed by the ZFS snapshot's name) through the tool
`zfs_snapshots_record`. It takes none it cannot record, none with the ledger unreadable or full, and none while the pool is more than 80 % full.
It removes the oldest past a keep count only by their own record: name, `mesh:schedule` and ledger entry fit, no hold and no clone.
A snapshot taken by hand (`zfs_snapshot`, which refuses a `mesh-` name, event `snapshot.taken.person`) is never removed.
## Tools
The tools run as the operator account. What ZFS lets any account read runs as that account. What ZFS
keeps for root goes through `sudo -n` with a fixed argument list, never a shell: events, history, the
snapshot and the scrub acts. Sizes are whole bytes, read under `-p`, except where status prints human sizes
even under `-p` (a scan's sizes). Each answer also says its findings in lines under `said`.
The tools run as the operator account. What ZFS keeps for root goes through `sudo -n` with a fixed argument list, never a shell. A command that
runs out of time is ended with everything it started.
A command that runs out of time is ended with everything it started. It runs in a process group of its
own. At the deadline the group gets SIGTERM, which sudo passes on to the command it runs as root, and
after five seconds SIGKILL. An act that ran out of time is answered from what the machine shows afterwards,
or as "outcome unknown".
| tool | what |
|---|---|
| `zfs_datasets`, `zfs_snapshots`, `zfs_pools`, `zfs_space`, `zfs_pool_layout`, `zfs_scrub`, `zfs_io`, `zfs_arc`, `zfs_events`, `zfs_properties`, `zfs_health` | the reads (below) |
| `zfs_snapshot` | takes `dataset@name`, a person's act with why |
| `zfs_scrub_start`, `zfs_scrub_pause`, `zfs_scrub_stop` | a person's scrub acts |
| `zfs_config` | every value with its config value, what runs, how it stands, the package's files, and what the module says before it leaves |
| `zfs_config_apply` | converge now, as the apply does |
| `zfs_snapshots_schedule` | the keep counts, next moments, what the next run would remove, and the ledger |
| `zfs_snapshots_run_now`, `zfs_snapshots_prune_now`, `zfs_snapshots_pause`, `zfs_snapshots_resume` | a person's acts on the scheduled ZFS snapshots; the hand acts wait up to 30 seconds for their event, else answer "started" |
| `zfs_module_events` | the module's own events, newest first (`zfs_events` reads ZFS's) |
| `zfs_snapshots_check` | the ledger against what ZFS holds; changes nothing |
| `zfs_findings` | the follower's findings |
| `zfs_snapshots_record` | the run's own record of a `taken` or `removed` ZFS snapshot |
| tool | what | replaces |
|---|---|---|
| `zfs_datasets` | Every filesystem and volume: mountpoint, used, available, referenced, compression ratio, sharenfs, and the number of snapshots with the space they hold together | `zfs list` |
| `zfs_snapshots` | The snapshots of one `dataset` (optional), oldest first: creation time, used (the space only that snapshot holds) and referenced | `zfs list -t snapshot` |
| `zfs_snapshot` | Takes `dataset@name` (see below) | `zfs snapshot` |
| `zfs_pools` | Each pool's state with status's reason and action, space, capacity and fragmentation %, device error count, data errors and scans | `zpool list`, `zpool status -x` |
| `zfs_space` | Each pool's space, then every dataset, biggest first, with what uses its space, quotas and reservations, compression and the savings | `zfs list -o space`, `zpool list -v` |
| `zfs_pool_layout` | Vdevs and disks, with state, error counts and status's note; log, cache, special and dedup devices; spares | `zpool status`, `zpool status -v` |
| `zfs_scrub` | Each pool's scans: its scrub or error scrub, and its resilver (pool-wide, or each vdev's rebuild), with progress and time to go while one runs; the days since the last finished scrub; and the module's schedule with its next run. `history: true` also reads the pool's history | `zpool history -i` |
| `zfs_scrub_start` | Starts a scrub of `pool`, or resumes a paused one, with `why` | `zpool scrub` |
| `zfs_scrub_pause` | Pauses one, keeping its place, with `why` | `zpool scrub -p` |
| `zfs_scrub_stop` | Stops one, losing its place, with `why` | `zpool scrub -s` |
| `zfs_io` | A sample of `seconds` (default 3, from 1 to 10): operations and bytes per second per pool, vdev and disk | `zpool iostat -v` |
| `zfs_arc` | ARC size, target, minimum and maximum, hit ratio since boot, and L2ARC where present | `arcstat`, `arc_summary` |
| `zfs_events` | Recent events, newest first, as root. By default it shows only errors and changes of state, and `all: true` shows every event | `zpool events -v` |
| `zfs_properties` | Every property of one `dataset` with its source. The value of `keylocation` is withheld | `zfs get all` |
| `zfs_health` | The health check (below) | |
When the module leaves a node, the next load of the kernel module uses ZFS's defaults, `zfs-import-cache`, `zfs-mount` and `zfs.target` stay
running, `/etc/hostid` stays, and pool and dataset properties and devices stay as the last apply set them.
## The snapshot
@@ -75,17 +115,17 @@ It runs `sudo -n zfs snapshot -- <dataset>@<name>`. It answers from `zfs list` a
it ran: the snapshot made, its creation time, and any other snapshot that came or went during the call. It
also gives the dataset's free space and the pool's capacity. **From then on, the snapshot holds every block
of the dataset that is deleted or overwritten, so that space is not freed until the snapshot is destroyed by
hand.** No tool of the mesh destroys a snapshot.
hand.** The scheduled run destroys only the scheduled ZFS snapshots it took and recorded, by their own record.
The act is recorded as the event `snapshot.taken`, and ZFS keeps the command in the pool's own history.
The act is recorded as the event `snapshot.taken.person`, and ZFS keeps the command in the pool's own history.
## The scrub schedule
The module owns the schedule of the pool its `scrub-pool` setting names.
The module owns the schedule of the pool its `pool` setting names (before this version `scrub-pool`, which mesh-config renames).
| setting | default | what |
|---|---|---|
| `scrub-pool` | none: the operator's value | the pool to scrub |
| `pool` | none: the operator's value | the pool to scrub |
| `scrub-cadence` | `weekly` | `weekly`, or `monthly` (the first given weekday of each month) |
| `scrub-day` | `Mon` | `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat` or `Sun`, nothing else |
| `scrub-time` | `00:00` | `HH:MM` from `00:00` to `23:59`, in the machine's time zone |
@@ -93,17 +133,13 @@ The module owns the schedule of the pool its `scrub-pool` setting names.
The defaults are what the distribution's weekly timer did on the home server before the module, so taking
it over does not move the schedule. The operator chose monthly for the home server.
**The scheduled run.** The node-engine runs the module's scheduled process `zfs-mesh-scrub` (`zfs-tools
scrub-run`, as root) from its own timer `zfs-mesh-scrub.timer`, every quarter of an hour. The run reads the
configuration the mesh writes from the settings (`/etc/zfs-tools/scrub.conf`). It finds the slot, which is
the last scheduled time not after now. If the slot is not yet served, the run acts as upstream's
`zfs-scrub@.service` does:
**The scheduled run.** The node-engine runs the module's scheduled process `zfs-mesh-run` (`zfs-tools run`, as root) from its
own timer `zfs-mesh-run.timer`, every quarter of an hour. The run reads `/etc/zfs-tools/config.json`. It finds the slot, which
is the last scheduled time not after now. If the slot is not yet served, the run starts the scrub with `zpool scrub -- <pool>`
and returns at once: it never waits for it, and the event follower hears the scrub end. If a scrub or resilver runs, or the pool
is not imported, the run does nothing and says `run.skipped`.
- if a scrub is running, it waits for it with `zpool wait -t scrub -- <pool>`;
- otherwise it starts one with `zpool scrub -- <pool>` and waits for it the same way;
- if the start is refused because a scrub has just begun, it waits for that one.
It never pauses or stops a scrub. The unit has no ExecStop, so stopping the unit ends only the wait. A
It never pauses or stops a scrub. A
resilver that is running defers the scrub to the next run after it. The served slot is kept in
`/var/lib/zfs-scrub/scrub-served.json`, in a folder of root's alone (0700), apart from the account's state.
The run writes it only into a real folder of its own and never through a link: a new file opened with
@@ -119,7 +155,7 @@ the names of the two distribution timers. The controller fills it there and refu
digits, `:`, `_`, `.` and `-` (mesh-controller #171).
**Exactly one schedule per pool.** The distribution's `zfs-scrub-weekly@<pool>.timer` and
`zfs-scrub-monthly@<pool>.timer` are held stopped and disabled. They come after the scheduled run in the
`zfs-scrub-monthly@<pool>.timer` are held stopped and disabled, and the trim timers with them while `trim-schedule` is set. They come after the scheduled run in the
declaration, so a run that cannot be installed stops the apply before the timer that scrubs the pool today
is touched. On the home server the weekly one was enabled by hand, outside the mesh. A scrub already
running is not stopped by this; only its timer is.
@@ -161,7 +197,7 @@ The module runs two long-running processes of its own binary, and the scheduled
their health changes.
- **The event follower** (`zfs-tools events`) runs as root and follows `zpool events -f`. It is described
in the next section.
- **The scheduled run** (`zfs-tools scrub-run`) is not long-running and has no health check of its own;
- **The scheduled run** (`zfs-tools run`) is not long-running and has no health check of its own;
`zfs_health` says when it failed or did not start a due scrub.
`zfs_health` is **unhealthy** for a fault now:
@@ -216,12 +252,17 @@ The tests cover:
paused scrubs) and `iostat -H` without section lines;
- a timed-out command ended with its children;
- the snapshot's refusals, read-back, space statement and unknown outcome;
- the scrub acts, the schedule's slots, and the scheduled run (start, wait, never pause, defer to a
- the scrub acts, the schedule's slots, and the scheduled run (start and return, never pause, defer to a
resilver, refuse an invalid setting);
- the config's parse, one table per value; each convergence plan (properties, devices, the node's ZFS id, the ARC); the lock; the
scheduled ZFS snapshots' taking and removal; the hand acts; the keeper's closed list of calls and its socket; the tools; the health;
- the follower's cursor, dropped events, kill before wait, and exit after repeated failures;
- the health verdicts, past against new errors;
- a machine without ZFS;
- the manifest against the tools the bundle serves.
- the manifest (values, files, units, order, bus state, keeper) against the tools the bundle serves and the call-site test.
The module adds no integration test (its test budget is 500 unit tests and 50 integration tests): the main design's live tests stand for
the module whole.
## Files it does not know
@@ -231,3 +272,6 @@ Each directory or archive this module declares says how it treats a file it did
- `/run/zfs-events`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/etc/zfs-tools`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/var/lib/zfs-scrub`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/run/zfs-tools`: everything in it is the program's (the lock); the mesh moves nothing out.
- the keeper's folder: everything in it is the program's (its socket and its credential).
- `/etc/zfs/mesh-zed.d`: the module's own, while `zed` is on.
modules/zfs/cmd/zfs-tools/manifest_test.go
+16
View File
@@ -540,3 +540,19 @@ func TestTheManifestsBusStateKeeperAndTools(t *testing.T) {
}
}
}
// §31, the test budget: the module stays within 500 unit tests and adds no integration test (its live tests are the main design's).
func TestTheModuleStaysWithinItsTestBudget(t *testing.T) {
files, _ := filepath.Glob("*_test.go")
n := 0
for _, f := range files {
b, _ := os.ReadFile(f)
if strings.Contains(string(b), "//go:build integration") {
t.Errorf("%s is an integration test: the module adds none", f)
}
n += len(regexp.MustCompile(`(?m)^func Test`).FindAllString(string(b), -1))
}
if n > 500 {
t.Errorf("%d unit tests, over the budget of 500", n)
}
}