From 7831f14a6e8ec1ee089be19b364e4a78fcb0ffb9 Mon Sep 17 00:00:00 2001 From: jochens Date: Fri, 13 Nov 2026 01:06:04 +0100 Subject: [PATCH] Z19 zfs-readme-budget: the README says what the module owns and how it is checked, and the test budget is a test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Builds: design 42 zfs-owns-zfs §31, §7, §19, §23 --- modules/zfs/README.md | 172 +++++++++++++-------- modules/zfs/cmd/zfs-tools/manifest_test.go | 16 ++ 2 files changed, 124 insertions(+), 64 deletions(-) diff --git a/modules/zfs/README.md b/modules/zfs/README.md index ecb0b5dc3..97609eb82 100644 --- a/modules/zfs/README.md +++ b/modules/zfs/README.md @@ -1,65 +1,105 @@ # zfs -The mesh's view of ZFS on a machine: every pool's health, layout, space, scrubs, I/O, cache and events, -and every dataset with its space and snapshots. It owns the pool's scrub schedule, run by a scheduled process of its own. It has four acts: a named -snapshot of one dataset, and a scrub's start, pause and stop. It is for the home server, whose pool holds -the operator's media library, which the mesh does not own (novox/hq ADR 0051). So the module reads, and -what it does only adds a snapshot or reads the pool through. +The mesh's view of ZFS on a machine, and what it owns of it: the packages, the pool's properties, cache and log devices, +the node's ZFS id, the ARC, the event daemon, the scrub and trim schedule and scheduled ZFS snapshots. It changes only what +its config values set: a value that is not set, or an empty list, is not managed, and nothing is ever removed by a blank +value (ADR 0679). The pool's data is the operator's, which the mesh does not own (novox/hq ADR 0051). + +## What it installs + +`zfs-utils` and `zfs-dkms` are declared present: the node-engine installs a missing one, upgrades none, removes none. +`zfs-utils-debug` is not the module's and is left as found. The module never writes the package's own `/etc/default/zfs` +or `/etc/zfs/zed.d`; a `.pacnew` beside `/etc/default/zfs` or the package's `zed.rc` is a warning. ## What it does not do -There is no destroy, rollback, set, send, receive, rename, clone, replace, attach, detach or clear. Each of -those changes or removes the operator's data, or the pool's layout, and needs a decision of its own first. -A resilver is ZFS's own: it starts one when a disk is replaced or comes back. The module watches and -reports resilvers, and starts none. +It never destroys a dataset, rolls back, renames, clears, sends, receives, replaces, attaches or detaches, and it adds +no data vdev, spare, special or dedup device. Each of those changes or removes the operator's data or the pool's layout +and needs a decision of its own first. A dataset's `sharenfs` is the seat `node-nfs-server`'s and `sharesmb` the SMB module's: +a value naming either is refused. -This is checked by `TestTheBundleRunsOnlyReadingVerbsSnapshotAndScrub`. Every command the bundle runs is -built by one of its argument functions, and each function is held to its program and verb: +This is checked by `TestTheBundleRunsOnlyTheFormsOfADR0679`. Every command the bundle runs is built by one of its argument +functions, and each function is held to its program, its verb and its whole form: -- `zfs`: `list`, `get` and `snapshot`; -- `zpool`: `list`, `status`, `iostat`, `events`, `history`, `wait -t scrub` and `scrub` (with `-p` or `-s` - only, and those only from the person's acts, never from the scheduled run); -- `systemctl`: `show`. +- `zpool`: `add cache|log` (never `-f`), `remove` of a cache or log device, `set` of a listed pool property, `scrub` + (with `-p` or `-s` only from a person's act), `trim` of the pool, and the reads `status`, `list`, `get`, `iostat`, `events` + and `history`; +- `zfs`: `snapshot`, `set mesh:schedule=` on a snapshot named `mesh-`, `destroy -- @`, `set` and + `inherit` of a listed dataset property, and the reads `list`, `get` and `holds`; +- `wipefs --no-act `, `zdb -l ` and `zdb -C `, `zgenhostid -f ` only with the value the parse + validated, `udevadm monitor --udev --subsystem-match=block`, and `systemctl show`. -The test fails on any call site that builds its own argument list, through `ask`, `AsRoot`, `Escalated`, -`.Run(`, `exec.Command(`, `exec.CommandContext(`, `os.StartProcess` or `syscall.Exec`. +Any other `zpool` or `zfs` form, and every `-f`, `-r`, `-R` and `-d`, is refused; so is any call site that builds its own argument +list. -It installs nothing. ZFS belongs to the machine, and its kernel module is built against that machine's -kernel. On a machine without ZFS, every tool answers `applicable: false` with the reason, and the health -check is unhealthy and says to unassign the module, because a module put where it does nothing must not -look healthy. +On a machine without ZFS, every tool answers `applicable: false` with the reason, and the health check is unhealthy and says +to unassign the module. It holds no seat. A new `node-*` seat is the controller's to define (novox/hq ADR 0121, ADR 0246). +## Config values + +| value | form | managed by | +|---|---|---| +| `pool` (was `scrub-pool`) | the pool's name, trusted, required | the scrub, trim, ZFS snapshots and the apply | +| `scrub-cadence`, `scrub-day`, `scrub-time` | as before, with their defaults | the scheduled run | +| `arc-max`, `arc-min` | whole MiB | written live and into `/etc/modprobe.d/zfs.conf`, read back | +| `module-options` | `name=digits` from a fixed list | the options file; a live parameter also live, others at the next load of the kernel module | +| `hostid` | eight hexadecimal digits | `/etc/hostid`, by `zgenhostid -f`, only when the pool's label holds 0 or this value | +| `zed`, `zed-handlers` | `on` or `off`; `all-syslog`, `resilver_finish-start-scrub` | the module's own ZED folder, without links | +| `trim-schedule`, `trim-day`, `trim-time` | `none`, `weekly` or `monthly` with a day and a time | the scheduled run | +| `cache-devices`, `log-devices`, `log-mirror` | names under `/dev/disk/by-id` | the apply, after its checks | +| `pool-properties`, `dataset-properties` | the listed properties | the apply, read back | +| `snapshots` | `{"none": true}` or keep counts `hourly`, `daily`, `weekly`, `monthly`, 0 to 168 | the scheduled run | + +A value of another form is refused before anything is written or run (`config.refused`); one refused value refuses the whole +config. A keep count of 0 is that kind's none: none is taken, and its existing scheduled ZFS snapshots are kept with a warning. +A trim schedule of `weekly` or `monthly` without a `trim-day` or `trim-time` is refused. Changing `hostid` needs a check of the +pool's import at the next boot; the module never reboots. + +## The apply, the run, the keeper and the follower + +The module writes `/etc/zfs-tools/config.json` (all its values as one JSON object). The apply (`zfs-tools apply`, root, +run-once, started again by the config file's change) converges the node under an exclusive lock, waits (`config.waiting`) while a scrub +or resilver runs or before the pool is imported, and exits non-zero only when a command it ran failed. The event follower +(`zfs-tools events`, root) starts the apply at its own start and when a scrub or resilver ends or a declared device appears, and keeps the +findings of a pool or device not ONLINE in its state `findings`. The scheduled run (`zfs-tools run`, root, every quarter of an hour) takes +the same lock, starts a due scrub or trim and returns without waiting, takes and removes the scheduled ZFS snapshots and acts on a person's +hand acts. The three root processes hold no bus account: the keeper (`zfs-tools keeper`, the module's user `zfs-keeper`) holds the +module's one, and serves a unix socket only root may open, with a closed list of calls: emit an event of the module, put, delete, get or list a key of the states `applied`, +`schedule` and `findings`, and invoke `zfs_snapshots_record`, `zfs_snapshots_schedule` or the service manager's `start` of the module's two +units. With the keeper not answering they act as with the bus down and warn. + +## Scheduled ZFS snapshots + +Each kind with a keep count above 0 is due at the first run of the hour, day, Monday or month, against the ledger's last entry. For every +filesystem of the pool the run takes `zfs snapshot -- @mesh--`, never recursive, sets `mesh:schedule`, and records +`taken` in the ledger `snapshots` (one instance per dataset and kind, its state a map keyed by the ZFS snapshot's name) through the tool +`zfs_snapshots_record`. It takes none it cannot record, none with the ledger unreadable or full, and none while the pool is more than 80 % full. +It removes the oldest past a keep count only by their own record: name, `mesh:schedule` and ledger entry fit, no hold and no clone. +A snapshot taken by hand (`zfs_snapshot`, which refuses a `mesh-` name, event `snapshot.taken.person`) is never removed. + ## Tools -The tools run as the operator account. What ZFS lets any account read runs as that account. What ZFS -keeps for root goes through `sudo -n` with a fixed argument list, never a shell: events, history, the -snapshot and the scrub acts. Sizes are whole bytes, read under `-p`, except where status prints human sizes -even under `-p` (a scan's sizes). Each answer also says its findings in lines under `said`. +The tools run as the operator account. What ZFS keeps for root goes through `sudo -n` with a fixed argument list, never a shell. A command that +runs out of time is ended with everything it started. -A command that runs out of time is ended with everything it started. It runs in a process group of its -own. At the deadline the group gets SIGTERM, which sudo passes on to the command it runs as root, and -after five seconds SIGKILL. An act that ran out of time is answered from what the machine shows afterwards, -or as "outcome unknown". +| tool | what | +|---|---| +| `zfs_datasets`, `zfs_snapshots`, `zfs_pools`, `zfs_space`, `zfs_pool_layout`, `zfs_scrub`, `zfs_io`, `zfs_arc`, `zfs_events`, `zfs_properties`, `zfs_health` | the reads (below) | +| `zfs_snapshot` | takes `dataset@name`, a person's act with why | +| `zfs_scrub_start`, `zfs_scrub_pause`, `zfs_scrub_stop` | a person's scrub acts | +| `zfs_config` | every value with its config value, what runs, how it stands, the package's files, and what the module says before it leaves | +| `zfs_config_apply` | converge now, as the apply does | +| `zfs_snapshots_schedule` | the keep counts, next moments, what the next run would remove, and the ledger | +| `zfs_snapshots_run_now`, `zfs_snapshots_prune_now`, `zfs_snapshots_pause`, `zfs_snapshots_resume` | a person's acts on the scheduled ZFS snapshots; the hand acts wait up to 30 seconds for their event, else answer "started" | +| `zfs_module_events` | the module's own events, newest first (`zfs_events` reads ZFS's) | +| `zfs_snapshots_check` | the ledger against what ZFS holds; changes nothing | +| `zfs_findings` | the follower's findings | +| `zfs_snapshots_record` | the run's own record of a `taken` or `removed` ZFS snapshot | -| tool | what | replaces | -|---|---|---| -| `zfs_datasets` | Every filesystem and volume: mountpoint, used, available, referenced, compression ratio, sharenfs, and the number of snapshots with the space they hold together | `zfs list` | -| `zfs_snapshots` | The snapshots of one `dataset` (optional), oldest first: creation time, used (the space only that snapshot holds) and referenced | `zfs list -t snapshot` | -| `zfs_snapshot` | Takes `dataset@name` (see below) | `zfs snapshot` | -| `zfs_pools` | Each pool's state with status's reason and action, space, capacity and fragmentation %, device error count, data errors and scans | `zpool list`, `zpool status -x` | -| `zfs_space` | Each pool's space, then every dataset, biggest first, with what uses its space, quotas and reservations, compression and the savings | `zfs list -o space`, `zpool list -v` | -| `zfs_pool_layout` | Vdevs and disks, with state, error counts and status's note; log, cache, special and dedup devices; spares | `zpool status`, `zpool status -v` | -| `zfs_scrub` | Each pool's scans: its scrub or error scrub, and its resilver (pool-wide, or each vdev's rebuild), with progress and time to go while one runs; the days since the last finished scrub; and the module's schedule with its next run. `history: true` also reads the pool's history | `zpool history -i` | -| `zfs_scrub_start` | Starts a scrub of `pool`, or resumes a paused one, with `why` | `zpool scrub` | -| `zfs_scrub_pause` | Pauses one, keeping its place, with `why` | `zpool scrub -p` | -| `zfs_scrub_stop` | Stops one, losing its place, with `why` | `zpool scrub -s` | -| `zfs_io` | A sample of `seconds` (default 3, from 1 to 10): operations and bytes per second per pool, vdev and disk | `zpool iostat -v` | -| `zfs_arc` | ARC size, target, minimum and maximum, hit ratio since boot, and L2ARC where present | `arcstat`, `arc_summary` | -| `zfs_events` | Recent events, newest first, as root. By default it shows only errors and changes of state, and `all: true` shows every event | `zpool events -v` | -| `zfs_properties` | Every property of one `dataset` with its source. The value of `keylocation` is withheld | `zfs get all` | -| `zfs_health` | The health check (below) | | +When the module leaves a node, the next load of the kernel module uses ZFS's defaults, `zfs-import-cache`, `zfs-mount` and `zfs.target` stay +running, `/etc/hostid` stays, and pool and dataset properties and devices stay as the last apply set them. ## The snapshot @@ -75,17 +115,17 @@ It runs `sudo -n zfs snapshot -- @`. It answers from `zfs list` a it ran: the snapshot made, its creation time, and any other snapshot that came or went during the call. It also gives the dataset's free space and the pool's capacity. **From then on, the snapshot holds every block of the dataset that is deleted or overwritten, so that space is not freed until the snapshot is destroyed by -hand.** No tool of the mesh destroys a snapshot. +hand.** The scheduled run destroys only the scheduled ZFS snapshots it took and recorded, by their own record. -The act is recorded as the event `snapshot.taken`, and ZFS keeps the command in the pool's own history. +The act is recorded as the event `snapshot.taken.person`, and ZFS keeps the command in the pool's own history. ## The scrub schedule -The module owns the schedule of the pool its `scrub-pool` setting names. +The module owns the schedule of the pool its `pool` setting names (before this version `scrub-pool`, which mesh-config renames). | setting | default | what | |---|---|---| -| `scrub-pool` | none: the operator's value | the pool to scrub | +| `pool` | none: the operator's value | the pool to scrub | | `scrub-cadence` | `weekly` | `weekly`, or `monthly` (the first given weekday of each month) | | `scrub-day` | `Mon` | `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat` or `Sun`, nothing else | | `scrub-time` | `00:00` | `HH:MM` from `00:00` to `23:59`, in the machine's time zone | @@ -93,17 +133,13 @@ The module owns the schedule of the pool its `scrub-pool` setting names. The defaults are what the distribution's weekly timer did on the home server before the module, so taking it over does not move the schedule. The operator chose monthly for the home server. -**The scheduled run.** The node-engine runs the module's scheduled process `zfs-mesh-scrub` (`zfs-tools -scrub-run`, as root) from its own timer `zfs-mesh-scrub.timer`, every quarter of an hour. The run reads the -configuration the mesh writes from the settings (`/etc/zfs-tools/scrub.conf`). It finds the slot, which is -the last scheduled time not after now. If the slot is not yet served, the run acts as upstream's -`zfs-scrub@.service` does: +**The scheduled run.** The node-engine runs the module's scheduled process `zfs-mesh-run` (`zfs-tools run`, as root) from its +own timer `zfs-mesh-run.timer`, every quarter of an hour. The run reads `/etc/zfs-tools/config.json`. It finds the slot, which +is the last scheduled time not after now. If the slot is not yet served, the run starts the scrub with `zpool scrub -- ` +and returns at once: it never waits for it, and the event follower hears the scrub end. If a scrub or resilver runs, or the pool +is not imported, the run does nothing and says `run.skipped`. -- if a scrub is running, it waits for it with `zpool wait -t scrub -- `; -- otherwise it starts one with `zpool scrub -- ` and waits for it the same way; -- if the start is refused because a scrub has just begun, it waits for that one. - -It never pauses or stops a scrub. The unit has no ExecStop, so stopping the unit ends only the wait. A +It never pauses or stops a scrub. A resilver that is running defers the scrub to the next run after it. The served slot is kept in `/var/lib/zfs-scrub/scrub-served.json`, in a folder of root's alone (0700), apart from the account's state. The run writes it only into a real folder of its own and never through a link: a new file opened with @@ -119,7 +155,7 @@ the names of the two distribution timers. The controller fills it there and refu digits, `:`, `_`, `.` and `-` (mesh-controller #171). **Exactly one schedule per pool.** The distribution's `zfs-scrub-weekly@.timer` and -`zfs-scrub-monthly@.timer` are held stopped and disabled. They come after the scheduled run in the +`zfs-scrub-monthly@.timer` are held stopped and disabled, and the trim timers with them while `trim-schedule` is set. They come after the scheduled run in the declaration, so a run that cannot be installed stops the apply before the timer that scrubs the pool today is touched. On the home server the weekly one was enabled by hand, outside the mesh. A scrub already running is not stopped by this; only its timer is. @@ -161,7 +197,7 @@ The module runs two long-running processes of its own binary, and the scheduled their health changes. - **The event follower** (`zfs-tools events`) runs as root and follows `zpool events -f`. It is described in the next section. -- **The scheduled run** (`zfs-tools scrub-run`) is not long-running and has no health check of its own; +- **The scheduled run** (`zfs-tools run`) is not long-running and has no health check of its own; `zfs_health` says when it failed or did not start a due scrub. `zfs_health` is **unhealthy** for a fault now: @@ -216,12 +252,17 @@ The tests cover: paused scrubs) and `iostat -H` without section lines; - a timed-out command ended with its children; - the snapshot's refusals, read-back, space statement and unknown outcome; -- the scrub acts, the schedule's slots, and the scheduled run (start, wait, never pause, defer to a +- the scrub acts, the schedule's slots, and the scheduled run (start and return, never pause, defer to a resilver, refuse an invalid setting); +- the config's parse, one table per value; each convergence plan (properties, devices, the node's ZFS id, the ARC); the lock; the + scheduled ZFS snapshots' taking and removal; the hand acts; the keeper's closed list of calls and its socket; the tools; the health; - the follower's cursor, dropped events, kill before wait, and exit after repeated failures; - the health verdicts, past against new errors; - a machine without ZFS; -- the manifest against the tools the bundle serves. +- the manifest (values, files, units, order, bus state, keeper) against the tools the bundle serves and the call-site test. + +The module adds no integration test (its test budget is 500 unit tests and 50 integration tests): the main design's live tests stand for +the module whole. ## Files it does not know @@ -231,3 +272,6 @@ Each directory or archive this module declares says how it treats a file it did - `/run/zfs-events`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2). - `/etc/zfs-tools`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2). - `/var/lib/zfs-scrub`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2). +- `/run/zfs-tools`: everything in it is the program's (the lock); the mesh moves nothing out. +- the keeper's folder: everything in it is the program's (its socket and its credential). +- `/etc/zfs/mesh-zed.d`: the module's own, while `zed` is on. diff --git a/modules/zfs/cmd/zfs-tools/manifest_test.go b/modules/zfs/cmd/zfs-tools/manifest_test.go index ea7021441..4d19bf8ad 100644 --- a/modules/zfs/cmd/zfs-tools/manifest_test.go +++ b/modules/zfs/cmd/zfs-tools/manifest_test.go @@ -540,3 +540,19 @@ func TestTheManifestsBusStateKeeperAndTools(t *testing.T) { } } } + +// §31, the test budget: the module stays within 500 unit tests and adds no integration test (its live tests are the main design's). +func TestTheModuleStaysWithinItsTestBudget(t *testing.T) { + files, _ := filepath.Glob("*_test.go") + n := 0 + for _, f := range files { + b, _ := os.ReadFile(f) + if strings.Contains(string(b), "//go:build integration") { + t.Errorf("%s is an integration test: the module adds none", f) + } + n += len(regexp.MustCompile(`(?m)^func Test`).FindAllString(string(b), -1)) + } + if n > 500 { + t.Errorf("%d unit tests, over the budget of 500", n) + } +}