jochen
7c8497225d
mesh/merge-gate pass: builds new: modules/zfs, sent nowhere; no bus step; every machine composes with the change as it did without (4 of 4 compose)
mesh/repo-check fail: its merge-check.sh failed: --- FAIL: TestASnapshotIsAnsweredFromTheListAfterwardsAndRecorded (0.00s)
mesh/delivery superseded: a newer head of the same pull request
The home server's pool holds the operator's media library, and no mesh tool could say whether it is healthy, how full it is, when it was last scrubbed or what snapshots exist; the only way was a login. This module reads all of it with fixed zfs and zpool commands (list, get, status, iostat, events, history) and has one act, zfs_snapshot, which only adds a snapshot: refused for an unknown dataset, a name outside [A-Za-z0-9._-], an existing snapshot or no why; run as sudo -n zfs snapshot -- ds@name; answered from zfs list afterwards, with any other snapshot that came or went meanwhile; recorded as the event snapshot.taken. No destroy, rollback, set or send exists, and a test holds every command to its verb. Health (ADR 0240): zfs_health on the watch process, unhealthy for a pool not ONLINE, device or data errors, a pool past 90 % full or no ZFS at all; past 80 % and a scrub older than 35 days are warnings. Beside it, the event follower (zpool events -f, as root) keeps ZFS's errors in its journal past a reboot and is judged by its unit, the check the module does not run itself. No seat (ADR 0121), no package: ZFS is the machine's.
108 lines
6.9 KiB
Markdown
108 lines
6.9 KiB
Markdown
# zfs
|
|
|
|
The mesh's view of ZFS on a machine: every pool's health, layout, space, scrubs, I/O, cache and events,
|
|
every dataset with its space and snapshots, and one act, a named snapshot of one dataset. It is for the
|
|
home server, whose pool holds the operator's media library, which the mesh does not own (novox/hq ADR
|
|
0051). So the module reads, and the one thing it does only adds.
|
|
|
|
## What it does not do
|
|
|
|
There is no destroy, rollback, set, send, receive, rename, clone or any other change. Each of those changes
|
|
or removes the operator's data and needs a decision of its own first. `zfs_snapshot` is the only tool that
|
|
changes anything, and it only adds a snapshot.
|
|
|
|
This is checked by `TestTheBundleRunsOnlyReadingVerbsAndSnapshot`. Every command the bundle runs is built
|
|
by one of its argument functions, and each function is held to its verb: `list`, `get`, `status`,
|
|
`iostat`, `events`, `history` or `snapshot`. A call site that builds its own argument list fails the test.
|
|
|
|
It installs nothing. ZFS belongs to the machine, and its kernel module is built against that machine's
|
|
kernel. On a machine without ZFS, every tool answers `applicable: false` with the reason, and the health
|
|
check is unhealthy and says to unassign the module, because a module put where it does nothing must not
|
|
look healthy.
|
|
|
|
It holds no seat. A new `node-*` seat is the controller's to define (novox/hq ADR 0121, ADR 0246). This
|
|
is a module with tools, like `disk-load` and `sensors`. If the mesh later gives a machine's storage a
|
|
seat, these tools become its verbs.
|
|
|
|
## Tools
|
|
|
|
The tools run as the operator account. What ZFS lets any account read runs as that account. What ZFS
|
|
keeps for root (events, history and the snapshot itself) goes through `sudo -n` with a fixed argument
|
|
list, never a shell. Sizes are whole bytes, read under `-p`, and each answer also says its findings in
|
|
lines under `said`.
|
|
|
|
| tool | what | replaces |
|
|
|---|---|---|
|
|
| `zfs_datasets` | Every filesystem and volume: mountpoint, used, available, referenced, compression ratio, sharenfs, and the number of snapshots with the space they hold together (`usedbysnapshots`) | `zfs list` |
|
|
| `zfs_snapshots` | The snapshots of one dataset (`dataset`, optional), oldest first: creation time, used (the space only that snapshot holds) and referenced | `zfs list -t snapshot` |
|
|
| `zfs_snapshot` | Takes `dataset@name` (see below) | `zfs snapshot` |
|
|
| `zfs_pools` | Each pool's state with status's explanation and action, space, capacity and fragmentation %, device error count, data errors and last scan | `zpool list`, `zpool status -x` |
|
|
| `zfs_space` | Each pool's space, then every dataset, biggest first: usedbydataset, usedbysnapshots, usedbychildren and usedbyrefreservation, quotas and reservations where set, compression, logical size against physical size, and the savings | `zfs list -o space`, `zpool list -v` |
|
|
| `zfs_pool_layout` | Vdevs and disks by the names the pool knows them by, with state, read, write and checksum errors and status's note; log, cache, special and dedup devices; spares | `zpool status`, `zpool status -v` |
|
|
| `zfs_scrub` | The last scan (scrub or resilver): when, how long, what it repaired, the errors, or its progress while it runs; the days since the last finished scrub, with a warning past 35. With `history: true`, it also reads the pool's history (as root) for the scrub and resilver that status no longer shows | `zpool history -i` |
|
|
| `zfs_io` | A sample of `seconds` (default 3, from 1 to 10): operations and bytes per second per pool, vdev and disk | `zpool iostat -v` |
|
|
| `zfs_arc` | ARC size, target, minimum and maximum, hit ratio since boot, demand hit ratio, and L2ARC where present | `arcstat`, `arc_summary` |
|
|
| `zfs_events` | Recent events, newest first, as root. By default it shows only errors and changes of state, and `all: true` shows every event (`limit`, default 50) | `zpool events -v` |
|
|
| `zfs_properties` | Every property of one `dataset` with its source, the usual ones in lines first. The value of `keylocation` is withheld, and ZFS never gives a key itself | `zfs get all` |
|
|
| `zfs_health` | The health check (below) | |
|
|
|
|
## The snapshot
|
|
|
|
`zfs_snapshot {dataset, name, why, recursive?}` refuses all of these before anything runs:
|
|
|
|
- a dataset that `zfs list` does not show on this machine;
|
|
- a name with anything but letters, digits, `.`, `_` and `-`;
|
|
- a whole name longer than ZFS's 255 characters;
|
|
- a snapshot that already exists (with `recursive`, on any child too);
|
|
- a missing `why`, or a `why` longer than one line.
|
|
|
|
It then runs `sudo -n zfs snapshot -- <dataset>@<name>`, with `-r` only when `recursive` is true. It
|
|
answers from `zfs list` afterwards, not from what it ran: the snapshot made and its creation time, and any
|
|
other snapshot that came or went during the call (another schedule's, for example). A snapshot the list
|
|
does not show is an error, whatever the command said.
|
|
|
|
The act is recorded as the event `snapshot.taken` (the dataset, the name, why, whether it was done, and
|
|
when), and ZFS keeps the command in the pool's own history. An event the bus does not accept is said in
|
|
the answer as NOT RECORDED.
|
|
|
|
## Health
|
|
|
|
The module runs two long-running processes of its own binary, and neither writes a file or changes
|
|
anything:
|
|
|
|
- **The watch** (`zfs-tools watch`) carries the module's health check, `zfs_health` (novox/hq ADR 0240).
|
|
Each minute it reads the pools and writes a line in its journal whenever their health changes.
|
|
- **The event follower** (`zfs-tools events`) runs as root and follows `zpool events -f`. It writes each
|
|
error and change of state to its journal as ZFS emits it. ZFS keeps its events in memory only, so they
|
|
are lost at a reboot, and the journal keeps them past one. The node-engine judges it by its unit. That
|
|
is the check the module does not run itself, which a tool check needs beside it (ADR 0240 rule 2).
|
|
|
|
`zfs_health` is **unhealthy** when:
|
|
|
|
- a pool is not ONLINE (the answer gives status's reason and action);
|
|
- a device has read, write or checksum errors, or is not online;
|
|
- a pool reports data errors;
|
|
- a pool is more than 90 % full;
|
|
- no pool is imported;
|
|
- ZFS is not on the machine.
|
|
|
|
These are **warnings**, which leave the module healthy:
|
|
|
|
- a pool is more than 80 % full;
|
|
- a pool's last scrub finished more than 35 days ago, or the pool was never scrubbed.
|
|
|
|
The health check judges only the last scan that status shows. If that scan was a resilver, the last
|
|
scrub's age is in `zfs_scrub` with `history: true`.
|
|
|
|
## Tests
|
|
|
|
`go test ./...` runs against OpenZFS 2.4 output in `cmd/zfs-tools/testdata`. Each file has the exact form
|
|
the command prints, with documentation names in place of the real pools and disks. The tests cover:
|
|
|
|
- parsing every command's output;
|
|
- the snapshot's argument list, every refusal (none runs a command), the read-back, a snapshot the list
|
|
does not show, a refused sudo, an event the bus does not accept, and another snapshot taken meanwhile;
|
|
- the health verdicts;
|
|
- a machine without ZFS;
|
|
- the manifest against the tools the bundle serves.
|