diff --git a/02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md b/02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md index d7917bc..9b9b21a 100644 --- a/02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md +++ b/02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md @@ -118,10 +118,22 @@ anything irreplaceable depends on the machine's `node-backup` holder (ADR 0207) up or watches its array — so it cannot be assigned where nothing would. Valuable and rebuildable data is backed up where a holder is and refuses no machine for want of one. -**5. The holder measures; the controller judges.** Once an hour, and after every night, the holder -measures every item that is not a cache — the size of its files and its newest change, in one walk as -root — and reads the redundant storage it is on: a ZFS pool's health and its own verdict, an md array's -members, a btrfs filesystem's error counters. It answers this, with each item's newest good backup, in +**5. The holder measures; the controller judges — and nothing large is ever walked.** The holder +measures every item that is not a cache by the item's own `measure`: + +- `walk` (the default, for small items): every file summed and the newest change found, in one walk as + root — **at most once a day**, after the night, and **stopped after ten minutes or two million + files**, said as "measured partially": a partial size is a lower bound, shown and never compared; +- `dataset`: the ZFS dataset holding the path — its size from the filesystem's own counters — and the + newest change among the path's top-level entries; hourly, nothing walked. Items on one dataset share + its size, so a shrink of it is one condition naming them all; +- `shallow`: the newest change among the top-level entries only, and no size; hourly. + +A large item never says `walk`: the media library is `dataset`, Plex's metadata and previews, the +artifact store and the models are `shallow`. A file walk over a pool of that size every hour is load on +the array that protects the library, and competes with the server reading it. The holder also reads the +redundant storage each item is on: a ZFS pool's health and its own verdict, an md array's members, a +btrfs filesystem's error counters. It answers this, with each item's newest good backup, in `node-backup.backed-up`; it decides nothing. **The holder is the measurer**, not the node-engine: it already reads every declared path as root, it is a module that can be replaced, and the core gains no work. The controller's self-check asks every holder on every run. @@ -144,7 +156,7 @@ class kept in it. | Condition | Raised when | |---|---| -| `data-shrank` | an item holds less than half of its largest size in seven days, and at least 16 MiB less | +| `data-shrank` | an item holds less than half of its largest size in seven days, and at least 16 MiB less; one condition per dataset for items measured from a dataset's counters; never from a partial size | | `data-missing` | an item's path is gone | | `empty-replacement` | a copy in use is less than half the size (and 16 MiB less) of a copy of the same thing kept elsewhere: an item retired on another machine, or a consumer's data at another provider — issue 273's shape | | `data-held-twice` (warning) | a consumer has active data at two providers whose sizes cannot be compared | @@ -169,7 +181,7 @@ the media library and the photo sites' storage are irreplaceable; everything peo | Module | Own data | Its consumers' | Kept with a provider | |---|---|---|---| -| plex | the media library, eight operator paths: **irreplaceable**, by **redundancy**; metadata, previews, working data: rebuildable (backed up); transcode: cache | | | +| plex | the media library, eight operator paths: **irreplaceable**, by **redundancy**, measured from its dataset; metadata: rebuildable, backed up, measured shallow; **previews: rebuildable, not backed up** (Plex makes them again; too large to copy nightly); working data: rebuildable; transcode: cache | | | | photos | | | object store and document store: **irreplaceable** | | postgres, mssql, mongodb | the store: valuable, by dump; the dumps: rebuildable | valuable | | | minio, influxdb | data (and influx's config): valuable | valuable | | @@ -184,28 +196,33 @@ the media library and the photo sites' storage are irreplaceable; everything peo | home-assistant | its configuration and history: valuable, written all the time (a day) | | | | nats | the bus's streams and buckets: valuable, written all the time (a day) | | | | n8n | data: valuable; scratch: cache | | | -| supabase | database, storage, functions: valuable; its seeded configuration: rebuildable | | | +| supabase | database: valuable, **by `pg_dumpall`** inside its container; storage, functions: valuable; the dump and its seeded configuration: rebuildable | | | | claude-code | the operator's agent's home: valuable | | | | ssh-client | the operator's keys: valuable | | | | radarr, sonarr, lidarr, bazarr | the database: valuable, by sqlite's own backup; settings: valuable; covers and the night's copy: rebuildable; lidarr's start-up scripts: cache | | | | bookshelf, jackett, kometa, nzbget, ombi, qbittorrent, tautulli | configuration: valuable | | | -| distribution, ollama | the artifact store, the models: rebuildable, **not backed up** (too large; made again) | | | +| distribution, ollama | the artifact store, the models: rebuildable, **not backed up** (too large; made again), measured shallow | | | | only-office | its working data, database, fonts: rebuildable; logs, queue, cache: cache | | | | records, route-proxy | the record's clone, issued certificates: rebuildable; the authority's roots: cache | | | | restic | the repository: rebuildable, not backed up — it is the copy, never the only copy of anything; its state: cache | | | | build-agent, icecast, lab, model-usage, searxng | cache | | | -**Uncertain, for the operator:** supabase's database is copied as live files, which may not restore — a -dump is wanted; the bus's streams are copied live; mailu's queue is valuable though transient; Plex's -previews may be large to copy nightly; the agent's home and the operator's keys are valuable, and are -backed up only on machines that hold `node-backup` (today, neither workstation does). +**Uncertain, for the operator:** the bus's streams are copied as live files — the bus's image carries +no tool to snapshot a stream safely, so a consistent copy needs the bus's own backup command run from +somewhere that has it, which is not built; mailu's queue is valuable though transient; the agent's home +and the operator's keys are valuable, and are backed up only on machines that hold `node-backup` +(today, neither workstation does). ## Consequences - **The first run of the self-check records what every machine holds**, and the first night backs up - what was not backed up before: on the home server, every application's own data and Plex's metadata - and previews; on the control node, the bus's streams and the certificate authority. The first night - there costs that data's full size, later nights what changed. + what was not backed up before: on the home server, every application's own data and Plex's metadata; + on the control node, the bus's streams and the certificate authority. Measured read-only on + 2026-10-06, the home server's new nightly data is about 11 GB of application data (the chat server's + database 5.6 GB, the database platform's 2.1 GB before its dump, the network controller 1.1 GB, the + viewing history 0.9 GB, dashboards 0.5 GB, the rest under 0.2 GB each) and Plex's metadata, at most + the 136 GB its disk holds — under 150 GB against 33.6 TB free where the repository is: under half a + percent. The previews, on the media pool, are left out. Later nights cost what changed. - **The media library is watched, not copied.** Its array is read every hour; a degraded pool is urgent. A pool that loses a vdev still loses the library — the operator's accepted risk, now said rather than silent. SMART pre-failure warnings are not read: the pool's own device error counters are, diff --git a/03-DESIGN/01-to-be/32-what-a-module-declares.md b/03-DESIGN/01-to-be/32-what-a-module-declares.md index 4df1426..40ddb42 100644 --- a/03-DESIGN/01-to-be/32-what-a-module-declares.md +++ b/03-DESIGN/01-to-be/32-what-a-module-declares.md @@ -273,7 +273,7 @@ mesh protects only what it is told about. One section, `data`, says all of it: | `data.` | says | |---|---| -| `own` | each item the module keeps: `id`, `path` (`${dir:}` or `${access:}`, or beneath one), `class`, and its protection — `backup` (`copy`, `none`, or `{dump, into}`) or `redundancy` with why that is enough — with optional `within`, `active`, `why` | +| `own` | each item the module keeps: `id`, `path` (`${dir:}` or `${access:}`, or beneath one), `class`, and its protection — `backup` (`copy`, `none`, or `{dump, into}`) or `redundancy` with why that is enough — with optional `measure` (`walk`, `dataset`, `shallow`), `within`, `active`, `why` | | `consumers` | per provision it grants: the `class` of what it keeps for each consumer, and `in` — the own item, or the required provision, holding it | | `kept-by` | per provision it requires: the `class` of what of its own the provider keeps | diff --git a/03-DESIGN/01-to-be/43-backups-against-mistakes.md b/03-DESIGN/01-to-be/43-backups-against-mistakes.md index 8dd535e..5da123d 100644 --- a/03-DESIGN/01-to-be/43-backups-against-mistakes.md +++ b/03-DESIGN/01-to-be/43-backups-against-mistakes.md @@ -50,8 +50,10 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214). *Added 2026-10-06, [ADR 0233](../../02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md).* The mesh composes a second file for the holder: every data item every module on the machine declares, with its class, its path, the path that covers it and its protection. Once an hour and after every -night the holder measures each item that is not a cache — the size of its files and its newest change, -in one walk as root — and reads the redundant storage it is on (a ZFS pool's health, an md array's +night the holder measures each item that is not a cache, by the item's `measure`: a bounded walk at +most daily for a small item (stopped after ten minutes or two million files and said as partial), a ZFS +dataset's own counters hourly, or the top-level entries only — never an hourly walk of anything large — +and reads the redundant storage it is on (a ZFS pool's health, an md array's members, a btrfs filesystem's error counters). `backed-up` says each item with that measurement and its newest good backup. The holder judges nothing: the controller's self-check (to-be 45, D13) keeps the readings and raises what they show.