ADR 0233: a module declares the data it holds, and the mesh protects and watches it

The operator's direction after issue 273: what data a module keeps, how precious it is and how it
is protected is said once, in the manifest, and backups, sticky bindings, retirement on unassign
and the self-check's watch are derived from it. Amends to-be 18, 32, 43 and 45.

02-DECISIONS/0214-backups-guard-against-mistakes-and-stay-on-the-machine.md

@@ -46,6 +46,13 @@ filesystem where there is one. Its key is a mesh secret.
 **A night that fails reaches the operator,** and a machine with data and no good backup in 48 hours
 shows in the mesh's status.
 
+> **The mechanism changed — 2026-10-06, by [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md).** The decision stands: backups guard
+> against mistakes and stay on the machine. What moved is the declaration: a module declares its data in
+> its `data` section, with a class and how each item is protected, and the holder's lines are derived
+> from it; a line written by hand is refused. The media library, left out here, is now declared
+> irreplaceable and protected by the redundancy of its array, which the holder watches, because there
+> is no room to copy it.
+
 ## Consequences
 
 - Adding a store provider means declaring its dump; the catalogue check can refuse a store provider

02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md

@@ -92,6 +92,13 @@ checks every machine every run. It raises:
 - any consumer about to be sent another provider than the one on record with no pin naming it, as
   **urgent**. The resolver makes this impossible; the probe exists for the day it is not.
 
+> **The mechanism changed — 2026-10-06, by [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md).** The decision stands: a binding to
+> a consumer's data moves only by a person. What moved is how a provision is known to keep data: a
+> provider now says it in its `data` section, as a class for its consumers — irreplaceable, valuable or
+> rebuildable keep data, cache and none do not — and `module check` refuses a provider that grants
+> without saying it. `keeps-consumer-data` on the offer and the inference from `grants` are read only
+> from a definition already on the shelf.
+
 ## Consequences
 
 - **The first push after rollout records what each machine is bound to then.** A consumer moved

02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md

@@ -0,0 +1,255 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-10-06
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md
+---
+
+# 233. A module declares the data it holds, and the mesh protects and watches it from that declaration
+
+## Context
+
+**This is the operator's decision, taken on 2026-10-06**, the day after five applications ran for
+twenty hours on empty databases ([issue 273](../04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md)).
+In the operator's words: modules must indicate the "data storage which should be monitored/protected",
+"in the module's manifest", "in a generic, configurable way". The ranking of what is precious is also the
+operator's, given the same day and recorded as given:
+
+> "data loss would be painful but not the end of the world. Only my plex storage is sacred and the photo
+> sites as well (I'm not sure everyone has a good backup themselves but it's their own responsibility)."
+
+> "we can't backup the plex storage, we don't have any storage for it, it's on a RAID5 which should be
+> safe as far as we can afford it. The plex metadata storage is not important" — and the metadata
+> "should be part of the standard backup plan of course".
+
+What the mesh knew about data before this record, measured on the code and the catalogue that day:
+
+- **Where a consumer's data is** was inferred, not said. [ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md)
+  made a binding sticky when the provider *grants* a credential; no manifest in the catalogue stated
+  `keeps-consumer-data`. A provider whose grants hold nothing — a cache — was sticky all the same.
+- **Backups were a second list.** [ADR 0214](0214-backups-guard-against-mistakes-and-stay-on-the-machine.md)
+  had a module write `run` and `path` lines to the `node-backup` seat by hand; thirteen modules did,
+  across two catalogues. The rule that "the catalogue check refuses a store provider that declares none"
+  was never in `module check`: one controller test applied it to the catalogue checked out beside it,
+  against a list of six provisions kept by hand.
+- **Unassigning a module kept its data only by accident of the host's rule.** The node-engine removes an
+  orphaned directory only when it is empty; a directory with anything in it is **kept and forgotten** —
+  its record dropped from the machine's store, the report saying "kept … remove it by hand once you know
+  what it is". Nothing anywhere then knew the data was there, and the one instruction given was the one
+  that destroys it.
+- **Nothing was measured.** No size, no last write, no check that a backup covered what mattered, and
+  nothing that could tell an empty copy of a database from the full one beside it — exactly the shape
+  of issue 273.
+- **The media library was out of every design** (ADR 0214 named it as not backed up), and the photo
+  sites' storage lives in two providers, the object store and the document store, as consumer data
+  whose preciousness only the photo application knows.
+
+## Considered Options
+
+1. **Keep ADR 0232's inference and add a list of protected paths in the backup module.** Rejected: a
+   second list is the failure ADR 0214 rejected ("whatever is not listed is unprotected, silently"), and
+   an inference cannot say that a grant holds a cache.
+2. **Keep `keeps-consumer-data` on the offer, required, and add a separate `backups` field.** Rejected:
+   two places for one fact, and a yes/no cannot carry the operator's ranking — a cache's consumers and a
+   photo library's both "keep data".
+3. **Every declared item backed up, irreplaceable or not; the media library copied too.** Rejected by
+   the operator: there is no storage for a copy of the media library. A class that forces a backup would
+   make the most precious data undeclarable.
+4. **A class per item decides everything, including the protection.** Rejected for the same reason:
+   how precious data is and how it is protected are two facts. The media library is the most precious
+   thing on the mesh and the one thing that cannot be copied.
+5. **A per-module check in code ("postgres is a store").** Rejected: never per-module code; a new
+   module would be unprotected until someone remembered it.
+6. **Array health watched by a new `node-storage` seat.** Deferred, not adopted: the only thing needed
+   today is to *read* an array's health under declared data, and the backup holder already reads that
+   data on every machine that keeps any. A seat for storage is right the day the mesh *acts* on disks
+   (scrubs, replacing a member); that is not decided here.
+7. **One `data` section, a class for how precious, a protection for how it is kept, and everything
+   derived from them.** Chosen.
+
+## Decision
+
+**1. One section, `data`, says every kind of data a module keeps.**
+
+- `own`: a list of items, each `id`, `path`, `class`, and how it is protected. `path` names one of the
+  module's directories, `${dir:<id>}`, or an operator's path it was given, `${access:<id>}`, or a path
+  beneath either — never a machine path (ADR 0112).
+- `consumers`: per provision the module grants, the `class` of what it keeps for each consumer and
+  `in` — the own item that holds it, or a provision the module itself requires (an identity provider
+  keeps its clients in its database). **Any provider that `grants` must say this; `module check`
+  refuses it otherwise.** The offer's `keeps-consumer-data` (ADR 0232) is still read from a definition
+  already on the shelf, and refused by `module check` in a new one: it is said once, here.
+- `kept-by`: per provision the module requires, the `class` of what of its own lives with that
+  provider. The photo sites' application says its objects and its database are irreplaceable; the
+  object store and the document store are then held to that class for the machine and consumer
+  concerned.
+- Each entry may carry `why`, a line for the reviewer.
+
+**2. Four classes, ranked by the operator.** A fixed vocabulary; a fifth is a decision.
+
+| Class | Is | Backed up | On unassign | Alerts |
+|---|---|---|---|---|
+| `irreplaceable` | what must never be lost: the media library, the photo sites' storage | a protection is **required**: a backup, or a declared redundancy | retired, never removed | **urgent** |
+| `valuable` | anybody's work, painful to lose: every store, mailboxes, repositories, the vault, the agent's home | **yes, by default** — the standard nightly plan, where the machine has a backup holder | retired, never removed | warning |
+| `rebuildable` | made again from elsewhere: a dump, a clone, thumbnails, Plex's metadata | **yes, by default**; may say `none` (the artifact store, downloaded models) | forgotten | none |
+| `cache` | disposable | never | forgotten | none |
+
+For consumers, `none` also exists: the provision keeps nothing of anybody's (a resolver). A binding to
+a consumer's data is sticky (ADR 0232) when the class is irreplaceable, valuable or rebuildable — not
+for a cache or none. **Redis is `cache`**: no module in this catalogue requires its provision, its name
+says what it is for, and a consumer keeping data in a cache would be the consumer's mistake, not the
+mesh's to make sticky.
+
+**3. Protection is its own field.** `backup` is `copy` (the holder reads the path as it stands),
+`none`, or `{dump, into}` — a command that writes a consistent copy into another item, which is copied
+(a running store's files are not a consistent copy). Unsaid, anything but a cache is copied. Or
+`redundancy: "<why that is enough>"`: the item is protected by the redundancy of the storage it is on,
+and the mesh watches that storage instead. **An irreplaceable item has a backup or a redundancy;
+`module check` refuses it with neither.** `within` bounds the age of its last good backup (48 hours
+unsaid, ADR 0214); `active` says it is written all the time and how long quiet is a fault.
+
+**4. The backup holder's lines are derived from the section.** The controller composes, from every
+module on a machine, the `backup` lines it always did and a second kind, `data`: every item with its
+class, its path, the path that covers it and its protection. **A backup line written by hand is
+refused by `module check`** and, beside a data section, not placed: one list. A module keeping
+anything irreplaceable depends on the machine's `node-backup` holder (ADR 0207) — the holder backs it
+up or watches its array — so it cannot be assigned where nothing would. Valuable and rebuildable data
+is backed up where a holder is and refuses no machine for want of one.
+
+**5. The holder measures; the controller judges.** Once an hour, and after every night, the holder
+measures every item that is not a cache — the size of its files and its newest change, in one walk as
+root — and reads the redundant storage it is on: a ZFS pool's health and its own verdict, an md array's
+members, a btrfs filesystem's error counters. It answers this, with each item's newest good backup, in
+`node-backup.backed-up`; it decides nothing. **The holder is the measurer**, not the node-engine: it
+already reads every declared path as root, it is a module that can be replaced, and the core gains no
+work. The controller's self-check asks every holder on every run.
+
+**6. Unassigning keeps the data, and the mesh remembers it.** An irreplaceable or valuable item in a
+module's own directory that its machine no longer declares is **retired** (ADR 0230's meaning): kept,
+recorded with when and why, listed by `cleanup list`, said by `unassign` when it is done, and a warning
+after thirty days. Assigned again, it is no longer retired. It is deleted only by `cleanup delete
+<machine> <module> <item> --why`, a hand act, which the machine's backup holder executes **only after
+taking a last restore point of it**, tagged as retired, so the deletion can be undone until a person
+forgets that restore point; the holder refuses any path declared now. An item on an operator's path —
+the media library — is **never** retired and never deleted: the mesh does not own it. The node-engine's
+rule stays (a directory with anything in it is never removed); its report no longer says "remove it by
+hand".
+
+**7. What is watched, and how loud.** Each condition is **urgent for irreplaceable data and a warning
+for valuable data**; rebuildable data and caches raise none. A consumer's data is as precious as the
+stricter of its provider's class and its own `kept-by`; a provider's item is held to the strictest
+class kept in it.
+
+| Condition | Raised when |
+|---|---|
+| `data-shrank` | an item holds less than half of its largest size in seven days, and at least 16 MiB less |
+| `data-missing` | an item's path is gone |
+| `empty-replacement` | a copy in use is less than half the size (and 16 MiB less) of a copy of the same thing kept elsewhere: an item retired on another machine, or a consumer's data at another provider — issue 273's shape |
+| `data-held-twice` (warning) | a consumer has active data at two providers whose sizes cannot be compared |
+| `data-quiet` | an item that says `active` has not been written within it |
+| `backup-stale` | an item's newest good backup is older than its bound, or there is none — for valuable data, only where a holder measured it |
+| `array-degraded` | the redundant storage under watched data is not healthy, or cannot be read; one per array |
+| `protection-missing` | an item says `redundancy` and is on storage the holder cannot read as redundant |
+| `data-unmeasured` (warning) | a machine's holder did not answer |
+| `cleanup-waiting` (warning) | an item retired more than thirty days |
+
+**8. A verb reads it.** `data` lists every item on every machine with its class, protection, path,
+size, newest write, newest good backup and its bound, the array under it, and what is retired.
+
+**9. Every catalogue module that holds data declares it** (the table below), and the catalogue check
+refuses a container that writes one of its directories, mounted whole, without a data item covering
+it — a directory the mesh does not know about is data the mesh cannot protect.
+
+### The classes, per module
+
+The operator's ranking applied to every module that holds data, **for the operator to correct**: only
+the media library and the photo sites' storage are irreplaceable; everything people wrote is valuable.
+
+| Module | Own data | Its consumers' | Kept with a provider |
+|---|---|---|---|
+| plex | the media library, eight operator paths: **irreplaceable**, by **redundancy**; metadata, previews, working data: rebuildable (backed up); transcode: cache | | |
+| photos | | | object store and document store: **irreplaceable** |
+| postgres, mssql, mongodb | the store: valuable, by dump; the dumps: rebuildable | valuable | |
+| minio, influxdb | data (and influx's config): valuable | valuable | |
+| mesh-vault | state, ledger, root: valuable | valuable | |
+| mailu | mail, signing keys, data, calendars, webmail, the queue: valuable; filter, certificates, autoconfiguration, fetch state: rebuildable; virus signatures, its redis: cache | valuable | |
+| gitea | data: valuable | the package registry: rebuildable | |
+| keycloak | | clients: rebuildable, in its database | |
+| umami | | page views: valuable, in its database | |
+| mosquitto | retained messages: rebuildable | rebuildable | |
+| redis | its snapshot: cache | **cache** | |
+| nextcloud, baserow, grafana, matrix, nodered, unifi, step-ca, audit-logger | their data: valuable | | |
+| home-assistant | its configuration and history: valuable, written all the time (a day) | | |
+| nats | the bus's streams and buckets: valuable, written all the time (a day) | | |
+| n8n | data: valuable; scratch: cache | | |
+| supabase | database, storage, functions: valuable; its seeded configuration: rebuildable | | |
+| claude-code | the operator's agent's home: valuable | | |
+| ssh-client | the operator's keys: valuable | | |
+| radarr, sonarr, lidarr, bazarr | the database: valuable, by sqlite's own backup; settings: valuable; covers and the night's copy: rebuildable; lidarr's start-up scripts: cache | | |
+| bookshelf, jackett, kometa, nzbget, ombi, qbittorrent, tautulli | configuration: valuable | | |
+| distribution, ollama | the artifact store, the models: rebuildable, **not backed up** (too large; made again) | | |
+| only-office | its working data, database, fonts: rebuildable; logs, queue, cache: cache | | |
+| records, route-proxy | the record's clone, issued certificates: rebuildable; the authority's roots: cache | | |
+| restic | the repository: rebuildable, not backed up — it is the copy, never the only copy of anything; its state: cache | | |
+| build-agent, icecast, lab, model-usage, searxng | cache | | |
+
+**Uncertain, for the operator:** supabase's database is copied as live files, which may not restore — a
+dump is wanted; the bus's streams are copied live; mailu's queue is valuable though transient; Plex's
+previews may be large to copy nightly; the agent's home and the operator's keys are valuable, and are
+backed up only on machines that hold `node-backup` (today, neither workstation does).
+
+## Consequences
+
+- **The first run of the self-check records what every machine holds**, and the first night backs up
+  what was not backed up before: on the home server, every application's own data and Plex's metadata
+  and previews; on the control node, the bus's streams and the certificate authority. The first night
+  there costs that data's full size, later nights what changed.
+- **The media library is watched, not copied.** Its array is read every hour; a degraded pool is
+  urgent. A pool that loses a vdev still loses the library — the operator's accepted risk, now said
+  rather than silent. SMART pre-failure warnings are not read: the pool's own device error counters are,
+  and reading SMART is the work of a storage seat when one is decided.
+- **A provider that grants and a container that writes a directory must declare their data** before
+  `module check` passes; a module outside the catalogue meets this in its own repository, and a module
+  already on the shelf is read as it is.
+- **Retired data takes space until a person deletes it**, as ADR 0230's retired consumers do.
+- **An empty replacement of a store's consumers is a warning**, not urgent: the stores' data is
+  valuable by the operator's ranking. A consumer that says `kept-by` irreplaceable — the photo sites —
+  makes it urgent.
+- **Rollout order**: the controller first (it reads the new section and composes the holder's new file;
+  an older holder ignores what it is not given), then the catalogues and the photo application, then a
+  push of every machine. The controller's grant gains `node-backup.backed-up`, and the installer's first
+  user list with it.
+
+## How it is checked
+
+| What | Checked by |
+|---|---|
+| the section parses and refuses what it must: unknown class, a machine path, an undeclared directory or access, irreplaceable with no protection, a cache backed up, a dump into nothing | mesh-controller `internal/catalogue/data_test.go` |
+| a granting provider says its consumers' class; nobody says it on the offer; no backup line by hand; a written directory is declared; data kept with a provider as irreplaceable is protected there | `data_test.go`, and `module check` over the whole catalogue (`TestModuleCheckPassesTheCatalogue`, `TestEveryCatalogueModuleDeclaresItsData`) |
+| stickiness follows the class; an older definition still follows its grant | `data_test.go` (`TestKeepingConsumerDataFollowsTheClass`) |
+| the holder's lines are derived — dumps, copies, redundancy, a home directory, an operator's path — and only irreplaceable data needs a holder | `data_test.go` (`TestTheBackupHoldersLinesAreDerivedFromTheData`) |
+| unassigning keeps irreplaceable and valuable data retired, says so, and lists it; an empty replacement of it elsewhere is said — through the real stores and a real bus | `cmd/mesh-controller/data_test.go` (`TestNatsUnassigningIrreplaceableDataRetiresItAndAnEmptyReplacementIsUrgent`), `internal/inventory/data_test.go` |
+| **issue 273 replayed**: five consumers' real databases at one provider, empty ones at another — each an empty replacement, urgent where the consumer keeps irreplaceable data there; a deliberate move is silent | `data_test.go` (`TestAnEmptyReplacementOfAConsumersDataIsSaid`, `TestADeliberateMoveIsNoEmptyReplacement`) |
+| shrink, missing, quiet, backup age, array health and missing protection, by class | `data_test.go` |
+| the holder reads the data file, measures, reads ZFS and md, and deletes only after a last restore point, never a declared path | mesh-catalog `modules/restic/cmd/restic-backups/data_test.go` |
+| the node-engine keeps a directory with anything in it and says so without "by hand" | mesh-host `internal/apply/apply_test.go` |
+| the controller's grant names `node-backup.backed-up`, and the installer's carries it | mesh-controller `internal/broker` (`TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose`, the golden) |
+| live, after rollout | `data` lists every declared item with a measurement and a recent backup; `doctor` passes D13; `conditions` holds no `array-degraded` while the pool is healthy |
+
+## References
+
+- [Issue 273](../04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md) — the incident.
+- [ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md) — extended: stickiness now
+  follows the class, and `keeps-consumer-data` moved into the data section.
+- [ADR 0230](0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md) — the
+  meaning of retired, applied to a module's own data.
+- [ADR 0214](0214-backups-guard-against-mistakes-and-stay-on-the-machine.md) — its backup lines are now
+  derived; its exclusion of the media library is replaced by redundancy watched.
+- [ADR 0051](0051-shared-data-is-the-operators.md) — an operator's path is the operator's: never retired
+  or deleted.
+- [To-be 32](../03-DESIGN/01-to-be/32-what-a-module-declares.md), [to-be 43](../03-DESIGN/01-to-be/43-backups-against-mistakes.md),
+  [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — amended alongside.
+- mesh-controller `internal/catalogue/data.go`, `internal/inventory/data.go` (migration 0072),
+  `cmd/mesh-controller/data.go`; mesh-catalog `modules/restic`; mesh-host `internal/apply/apply.go`.

02-DECISIONS/README.md

@@ -331,6 +331,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0225** — [A consumer's identity is bounded by the provision it requires, judged before merge, and never refuses its provider](0225-a-consumers-identity-is-bounded-by-the-provision-it-requires.md)
 - **0228** — [A value given by hand lives only until its module's first good start](0228-a-value-given-by-hand-lives-only-until-its-modules-first-good-start.md)
 - **0232** — [A binding to a consumer's data moves only by a person](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md)
+- **0233** — [A module declares the data it holds, and the mesh protects and watches it from that declaration](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)
 
 ### How it is built