novox/hq
6
0
Fork 0
Code Issues Pull Requests 5 Releases Wiki Activity

Merge pull request 'ADR 0233: a module declares the data it holds, and the mesh protects and watches it' (#141) from feat/a-module-declares-the-data-it-holds into main

Browse Source
This commit was merged in pull request #141.
This commit is contained in:
mesh-admin
2026-10-06 15:11:21 +00:00
parent 3f62e2bcc1 082693fb25
commit 754e0fb8e0
8 changed files with 375 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
02-DECISIONS/0214-backups-guard-against-mistakes-and-stay-on-the-machine.md
+7
View File
@@ -46,6 +46,13 @@ filesystem where there is one. Its key is a mesh secret.
**A night that fails reaches the operator,** and a machine with data and no good backup in 48 hours
shows in the mesh's status.
> **The mechanism changed — 2026-10-06, by [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md).** The decision stands: backups guard
> against mistakes and stay on the machine. What moved is the declaration: a module declares its data in
> its `data` section, with a class and how each item is protected, and the holder's lines are derived
> from it; a line written by hand is refused. The media library, left out here, is now declared
> irreplaceable and protected by the redundancy of its array, which the holder watches, because there
> is no room to copy it.
## Consequences
- Adding a store provider means declaring its dump; the catalogue check can refuse a store provider
02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md
+7
View File
@@ -92,6 +92,13 @@ checks every machine every run. It raises:
- any consumer about to be sent another provider than the one on record with no pin naming it, as
**urgent**. The resolver makes this impossible; the probe exists for the day it is not.
> **The mechanism changed — 2026-10-06, by [ADR 0233](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md).** The decision stands: a binding to
> a consumer's data moves only by a person. What moved is how a provision is known to keep data: a
> provider now says it in its `data` section, as a class for its consumers — irreplaceable, valuable or
> rebuildable keep data, cache and none do not — and `module check` refuses a provider that grants
> without saying it. `keeps-consumer-data` on the offer and the inference from `grants` are read only
> from a definition already on the shelf.
## Consequences
- **The first push after rollout records what each machine is bound to then.** A consumer moved
02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md
+272
View File
@@ -0,0 +1,272 @@
---
topic: what runs on it
status: accepted
date: 2026-10-06
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md
---
# 233. A module declares the data it holds, and the mesh protects and watches it from that declaration
## Context
**This is the operator's decision, taken on 2026-10-06**, the day after five applications ran for
twenty hours on empty databases ([issue 273](../04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md)).
In the operator's words: modules must indicate the "data storage which should be monitored/protected",
"in the module's manifest", "in a generic, configurable way". The ranking of what is precious is also the
operator's, given the same day and recorded as given:
> "data loss would be painful but not the end of the world. Only my plex storage is sacred and the photo
> sites as well (I'm not sure everyone has a good backup themselves but it's their own responsibility)."
> "we can't backup the plex storage, we don't have any storage for it, it's on a RAID5 which should be
> safe as far as we can afford it. The plex metadata storage is not important" — and the metadata
> "should be part of the standard backup plan of course".
What the mesh knew about data before this record, measured on the code and the catalogue that day:
- **Where a consumer's data is** was inferred, not said. [ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md)
made a binding sticky when the provider *grants* a credential; no manifest in the catalogue stated
`keeps-consumer-data`. A provider whose grants hold nothing — a cache — was sticky all the same.
- **Backups were a second list.** [ADR 0214](0214-backups-guard-against-mistakes-and-stay-on-the-machine.md)
had a module write `run` and `path` lines to the `node-backup` seat by hand; thirteen modules did,
across two catalogues. The rule that "the catalogue check refuses a store provider that declares none"
was never in `module check`: one controller test applied it to the catalogue checked out beside it,
against a list of six provisions kept by hand.
- **Unassigning a module kept its data only by accident of the host's rule.** The node-engine removes an
orphaned directory only when it is empty; a directory with anything in it is **kept and forgotten** —
its record dropped from the machine's store, the report saying "kept … remove it by hand once you know
what it is". Nothing anywhere then knew the data was there, and the one instruction given was the one
that destroys it.
- **Nothing was measured.** No size, no last write, no check that a backup covered what mattered, and
nothing that could tell an empty copy of a database from the full one beside it — exactly the shape
of issue 273.
- **The media library was out of every design** (ADR 0214 named it as not backed up), and the photo
sites' storage lives in two providers, the object store and the document store, as consumer data
whose preciousness only the photo application knows.
## Considered Options
1. **Keep ADR 0232's inference and add a list of protected paths in the backup module.** Rejected: a
second list is the failure ADR 0214 rejected ("whatever is not listed is unprotected, silently"), and
an inference cannot say that a grant holds a cache.
2. **Keep `keeps-consumer-data` on the offer, required, and add a separate `backups` field.** Rejected:
two places for one fact, and a yes/no cannot carry the operator's ranking — a cache's consumers and a
photo library's both "keep data".
3. **Every declared item backed up, irreplaceable or not; the media library copied too.** Rejected by
the operator: there is no storage for a copy of the media library. A class that forces a backup would
make the most precious data undeclarable.
4. **A class per item decides everything, including the protection.** Rejected for the same reason:
how precious data is and how it is protected are two facts. The media library is the most precious
thing on the mesh and the one thing that cannot be copied.
5. **A per-module check in code ("postgres is a store").** Rejected: never per-module code; a new
module would be unprotected until someone remembered it.
6. **Array health watched by a new `node-storage` seat.** Deferred, not adopted: the only thing needed
today is to *read* an array's health under declared data, and the backup holder already reads that
data on every machine that keeps any. A seat for storage is right the day the mesh *acts* on disks
(scrubs, replacing a member); that is not decided here.
7. **One `data` section, a class for how precious, a protection for how it is kept, and everything
derived from them.** Chosen.
## Decision
**1. One section, `data`, says every kind of data a module keeps.**
- `own`: a list of items, each `id`, `path`, `class`, and how it is protected. `path` names one of the
module's directories, `${dir:<id>}`, or an operator's path it was given, `${access:<id>}`, or a path
beneath either — never a machine path (ADR 0112).
- `consumers`: per provision the module grants, the `class` of what it keeps for each consumer and
`in` — the own item that holds it, or a provision the module itself requires (an identity provider
keeps its clients in its database). **Any provider that `grants` must say this; `module check`
refuses it otherwise.** The offer's `keeps-consumer-data` (ADR 0232) is still read from a definition
already on the shelf, and refused by `module check` in a new one: it is said once, here.
- `kept-by`: per provision the module requires, the `class` of what of its own lives with that
provider. The photo sites' application says its objects and its database are irreplaceable; the
object store and the document store are then held to that class for the machine and consumer
concerned.
- Each entry may carry `why`, a line for the reviewer.
**2. Four classes, ranked by the operator.** A fixed vocabulary; a fifth is a decision.
| Class | Is | Backed up | On unassign | Alerts |
|---|---|---|---|---|
| `irreplaceable` | what must never be lost: the media library, the photo sites' storage | a protection is **required**: a backup, or a declared redundancy | retired, never removed | **urgent** |
| `valuable` | anybody's work, painful to lose: every store, mailboxes, repositories, the vault, the agent's home | **yes, by default** — the standard nightly plan, where the machine has a backup holder | retired, never removed | warning |
| `rebuildable` | made again from elsewhere: a dump, a clone, thumbnails, Plex's metadata | **yes, by default**; may say `none` (the artifact store, downloaded models) | forgotten | none |
| `cache` | disposable | never | forgotten | none |
For consumers, `none` also exists: the provision keeps nothing of anybody's (a resolver). A binding to
a consumer's data is sticky (ADR 0232) when the class is irreplaceable, valuable or rebuildable — not
for a cache or none. **Redis is `cache`**: no module in this catalogue requires its provision, its name
says what it is for, and a consumer keeping data in a cache would be the consumer's mistake, not the
mesh's to make sticky.
**3. Protection is its own field.** `backup` is `copy` (the holder reads the path as it stands),
`none`, or `{dump, into}` — a command that writes a consistent copy into another item, which is copied
(a running store's files are not a consistent copy). Unsaid, anything but a cache is copied. Or
`redundancy: "<why that is enough>"`: the item is protected by the redundancy of the storage it is on,
and the mesh watches that storage instead. **An irreplaceable item has a backup or a redundancy;
`module check` refuses it with neither.** `within` bounds the age of its last good backup (48 hours
unsaid, ADR 0214); `active` says it is written all the time and how long quiet is a fault.
**4. The backup holder's lines are derived from the section.** The controller composes, from every
module on a machine, the `backup` lines it always did and a second kind, `data`: every item with its
class, its path, the path that covers it and its protection. **A backup line written by hand is
refused by `module check`** and, beside a data section, not placed: one list. A module keeping
anything irreplaceable depends on the machine's `node-backup` holder (ADR 0207) — the holder backs it
up or watches its array — so it cannot be assigned where nothing would. Valuable and rebuildable data
is backed up where a holder is and refuses no machine for want of one.
**5. The holder measures; the controller judges — and nothing large is ever walked.** The holder
measures every item that is not a cache by the item's own `measure`:
- `walk` (the default, for small items): every file summed and the newest change found, in one walk as
root — **at most once a day**, after the night, and **stopped after ten minutes or two million
files**, said as "measured partially": a partial size is a lower bound, shown and never compared;
- `dataset`: the ZFS dataset holding the path — its size from the filesystem's own counters — and the
newest change among the path's top-level entries; hourly, nothing walked. Items on one dataset share
its size, so a shrink of it is one condition naming them all;
- `shallow`: the newest change among the top-level entries only, and no size; hourly.
A large item never says `walk`: the media library is `dataset`, Plex's metadata and previews, the
artifact store and the models are `shallow`. A file walk over a pool of that size every hour is load on
the array that protects the library, and competes with the server reading it. The holder also reads the
redundant storage each item is on: a ZFS pool's health and its own verdict, an md array's members, a
btrfs filesystem's error counters. It answers this, with each item's newest good backup, in
`node-backup.backed-up`; it decides nothing. **The holder is the measurer**, not the node-engine: it
already reads every declared path as root, it is a module that can be replaced, and the core gains no
work. The controller's self-check asks every holder on every run.
**6. Unassigning keeps the data, and the mesh remembers it.** An irreplaceable or valuable item in a
module's own directory that its machine no longer declares is **retired** (ADR 0230's meaning): kept,
recorded with when and why, listed by `cleanup list`, said by `unassign` when it is done, and a warning
after thirty days. Assigned again, it is no longer retired. It is deleted only by `cleanup delete
<machine> <module> <item> --why`, a hand act, which the machine's backup holder executes **only after
taking a last restore point of it**, tagged as retired, so the deletion can be undone until a person
forgets that restore point; the holder refuses any path declared now. An item on an operator's path —
the media library — is **never** retired and never deleted: the mesh does not own it. The node-engine's
rule stays (a directory with anything in it is never removed); its report no longer says "remove it by
hand".
**7. What is watched, and how loud.** Each condition is **urgent for irreplaceable data and a warning
for valuable data**; rebuildable data and caches raise none. A consumer's data is as precious as the
stricter of its provider's class and its own `kept-by`; a provider's item is held to the strictest
class kept in it.
| Condition | Raised when |
|---|---|
| `data-shrank` | an item holds less than half of its largest size in seven days, and at least 16 MiB less; one condition per dataset for items measured from a dataset's counters; never from a partial size |
| `data-missing` | an item's path is gone |
| `empty-replacement` | a copy in use is less than half the size (and 16 MiB less) of a copy of the same thing kept elsewhere: an item retired on another machine, or a consumer's data at another provider — issue 273's shape |
| `data-held-twice` (warning) | a consumer has active data at two providers whose sizes cannot be compared |
| `data-quiet` | an item that says `active` has not been written within it |
| `backup-stale` | an item's newest good backup is older than its bound, or there is none — for valuable data, only where a holder measured it |
| `array-degraded` | the redundant storage under watched data is not healthy, or cannot be read; one per array |
| `protection-missing` | an item says `redundancy` and is on storage the holder cannot read as redundant |
| `data-unmeasured` (warning) | a machine's holder did not answer |
| `cleanup-waiting` (warning) | an item retired more than thirty days |
**8. A verb reads it.** `data` lists every item on every machine with its class, protection, path,
size, newest write, newest good backup and its bound, the array under it, and what is retired.
**9. Every catalogue module that holds data declares it** (the table below), and the catalogue check
refuses a container that writes one of its directories, mounted whole, without a data item covering
it — a directory the mesh does not know about is data the mesh cannot protect.
### The classes, per module
The operator's ranking applied to every module that holds data, **for the operator to correct**: only
the media library and the photo sites' storage are irreplaceable; everything people wrote is valuable.
| Module | Own data | Its consumers' | Kept with a provider |
|---|---|---|---|
| plex | the media library, eight operator paths: **irreplaceable**, by **redundancy**, measured from its dataset; metadata: rebuildable, backed up, measured shallow; **previews: rebuildable, not backed up** (Plex makes them again; too large to copy nightly); working data: rebuildable; transcode: cache | | |
| photos | | | object store and document store: **irreplaceable** |
| postgres, mssql, mongodb | the store: valuable, by dump; the dumps: rebuildable | valuable | |
| minio, influxdb | data (and influx's config): valuable | valuable | |
| mesh-vault | state, ledger, root: valuable | valuable | |
| mailu | mail, signing keys, data, calendars, webmail, the queue: valuable; filter, certificates, autoconfiguration, fetch state: rebuildable; virus signatures, its redis: cache | valuable | |
| gitea | data: valuable | the package registry: rebuildable | |
| keycloak | | clients: rebuildable, in its database | |
| umami | | page views: valuable, in its database | |
| mosquitto | retained messages: rebuildable | rebuildable | |
| redis | its snapshot: cache | **cache** | |
| nextcloud, baserow, grafana, matrix, nodered, unifi, step-ca, audit-logger | their data: valuable | | |
| home-assistant | its configuration and history: valuable, written all the time (a day) | | |
| nats | the bus's streams and buckets: valuable, written all the time (a day) | | |
| n8n | data: valuable; scratch: cache | | |
| supabase | database: valuable, **by `pg_dumpall`** inside its container; storage, functions: valuable; the dump and its seeded configuration: rebuildable | | |
| claude-code | the operator's agent's home: valuable | | |
| ssh-client | the operator's keys: valuable | | |
| radarr, sonarr, lidarr, bazarr | the database: valuable, by sqlite's own backup; settings: valuable; covers and the night's copy: rebuildable; lidarr's start-up scripts: cache | | |
| bookshelf, jackett, kometa, nzbget, ombi, qbittorrent, tautulli | configuration: valuable | | |
| distribution, ollama | the artifact store, the models: rebuildable, **not backed up** (too large; made again), measured shallow | | |
| only-office | its working data, database, fonts: rebuildable; logs, queue, cache: cache | | |
| records, route-proxy | the record's clone, issued certificates: rebuildable; the authority's roots: cache | | |
| restic | the repository: rebuildable, not backed up — it is the copy, never the only copy of anything; its state: cache | | |
| build-agent, icecast, lab, model-usage, searxng | cache | | |
**Uncertain, for the operator:** the bus's streams are copied as live files — the bus's image carries
no tool to snapshot a stream safely, so a consistent copy needs the bus's own backup command run from
somewhere that has it, which is not built; mailu's queue is valuable though transient; the agent's home
and the operator's keys are valuable, and are backed up only on machines that hold `node-backup`
(today, neither workstation does).
## Consequences
- **The first run of the self-check records what every machine holds**, and the first night backs up
what was not backed up before: on the home server, every application's own data and Plex's metadata;
on the control node, the bus's streams and the certificate authority. Measured read-only on
2026-10-06, the home server's new nightly data is about 11 GB of application data (the chat server's
database 5.6 GB, the database platform's 2.1 GB before its dump, the network controller 1.1 GB, the
viewing history 0.9 GB, dashboards 0.5 GB, the rest under 0.2 GB each) and Plex's metadata, at most
the 136 GB its disk holds — under 150 GB against 33.6 TB free where the repository is: under half a
percent. The previews, on the media pool, are left out. Later nights cost what changed.
- **The media library is watched, not copied.** Its array is read every hour; a degraded pool is
urgent. A pool that loses a vdev still loses the library — the operator's accepted risk, now said
rather than silent. SMART pre-failure warnings are not read: the pool's own device error counters are,
and reading SMART is the work of a storage seat when one is decided.
- **A provider that grants and a container that writes a directory must declare their data** before
`module check` passes; a module outside the catalogue meets this in its own repository, and a module
already on the shelf is read as it is.
- **Retired data takes space until a person deletes it**, as ADR 0230's retired consumers do.
- **An empty replacement of a store's consumers is a warning**, not urgent: the stores' data is
valuable by the operator's ranking. A consumer that says `kept-by` irreplaceable — the photo sites —
makes it urgent.
- **Rollout order**: the controller first (it reads the new section and composes the holder's new file;
an older holder ignores what it is not given), then the catalogues and the photo application, then a
push of every machine. The controller's grant gains `node-backup.backed-up`, and the installer's first
user list with it.
## How it is checked
| What | Checked by |
|---|---|
| the section parses and refuses what it must: unknown class, a machine path, an undeclared directory or access, irreplaceable with no protection, a cache backed up, a dump into nothing | mesh-controller `internal/catalogue/data_test.go` |
| a granting provider says its consumers' class; nobody says it on the offer; no backup line by hand; a written directory is declared; data kept with a provider as irreplaceable is protected there | `data_test.go`, and `module check` over the whole catalogue (`TestModuleCheckPassesTheCatalogue`, `TestEveryCatalogueModuleDeclaresItsData`) |
| stickiness follows the class; an older definition still follows its grant | `data_test.go` (`TestKeepingConsumerDataFollowsTheClass`) |
| the holder's lines are derived — dumps, copies, redundancy, a home directory, an operator's path — and only irreplaceable data needs a holder | `data_test.go` (`TestTheBackupHoldersLinesAreDerivedFromTheData`) |
| unassigning keeps irreplaceable and valuable data retired, says so, and lists it; an empty replacement of it elsewhere is said — through the real stores and a real bus | `cmd/mesh-controller/data_test.go` (`TestNatsUnassigningIrreplaceableDataRetiresItAndAnEmptyReplacementIsUrgent`), `internal/inventory/data_test.go` |
| **issue 273 replayed**: five consumers' real databases at one provider, empty ones at another — each an empty replacement, urgent where the consumer keeps irreplaceable data there; a deliberate move is silent | `data_test.go` (`TestAnEmptyReplacementOfAConsumersDataIsSaid`, `TestADeliberateMoveIsNoEmptyReplacement`) |
| shrink, missing, quiet, backup age, array health and missing protection, by class | `data_test.go` |
| the holder reads the data file, measures, reads ZFS and md, and deletes only after a last restore point, never a declared path | mesh-catalog `modules/restic/cmd/restic-backups/data_test.go` |
| the node-engine keeps a directory with anything in it and says so without "by hand" | mesh-host `internal/apply/apply_test.go` |
| the controller's grant names `node-backup.backed-up`, and the installer's carries it | mesh-controller `internal/broker` (`TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose`, the golden) |
| live, after rollout | `data` lists every declared item with a measurement and a recent backup; `doctor` passes D13; `conditions` holds no `array-degraded` while the pool is healthy |
## References
- [Issue 273](../04-ISSUES/273-a-rule-for-the-resolver-moved-a-machines-databases/00-report.md) — the incident.
- [ADR 0232](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md) — extended: stickiness now
follows the class, and `keeps-consumer-data` moved into the data section.
- [ADR 0230](0230-a-consumer-the-mesh-stops-asking-for-is-retired-and-deleted-only-by-a-person.md) — the
meaning of retired, applied to a module's own data.
- [ADR 0214](0214-backups-guard-against-mistakes-and-stay-on-the-machine.md) — its backup lines are now
derived; its exclusion of the media library is replaced by redundancy watched.
- [ADR 0051](0051-shared-data-is-the-operators.md) — an operator's path is the operator's: never retired
or deleted.
- [To-be 32](../03-DESIGN/01-to-be/32-what-a-module-declares.md), [to-be 43](../03-DESIGN/01-to-be/43-backups-against-mistakes.md),
[to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — amended alongside.
- mesh-controller `internal/catalogue/data.go`, `internal/inventory/data.go` (migration 0072),
`cmd/mesh-controller/data.go`; mesh-catalog `modules/restic`; mesh-host `internal/apply/apply.go`.
02-DECISIONS/README.md
+1
View File
@@ -332,6 +332,7 @@ python3 00-META/checks/index.py fail if stale
- **0225** — [A consumer's identity is bounded by the provision it requires, judged before merge, and never refuses its provider](0225-a-consumers-identity-is-bounded-by-the-provision-it-requires.md)
- **0228** — [A value given by hand lives only until its module's first good start](0228-a-value-given-by-hand-lives-only-until-its-modules-first-good-start.md)
- **0232** — [A binding to a consumer's data moves only by a person](0232-a-binding-to-a-consumers-data-moves-only-by-a-person.md)
- **0233** — [A module declares the data it holds, and the mesh protects and watches it from that declaration](0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)
### How it is built
03-DESIGN/01-to-be/18-building-a-module.md
+3 -1
View File
@@ -5,8 +5,9 @@ code:
- mesh-controller cmd/mesh-builder
- mesh-controller internal/builder
- mesh-catalog modules/build-agent
updated: 2026-10-05
updated: 2026-10-06
decisions:
- 02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md
- 02-DECISIONS/0219-the-build-queue-is-controlled-through-the-controller-and-the-build-seat.md
- 02-DECISIONS/0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md
- 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
@@ -188,6 +189,7 @@ disagrees with it.
| `accesses` | operator-owned paths it may use and must not own ([ADR 0051](../../02-DECISIONS/0051-shared-data-is-the-operators.md)) |
| `certificate` | a certificate for a name it serves |
| `grants` | credentials it must create for its consumers |
| `data` | the data it holds — its own, what it keeps for its consumers, what it keeps with a provider — each with a class and how it is protected; backups, retirement and the self-check's watch are derived from it ([ADR 0233](../../02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)) |
| `filtering` | rules beyond its own ports |
| `computed` | marks a module the controller generates rather than an author writing |
| `build.artifacts` | what it produces; an artifact's `context` may be a URL or a path on the git seat (`seat: git`), composed by the mesh that builds it |
03-DESIGN/01-to-be/32-what-a-module-declares.md
+20 -1
View File
@@ -12,8 +12,9 @@ code:
- mesh-tools src/main.ts
- mesh-catalog modules/mesh-catalog
- mesh-tools node-tools/internal/runtime (a module's state, ADR 0201)
updated: 2026-10-04
updated: 2026-10-06
decisions:
- 02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
- 02-DECISIONS/0126-a-module-declares-its-own-seats.md
@@ -265,6 +266,24 @@ module's assignment** — what a module stored is data
([ADR 0030](../../02-DECISIONS/0030-data-outlives-the-mesh-that-declared-it.md)) — and one whose
declaration is gone is reported, never removed by the mesh.
**A module declares the data it holds.** *Added 2026-10-06,
[ADR 0233](../../02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md).*
State on the bus is one kind of data; the rest lives on a machine's disks or with a provider, and the
mesh protects only what it is told about. One section, `data`, says all of it:
| `data.` | says |
|---|---|
| `own` | each item the module keeps: `id`, `path` (`${dir:<id>}` or `${access:<id>}`, or beneath one), `class`, and its protection — `backup` (`copy`, `none`, or `{dump, into}`) or `redundancy` with why that is enough — with optional `measure` (`walk`, `dataset`, `shallow`), `within`, `active`, `why` |
| `consumers` | per provision it grants: the `class` of what it keeps for each consumer, and `in` — the own item, or the required provision, holding it |
| `kept-by` | per provision it requires: the `class` of what of its own the provider keeps |
The classes are `irreplaceable`, `valuable`, `rebuildable` and `cache` (and `none`, for consumers),
ranked by the operator; the protections, the bindings that do not move, the backup holder's lines, what
an unassignment retires and what the self-check raises are all derived from them, never written per
module. The backup holder's composed `backup` and `data` lines are the bus-free half: a file the
holder reads, like every contribution (ADR 0212). The rules are in the record; `module check` holds
them.
## 5. Seats
A module declares a seat with its protocol, and the mesh enforces one holder at its scope
03-DESIGN/01-to-be/43-backups-against-mistakes.md
+40 -7
View File
@@ -3,9 +3,11 @@ layer: to-be
status: in-progress
code:
- mesh-controller: internal/catalogue/seats.go (node-backup), internal/catalogue/seat_contributions.go (BackupSeat, CheckBackup, a contribution's directories)
- mesh-catalog: modules/restic (the holder), modules/postgres, modules/mssql, modules/mongodb, modules/minio, modules/influxdb, modules/mesh-vault, modules/mailu, modules/gitea, modules/nextcloud (backup contributions)
updated: 2026-10-05
- mesh-catalog: modules/restic (the holder; it measures the declared data and deletes a retired item, ADR 0233), every module's `data` section
- mesh-controller: internal/catalogue/data.go (the lines derived from a module's data, ADR 0233)
updated: 2026-10-06
decisions:
- 02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md
- 02-DECISIONS/0214-backups-guard-against-mistakes-and-stay-on-the-machine.md
- 02-DECISIONS/0053-a-step-that-runs-on-a-schedule.md
- 02-DECISIONS/0085-a-secret-is-a-provision.md
@@ -25,8 +27,13 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214).
It keeps one encrypted, deduplicating repository on the machine and runs a nightly scheduled step
(ADR 0053). The repository's key is a secret provisioned to the holder (ADR 0085), held in the
vault, so a person can open the repository without the holder running.
- **A module declares its data in its manifest,** naming no node and no absolute path (ADR 0112),
in one of two forms:
- **A module declares its data in its manifest,** naming no node and no absolute path (ADR 0112).
*Amended 2026-10-06 ([ADR 0233](../../02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md)):*
a module no longer writes backup lines; it declares its data in its `data` section — each item's
class and how it is protected — and the lines below are **derived** from that, one list. Anything
but a cache is in the nightly backup by default; an item may say `none`, or, if it is irreplaceable
and cannot be copied, `redundancy` with why, and the holder then watches the array it is on. A
backup line written by hand is refused by `module check`. The two forms the derivation produces:
- **a dump** — for a store provider: the command that writes a consistent, logical copy of each
database it serves, run by the provider in its own container, its output handed to the holder.
The provider covers every consumer it provisions, so a module that only *uses* a database
@@ -38,6 +45,25 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214).
module assigned is covered the next night; a module unassigned stops being backed up, and its
restore points age out by the rotation, never at once.
## The holder measures what is declared
*Added 2026-10-06, [ADR 0233](../../02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md).*
The mesh composes a second file for the holder: every data item every module on the machine declares,
with its class, its path, the path that covers it and its protection. Once an hour and after every
night the holder measures each item that is not a cache, by the item's `measure`: a bounded walk at
most daily for a small item (stopped after ten minutes or two million files and said as partial), a ZFS
dataset's own counters hourly, or the top-level entries only — never an hourly walk of anything large —
and reads the redundant storage it is on (a ZFS pool's health, an md array's
members, a btrfs filesystem's error counters). `backed-up` says each item with that measurement and its
newest good backup. The holder judges nothing: the controller's self-check (to-be 45, D13) keeps the
readings and raises what they show.
**It deletes a retired item, when a person has decided.** An item the mesh retired — its module gone
from the machine — is removed by the holder only when the controller's `cleanup delete` asks, and only
after it has taken a last restore point of it, tagged as retired: a deletion can be undone until that
restore point is forgotten. It refuses any path declared on the machine now, its own repository, and
anything too near the root.
## A night
Each declared dump runs and writes a full logical copy; each declared path is read as it stands. All
@@ -74,10 +100,17 @@ a manifest. Not off-site: a lost machine loses its backups with its data, by the
## Not in this design
- Copies off the machine, and encryption to anyone but the mesh's own vault.
- The media library and anything else a module declares no data for.
- Recreatable things: container images, the artifact registry, caches.
- Copying the media library: there is no room for it. It is declared irreplaceable and protected by the
redundancy of the array it is on, which the holder watches (ADR 0233); a lost array loses it, by the
operator's accepted risk.
- Anything a module declares no data for.
- Recreatable things a module says `backup: none` of — container images, the artifact registry,
downloaded models — and caches.
## How it is checked
The catalogue check refuses a module that provides a store seat and declares no dump. The weekly
The catalogue check (`module check`, and the catalogue-wide test beside it) refuses a provider that
grants without saying what it keeps for its consumers, an irreplaceable item with neither a backup nor a
redundancy, and any backup line written by hand (ADR 0233). The store rule this design stated before
was held only by a controller test over the catalogue checked out beside it, never by `module check`. The weekly
restore test above, and the 48-hour status line, are the running checks.
03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
+25
View File
@@ -5,6 +5,7 @@ code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
updated: 2026-10-06
decisions:
- 02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md
- 02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md
- 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
- 02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md
- 02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md
@@ -73,6 +74,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
| a provider's standing | the provider | the provider's events | the controller keeps the newest word as a condition |
| a consumer's retirement: active, retired (when, why), deleted | the provider | its own backend's mark; said on `provisioner.retirement` | the controller asks the provider's tools; a person approves, rejects and deletes through the controller's verbs |
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
A bus subject two components may publish on is refused when the controller composes grants, unless
@@ -156,6 +158,17 @@ outlive the history.
`cleanup-waiting` (warning), from the probe D11, while anything retired is older than thirty days. The
first two clear on the provider's `retired`, `settled` or `approved` word, and all three when the
provider is no longer assigned there.
- **Data** ([ADR 0233](../../02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md))
raises, from D13, each **urgent for irreplaceable data and a warning for valuable data** — the
operator's ranking — and none for rebuildable data or caches: `data-shrank` (less than half its
largest size in seven days, and 16 MiB less), `data-missing`, `empty-replacement` (a copy in use
less than half a copy of the same thing kept elsewhere — issue 273's shape — for an item or for a
consumer's data at a provider), `data-quiet` (an item that says `active`, unwritten past it),
`backup-stale`, `array-degraded` (one per array under watched data) and `protection-missing` (an item
said to be on redundancy, found on plain storage); and as warnings `data-held-twice`,
`data-unmeasured` and `cleanup-waiting` for an item retired over thirty days. Keyed by machine,
module and item, or by the array. A shrink a person meant is silenced with why; everything else
clears on observation.
## 3. The signals table (rule 5)
@@ -208,6 +221,8 @@ itself — an unanswered probe is never a pass.
| D9 | `status` answers in full within ten seconds | 265 |
| D10 | every machine runs the node-engine and node tools builds its plan says, or is inside a plan's window | version split |
| D11 | no provider holds a consumer retired more than thirty days: each provider assigned is asked `provisioner_retirement` on its machine (ADR 0230); one that does not serve it yet is named, not failed | ADR 0230 |
| D12 | every consumer of a provision that keeps its data is bound where it was last sent, or moves by a pin (`binding-kept`, `binding-moving`, `binding-moved`) | ADR 0232 |
| D13 | every item of data a machine declares is measured, is there, holds what it held, is written where it says it is, is backed up within its bound or sits on healthy redundant storage, and is no empty replacement of a copy kept elsewhere; an irreplaceable or valuable item a machine no longer declares is retired, not forgotten. Each machine's `node-backup` holder is asked `backed-up`; every provider of kept consumer data `provisioner_retirement` (with each held consumer's size) — see §2 for the kinds | ADR 0233, issue 273 |
| DW | the watchdogs of §3 ran within three of their intervals: the watchers are watched, and raise S10 when the self-check stops | rule 6 |
| H-* | the health probes of §8, run for every core component on every machine | rule 8 |
@@ -514,6 +529,16 @@ controller's tests against a real store, and H3 and H4 against a real bus: induc
the probe's own next run, and handed to the operator after the budget. **Not yet:** the induced failures
on a lab mesh (mesh-lab), and so the phase's *done when*; the live week.
**Amended 2026-10-06** ([ADR 0233](../../02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md),
the operator's decision, before Phase 4): a module declares the data it holds; D13 and its conditions;
the `data` verb; `cleanup` extended to a module's own retired data, deleted by the machine's backup
holder after a last restore point; the controller's grant names `node-backup.backed-up`. Built on
branches in mesh-controller (migration 0072), mesh-host (the installer's user list; the kept-directory
report), mesh-catalog (the holder's measuring and deletion, every module's data, the providers' held
sizes), the media catalogue and the photo application, not yet merged. **Not yet:** the TypeScript
providers saying their consumers' sizes (until then a consumer at two of them is `data-held-twice`, a
warning, not compared), and SMART read under an array.
### Phase 4 — Core upgrades that roll back
| Repository | Delivers |