ADR 0235: back the bus up by the server's own snapshot of each stream

Resolves ADR 0233's flagged item (the bus's streams copied as live files) as
a progressive insight pointing here. The bus module's own account is granted
the snapshot API and nothing else; restore builds a new store beside the live
one for a person to swap in. To-be 43 gains the bus's section and its restore
steps; design 25 and to-be 45 point to it.
This commit is contained in:
jochen
2026-10-06 18:23:40 +02:00
parent 40e7911da2
commit e65daf6f4d
6 changed files with 224 additions and 2 deletions
+9 -1
View File
@@ -8,8 +8,9 @@ code:
- mesh-catalog modules/nats (to be written)
- mesh-sdk src (the protocol's NATS binding, step 3)
- mesh-tools node-tools/internal/bus (a module's state, ADR 0201)
updated: 2026-10-04
updated: 2026-10-06
decisions:
- 02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md
- 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md
- 02-DECISIONS/0167-a-membership-carries-what-its-module-receives-and-who-the-mesh-is.md
- 02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md
@@ -175,6 +176,13 @@ call is a timeout the caller already handles.
Streams and consumers are objects the controller creates at genesis and asserts on start; a module
declares nothing about them. The controller is the only writer of stream definitions.
*Added 2026-10-06, [ADR 0235](../../02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md):*
one other user reads the streams whole, and writes none of them. The module that is the bus — the one
holding `mesh-broker` — has an account of its own whose only grant is the snapshot API: the stream
names, a stream's information, the snapshot request and its acknowledgements, and its own inbox. The
night's backup takes every stream through it ([to-be 43](43-backups-against-mistakes.md)). The
controller stays the only writer of stream definitions; the writers table holds that at composition.
### The store window, and what moving it into the server changes
The guarantee ([ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md)) is
@@ -5,8 +5,11 @@ code:
- mesh-controller: internal/catalogue/seats.go (node-backup), internal/catalogue/seat_contributions.go (BackupSeat, CheckBackup, a contribution's directories)
- mesh-catalog: modules/restic (the holder; it measures the declared data and deletes a retired item, ADR 0233), every module's `data` section
- mesh-controller: internal/catalogue/data.go (the lines derived from a module's data, ADR 0233)
- mesh-catalog: modules/nats/snapshot (the bus's snapshot and its restore, ADR 0235)
- mesh-controller: internal/broker (the bus module's snapshot-only account, ADR 0235)
updated: 2026-10-06
decisions:
- 02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md
- 02-DECISIONS/0233-a-module-declares-the-data-it-holds-and-the-mesh-protects-and-watches-it-from-that.md
- 02-DECISIONS/0214-backups-guard-against-mistakes-and-stay-on-the-machine.md
- 02-DECISIONS/0053-a-step-that-runs-on-a-schedule.md
@@ -72,6 +75,37 @@ has not seen, so the object store's first night costs its full size and later ni
changed. Then the rotation prunes to 14 daily, 8 weekly and 6 monthly snapshots. A dump that fails
fails the night for that module only; the others are still taken.
## The bus: the server's own snapshot, never its files
*Added 2026-10-06, [ADR 0235](../../02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md).*
The bus keeps everything it holds in streams — its key-value buckets are streams too — and its store
is written all the time, so a copy of the store's files may not restore. Its streams are protected by a
dump like a database's: in the bus's own container, a program built into the bus's image asks the
server for each stream through the server's snapshot API, one at a time, acknowledging each chunk so
the server's flow control keeps moving; the server goes on taking writes, and no stream is paused or
reconfigured. It runs under the bus module's own account, which may take snapshots and do nothing
else. It writes one archive into the module's snapshots directory, which the night copies: a manifest
— when, how long, and per stream its messages, bytes, sequences, consumers and the checksum of each
file — then each stream's configuration and the server's archive of it. The live store is not copied.
Every message up to the last sequence the manifest gives for a stream is in the archive, exactly; a
stream written during its snapshot may carry part of what arrived while its blocks were read, after
that sequence. A failure fails the bus's night, keeps the previous archive, and is said as
`backup-stale` like any other.
**Restoring the bus** is done beside it, never over it — a stream is restored only where it does not
exist, and on the live bus every one exists:
1. `restore` the bus module's snapshots from a named night, beside the live ones;
2. check the archive against its manifest (`mesh-nats-snapshot verify`);
3. build a new store from it with the bus's own image: the program starts the bus's server on loopback
with the bus's one account, restores every stream, holds each to the manifest and stops it;
4. swap the new store in for the live one with the bus stopped — a person's act and a planned bus step
(to-be 45), in one command line, because the machine's node-engine starts a stopped container again
at its next reconcile. The store kept aside is removed by a person once the bus is seen whole.
The module's README has the commands.
## The verbs
On the seat, for a person or an agent:
@@ -92,6 +126,9 @@ a manifest. Not off-site: a lost machine loses its backups with its data, by the
## Proving it
- A night that fails, or does not run, reaches the operator's output channel, naming the module.
- The bus's snapshot is proven by restoring it, in tests against the bus's own release: a server
written to throughout its snapshot, restored into a fresh server and into a new store served by a
third, compared message by message (ADR 0235).
- Weekly, the holder checks the repository's integrity and restores the newest dump of one database,
in rotation, into a throwaway instance with no network, comparing table row counts with the live
database.
@@ -408,7 +408,11 @@ controller: the streams are snapshotted; a `bus-maintenance` condition is open f
the bus is replaced; afterwards D6, D7 and the round trip must pass, or the step is reported failed and
the snapshot is the way back. A step whose new version cannot be reverted (the bus's 2.10 → 2.11 is one)
says so before it starts and runs only on a person's explicit word, recorded as a hand act. Whether the
bus becomes a cluster that can be upgraded live is left to its own effort.
bus becomes a cluster that can be upgraded live is left to its own effort. *2026-10-06:* the snapshot
and the way back from it exist — the bus image's own snapshot program, the same one the night's backup
runs, and a restore that builds a new store beside the live one for a person to swap in
([ADR 0235](../../02-DECISIONS/0235-the-bus-is-backed-up-by-its-own-snapshot-of-each-stream.md),
[to-be 43](43-backups-against-mistakes.md)).
## 9. Before merge: facts and replays (rule 9)