Back up the bus by the server's own snapshot of each stream, not its live files (hq ADR 0235) #93

2026-10-06T16:24:06Z

mesh-admin commented

2026-10-06 16:24:06 +00:00

Resolves ADR 0233's flagged item: the restic holder copied JetStream's store while NATS wrote it.

modules/nats/snapshot/ (own Go module, nats.go + s2 only): mesh-nats-snapshot snapshot|verify|restore. Snapshot: every stream one at a time via $JS.API.STREAM.SNAPSHOT.<s>, each chunk acked as it arrives (server window 8 MiB), 10 min/stream and 30 min total bounds, pause between streams, never pauses/reconfigures a stream; memory streams listed as skipped. Writes one tar to stdout: manifest.json (time, duration, server version, per stream subjects/messages/bytes/sequences/consumers, sizes + SHA-256) then streams/<name>/backup.json + stream.tar.s2 (nats CLI layout). Any failure fails the whole.
restore --into DIR: starts the image's own nats-server on loopback with account MESH, restores every stream, holds each to the manifest, stops; leaves jetstream/MESH/streams/... to swap in. restore --server: into a server holding none of the streams. Nothing is restored over a live stream.
Dockerfile: a GO_BASE stage builds the program into the bus image (/usr/local/bin/mesh-nats-snapshot). The image changes, so the bus restarts once — a planned bus step.
module.json: own secret broker (${dir:mesh-state}/broker); jetstream item backup: {dump, into: snapshots} — docker exec -i mesh-broker-nats mesh-nats-snapshot snapshot < broker > snapshots/bus.tar.partial && mv … (partial removed on failure, last good kept); new snapshots item (rebuildable). The live store is no longer copied.
README.md: the backup and the restore procedure.

Found by the restore test: a stream written during its snapshot carries part of what arrived while its blocks were read, past the manifest's last sequence. The promise is therefore: every message up to the manifest's last sequence, exactly; a restore ending before it is refused.

Tests: MESH_TEST_DOCKER=1 go test -race ./... in snapshot/ (throwaway nats:2.11 filled like the bus and written every 2ms during the snapshot; restored into a fresh server and into a new store served by a third, compared message by message; damaged archive refused; snapshot user refused every write); MESH_TEST_NATS_IMAGE=<built image> runs the manifest's dump exactly. At 105 MiB / 90k messages: snapshot 0.5 s, restore of the largest stream 0.3 s, 0 writes refused. cmd/nats-tools and restic tests green; controller catalogue checks pass against this branch.

Requires mesh-controller feat/bus-snapshot merged and deployed first. Rollout: after merge, module issue nats --node <bus machine> before that machine's next push (a push refuses an unissued declared account), then push as a planned bus step.

Resolves ADR 0233's flagged item: the restic holder copied JetStream's store while NATS wrote it. - `modules/nats/snapshot/` (own Go module, nats.go + s2 only): `mesh-nats-snapshot snapshot|verify|restore`. Snapshot: every stream one at a time via `$JS.API.STREAM.SNAPSHOT.<s>`, each chunk acked as it arrives (server window 8 MiB), 10 min/stream and 30 min total bounds, pause between streams, never pauses/reconfigures a stream; memory streams listed as skipped. Writes one tar to stdout: `manifest.json` (time, duration, server version, per stream subjects/messages/bytes/sequences/consumers, sizes + SHA-256) then `streams/<name>/backup.json` + `stream.tar.s2` (nats CLI layout). Any failure fails the whole. - `restore --into DIR`: starts the image's own nats-server on loopback with account MESH, restores every stream, holds each to the manifest, stops; leaves `jetstream/MESH/streams/...` to swap in. `restore --server`: into a server holding none of the streams. Nothing is restored over a live stream. - `Dockerfile`: a GO_BASE stage builds the program into the bus image (`/usr/local/bin/mesh-nats-snapshot`). **The image changes, so the bus restarts once — a planned bus step.** - `module.json`: own secret `broker` (`${dir:mesh-state}/broker`); `jetstream` item `backup: {dump, into: snapshots}` — `docker exec -i mesh-broker-nats mesh-nats-snapshot snapshot < broker > snapshots/bus.tar.partial && mv …` (partial removed on failure, last good kept); new `snapshots` item (rebuildable). The live store is no longer copied. - `README.md`: the backup and the restore procedure. Found by the restore test: a stream written during its snapshot carries part of what arrived while its blocks were read, past the manifest's last sequence. The promise is therefore: every message up to the manifest's last sequence, exactly; a restore ending before it is refused. Tests: `MESH_TEST_DOCKER=1 go test -race ./...` in `snapshot/` (throwaway nats:2.11 filled like the bus and written every 2ms during the snapshot; restored into a fresh server and into a new store served by a third, compared message by message; damaged archive refused; snapshot user refused every write); `MESH_TEST_NATS_IMAGE=<built image>` runs the manifest's dump exactly. At 105 MiB / 90k messages: snapshot 0.5 s, restore of the largest stream 0.3 s, 0 writes refused. `cmd/nats-tools` and `restic` tests green; controller catalogue checks pass against this branch. Requires mesh-controller `feat/bus-snapshot` merged and deployed first. Rollout: after merge, `module issue nats --node <bus machine>` before that machine's next push (a push refuses an unissued declared account), then push as a planned bus step.

mesh-admin added 1 commit 2026-10-06 16:24:06 +00:00

Back up the bus by the server's own snapshot of each stream, not its live files (hq ADR 0235) 419e82cded

The restic holder copied JetStream's store while the server wrote it; such a
copy may not restore. The nats image now carries mesh-nats-snapshot, run by
the declared dump under the module's own bus account (snapshot API only):
every stream one at a time, flow-controlled, into one tar with a manifest of
counts, sequences and checksums. Restore builds a new store beside the live
one with the bus's own server; a person swaps it in. Proven against
throwaway nats 2.11 servers being written to during the snapshot.

mesh-admin merged commit 9d407248b3 into main

2026-10-06 17:33:13 +00:00

mesh-admin deleted branch feat/bus-snapshot

2026-10-06 17:33:13 +00:00

mesh-admin commented

2026-10-06 23:08:58 +00:00

Delivery novox/mesh-catalog@9d407248b369 — superseded since 2026-10-06T23:05:28Z

Transitions

2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#93 merged; its head was not heard by this owner, so it is made from the forge's word
2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on
2026-10-06 23:05 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's plans go
2026-10-06 23:05 delivering → superseded (superseded): a newer delivery to the same trunk took over its walk

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-catalog@9d407248b369` — **superseded** since 2026-10-06T23:05:28Z **Transitions** - 2026-10-06 23:05 (new) → proposed (announced): novox/mesh-catalog#93 merged; its head was not heard by this owner, so it is made from the forge's word - 2026-10-06 23:05 proposed → held (merged-unchecked): merged without a passing check — a verdict known with it is taken first: only a person decides that it goes on - 2026-10-06 23:05 held → delivering (go): its walk started on a word that was not this owner's: the controller's own path, which waits for nobody, or a person's `plans go` - 2026-10-06 23:05 delivering → superseded (superseded): a newer delivery to the same trunk took over its walk The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-catalog#93