jochen
950e37a46e
A watch that panicked stopped for good, and one that could not read said so only to stderr, so a dead watch read as no slow consumers. A faulted look is now recovered and the loop goes on; every look ends with a bus.watch heartbeat saying whether the watch is healthy, which the controller raises when it stops or says it cannot see. The record is read newest first back to the last look, and closures that could not be published are forgotten after the hour.
nats — the mesh's bus
The bus server (novox/hq design 25), the configuration the module owns (nats.conf: ports, TLS,
JetStream's store) and the user list the controller composes (accounts.conf), reloaded in place by
the entrypoint. Its tools (cmd/nats-tools) are served by the node's runtime and only read.
Its data, and how it is backed up
Everything the bus keeps is in JetStream: every stream, and every key-value bucket (a stream named
KV_<bucket>) — conditions and their history, calls, hand-acts, the controller's lease, assignments,
events, every module's state.
The store's files are never what is backed up. They are written all the time, and a copy taken
while the server writes them may not restore. The jetstream data item is protected by a dump
instead (novox/hq ADR 0235, to-be 43): each night the machine's backup holder runs
docker exec -i mesh-broker-nats mesh-nats-snapshot snapshot < <the module's broker credential> > <snapshots>/bus.tar
and backs up the snapshots directory. mesh-nats-snapshot (snapshot/, built into this image)
asks the server for each stream through JetStream's snapshot API, one at a time, acknowledging each
chunk so the server's flow control keeps moving; the server goes on taking writes and no stream is
paused or reconfigured. It writes one tar on stdout:
manifest.json— when it was taken and how long it took, the server's version, and per stream its subjects, messages, bytes, first and last sequence, consumers, and the size and SHA-256 of its files; a memory stream is listed as skipped, with why;streams/<name>/backup.json— the stream's configuration and state, as the server gave them;streams/<name>/stream.tar.s2— the server's archive of the stream, consumers included.
It runs as the module's own bus account, which the controller grants the snapshot API and nothing
else — no stream may be defined, changed, purged or written by it (mesh-controller
internal/broker, BusSnapshotGrants). A failure fails that night for the module, leaves the
previous bus.tar in place, and reaches the operator as backup-stale once the last good night is
past its bound.
What a snapshot promises. Every message up to the last sequence its manifest gives for a stream is in it, exactly. A stream written to while it is read out may hold some messages past that sequence too — the server reads its blocks a moment after it states the stream — and a restore of it ends at or after the manifest's sequence, never before.
Restoring
Nothing is restored over the live bus: a stream is only ever restored where it does not exist, and the controller asserts every stream on start. So the bus is restored into a new store beside the live one, which a person swaps in.
-
Take the night back, beside the live data.
node-backup.restorefornatson the bus's machine (optionally naming a restore point) restores the snapshots directory as<snapshots>.restored-<stamp>/, holding that night'sbus.tar. -
Check it.
docker exec -i mesh-broker-nats mesh-nats-snapshot verify < <restored>/bus.tar— every file against the manifest's checksum, every archive read to its end. -
Build the new store, with the bus's own image (the server that fills it is the same release the bus runs):
image=$(docker inspect -f '{{.Config.Image}}' mesh-broker-nats) docker run --rm -v <restored>:/in:ro -v <new-store>:/out \ --entrypoint mesh-nats-snapshot "$image" restore --into /out --from /in/bus.tarIt starts a server of its own on the container's loopback with the bus's one account (
MESH), restores every stream, holds each to the manifest, stops it, and leaves<new-store>/jetstream/MESH/streams/…— the bus's layout. -
Swap it in — a person's act, a planned bus step (novox/hq to-be 45): say it as a hand act, then in one command line, because the node-engine starts a stopped container again at its next reconcile (within five minutes):
docker stop mesh-broker-nats && \ mv <jetstream-data>/jetstream <jetstream-data>/jetstream.before-restore-<stamp> && \ mv <new-store>/jetstream <jetstream-data>/jetstream && \ docker start mesh-broker-natsThe store kept aside is removed by a person once the bus is seen whole (
nats_streams, the controller'sdoctor).
A single stream can be restored into any server that does not hold it with
mesh-nats-snapshot restore --server <url> --credential <file>; the archive is also what the nats CLI
reads (nats stream restore <dir> on an unpacked streams/<name>/).
Tests
snapshot/:MESH_TEST_DOCKER=1 go test -race ./...— a throwawaynats:2.11filled like the mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the snapshot restored into a fresh server and into a new store served by a third, each compared message by message; a damaged archive refused; and the snapshot user refused every write.MESH_TEST_NATS_IMAGE=<this image>also runs the manifest's declared dump, exactly, against the module's image and configuration.cmd/nats-tools:go test ./....