jochen
419e82cded
The restic holder copied JetStream's store while the server wrote it; such a copy may not restore. The nats image now carries mesh-nats-snapshot, run by the declared dump under the module's own bus account (snapshot API only): every stream one at a time, flow-controlled, into one tar with a manifest of counts, sequences and checksums. Restore builds a new store beside the live one with the bus's own server; a person swaps it in. Proven against throwaway nats 2.11 servers being written to during the snapshot.
88 lines
5.2 KiB
Markdown
88 lines
5.2 KiB
Markdown
# nats — the mesh's bus
|
|
|
|
The bus server (novox/hq design 25), the configuration the module owns (`nats.conf`: ports, TLS,
|
|
JetStream's store) and the user list the controller composes (`accounts.conf`), reloaded in place by
|
|
the entrypoint. Its tools (`cmd/nats-tools`) are served by the node's runtime and only read.
|
|
|
|
## Its data, and how it is backed up
|
|
|
|
Everything the bus keeps is in JetStream: every stream, and every key-value bucket (a stream named
|
|
`KV_<bucket>`) — conditions and their history, calls, hand-acts, the controller's lease, assignments,
|
|
events, every module's state.
|
|
|
|
**The store's files are never what is backed up.** They are written all the time, and a copy taken
|
|
while the server writes them may not restore. The `jetstream` data item is protected by a dump
|
|
instead (novox/hq ADR 0235, to-be 43): each night the machine's backup holder runs
|
|
|
|
docker exec -i mesh-broker-nats mesh-nats-snapshot snapshot < <the module's broker credential> > <snapshots>/bus.tar
|
|
|
|
and backs up the `snapshots` directory. `mesh-nats-snapshot` (`snapshot/`, built into this image)
|
|
asks the server for each stream through JetStream's snapshot API, one at a time, acknowledging each
|
|
chunk so the server's flow control keeps moving; the server goes on taking writes and no stream is
|
|
paused or reconfigured. It writes one tar on stdout:
|
|
|
|
- `manifest.json` — when it was taken and how long it took, the server's version, and per stream its
|
|
subjects, messages, bytes, first and last sequence, consumers, and the size and SHA-256 of its files;
|
|
a memory stream is listed as skipped, with why;
|
|
- `streams/<name>/backup.json` — the stream's configuration and state, as the server gave them;
|
|
- `streams/<name>/stream.tar.s2` — the server's archive of the stream, consumers included.
|
|
|
|
It runs as the module's own bus account, which the controller grants the snapshot API and nothing
|
|
else — no stream may be defined, changed, purged or written by it (mesh-controller
|
|
`internal/broker`, `BusSnapshotGrants`). A failure fails that night for the module, leaves the
|
|
previous `bus.tar` in place, and reaches the operator as `backup-stale` once the last good night is
|
|
past its bound.
|
|
|
|
**What a snapshot promises.** Every message up to the last sequence its manifest gives for a stream is
|
|
in it, exactly. A stream written to while it is read out may hold some messages past that sequence
|
|
too — the server reads its blocks a moment after it states the stream — and a restore of it ends at
|
|
or after the manifest's sequence, never before.
|
|
|
|
## Restoring
|
|
|
|
Nothing is restored over the live bus: a stream is only ever restored where it does not exist, and
|
|
the controller asserts every stream on start. So the bus is restored into a **new store beside the
|
|
live one**, which a person swaps in.
|
|
|
|
1. **Take the night back, beside the live data.** `node-backup.restore` for `nats` on the bus's
|
|
machine (optionally naming a restore point) restores the snapshots directory as
|
|
`<snapshots>.restored-<stamp>/`, holding that night's `bus.tar`.
|
|
2. **Check it.** `docker exec -i mesh-broker-nats mesh-nats-snapshot verify < <restored>/bus.tar` —
|
|
every file against the manifest's checksum, every archive read to its end.
|
|
3. **Build the new store**, with the bus's own image (the server that fills it is the same release
|
|
the bus runs):
|
|
|
|
image=$(docker inspect -f '{{.Config.Image}}' mesh-broker-nats)
|
|
docker run --rm -v <restored>:/in:ro -v <new-store>:/out \
|
|
--entrypoint mesh-nats-snapshot "$image" restore --into /out --from /in/bus.tar
|
|
|
|
It starts a server of its own on the container's loopback with the bus's one account (`MESH`),
|
|
restores every stream, holds each to the manifest, stops it, and leaves
|
|
`<new-store>/jetstream/MESH/streams/…` — the bus's layout.
|
|
4. **Swap it in — a person's act, a planned bus step** (novox/hq to-be 45): say it as a hand act,
|
|
then in one command line, because the node-engine starts a stopped container again at its next
|
|
reconcile (within five minutes):
|
|
|
|
docker stop mesh-broker-nats && \
|
|
mv <jetstream-data>/jetstream <jetstream-data>/jetstream.before-restore-<stamp> && \
|
|
mv <new-store>/jetstream <jetstream-data>/jetstream && \
|
|
docker start mesh-broker-nats
|
|
|
|
The store kept aside is removed by a person once the bus is seen whole (`nats_streams`, the
|
|
controller's `doctor`).
|
|
|
|
A single stream can be restored into any server that does not hold it with
|
|
`mesh-nats-snapshot restore --server <url> --credential <file>`; the archive is also what the nats CLI
|
|
reads (`nats stream restore <dir>` on an unpacked `streams/<name>/`).
|
|
|
|
## Tests
|
|
|
|
- `snapshot/`: `MESH_TEST_DOCKER=1 go test -race ./...` — a throwaway `nats:2.11` filled like the
|
|
mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a
|
|
purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the
|
|
snapshot restored into a fresh server and into a new store served by a third, each compared message
|
|
by message; a damaged archive refused; and the snapshot user refused every write.
|
|
`MESH_TEST_NATS_IMAGE=<this image>` also runs the manifest's declared dump, exactly, against the
|
|
module's image and configuration.
|
|
- `cmd/nats-tools`: `go test ./...`.
|