mesh-catalog/modules/nats

nats — the mesh's bus

The bus server (novox/hq design 25), the configuration the module owns (nats.conf: ports, TLS, JetStream's store) and the user list the controller composes (accounts.conf), reloaded in place by the entrypoint. Its tools (cmd/nats-tools) are served by the node's runtime and only read.

Its data, and how it is backed up

Everything the bus keeps is in JetStream: every stream, and every key-value bucket (a stream named KV_<bucket>) — conditions and their history, calls, hand-acts, the controller's lease, assignments, events, every module's state.

The store's files are never what is backed up. They are written all the time, and a copy taken while the server writes them may not restore. The jetstream data item is protected by a dump instead (novox/hq ADR 0235, to-be 43): each night the machine's backup holder runs

docker exec -i mesh-broker-nats mesh-nats-snapshot snapshot < <the module's broker credential> > <snapshots>/bus.tar

and backs up the snapshots directory. mesh-nats-snapshot (snapshot/, built into this image) asks the server for each stream through JetStream's snapshot API, one at a time, acknowledging each chunk so the server's flow control keeps moving; the server goes on taking writes and no stream is paused or reconfigured. It writes one tar on stdout:

• manifest.json — when it was taken and how long it took, the server's version, and per stream its subjects, messages, bytes, first and last sequence, consumers, and the size and SHA-256 of its files; a memory stream is listed as skipped, with why;
• streams/<name>/backup.json — the stream's configuration and state, as the server gave them;
• streams/<name>/stream.tar.s2 — the server's archive of the stream, consumers included.

It runs as the module's own bus account, which the controller grants the snapshot API and nothing else — no stream may be defined, changed, purged or written by it (mesh-controller internal/broker, BusSnapshotGrants). A failure fails that night for the module, leaves the previous bus.tar in place, and reaches the operator as backup-stale once the last good night is past its bound.

What a snapshot promises. Every message up to the last sequence its manifest gives for a stream is in it, exactly. A stream written to while it is read out may hold some messages past that sequence too — the server reads its blocks a moment after it states the stream — and a restore of it ends at or after the manifest's sequence, never before.

Restoring

Nothing is restored over the live bus: a stream is only ever restored where it does not exist, and the controller asserts every stream on start. So the bus is restored into a new store beside the live one, which a person swaps in.

1. Take the night back, beside the live data. node-backup.restore for nats on the bus's machine (optionally naming a restore point) restores the snapshots directory as <snapshots>.restored-<stamp>/, holding that night's bus.tar.

2. Check it. docker exec -i mesh-broker-nats mesh-nats-snapshot verify < <restored>/bus.tar — every file against the manifest's checksum, every archive read to its end.

3. Build the new store, with the bus's own image (the server that fills it is the same release the bus runs):

   image=$(docker inspect -f '{{.Config.Image}}' mesh-broker-nats)
   docker run --rm -v <restored>:/in:ro -v <new-store>:/out \
     --entrypoint mesh-nats-snapshot "$image" restore --into /out --from /in/bus.tar
   

   It starts a server of its own on the container's loopback with the bus's one account (MESH), restores every stream, holds each to the manifest, stops it, and leaves <new-store>/jetstream/MESH/streams/… — the bus's layout.

4. Swap it in — a person's act, a planned bus step (novox/hq to-be 45): say it as a hand act, then in one command line, because the node-engine starts a stopped container again at its next reconcile (within five minutes):

   docker stop mesh-broker-nats && \
     mv <jetstream-data>/jetstream <jetstream-data>/jetstream.before-restore-<stamp> && \
     mv <new-store>/jetstream <jetstream-data>/jetstream && \
     docker start mesh-broker-nats
   

   The store kept aside is removed by a person once the bus is seen whole (nats_streams, the controller's doctor).

A single stream can be restored into any server that does not hold it with mesh-nats-snapshot restore --server <url> --credential <file>; the archive is also what the nats CLI reads (nats stream restore <dir> on an unpacked streams/<name>/).

Tests

• snapshot/: MESH_TEST_DOCKER=1 go test -race ./... — a throwaway nats:2.11 filled like the mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the snapshot restored into a fresh server and into a new store served by a third, each compared message by message; a damaged archive refused; and the snapshot user refused every write. MESH_TEST_NATS_IMAGE=<this image> also runs the manifest's declared dump, exactly, against the module's image and configuration.
• cmd/nats-tools: go test ./....