mesh-catalog/modules/nats

nats — the mesh's bus

The bus server (novox/hq design 25), the configuration the module owns (nats.conf: ports, TLS, JetStream's store) and the user list the controller composes (accounts.conf), reloaded in place by the entrypoint. The server is the upstream nats image, pinned by digest in the manifest's server artifact (nats:<release>-alpine@sha256:<digest>: the digest names the image, the tag says the release, and cmd/nats-tools' release test keeps the tag equal to the server its tests run). The module's own two programs are mounted into it read-only (novox/hq ADR 0480): the snapshot program (a Go bundle of cmd/mesh-nats-snapshot) at /opt/mesh, and the entrypoint (program/entrypoint.sh, an archive) at /opt/mesh-entrypoint, run through sh. Both archive resources are in the container's restart-on. Its tools (cmd/nats-tools) are served by the node's runtime and only read.

Its data, and how it is backed up

Everything the bus keeps is in JetStream: every stream, and every key-value bucket (a stream named KV_<bucket>) — conditions and their history, calls, hand-acts, the controller's lease, assignments, events, every module's state.

The store's files are never what is backed up. They are written all the time, and a copy taken while the server writes them may not restore. The jetstream data item is protected by a dump instead (novox/hq ADR 0235, to-be 43): each night the machine's backup holder runs

docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot snapshot < <the module's broker credential> > <snapshots>/bus.tar

and backs up the snapshots directory. mesh-nats-snapshot (cmd/mesh-nats-snapshot, mounted into the server's container) asks the server for each stream through JetStream's snapshot API, one at a time, acknowledging each chunk so the server's flow control keeps moving; the server goes on taking writes and no stream is paused or reconfigured. It writes one tar on stdout:

• manifest.json — when it was taken and how long it took, the server's version, and per stream its subjects, messages, bytes, first and last sequence, consumers, and the size and SHA-256 of its files; a memory stream is listed as skipped, with why;
• streams/<name>/backup.json — the stream's configuration and state, as the server gave them;
• streams/<name>/stream.tar.s2 — the server's archive of the stream, consumers included.

It runs as the module's own bus account, which the controller grants the snapshot API and nothing else — no stream may be defined, changed, purged or written by it (mesh-controller internal/broker, BusSnapshotGrants). A failure fails that night for the module, leaves the previous bus.tar in place, and reaches the operator as backup-stale once the last good night is past its bound.

What a snapshot promises. Every message up to the last sequence its manifest gives for a stream is in it, exactly. A stream written to while it is read out may hold some messages past that sequence too — the server reads its blocks a moment after it states the stream — and a restore of it ends at or after the manifest's sequence, never before.

Restoring

Nothing is restored over the live bus: a stream is only ever restored where it does not exist, and the controller asserts every stream on start. So the bus is restored into a new store beside the live one, which a person swaps in.

1. Take the night back, beside the live data. node-backup.restore for nats on the bus's machine (optionally naming a restore point) restores the snapshots directory as <snapshots>.restored-<stamp>/, holding that night's bus.tar.

2. Check it. The tool <node>/nats.nats_snapshot_verify checks the newest bus.tar of the live snapshots directory (or one file named in it) — every file against the manifest's checksum, every archive read to its end. It reads nothing outside that directory, so a restored copy (<snapshots>.restored-<stamp>/) is checked by docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot verify < <restored>/bus.tar, and step 3's restore holds the archive to its manifest first and refuses on a fault.

3. Build the new store, with the bus's own image and the mounted program (the server that fills it is the same release the bus runs):

   image=$(docker inspect -f '{{.Config.Image}}' mesh-broker-nats)
   docker run --rm -v <restored>:/in:ro -v <new-store>:/out \
     -v <the module's state directory>/snapshot:/opt/mesh:ro \
     --entrypoint /opt/mesh/mesh-nats-snapshot "$image" restore --into /out --from /in/bus.tar
   

   It starts a server of its own on the container's loopback with the bus's one account (MESH), restores every stream, holds each to the manifest, stops it, and leaves <new-store>/jetstream/MESH/streams/… — the bus's layout.

4. Swap it in — a person's act, a planned bus step (novox/hq to-be 45): say it as a hand act, then in one command line, because the node-engine starts a stopped container again at its next reconcile (within five minutes):

   docker stop mesh-broker-nats && \
     mv <jetstream-data>/jetstream <jetstream-data>/jetstream.before-restore-<stamp> && \
     mv <new-store>/jetstream <jetstream-data>/jetstream && \
     docker start mesh-broker-nats
   

   The store kept aside is removed by a person once the bus is seen whole (nats_streams, nats_backlog).

A single stream can be restored into any server that does not hold it with mesh-nats-snapshot restore --server <url> --credential <file>; the archive is also what the nats CLI reads (nats stream restore <dir> on an unpacked streams/<name>/).

Upgrading the bus, and going back

The bus is replaced only as a planned step a person starts (novox/hq ADR 0236, ADR 0480 clause 6): the merge of a change to this module builds the bus and sends nothing, and no rule of the assigner moves it (mesh-wants.upgrade gives the bus the policy record, and refuses a person's other choice). The step is a put of the assigner (novox/hq ADR 0546, ADR 0565 clauses 8 and 9). In this order:

1. Read what it would do, and that nothing else moves. mesh-wants.bus answers the fingerprint of the bus module each node runs against the indexer's newest, and the last bus steps (from the hand-act log and the assigner's puts of the bus). mesh-delivery-assigner.assigner_puts shows no other put still open.
2. Snapshot with the dump the bus runs now, through the machine's backup holder (<node>/node-backup.now for nats on the bus's machine), and verify the bus.tar it wrote with <node>/nats.nats_snapshot_verify (it answers the file, its time and size, each stream's last sequence, and passed or the first fault; run it after the dump and read taken). This is the known-good snapshot to come back to.
3. Tell the operator what the step raises before starting it: bus maintenance is open while the step runs; every bus connection is dropped for seconds and reconnects; conditions of other modules may flash and clear.
4. Start it: mesh-delivery-assigner.assigner_put with node (the bus's machine), module nats, version (newest or the fingerprint mesh-wants.bus named) and why. The put asks the operator first, on the proven channel, with the why; only on their yes does it go on. After the yes it has the bus node's backup holder take the bus's snapshot again (it reads the holder's record back for at most 15 minutes), then writes the composition, the grant and the machine's want, and records the warrant in the hand-act log. Each step is said as the event mesh-delivery-assigner.put; wait on it, and assigner_puts says where the put stands. The step is reversible because the server binary (the same digest), the data directory and the configuration do not change; TestTheStoreSurvivesTheNewDeclarationTheOldAndTheNewAgain shows it: the new declaration, then the old one, then the new again run on one store, each reading what the one before wrote.
5. After: nats_server (the new release) and nats_streams (every stream and bucket as before). The controller's doctor, which checked every stream, every consumer and a round trip (ADR 0236 section 40), ended with the controller, and no verb has taken that check over yet: read nats_streams and nats_backlog by hand for it. Then wait for the next nightly dump and verify it: the upgrade is finished only once that dump verified.
6. Back: a put of the previous build (assigner_put with its fingerprint) puts the old declaration back on the same store; if the store itself is hurt, restore the snapshot as described above.

Tests

• cmd/mesh-nats-snapshot: go test -tags integration -race ./..., run by the pull request's check (they start throwaway containers; nobody runs them by hand) — a throwaway nats:2.11 filled like the mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the snapshot restored into a fresh server and into a new store served by a third, each compared message by message; a damaged archive refused; and the snapshot user refused every write. The same run starts the manifest's pinned upstream image with the program and the entrypoint mounted as the manifest declares them: the declared dump, exactly, writes a bus.tar that verifies; a change of the user list reloads the server and a connection open meanwhile survives; and the new declaration, the old one and the new again run on one store.
• cmd/nats-tools: go test ./... — the tools, the release test (the manifest's tag equals the server tested) and the manifest's shape (programs mounted read-only, named in restart-on, no image built from a Dockerfile).

Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

• /var/lib/mesh-broker-nats: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• /var/lib/nats-module/conf: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• ${dir:snapshots}: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• ${dir:mesh-state}: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• ${dir:state}/snapshot: the unpacked snapshot program; everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
• ${dir:state}/entrypoint: the unpacked entrypoint; everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).