mesh-catalog/modules/nats/README.md

# nats — the mesh's bus

The bus server (novox/hq design 25), the configuration the module owns (`nats.conf`: ports, TLS,
JetStream's store) and the user list the controller composes (`accounts.conf`), reloaded in place by
the entrypoint. Its tools (`cmd/nats-tools`) are served by the node's runtime and only read.

## Its data, and how it is backed up

Everything the bus keeps is in JetStream: every stream, and every key-value bucket (a stream named
`KV_<bucket>`) — conditions and their history, calls, hand-acts, the controller's lease, assignments,
events, every module's state.

**The store's files are never what is backed up.** They are written all the time, and a copy taken
while the server writes them may not restore. The `jetstream` data item is protected by a dump
instead (novox/hq ADR 0235, to-be 43): each night the machine's backup holder runs

    docker exec -i mesh-broker-nats mesh-nats-snapshot snapshot < <the module's broker credential> > <snapshots>/bus.tar

and backs up the `snapshots` directory. `mesh-nats-snapshot` (`snapshot/`, built into this image)
asks the server for each stream through JetStream's snapshot API, one at a time, acknowledging each
chunk so the server's flow control keeps moving; the server goes on taking writes and no stream is
paused or reconfigured. It writes one tar on stdout:

- `manifest.json` — when it was taken and how long it took, the server's version, and per stream its
  subjects, messages, bytes, first and last sequence, consumers, and the size and SHA-256 of its files;
  a memory stream is listed as skipped, with why;
- `streams/<name>/backup.json` — the stream's configuration and state, as the server gave them;
- `streams/<name>/stream.tar.s2` — the server's archive of the stream, consumers included.

It runs as the module's own bus account, which the controller grants the snapshot API and nothing
else — no stream may be defined, changed, purged or written by it (mesh-controller
`internal/broker`, `BusSnapshotGrants`). A failure fails that night for the module, leaves the
previous `bus.tar` in place, and reaches the operator as `backup-stale` once the last good night is
past its bound.

**What a snapshot promises.** Every message up to the last sequence its manifest gives for a stream is
in it, exactly. A stream written to while it is read out may hold some messages past that sequence
too — the server reads its blocks a moment after it states the stream — and a restore of it ends at
or after the manifest's sequence, never before.

## Restoring

Nothing is restored over the live bus: a stream is only ever restored where it does not exist, and
the controller asserts every stream on start. So the bus is restored into a **new store beside the
live one**, which a person swaps in.

1. **Take the night back, beside the live data.** `node-backup.restore` for `nats` on the bus's
   machine (optionally naming a restore point) restores the snapshots directory as
   `<snapshots>.restored-<stamp>/`, holding that night's `bus.tar`.
2. **Check it.** `docker exec -i mesh-broker-nats mesh-nats-snapshot verify < <restored>/bus.tar` —
   every file against the manifest's checksum, every archive read to its end.
3. **Build the new store**, with the bus's own image (the server that fills it is the same release
   the bus runs):

       image=$(docker inspect -f '{{.Config.Image}}' mesh-broker-nats)
       docker run --rm -v <restored>:/in:ro -v <new-store>:/out \
         --entrypoint mesh-nats-snapshot "$image" restore --into /out --from /in/bus.tar

   It starts a server of its own on the container's loopback with the bus's one account (`MESH`),
   restores every stream, holds each to the manifest, stops it, and leaves
   `<new-store>/jetstream/MESH/streams/…` — the bus's layout.
4. **Swap it in — a person's act, a planned bus step** (novox/hq to-be 45): say it as a hand act,
   then in one command line, because the node-engine starts a stopped container again at its next
   reconcile (within five minutes):

       docker stop mesh-broker-nats && \
         mv <jetstream-data>/jetstream <jetstream-data>/jetstream.before-restore-<stamp> && \
         mv <new-store>/jetstream <jetstream-data>/jetstream && \
         docker start mesh-broker-nats

   The store kept aside is removed by a person once the bus is seen whole (`nats_streams`, the
   controller's `doctor`).

A single stream can be restored into any server that does not hold it with
`mesh-nats-snapshot restore --server <url> --credential <file>`; the archive is also what the nats CLI
reads (`nats stream restore <dir>` on an unpacked `streams/<name>/`).

## Tests

- `snapshot/`: `MESH_TEST_DOCKER=1 go test -race ./...` — a throwaway `nats:2.11` filled like the
  mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a
  purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the
  snapshot restored into a fresh server and into a new store served by a third, each compared message
  by message; a damaged archive refused; and the snapshot user refused every write.
  `MESH_TEST_NATS_IMAGE=<this image>` also runs the manifest's declared dump, exactly, against the
  module's image and configuration.
- `cmd/nats-tools`: `go test ./...`.

## Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

- `/var/lib/mesh-broker-nats`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/var/lib/nats-module/conf`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:snapshots}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:mesh-state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).