# nats — the mesh's bus

The bus server (novox/hq design 25), the configuration the module owns (`nats.conf`: ports, TLS,
JetStream's store) and the user list the controller composes (`accounts.conf`), reloaded in place by
the entrypoint. The server is the upstream nats image, pinned by digest in the manifest's `server`
artifact (`nats:<release>-alpine@sha256:<digest>`: the digest names the image, the tag says the release,
and `cmd/nats-tools`' release test keeps the tag equal to the server its tests run). The module's own two
programs are mounted into it read-only (novox/hq ADR 0480): the snapshot program (a Go bundle of
`cmd/mesh-nats-snapshot`) at `/opt/mesh`, and the entrypoint (`program/entrypoint.sh`, an archive) at
`/opt/mesh-entrypoint`, run through `sh`. Both archive resources are in the container's `restart-on`. Its tools (`cmd/nats-tools`) are served by the node's runtime and only read.

## Its data, and how it is backed up

Everything the bus keeps is in JetStream: every stream, and every key-value bucket (a stream named
`KV_<bucket>`) — conditions and their history, calls, hand-acts, the controller's lease, assignments,
events, every module's state.

**The store's files are never what is backed up.** They are written all the time, and a copy taken
while the server writes them may not restore. The `jetstream` data item is protected by a dump
instead (novox/hq ADR 0235, to-be 43): each night the machine's backup holder runs

    docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot snapshot < <the module's broker credential> > <snapshots>/bus.tar

and backs up the `snapshots` directory. `mesh-nats-snapshot` (`cmd/mesh-nats-snapshot`, mounted into the server's container)
asks the server for each stream through JetStream's snapshot API, one at a time, acknowledging each
chunk so the server's flow control keeps moving; the server goes on taking writes and no stream is
paused or reconfigured. It writes one tar on stdout:

- `manifest.json` — when it was taken and how long it took, the server's version, and per stream its
  subjects, messages, bytes, first and last sequence, consumers, and the size and SHA-256 of its files;
  a memory stream is listed as skipped, with why;
- `streams/<name>/backup.json` — the stream's configuration and state, as the server gave them;
- `streams/<name>/stream.tar.s2` — the server's archive of the stream, consumers included.

It runs as the module's own bus account, which the controller grants the snapshot API and nothing
else — no stream may be defined, changed, purged or written by it (mesh-controller
`internal/broker`, `BusSnapshotGrants`). A failure fails that night for the module, leaves the
previous `bus.tar` in place, and reaches the operator as `backup-stale` once the last good night is
past its bound.

**What a snapshot promises.** Every message up to the last sequence its manifest gives for a stream is
in it, exactly. A stream written to while it is read out may hold some messages past that sequence
too — the server reads its blocks a moment after it states the stream — and a restore of it ends at
or after the manifest's sequence, never before.

## Restoring

Nothing is restored over the live bus: a stream is only ever restored where it does not exist, and
the controller asserts every stream on start. So the bus is restored into a **new store beside the
live one**, which a person swaps in.

1. **Take the night back, beside the live data.** `node-backup.restore` for `nats` on the bus's
   machine (optionally naming a restore point) restores the snapshots directory as
   `<snapshots>.restored-<stamp>/`, holding that night's `bus.tar`.
2. **Check it.** `docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot verify < <restored>/bus.tar` —
   every file against the manifest's checksum, every archive read to its end.
3. **Build the new store**, with the bus's own image and the mounted program (the server that fills
   it is the same release the bus runs):

       image=$(docker inspect -f '{{.Config.Image}}' mesh-broker-nats)
       docker run --rm -v <restored>:/in:ro -v <new-store>:/out \
         -v <the module's state directory>/snapshot:/opt/mesh:ro \
         --entrypoint /opt/mesh/mesh-nats-snapshot "$image" restore --into /out --from /in/bus.tar

   It starts a server of its own on the container's loopback with the bus's one account (`MESH`),
   restores every stream, holds each to the manifest, stops it, and leaves
   `<new-store>/jetstream/MESH/streams/…` — the bus's layout.
4. **Swap it in — a person's act, a planned bus step** (novox/hq to-be 45): say it as a hand act,
   then in one command line, because the node-engine starts a stopped container again at its next
   reconcile (within five minutes):

       docker stop mesh-broker-nats && \
         mv <jetstream-data>/jetstream <jetstream-data>/jetstream.before-restore-<stamp> && \
         mv <new-store>/jetstream <jetstream-data>/jetstream && \
         docker start mesh-broker-nats

   The store kept aside is removed by a person once the bus is seen whole (`nats_streams`, the
   controller's `doctor`).

A single stream can be restored into any server that does not hold it with
`mesh-nats-snapshot restore --server <url> --credential <file>`; the archive is also what the nats CLI
reads (`nats stream restore <dir>` on an unpacked `streams/<name>/`).

## Upgrading the bus, and going back

The bus is replaced only as a planned step a person starts (novox/hq ADR 0236, ADR 0480 clause 6): the
merge of a change to this module builds the bus and sends nothing. The steps, in this order:

1. **Nothing else runs.** No delivery plan is open (`mesh-controller.plans`).
2. **Snapshot with the dump the bus runs now**, through the machine's backup holder
   (`<node>/node-backup.now` for `nats` on the bus's machine), and verify the `bus.tar` it wrote
   (`docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot verify`, or the old path
   `mesh-nats-snapshot verify` on the declaration before ADR 0480). Note where it is: that is `snapshot-taken`.
3. **Ask the operator for the moment** on the proven channel, naming what it raises: bus maintenance is
   open while the step runs; every bus connection is dropped for seconds and reconnects; conditions of
   other modules may flash and clear. Go on only on their yes.
4. **Start it:** `mesh-controller.bus` with `upgrade` true, `why`, `reversible` true and `snapshot-taken`.
   `reversible` is true because the server binary (the same digest), the data directory and the
   configuration do not change; `TestTheStoreSurvivesTheNewDeclarationTheOldAndTheNewAgain` shows it:
   the new declaration, then the old one, then the new again run on one store, each reading what the
   one before wrote.
5. **After:** `nats_server`, `nats_streams` (every stream and bucket as before) and the controller's
   `doctor`; the step ends done when the controller has checked every stream, every consumer and a
   round trip (ADR 0236 section 40). Then wait for the next nightly dump and verify it: the upgrade is
   finished only once that dump verified.
6. **Back:** a bus upgrade to the previous build puts the old declaration back on the same store; if the
   store itself is hurt, restore the snapshot as described above.

## Tests

- `cmd/mesh-nats-snapshot`: `go test -tags integration -race ./...`, run by `mesh-controller.try` (they start throwaway containers; nobody runs them by hand) — a throwaway `nats:2.11` filled like the
  mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a
  purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the
  snapshot restored into a fresh server and into a new store served by a third, each compared message
  by message; a damaged archive refused; and the snapshot user refused every write. The same run starts
  the manifest's pinned upstream image with the program and the entrypoint mounted as the manifest
  declares them: the declared dump, exactly, writes a `bus.tar` that verifies; a change of the user list
  reloads the server and a connection open meanwhile survives; and the new declaration, the old one and
  the new again run on one store.
- `cmd/nats-tools`: `go test ./...` — the tools, the release test (the manifest's tag equals the server
  tested) and the manifest's shape (programs mounted read-only, named in `restart-on`, no image built from
  a Dockerfile).

## Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

- `/var/lib/mesh-broker-nats`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `/var/lib/nats-module/conf`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:snapshots}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:mesh-state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:state}/snapshot`: the unpacked snapshot program; everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
- `${dir:state}/entrypoint`: the unpacked entrypoint; everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).