The bus server (novox/hq design 25), the configuration the module owns (nats.conf: ports, TLS, JetStream's store) and the user list the controller composes (accounts.conf), reloaded in place by the entrypoint. The server is the upstream nats image, pinned by digest in the manifest's server artifact (nats:<release>-alpine@sha256:<digest>: the digest names the image, the tag says the release, and cmd/nats-tools' release test keeps the tag equal to the server its tests run). The module's own two programs are mounted into it read-only (novox/hq ADR 0480): the snapshot program (a Go bundle of cmd/mesh-nats-snapshot) at /opt/mesh, and the entrypoint (program/entrypoint.sh, an archive) at /opt/mesh-entrypoint, run through sh. Both archive resources are in the container's restart-on. Its tools (cmd/nats-tools) are served by the node's runtime and only read.

Its data, and how it is backed up

Everything the bus keeps is in JetStream: every stream, and every key-value bucket (a stream named KV_<bucket>) — conditions and their history, calls, hand-acts, the controller's lease, assignments, events, every module's state.

The store's files are never what is backed up. They are written all the time, and a copy taken while the server writes them may not restore. The jetstream data item is protected by a dump instead (novox/hq ADR 0235, to-be 43): each night the machine's backup holder runs

docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot snapshot < <the module's broker credential> > <snapshots>/bus.tar

and backs up the snapshots directory. mesh-nats-snapshot (cmd/mesh-nats-snapshot, mounted into the server's container) asks the server for each stream through JetStream's snapshot API, one at a time, acknowledging each chunk so the server's flow control keeps moving; the server goes on taking writes and no stream is paused or reconfigured. It writes one tar on stdout:

manifest.json — when it was taken and how long it took, the server's version, and per stream its subjects, messages, bytes, first and last sequence, consumers, and the size and SHA-256 of its files; a memory stream is listed as skipped, with why;
streams/<name>/backup.json — the stream's configuration and state, as the server gave them;
streams/<name>/stream.tar.s2 — the server's archive of the stream, consumers included.

It runs as the module's own bus account, which the controller grants the snapshot API and nothing else — no stream may be defined, changed, purged or written by it (mesh-controller internal/broker, BusSnapshotGrants). A failure fails that night for the module, leaves the previous bus.tar in place, and reaches the operator as backup-stale once the last good night is past its bound.

What a snapshot promises. Every message up to the last sequence its manifest gives for a stream is in it, exactly. A stream written to while it is read out may hold some messages past that sequence too — the server reads its blocks a moment after it states the stream — and a restore of it ends at or after the manifest's sequence, never before.

Restoring

Nothing is restored over the live bus: a stream is only ever restored where it does not exist, and the controller asserts every stream on start. So the bus is restored into a new store beside the live one, which a person swaps in.

Take the night back, beside the live data. node-backup.restore for nats on the bus's machine (optionally naming a restore point) restores the snapshots directory as <snapshots>.restored-<stamp>/, holding that night's bus.tar.
Check it. docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot verify < <restored>/bus.tar — every file against the manifest's checksum, every archive read to its end.
Build the new store, with the bus's own image and the mounted program (the server that fills it is the same release the bus runs):
```
image=$(docker inspect -f '{{.Config.Image}}' mesh-broker-nats)
docker run --rm -v <restored>:/in:ro -v <new-store>:/out \
  -v /var/lib/nats-module/snapshot:/opt/mesh:ro \
  --entrypoint /opt/mesh/mesh-nats-snapshot "$image" restore --into /out --from /in/bus.tar
```
It starts a server of its own on the container's loopback with the bus's one account (MESH), restores every stream, holds each to the manifest, stops it, and leaves <new-store>/jetstream/MESH/streams/… — the bus's layout.
Swap it in — a person's act, a planned bus step (novox/hq to-be 45): say it as a hand act, then in one command line, because the node-engine starts a stopped container again at its next reconcile (within five minutes):
```
docker stop mesh-broker-nats && \
  mv <jetstream-data>/jetstream <jetstream-data>/jetstream.before-restore-<stamp> && \
  mv <new-store>/jetstream <jetstream-data>/jetstream && \
  docker start mesh-broker-nats
```
The store kept aside is removed by a person once the bus is seen whole (nats_streams, the controller's doctor).

A single stream can be restored into any server that does not hold it with mesh-nats-snapshot restore --server <url> --credential <file>; the archive is also what the nats CLI reads (nats stream restore <dir> on an unpacked streams/<name>/).

Upgrading the bus, and going back

The bus is replaced only as a planned step a person starts (novox/hq ADR 0236, ADR 0480 clause 6): the merge of a change to this module builds the bus and sends nothing. The steps, in this order:

Nothing else runs. No delivery plan is open (mesh-controller.plans).
Snapshot with the dump the bus runs now, through the machine's backup holder (<node>/node-backup.now for nats on the bus's machine), and verify the bus.tar it wrote (docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot verify, or the old path mesh-nats-snapshot verify on the declaration before ADR 0480). Note where it is: that is snapshot-taken.
Ask the operator for the moment on the proven channel, naming what it raises: bus maintenance is open while the step runs; every bus connection is dropped for seconds and reconnects; conditions of other modules may flash and clear. Go on only on their yes.
Start it: mesh-controller.bus with upgrade true, why, reversible true and snapshot-taken. reversible is true because the server binary (the same digest), the data directory and the configuration do not change; TestTheStoreSurvivesTheNewDeclarationTheOldAndTheNewAgain shows it: the new declaration, then the old one, then the new again run on one store, each reading what the one before wrote.
After: nats_server, nats_streams (every stream and bucket as before) and the controller's doctor; the step ends done when the controller has checked every stream, every consumer and a round trip (ADR 0236 section 40). Then wait for the next nightly dump and verify it: the upgrade is finished only once that dump verified.
Back: a bus upgrade to the previous build puts the old declaration back on the same store; if the store itself is hurt, restore the snapshot as described above.

Tests

cmd/mesh-nats-snapshot: go test -tags integration -race ./..., run by mesh-controller.try (they start throwaway containers; nobody runs them by hand) — a throwaway nats:2.11 filled like the mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the snapshot restored into a fresh server and into a new store served by a third, each compared message by message; a damaged archive refused; and the snapshot user refused every write. The same run starts the manifest's pinned upstream image with the program and the entrypoint mounted as the manifest declares them: the declared dump, exactly, writes a bus.tar that verifies; a change of the user list reloads the server and a connection open meanwhile survives; and the new declaration, the old one and the new again run on one store.
cmd/nats-tools: go test ./... — the tools, the release test (the manifest's tag equals the server tested) and the manifest's shape (programs mounted read-only, named in restart-on, no image built from a Dockerfile).

Files it does not know

Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).

/var/lib/mesh-broker-nats: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
/var/lib/nats-module/conf: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
${dir:snapshots}: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
${dir:mesh-state}: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
/var/lib/nats-module/snapshot: the unpacked snapshot program; everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
/var/lib/nats-module/entrypoint: the unpacked entrypoint; everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).