jschoubben
98ca97fc5b
mesh/check failed: ace: novox/mesh-catalog#442: checks/particulars/go.mod cannot be read to check its mesh-sdk pin: Gitea API /repos/novox/mesh-catalo…
mesh/merge-gate head 98ca97fc5b3f merged into trunk dcf6e86d0236 — pass: builds nats, sent nowhere; merging this needs a person's bus upgrade; 1 wait(s) fo…
mesh/repo-check head 98ca97fc5b3f merged into trunk dcf6e86d0236 — pass: its merge-check.sh passed, each part in its toolchain: merge-check.sh (go), merge-…
mesh/delivery delivered
The upgrade and restore steps told a person to docker exec the snapshot program; no mesh tool did it. nats_snapshot_verify reads the newest bus.tar of the snapshots directory only.
137 lines
9.5 KiB
Markdown
137 lines
9.5 KiB
Markdown
# nats — the mesh's bus
|
|
|
|
The bus server (novox/hq design 25), the configuration the module owns (`nats.conf`: ports, TLS,
|
|
JetStream's store) and the user list the controller composes (`accounts.conf`), reloaded in place by
|
|
the entrypoint. The server is the upstream nats image, pinned by digest in the manifest's `server`
|
|
artifact (`nats:<release>-alpine@sha256:<digest>`: the digest names the image, the tag says the release,
|
|
and `cmd/nats-tools`' release test keeps the tag equal to the server its tests run). The module's own two
|
|
programs are mounted into it read-only (novox/hq ADR 0480): the snapshot program (a Go bundle of
|
|
`cmd/mesh-nats-snapshot`) at `/opt/mesh`, and the entrypoint (`program/entrypoint.sh`, an archive) at
|
|
`/opt/mesh-entrypoint`, run through `sh`. Both archive resources are in the container's `restart-on`. Its tools (`cmd/nats-tools`) are served by the node's runtime and only read.
|
|
|
|
## Its data, and how it is backed up
|
|
|
|
Everything the bus keeps is in JetStream: every stream, and every key-value bucket (a stream named
|
|
`KV_<bucket>`) — conditions and their history, calls, hand-acts, the controller's lease, assignments,
|
|
events, every module's state.
|
|
|
|
**The store's files are never what is backed up.** They are written all the time, and a copy taken
|
|
while the server writes them may not restore. The `jetstream` data item is protected by a dump
|
|
instead (novox/hq ADR 0235, to-be 43): each night the machine's backup holder runs
|
|
|
|
docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot snapshot < <the module's broker credential> > <snapshots>/bus.tar
|
|
|
|
and backs up the `snapshots` directory. `mesh-nats-snapshot` (`cmd/mesh-nats-snapshot`, mounted into the server's container)
|
|
asks the server for each stream through JetStream's snapshot API, one at a time, acknowledging each
|
|
chunk so the server's flow control keeps moving; the server goes on taking writes and no stream is
|
|
paused or reconfigured. It writes one tar on stdout:
|
|
|
|
- `manifest.json` — when it was taken and how long it took, the server's version, and per stream its
|
|
subjects, messages, bytes, first and last sequence, consumers, and the size and SHA-256 of its files;
|
|
a memory stream is listed as skipped, with why;
|
|
- `streams/<name>/backup.json` — the stream's configuration and state, as the server gave them;
|
|
- `streams/<name>/stream.tar.s2` — the server's archive of the stream, consumers included.
|
|
|
|
It runs as the module's own bus account, which the controller grants the snapshot API and nothing
|
|
else — no stream may be defined, changed, purged or written by it (mesh-controller
|
|
`internal/broker`, `BusSnapshotGrants`). A failure fails that night for the module, leaves the
|
|
previous `bus.tar` in place, and reaches the operator as `backup-stale` once the last good night is
|
|
past its bound.
|
|
|
|
**What a snapshot promises.** Every message up to the last sequence its manifest gives for a stream is
|
|
in it, exactly. A stream written to while it is read out may hold some messages past that sequence
|
|
too — the server reads its blocks a moment after it states the stream — and a restore of it ends at
|
|
or after the manifest's sequence, never before.
|
|
|
|
## Restoring
|
|
|
|
Nothing is restored over the live bus: a stream is only ever restored where it does not exist, and
|
|
the controller asserts every stream on start. So the bus is restored into a **new store beside the
|
|
live one**, which a person swaps in.
|
|
|
|
1. **Take the night back, beside the live data.** `node-backup.restore` for `nats` on the bus's
|
|
machine (optionally naming a restore point) restores the snapshots directory as
|
|
`<snapshots>.restored-<stamp>/`, holding that night's `bus.tar`.
|
|
2. **Check it.** The tool `<node>/nats.nats_snapshot_verify` checks the newest `bus.tar` of the live snapshots
|
|
directory (or one `file` named in it) — every file against the manifest's checksum, every archive read to
|
|
its end. It reads nothing outside that directory, so a restored copy (`<snapshots>.restored-<stamp>/`) is
|
|
checked by `docker exec -i mesh-broker-nats /opt/mesh/mesh-nats-snapshot verify < <restored>/bus.tar`, and
|
|
step 3's `restore` holds the archive to its manifest first and refuses on a fault.
|
|
3. **Build the new store**, with the bus's own image and the mounted program (the server that fills
|
|
it is the same release the bus runs):
|
|
|
|
image=$(docker inspect -f '{{.Config.Image}}' mesh-broker-nats)
|
|
docker run --rm -v <restored>:/in:ro -v <new-store>:/out \
|
|
-v <the module's state directory>/snapshot:/opt/mesh:ro \
|
|
--entrypoint /opt/mesh/mesh-nats-snapshot "$image" restore --into /out --from /in/bus.tar
|
|
|
|
It starts a server of its own on the container's loopback with the bus's one account (`MESH`),
|
|
restores every stream, holds each to the manifest, stops it, and leaves
|
|
`<new-store>/jetstream/MESH/streams/…` — the bus's layout.
|
|
4. **Swap it in — a person's act, a planned bus step** (novox/hq to-be 45): say it as a hand act,
|
|
then in one command line, because the node-engine starts a stopped container again at its next
|
|
reconcile (within five minutes):
|
|
|
|
docker stop mesh-broker-nats && \
|
|
mv <jetstream-data>/jetstream <jetstream-data>/jetstream.before-restore-<stamp> && \
|
|
mv <new-store>/jetstream <jetstream-data>/jetstream && \
|
|
docker start mesh-broker-nats
|
|
|
|
The store kept aside is removed by a person once the bus is seen whole (`nats_streams`, the
|
|
controller's `doctor`).
|
|
|
|
A single stream can be restored into any server that does not hold it with
|
|
`mesh-nats-snapshot restore --server <url> --credential <file>`; the archive is also what the nats CLI
|
|
reads (`nats stream restore <dir>` on an unpacked `streams/<name>/`).
|
|
|
|
## Upgrading the bus, and going back
|
|
|
|
The bus is replaced only as a planned step a person starts (novox/hq ADR 0236, ADR 0480 clause 6): the
|
|
merge of a change to this module builds the bus and sends nothing. The steps, in this order:
|
|
|
|
1. **Nothing else runs.** No delivery plan is open (`mesh-controller.plans`).
|
|
2. **Snapshot with the dump the bus runs now**, through the machine's backup holder
|
|
(`<node>/node-backup.now` for `nats` on the bus's machine), and verify the `bus.tar` it wrote
|
|
with `<node>/nats.nats_snapshot_verify` (it answers the file, its time and size, each stream's last
|
|
sequence, and `passed` or the first fault; run it after the dump and read `taken`). Note where it is: that is `snapshot-taken`.
|
|
3. **Ask the operator for the moment** on the proven channel, naming what it raises: bus maintenance is
|
|
open while the step runs; every bus connection is dropped for seconds and reconnects; conditions of
|
|
other modules may flash and clear. Go on only on their yes.
|
|
4. **Start it:** `mesh-controller.bus` with `upgrade` true, `why`, `reversible` true and `snapshot-taken`.
|
|
`reversible` is true because the server binary (the same digest), the data directory and the
|
|
configuration do not change; `TestTheStoreSurvivesTheNewDeclarationTheOldAndTheNewAgain` shows it:
|
|
the new declaration, then the old one, then the new again run on one store, each reading what the
|
|
one before wrote.
|
|
5. **After:** `nats_server`, `nats_streams` (every stream and bucket as before) and the controller's
|
|
`doctor`; the step ends done when the controller has checked every stream, every consumer and a
|
|
round trip (ADR 0236 section 40). Then wait for the next nightly dump and verify it: the upgrade is
|
|
finished only once that dump verified.
|
|
6. **Back:** a bus upgrade to the previous build puts the old declaration back on the same store; if the
|
|
store itself is hurt, restore the snapshot as described above.
|
|
|
|
## Tests
|
|
|
|
- `cmd/mesh-nats-snapshot`: `go test -tags integration -race ./...`, run by `mesh-controller.try` (they start throwaway containers; nobody runs them by hand) — a throwaway `nats:2.11` filled like the
|
|
mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a
|
|
purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the
|
|
snapshot restored into a fresh server and into a new store served by a third, each compared message
|
|
by message; a damaged archive refused; and the snapshot user refused every write. The same run starts
|
|
the manifest's pinned upstream image with the program and the entrypoint mounted as the manifest
|
|
declares them: the declared dump, exactly, writes a `bus.tar` that verifies; a change of the user list
|
|
reloads the server and a connection open meanwhile survives; and the new declaration, the old one and
|
|
the new again run on one store.
|
|
- `cmd/nats-tools`: `go test ./...` — the tools, the release test (the manifest's tag equals the server
|
|
tested) and the manifest's shape (programs mounted read-only, named in `restart-on`, no image built from
|
|
a Dockerfile).
|
|
|
|
## Files it does not know
|
|
|
|
Each directory or archive this module declares says how it treats a file it did not write (ADR 0422, ADR 0416).
|
|
|
|
- `/var/lib/mesh-broker-nats`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
|
- `/var/lib/nats-module/conf`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
|
- `${dir:snapshots}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
|
- `${dir:mesh-state}`: everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
|
- `${dir:state}/snapshot`: the unpacked snapshot program; everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|
|
- `${dir:state}/entrypoint`: the unpacked entrypoint; everything in it is the program's; the mesh moves nothing out (ADR 0422 §2).
|