diff --git a/modules/nats/README.md b/modules/nats/README.md index 0994ea89..aeab2e77 100644 --- a/modules/nats/README.md +++ b/modules/nats/README.md @@ -77,8 +77,8 @@ live one**, which a person swaps in. mv /jetstream /jetstream && \ docker start mesh-broker-nats - The store kept aside is removed by a person once the bus is seen whole (`nats_streams`, the - controller's `doctor`). + The store kept aside is removed by a person once the bus is seen whole (`nats_streams`, + `nats_backlog`). A single stream can be restored into any server that does not hold it with `mesh-nats-snapshot restore --server --credential `; the archive is also what the nats CLI @@ -87,31 +87,43 @@ reads (`nats stream restore ` on an unpacked `streams//`). ## Upgrading the bus, and going back The bus is replaced only as a planned step a person starts (novox/hq ADR 0236, ADR 0480 clause 6): the -merge of a change to this module builds the bus and sends nothing. The steps, in this order: +merge of a change to this module builds the bus and sends nothing, and no rule of the assigner moves it +(`mesh-wants.upgrade` gives the bus the policy record, and refuses a person's other choice). The step is +a put of the assigner (novox/hq ADR 0546, ADR 0565 clauses 8 and 9). In this order: -1. **Nothing else runs.** No delivery plan is open (`mesh-controller.plans`). +1. **Read what it would do, and that nothing else moves.** `mesh-wants.bus` answers the fingerprint + of the bus module each node runs against the indexer's newest, and the last bus steps (from the + hand-act log and the assigner's puts of the bus). `mesh-delivery-assigner.assigner_puts` shows no + other put still open. 2. **Snapshot with the dump the bus runs now**, through the machine's backup holder (`/node-backup.now` for `nats` on the bus's machine), and verify the `bus.tar` it wrote with `/nats.nats_snapshot_verify` (it answers the file, its time and size, each stream's last - sequence, and `passed` or the first fault; run it after the dump and read `taken`). Note where it is: that is `snapshot-taken`. -3. **Ask the operator for the moment** on the proven channel, naming what it raises: bus maintenance is - open while the step runs; every bus connection is dropped for seconds and reconnects; conditions of - other modules may flash and clear. Go on only on their yes. -4. **Start it:** `mesh-controller.bus` with `upgrade` true, `why`, `reversible` true and `snapshot-taken`. - `reversible` is true because the server binary (the same digest), the data directory and the - configuration do not change; `TestTheStoreSurvivesTheNewDeclarationTheOldAndTheNewAgain` shows it: - the new declaration, then the old one, then the new again run on one store, each reading what the - one before wrote. -5. **After:** `nats_server`, `nats_streams` (every stream and bucket as before) and the controller's - `doctor`; the step ends done when the controller has checked every stream, every consumer and a - round trip (ADR 0236 section 40). Then wait for the next nightly dump and verify it: the upgrade is - finished only once that dump verified. -6. **Back:** a bus upgrade to the previous build puts the old declaration back on the same store; if the - store itself is hurt, restore the snapshot as described above. + sequence, and `passed` or the first fault; run it after the dump and read `taken`). This is the + known-good snapshot to come back to. +3. **Tell the operator what the step raises** before starting it: bus maintenance is open while the + step runs; every bus connection is dropped for seconds and reconnects; conditions of other modules + may flash and clear. +4. **Start it:** `mesh-delivery-assigner.assigner_put` with `node` (the bus's machine), `module` nats, + `version` (`newest` or the fingerprint `mesh-wants.bus` named) and `why`. The put asks the operator + first, on the proven channel, with the `why`; only on their yes does it go on. After the yes it has + the bus node's backup holder take the bus's snapshot again (it reads the holder's record back for + at most 15 minutes), then writes the composition, the grant and the machine's want, and records the + warrant in the hand-act log. Each step is said as the event `mesh-delivery-assigner.put`; wait on + it, and `assigner_puts` says where the put stands. The step is reversible because the server binary + (the same digest), the data directory and the configuration do not change; + `TestTheStoreSurvivesTheNewDeclarationTheOldAndTheNewAgain` shows it: the new declaration, then the + old one, then the new again run on one store, each reading what the one before wrote. +5. **After:** `nats_server` (the new release) and `nats_streams` (every stream and bucket as before). + The controller's `doctor`, which checked every stream, every consumer and a round trip (ADR 0236 + section 40), ended with the controller, and no verb has taken that check over yet: read + `nats_streams` and `nats_backlog` by hand for it. Then wait for the next nightly dump and verify it: + the upgrade is finished only once that dump verified. +6. **Back:** a put of the previous build (`assigner_put` with its fingerprint) puts the old declaration + back on the same store; if the store itself is hurt, restore the snapshot as described above. ## Tests -- `cmd/mesh-nats-snapshot`: `go test -tags integration -race ./...`, run by `mesh-controller.try` (they start throwaway containers; nobody runs them by hand) — a throwaway `nats:2.11` filled like the +- `cmd/mesh-nats-snapshot`: `go test -tags integration -race ./...`, run by the pull request's check (they start throwaway containers; nobody runs them by hand) — a throwaway `nats:2.11` filled like the mesh's bus (events with a gap deleted, a work queue part worked, buckets with history, deletes and a purge, an object, durable consumers, a memory stream) and written to throughout the snapshot; the snapshot restored into a fresh server and into a new store served by a third, each compared message