Adopt genesis's bus through the bus's planned step, its streams snapshotted first (issue 590, hq ADR 0333 §3) #108

2026-10-13T09:47:27Z

mesh-admin commented

2026-10-13 09:47:27 +00:00

Issue 590; decisions: hq ADR 0333 §3 (adoption replaces the bus's image once), ADR 0236 (the bus is replaced only as a planned step).

Seen: bed run 11:15 to 11:32, bed-590-5.log l.308. The broker step sent the nats module with a plain push anchor, and the controller refused it: "sending anchor would replace the bus (nats (not kno → 53e51864), which is a planned step: bus upgrade --why … snapshots its streams first and checks them after". Genesis stopped at step 18.

Fix: the step registers, builds, issues, sets the port and assigns as before. Then it sends through the bus's planned step (adoptTheBus, bus_adoption.go):

It snapshots every stream with the module's own snapshot program (mesh-nats-snapshot, ADR 0235), run from the image the build just made (the builds nats -n 1 reference, through this mesh's registry). The program connects as the controller's account, pinned to the bus's certificate fingerprint. The credential is written 0600 for that one run and removed. The snapshot goes to /var/lib/mesh-genesis/bus-before-adoption.tar. At genesis no backup holder exists for bus upgrade to ask, so the snapshot is handed over with --snapshot-taken.
It asks bus upgrade --why "genesis adopts its bus as the nats module (ADR 0333)" --reversible --snapshot-taken <file>. It says the restart, waits until mesh-broker-nats runs the module's image and /healthz answers, then waits until mesh-controller bus says the step ended done. The controller's doctor ends it done only once every stream, every durable consumer and a round trip answer (H-bus). A failed end is refused with the controller's words.
A re-run that finds the bus on the module's image sends it with a plain push, which the controller does not hold, because nothing moves.
The bundle's bus image moves from upstream nats 2.10.29 (b83efabe, behind the lab prefix) to 2.11.17 by the digest the module's image is built on (NATS_BASE e4bf19…). So the adoption changes no server release and is truly --reversible; it was 2.10 → 2.11 before, which ADR 0236 calls irreversible. The bundle names it upstream, as the store, so the lab's rewrite leaves it as it is.

Tests (fail on main):

TestTheBrokerStepAdoptsTheBusThroughItsPlannedStep: snapshot before upgrade, the image, server, credential and pin, no plain push, the credential removed, and the five lines said. With the step reverted to a push it fails.
TestABusStepThatEndsFailedIsRefused.
TestTheBundlesBusRunsTheReleaseTheModulesImageIsBuiltOn: reads the catalogue, and fails with main's bundle.
The phase-3 fakes answer builds, bus and the snapshot.
gofmt, vet and go test ./... pass, niced and without -race.

"not kno": cut short by the controller, not by the log. busHeld passes orNotKnown(...) through short(), which keeps 8 characters, so (not known) prints as (not kno. That is a cosmetic mesh-controller bug in cmd/mesh-controller/bus_step.go, not fixed here.

Not proven yet: whether the controller's account may take the stream snapshots (it acks chunks under allow_responses). The bed will show it.

Fixes issue 590 (bed proof outstanding).

Issue 590; decisions: hq ADR 0333 §3 (adoption replaces the bus's image once), ADR 0236 (the bus is replaced only as a planned step). **Seen**: bed run 11:15 to 11:32, bed-590-5.log l.308. The broker step sent the nats module with a plain `push anchor`, and the controller refused it: "sending anchor would replace the bus (nats (not kno → 53e51864), which is a planned step: `bus upgrade --why …` snapshots its streams first and checks them after". Genesis stopped at step 18. **Fix**: the step registers, builds, issues, sets the port and assigns as before. Then it sends through the bus's planned step (`adoptTheBus`, bus_adoption.go): - It snapshots every stream with the module's own snapshot program (`mesh-nats-snapshot`, ADR 0235), run from the image the build just made (the `builds nats -n 1` reference, through this mesh's registry). The program connects as the controller's account, pinned to the bus's certificate fingerprint. The credential is written 0600 for that one run and removed. The snapshot goes to /var/lib/mesh-genesis/bus-before-adoption.tar. At genesis no backup holder exists for `bus upgrade` to ask, so the snapshot is handed over with `--snapshot-taken`. - It asks `bus upgrade --why "genesis adopts its bus as the nats module (ADR 0333)" --reversible --snapshot-taken <file>`. It says the restart, waits until `mesh-broker-nats` runs the module's image and `/healthz` answers, then waits until `mesh-controller bus` says the step ended `done`. The controller's doctor ends it done only once every stream, every durable consumer and a round trip answer (H-bus). A `failed` end is refused with the controller's words. - A re-run that finds the bus on the module's image sends it with a plain push, which the controller does not hold, because nothing moves. - The bundle's bus image moves from upstream nats 2.10.29 (b83efabe, behind the lab prefix) to 2.11.17 by the digest the module's image is built on (`NATS_BASE` e4bf19…). So the adoption changes no server release and is truly `--reversible`; it was 2.10 → 2.11 before, which ADR 0236 calls irreversible. The bundle names it upstream, as the store, so the lab's rewrite leaves it as it is. **Tests (fail on main)**: - `TestTheBrokerStepAdoptsTheBusThroughItsPlannedStep`: snapshot before upgrade, the image, server, credential and pin, no plain push, the credential removed, and the five lines said. With the step reverted to a push it fails. - `TestABusStepThatEndsFailedIsRefused`. - `TestTheBundlesBusRunsTheReleaseTheModulesImageIsBuiltOn`: reads the catalogue, and fails with main's bundle. - The phase-3 fakes answer `builds`, `bus` and the snapshot. - gofmt, vet and `go test ./...` pass, niced and without -race. **"not kno"**: cut short by the controller, not by the log. `busHeld` passes `orNotKnown(...)` through `short()`, which keeps 8 characters, so `(not known)` prints as `(not kno`. That is a cosmetic mesh-controller bug in cmd/mesh-controller/bus_step.go, not fixed here. **Not proven yet**: whether the controller's account may take the stream snapshots (it acks chunks under `allow_responses`). The bed will show it. Fixes issue 590 (bed proof outstanding).

mesh-admin added 1 commit 2026-10-13 09:47:27 +00:00

Adopt genesis's bus through the bus's planned step, its streams snapshotted first (hq issue 590, ADR 0333 §3)

mesh/merge-gate head cdcfcd938016 merged into trunk 130c5d98a601 — pass: the change touches no module of the mesh's graph

Details

mesh/repo-check head cdcfcd938016 merged into trunk 130c5d98a601 — pass: its merge-check.sh passed

Details

mesh/delivery delivered

Details

cdcfcd9380

The broker step sent the nats module with a plain push, and the controller refuses a push that would
replace the bus: that is a planned step, bus upgrade, which snapshots the streams first and checks
them after (ADR 0236). Genesis stopped at step 18.

The step now snapshots every stream with the module's own snapshot program (no backup holder exists
yet to be asked), asks bus upgrade with that snapshot and why, says the one restart, and waits until
the bus runs the module's image, answers, and the controller ends the step done. The bundle's bus runs
the release the module's image is built on (2.11.17, by the same digest, where it ran 2.10.29), so the
step changes no server release and is said reversible.

mesh-admin commented

2026-10-13 09:48:05 +00:00

Delivery novox/mesh-host@cdcfcd938016 — delivered since 2026-10-13T11:12:12Z

Delivery plan — builds nothing: the change touches no module of the mesh's graph

Transitions

2026-10-13 09:47 (new) → proposed (announced): novox/mesh-host#108's head announced
2026-10-13 09:49 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-13 09:49 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-13 11:12 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-13 11:12 published → delivered (done): nothing for a walk to move

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@cdcfcd938016` — **delivered** since 2026-10-13T11:12:12Z **Delivery plan** — builds nothing: the change touches no module of the mesh's graph **Transitions** - 2026-10-13 09:47 (new) → proposed (announced): novox/mesh-host#108's head announced - 2026-10-13 09:49 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-13 09:49 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-13 11:12 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-13 11:12 published → delivered (done): nothing for a walk to move The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin reviewed

2026-10-13 11:11:17 +00:00

mesh-admin left a comment

Verdict: approve at cdcfcd9380

Wait bounds and loud failure (check 1). The sequence is adoptTheBus (snapshot, bounded by busStepWithin = 17m) → the "bus upgrade --why … --reversible --snapshot-taken " tell call (bounded by 2o.Wait, 6m by default) → waitForTheAdoptedBus (pre-existing, bounded by o.Wait, 3m) → waitForTheBusStep (bounded by busStepWithin = 17m, polling mesh-controller bus). I read mesh-controller's cmd/mesh-controller/bus_step.go: busCommand's "upgrade" subcommand does not block for the bus to become healthy — with --snapshot-taken given, it skips taking its own snapshot, starts the bus step record, calls sendRollout, prints the one-line notice ("sent …; bus-maintenance is open until the bus answers healthy again … within 15m0s, or the step is said failed with its snapshot as the way back") and returns. The actual judging happens asynchronously in probeBusStep (busStepBound = 15time.Minute), which waitForTheBusStep polls for with its own 17-minute bound (15m plus "its doctor's next look after that", matching ADR 0236 clause 40). So the mesh-host side's 2*o.Wait budget for the tell call is generous, not short, and every stage is bounded with a specific, descriptive error on failure (the credential missing, the certificate missing, the snapshot script's stdout on failure, "the bus's planned step was refused, so the bus is not replaced", or no bus step found at all). This matches ADR 0236 and ADR 0333 §3.

Snapshot-first and restore path (check 2). adoptTheBus calls snapshotTheBus (which writes busSnapshotFile before touching anything) before it ever calls "bus upgrade"; the already-adopted branch (bus already on the module's image) takes no snapshot and sends a plain push, correctly, since nothing changes. On a failed step, waitForTheBusStep returns the controller's own "bus" output verbatim, which (per bus_step.go's busStatus and probeBusStep) includes "last step N: … failed at …; snapshot " and, for the urgent condition, "the way back is to put the old build back ()" — so a failed step does say how to restore, in the controller's own words, not paraphrased.

nats version bump (check 3). The lock file changes the bundle's bus image from the upstream 2.10.29 digest (sha256:b83efabe…) to sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00. I read mesh-catalog's modules/nats/module.json: its build.on NATS_BASE is exactly that same digest, so the bundle's post-adoption image is byte-identical to what the nats module is built on — "it is what the module runs" holds, and the PR's claim that the step changes no server release (hence --reversible) checks out independently. This image swap happens only at genesis, once, on a bus no running mesh is on yet (ADR 0333 clause 1: every running mesh already runs the module's layout), so no existing bundle's user is exposed to a live 2.10→2.11 migration by this change.

Tests and checks (check 4). On cdcfcd9, gofmt -l . printed nothing, nice -n 15 env GOFLAGS=-p=2 GOMAXPROCS=4 go vet ./... printed nothing, and nice -n 15 env GOFLAGS=-p=2 GOMAXPROCS=4 go test ./... printed "ok" for all 23 test-bearing packages (cmd/mesh-bootstrap, cmd/mesh-host, internal/bootstrap, internal/witness, etc.), none skipped or failed. I ran these instead of merge-check.sh verbatim because gcc is present on this machine and the script would then run go test -race, which I was told to avoid; gofmt+vet+go test ./... is exactly what the script does on its non-race branch. To check the three named tests fail on main, I copied bus_adoption_test.go, phase3_test.go and registry_test.go onto a worktree of 130c5d9 (main) unchanged: the package fails to build ("undefined: busSnapshotCredential", "undefined: busSnapshotFile", "undefined: adoptionWhy", "undefined: certificateFingerprint", "undefined: waitForTheBusStep", …) — the tests exercise symbols that exist only in this PR's bus_adoption.go, so they fail on main as the PR claims.

Controller account's authority to snapshot (check 5, the PR's own "not proven yet"). I read mesh-catalog modules/nats/snapshot/{snapshot,connect}.go and mesh-controller internal/broker/nats.go and cmd/mesh-controller/bus_step.go. Two different mechanisms exist side by side:

The ordinary (post-genesis) bus-upgrade path never uses the controller's own account for this: bus upgrade asks the bus machine's node-backup seat holder to back the nats module up now (snapshotTheBusNow, via link.AskSeatTool to catalogue.BackupSeat), whose dump runs mesh-nats-snapshot inside/beside the module's own container, authenticated with the module's own secret (module.json's own-secrets.broker) — the dedicated principal nats.go calls SnapshotsTheBus, whose "whole authority is BusSnapshotGrants": Publish {$JS.API.STREAM.NAMES, $JS.API.STREAM.INFO., $JS.API.STREAM.SNAPSHOT., $JS.SNAPSHOT.ACK.>} and Subscribe {its own inbox only} — no AllowResponses.
Genesis's new code instead has the installer itself run mesh-nats-snapshot from outside the container, authenticated as busControllerUser ("controller"), because "at genesis no backup holder exists for bus upgrade to ask" (true, and reasonably handled with --snapshot-taken). The controller's own grant (KindController in PermissionsFor) is Publish {…, "$JS.API.>", …} (which covers STREAM.NAMES/INFO/SNAPSHOT, so the API calls go through), Subscribe {its own inbox, "_INBOX.controller.>"} (which mesh-catalog's connect.go matches via nats.CustomInboxPrefix("_INBOX."+c.User), so the deliver subject the program subscribes to is one the controller may legally hold), and AllowResponses: true. The controller's static Publish list does not include $JS.SNAPSHOT.ACK.>, so acking each chunk (snapshot.go's snapshotStream publishes to msg.Reply, the server's $JS.SNAPSHOT.ACK.. subject) depends entirely on AllowResponses granting a temporary reply permission for a message received with a Reply field on a subject it may subscribe to — which is the standard NATS responder pattern and, read against the server's model, should cover it. But I found no test anywhere in mesh-host or mesh-controller that exercises the controller account against a real server for this; mesh-controller's own snapshot_test.go holds BusSnapshotGrants to its purpose-built list, not to what KindController gets. That the dedicated SnapshotsTheBus principal was given an explicit static $JS.SNAPSHOT.ACK.> grant rather than AllowResponses, for the identical ack, is itself a sign the design did not treat AllowResponses as the obvious way to cover this subject. This is exactly what the PR's own body says: "Not proven yet: whether the controller's account may take the stream snapshots (it acks chunks under allow_responses). The bed will show it" — and the PR says "Fixes issue 590 (bed proof outstanding)", not that it is fully proven. My own reading agrees it is plausible but unconfirmed, and I recommend the mesh-lab bed (ADR 0333 clause 5) actually run and its snapshot step be checked (a stream present before the step still present after) before this is treated as settled — a live regression test of the controller account against a real nats-server, mirroring merge-check.sh's optional MESH_TEST_NATS block, would close this permanently rather than leaving it to the bed alone.

Nothing else blocks this: the diff is small and scoped to the broker step, the "already adopted" re-run path is unchanged in behaviour and still tested, the credential file is written 0600 and always removed (defer), and the lock file's comment correctly explains the one-time image swap. I approve at cdcfcd9380, with the controller-account authority question (above) flagged as the one real open risk, which the PR itself already discloses and defers to the bed.

Verdict: approve at cdcfcd938016324a3fe774db05c64137237afcf3 Wait bounds and loud failure (check 1). The sequence is adoptTheBus (snapshot, bounded by busStepWithin = 17m) → the "bus upgrade --why … --reversible --snapshot-taken <file>" tell call (bounded by 2*o.Wait, 6m by default) → waitForTheAdoptedBus (pre-existing, bounded by o.Wait, 3m) → waitForTheBusStep (bounded by busStepWithin = 17m, polling `mesh-controller bus`). I read mesh-controller's cmd/mesh-controller/bus_step.go: busCommand's "upgrade" subcommand does not block for the bus to become healthy — with --snapshot-taken given, it skips taking its own snapshot, starts the bus step record, calls sendRollout, prints the one-line notice ("sent …; `bus-maintenance` is open until the bus answers healthy again … within 15m0s, or the step is said failed with its snapshot as the way back") and returns. The actual judging happens asynchronously in probeBusStep (busStepBound = 15*time.Minute), which waitForTheBusStep polls for with its own 17-minute bound (15m plus "its doctor's next look after that", matching ADR 0236 clause 40). So the mesh-host side's 2*o.Wait budget for the tell call is generous, not short, and every stage is bounded with a specific, descriptive error on failure (the credential missing, the certificate missing, the snapshot script's stdout on failure, "the bus's planned step was refused, so the bus is not replaced", or no bus step found at all). This matches ADR 0236 and ADR 0333 §3. Snapshot-first and restore path (check 2). adoptTheBus calls snapshotTheBus (which writes busSnapshotFile before touching anything) before it ever calls "bus upgrade"; the already-adopted branch (bus already on the module's image) takes no snapshot and sends a plain push, correctly, since nothing changes. On a failed step, waitForTheBusStep returns the controller's own "bus" output verbatim, which (per bus_step.go's busStatus and probeBusStep) includes "last step N: … failed at …; snapshot <path>" and, for the urgent condition, "the way back is to put the old build back (<snapshot path>)" — so a failed step does say how to restore, in the controller's own words, not paraphrased. nats version bump (check 3). The lock file changes the bundle's bus image from the upstream 2.10.29 digest (sha256:b83efabe…) to sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00. I read mesh-catalog's modules/nats/module.json: its build.on NATS_BASE is exactly that same digest, so the bundle's post-adoption image is byte-identical to what the nats module is built on — "it is what the module runs" holds, and the PR's claim that the step changes no server release (hence --reversible) checks out independently. This image swap happens only at genesis, once, on a bus no running mesh is on yet (ADR 0333 clause 1: every running mesh already runs the module's layout), so no existing bundle's user is exposed to a live 2.10→2.11 migration by this change. Tests and checks (check 4). On cdcfcd9, `gofmt -l .` printed nothing, `nice -n 15 env GOFLAGS=-p=2 GOMAXPROCS=4 go vet ./...` printed nothing, and `nice -n 15 env GOFLAGS=-p=2 GOMAXPROCS=4 go test ./...` printed "ok" for all 23 test-bearing packages (cmd/mesh-bootstrap, cmd/mesh-host, internal/bootstrap, internal/witness, etc.), none skipped or failed. I ran these instead of merge-check.sh verbatim because gcc is present on this machine and the script would then run `go test -race`, which I was told to avoid; gofmt+vet+go test ./... is exactly what the script does on its non-race branch. To check the three named tests fail on main, I copied bus_adoption_test.go, phase3_test.go and registry_test.go onto a worktree of 130c5d9 (main) unchanged: the package fails to build ("undefined: busSnapshotCredential", "undefined: busSnapshotFile", "undefined: adoptionWhy", "undefined: certificateFingerprint", "undefined: waitForTheBusStep", …) — the tests exercise symbols that exist only in this PR's bus_adoption.go, so they fail on main as the PR claims. Controller account's authority to snapshot (check 5, the PR's own "not proven yet"). I read mesh-catalog modules/nats/snapshot/{snapshot,connect}.go and mesh-controller internal/broker/nats.go and cmd/mesh-controller/bus_step.go. Two different mechanisms exist side by side: - The ordinary (post-genesis) bus-upgrade path never uses the controller's own account for this: `bus upgrade` asks the bus machine's node-backup seat holder to back the nats module up now (snapshotTheBusNow, via link.AskSeatTool to catalogue.BackupSeat), whose dump runs mesh-nats-snapshot inside/beside the module's own container, authenticated with the module's own secret (module.json's own-secrets.broker) — the dedicated principal nats.go calls SnapshotsTheBus, whose "whole authority is BusSnapshotGrants": Publish {$JS.API.STREAM.NAMES, $JS.API.STREAM.INFO.*, $JS.API.STREAM.SNAPSHOT.*, $JS.SNAPSHOT.ACK.>} and Subscribe {its own inbox only} — no AllowResponses. - Genesis's new code instead has the installer itself run mesh-nats-snapshot from outside the container, authenticated as busControllerUser ("controller"), because "at genesis no backup holder exists for `bus upgrade` to ask" (true, and reasonably handled with --snapshot-taken). The controller's own grant (KindController in PermissionsFor) is Publish {…, "$JS.API.>", …} (which covers STREAM.NAMES/INFO/SNAPSHOT, so the API calls go through), Subscribe {its own inbox, "_INBOX.controller.>"} (which mesh-catalog's connect.go matches via nats.CustomInboxPrefix("_INBOX."+c.User), so the deliver subject the program subscribes to is one the controller may legally hold), and AllowResponses: true. The controller's static Publish list does not include $JS.SNAPSHOT.ACK.>, so acking each chunk (snapshot.go's snapshotStream publishes to msg.Reply, the server's $JS.SNAPSHOT.ACK.<stream>.<id> subject) depends entirely on AllowResponses granting a temporary reply permission for a message received with a Reply field on a subject it may subscribe to — which is the standard NATS responder pattern and, read against the server's model, should cover it. But I found no test anywhere in mesh-host or mesh-controller that exercises the controller account against a real server for this; mesh-controller's own snapshot_test.go holds BusSnapshotGrants to its purpose-built list, not to what KindController gets. That the dedicated SnapshotsTheBus principal was given an explicit static $JS.SNAPSHOT.ACK.> grant rather than AllowResponses, for the identical ack, is itself a sign the design did not treat AllowResponses as the obvious way to cover this subject. This is exactly what the PR's own body says: "Not proven yet: whether the controller's account may take the stream snapshots (it acks chunks under allow_responses). The bed will show it" — and the PR says "Fixes issue 590 (bed proof outstanding)", not that it is fully proven. My own reading agrees it is plausible but unconfirmed, and I recommend the mesh-lab bed (ADR 0333 clause 5) actually run and its snapshot step be checked (a stream present before the step still present after) before this is treated as settled — a live regression test of the controller account against a real nats-server, mirroring merge-check.sh's optional MESH_TEST_NATS block, would close this permanently rather than leaving it to the bed alone. Nothing else blocks this: the diff is small and scoped to the broker step, the "already adopted" re-run path is unchanged in behaviour and still tested, the credential file is written 0600 and always removed (defer), and the lock file's comment correctly explains the one-time image swap. I approve at cdcfcd938016324a3fe774db05c64137237afcf3, with the controller-account authority question (above) flagged as the one real open risk, which the PR itself already discloses and defers to the bed.

mesh-admin merged commit 12bc6a56a3 into main

2026-10-13 11:11:41 +00:00

mesh-admin deleted branch fix/590-adopt-the-bus-as-its-planned-step

2026-10-13 11:11:41 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#108