Place the bus's users after every push until the bus is adopted (issue 608) #103

2026-10-13T04:58:50Z

mesh-admin commented

2026-10-13 04:58:50 +00:00

Issue 608 (genesis stops at step 15, sdk: the build agent cannot reach the bus). Decisions: hq ADR 0331 §5, ADR 0333 §4.

Cause: until the broker step adopts the bus as the nats module, only the installer writes the bus's user list. placeTheBusUsers ran only in Enrol, at step 7. Step 13 minted the build agent's account and sealed it to the node with its push, but the list was never placed again. The agent restart-looped on nats: Authorization Violation, and the sdk step's build --wait waited until the installer's context killed the docker exec. That is the docker exited -1 and 137 (seen in the coordinator's diag run, 06:57).

Fix: pushNode places the composed list (placeUntilAdopted) after every successful push, so the bus re-reads it. Once nats is registered it places nothing, because the module writes the list from then on. deliverGenesisRegistry's push at step 14 now goes through pushNode as well.

Tests (two fail on main):

TestTheBusHearsTheBuildAgentsAccountAfterItsPush and TestTheGenesisDocumentsPushIsHeardByTheBusToo: the list is placed once after the push, at the module's bus-users path, with kill -s HUP mesh-broker-nats.
TestOnceTheBusIsTheNatsModulesThePushPlacesNothing (also holds on main).
The shared fakes answer broker accounts and the placing shell.
merge-check.sh passes. There is no nats-server here, so the bus tests are skipped, and the script says so.

The genesis document's absence: the build agent's manifest declares it at ${dir:mesh-state}/genesis-package-registry.json. The check that found no file looked for /var/lib/mesh/*/genesis-package-registry, without .json, so it would not have matched. I think it is not a second cause, but that is not proven. The next run should list /var/lib/mesh/build-agent/.

Fixes issue 608 (lab proof outstanding: genesis-keeps-its-bus bed from bare).

Issue 608 (genesis stops at step 15, sdk: the build agent cannot reach the bus). Decisions: hq ADR 0331 §5, ADR 0333 §4. **Cause**: until the broker step adopts the bus as the nats module, only the installer writes the bus's user list. `placeTheBusUsers` ran only in `Enrol`, at step 7. Step 13 minted the build agent's account and sealed it to the node with its push, but the list was never placed again. The agent restart-looped on `nats: Authorization Violation`, and the sdk step's `build --wait` waited until the installer's context killed the `docker exec`. That is the `docker exited -1` and `137` (seen in the coordinator's diag run, 06:57). **Fix**: `pushNode` places the composed list (`placeUntilAdopted`) after every successful push, so the bus re-reads it. Once `nats` is registered it places nothing, because the module writes the list from then on. `deliverGenesisRegistry`'s push at step 14 now goes through `pushNode` as well. **Tests (two fail on main)**: - `TestTheBusHearsTheBuildAgentsAccountAfterItsPush` and `TestTheGenesisDocumentsPushIsHeardByTheBusToo`: the list is placed once after the push, at the module's bus-users path, with `kill -s HUP mesh-broker-nats`. - `TestOnceTheBusIsTheNatsModulesThePushPlacesNothing` (also holds on main). - The shared fakes answer `broker accounts` and the placing shell. - merge-check.sh passes. There is no nats-server here, so the bus tests are skipped, and the script says so. **The genesis document's absence**: the build agent's manifest declares it at `${dir:mesh-state}/genesis-package-registry.json`. The check that found no file looked for `/var/lib/mesh/*/genesis-package-registry`, without `.json`, so it would not have matched. I think it is not a second cause, but that is not proven. The next run should list `/var/lib/mesh/build-agent/`. Fixes issue 608 (lab proof outstanding: genesis-keeps-its-bus bed from bare).

mesh-admin added 1 commit 2026-10-13 04:58:50 +00:00

Place the bus's users after every push until the bus is adopted (hq issue 608)

mesh/merge-gate head 7283209d1abc merged into trunk 64f0ab405101 — pass: the change touches no module of the mesh's graph

Details

mesh/repo-check head 7283209d1abc merged into trunk 64f0ab405101 — pass: its merge-check.sh passed

Details

mesh/delivery delivered

Details

7283209d1a

The bus reads its user list from a file that, until the broker step adopts it as the nats module,
only the installer writes, and the installer wrote it only at enrolment. The build agent's account,
minted with step 13's push, never reached the bus; the agent was refused (Authorization Violation)
and the sdk step's build waited until it was killed. Every push genesis makes now places the composed
list and has the bus re-read it, until the nats module is registered (ADR 0331 §5, ADR 0333 §4).

mesh-admin commented

2026-10-13 04:59:08 +00:00

Delivery novox/mesh-host@7283209d1abc — delivered since 2026-10-13T05:00:48Z

Delivery plan — builds nothing: the change touches no module of the mesh's graph

Transitions

2026-10-13 04:59 (new) → proposed (announced): novox/mesh-host#103's head announced
2026-10-13 05:00 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit
2026-10-13 05:00 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail
2026-10-13 05:00 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move
2026-10-13 05:00 published → delivered (done): nothing for a walk to move

The commit's note under refs/notes/mesh-plan keeps every transition: git log --notes=mesh-plan.

**Delivery** `novox/mesh-host@7283209d1abc` — **delivered** since 2026-10-13T05:00:48Z **Delivery plan** — builds nothing: the change touches no module of the mesh's graph **Transitions** - 2026-10-13 04:59 (new) → proposed (announced): novox/mesh-host#103's head announced - 2026-10-13 05:00 proposed → checked (checked): a verdict names this commit and the trunk's head as its trunk commit - 2026-10-13 05:00 checked → ready (accepted): the gate passed or warned, and the repository's own check did not fail - 2026-10-13 05:00 ready → published (merged): on the trunk its modules follow: merged there, and its walk opened — or nothing for a walk to move - 2026-10-13 05:00 published → delivered (done): nothing for a walk to move The commit's note under `refs/notes/mesh-plan` keeps every transition: `git log --notes=mesh-plan`.

mesh-admin reviewed

2026-10-13 05:00:32 +00:00

mesh-admin left a comment

Verdict: approve at 7283209d1a

Independent review against issue 608, ADR 0331 §5 and ADR 0333 §4, on a clone of the head.

Every push before adoption places the list. In internal/bootstrap the only controller push is control.tell(ctx, "push", o.Node) inside pushNode (module.go:140); the other match for "push" is publish.go:89, which is a docker push of an image to the registry and not a node push. Every caller now goes through pushNode: builder.go:81 (step 13, the build agent), control.go:108, module.go:76, phase2.go:95 and :150, phase3.go:135 (provider installs, including the broker step), and phase_packages.go:418 (step 14, the genesis document, newly routed by this PR). The other verbs the installer tells (assign, take, settings set, overlay place, secret accept, module add/issue) do not push. Enrolment keeps its own placings in enrol.go (lines 101, 123, 153).
Nothing is placed after adoption. placeUntilAdopted asks module list and returns without writing when a line's first word is exactly nats (talk.go mentions, an exact match, so no other module name containing "nats" can trip it). At the broker step installProvider registers nats before its push, so that push and every later one place nothing and the module's own write of bus-users stands, as ADR 0333 §4 says. TestOnceTheBusIsTheNatsModulesThePushPlacesNothing holds this.
The bus re-reads the list. placeTheBusUsers writes /var/lib/nats-module/conf/accounts.conf through a .next file and mv, then runs docker kill -s HUP mesh-broker-nats. The bundle mounts that directory (not the file) at /etc/nats, and its nats.conf ends with include accounts.conf, so the renamed file is visible in the container and nats-server reloads it on SIGHUP. The new tests assert both the module's path and the HUP of the module's container.
The new tests fail on main and pass at the head. With module.go and phase_packages.go reverted to main (64f0ab40) and the head's tests kept, TestTheBusHearsTheBuildAgentsAccountAfterItsPush and TestTheGenesisDocumentsPushIsHeardByTheBusToo fail ("the bus's users were placed 0 time(s) after ... push; once"), and TestOnceTheBusIsTheNatsModulesThePushPlacesNothing passes, as the PR body says. At the head all three pass.
go vet ./internal/bootstrap/ is clean and go test -race -count=1 ./internal/bootstrap/ passes (ok, 3.3s).

Observations, none blocking: the list is placed after the push, so a newly started agent may be refused once before the HUP lands and recover on its restart; placing before the push would avoid that, but the account is minted by assign, so after the push is correct and enough. A failure of module list inside placeUntilAdopted returns the bare error without saying it was the bus-users step; a sentence of context would help the operator. The lab proof from bare (the genesis-keeps-its-bus bed past step 15) is still outstanding, as the PR says.

Statuses at the time of review: mesh/merge-gate pending ("building on g14 since 06:59"), mesh/delivery pending ("checking"), and no mesh/repo-check status reported yet. This approval does not stand in for those checks; merge only once both required checks are green and read.

Verdict: approve at 7283209d1abc7d00c568f437c6c6b669ee2a50d5 Independent review against issue 608, ADR 0331 §5 and ADR 0333 §4, on a clone of the head. 1. Every push before adoption places the list. In internal/bootstrap the only controller push is `control.tell(ctx, "push", o.Node)` inside pushNode (module.go:140); the other match for "push" is publish.go:89, which is a `docker push` of an image to the registry and not a node push. Every caller now goes through pushNode: builder.go:81 (step 13, the build agent), control.go:108, module.go:76, phase2.go:95 and :150, phase3.go:135 (provider installs, including the broker step), and phase_packages.go:418 (step 14, the genesis document, newly routed by this PR). The other verbs the installer tells (assign, take, settings set, overlay place, secret accept, module add/issue) do not push. Enrolment keeps its own placings in enrol.go (lines 101, 123, 153). 2. Nothing is placed after adoption. placeUntilAdopted asks `module list` and returns without writing when a line's first word is exactly `nats` (talk.go mentions, an exact match, so no other module name containing "nats" can trip it). At the broker step installProvider registers nats before its push, so that push and every later one place nothing and the module's own write of bus-users stands, as ADR 0333 §4 says. TestOnceTheBusIsTheNatsModulesThePushPlacesNothing holds this. 3. The bus re-reads the list. placeTheBusUsers writes `/var/lib/nats-module/conf/accounts.conf` through a `.next` file and `mv`, then runs `docker kill -s HUP mesh-broker-nats`. The bundle mounts that directory (not the file) at /etc/nats, and its nats.conf ends with `include accounts.conf`, so the renamed file is visible in the container and nats-server reloads it on SIGHUP. The new tests assert both the module's path and the HUP of the module's container. 4. The new tests fail on main and pass at the head. With module.go and phase_packages.go reverted to main (64f0ab40) and the head's tests kept, TestTheBusHearsTheBuildAgentsAccountAfterItsPush and TestTheGenesisDocumentsPushIsHeardByTheBusToo fail ("the bus's users were placed 0 time(s) after ... push; once"), and TestOnceTheBusIsTheNatsModulesThePushPlacesNothing passes, as the PR body says. At the head all three pass. 5. `go vet ./internal/bootstrap/` is clean and `go test -race -count=1 ./internal/bootstrap/` passes (ok, 3.3s). Observations, none blocking: the list is placed after the push, so a newly started agent may be refused once before the HUP lands and recover on its restart; placing before the push would avoid that, but the account is minted by assign, so after the push is correct and enough. A failure of `module list` inside placeUntilAdopted returns the bare error without saying it was the bus-users step; a sentence of context would help the operator. The lab proof from bare (the genesis-keeps-its-bus bed past step 15) is still outstanding, as the PR says. Statuses at the time of review: mesh/merge-gate pending ("building on g14 since 06:59"), mesh/delivery pending ("checking"), and no mesh/repo-check status reported yet. This approval does not stand in for those checks; merge only once both required checks are green and read.

mesh-admin merged commit 7d1d6bffe3 into main

2026-10-13 05:00:43 +00:00

mesh-admin deleted branch fix/608-place-the-bus-users-after-every-push

2026-10-13 05:00:43 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-host#103