jschoubben
47909c6b71
Three issues named the branch that fixed them, and a branch is deleted when it merges — so every `fixed-by:` was a pointer that resolved to nothing by the time anyone followed it. They name commits and pull requests now, and playbook 03 says to. Two records were missing the thing a reader arrives for. 146 did not say that one of its fixes crash-looped the control plane on a running mesh, which is the whole reason the delivery subject carries the stream and the raise path was the only one exercised. 151 did not say that 152 removed the false reasons its roster moved, or that it stays open for the real ones. ADR 0080 enumerates what cycle.py enforces and named four things; it enforces five. A progressive insight names the fifth — the decision stands, the list had gone stale. The checks README and playbook 03 gained the same rule, and 155 points at all three.
4.5 KiB
status, opened, located-in
| status | opened | located-in | ||
|---|---|---|---|---|
| located | 2026-09-29 |
|
146 — the foundation cannot be raised on the bus the mesh runs on
What was observed
Raising a first node in the lab, to check a module against a real mesh, fails before any module is reached. Two separate faults, in the two bundles that exist:
The older bundle raises a control plane that cannot start. It brings up the previous broker, and the control plane it then starts says, once every few seconds, for ever:
mesh-controller: this control plane has no MESH_BUS_NATS, so it cannot reach the mesh's bus
That is the control plane being right. The mesh moved to one bus (ADR 0131) and the bundle did not. Every bed that raises a foundation raises this one, so every bed is in this state.
The newer bundle, written for the new bus, stops one step earlier. Its certificate step asks a container to make the broker's certificate:
docker run --rm --entrypoint sh -v <the broker's tls volume>:/tls <the bus image> \
-c "test -f /tls/tls.crt || (openssl req -x509 ... )"
...
failed bus-certificate: running the action: docker exited 127
127 is command not found. The bus's image has a shell and no openssl; the previous broker's
image had both, which is why the step worked when it was written against that one. Substituting the
store's image — the only other image the bundle carries — does not help: it has no openssl
either. So the step as written cannot succeed with anything the bundle names, and the fault is not
one image's: the bundle asks for a certificate to be made by a tool it never says must be there.
Measured 2026-09-29 on a fresh lab machine, both bundles, from bare.
Why this is here and not a note in the knowledge base
The mesh's own foundation is the one thing it cannot raise. Nothing reports that: the bundles are files in a repository, nothing applies them but a person raising a node, and the last thing that did was the hand-driven cut-over (ADR 0131, whose work was done on the machines rather than from a bundle). So the state where the mesh cannot make another one of itself is reachable, and was reached, without anything saying so.
It is also load-bearing for everything else: a lab bed proves a claim by raising a mesh, so while this holds, no bed can run, and every "checked in the lab" written from now on is a promise against a suite nobody can execute.
What would have prevented it
- Something raising the foundation on a schedule, from the bundle, as it is written — the bundle is the mesh's own installer and nothing installs from it. A bed that raises a first node is exactly that check, and it is the bed that cannot run.
- A step naming what it needs. The certificate step names an image and assumes a program inside it. An action that said which tool it requires would have failed at the declaration rather than at 127 on a machine.
Evidence to carry into diagnosis
mesh-host examples/foundation-first-node.lock— the previous broker, noMESH_BUS_NATS.mesh-host examples/foundation-first-node-nats.lock— the new bus;bus-certificateand itsverifyboth runopensslin the bus's image.- The bus image the bundle pins has
shand noopenssl; the store's image likewise. - The lab rewrites a bundle's registry-prefixed third-party references to upstream ones for a
machine with an uplink (
test/integration/harness.ts); the new bundle's bus reference needed that rule added, which is done and is not this issue.
What one of its fixes then did to a running mesh (2026-09-30)
The change that stopped the doubling — putting the stream into a push consumer's delivery subject — is correct on a foundation being raised and fatal on a mesh that is already running: the server will not move that subject while a subscriber is bound, and a node is bound to its declaration consumer the whole time it is up. The control plane crash-looped on the first build that carried it.
Recorded and fixed as issue 156. Noted here because this record is where somebody will arrive when reading why the subject carries the stream at all, and the answer is incomplete without it: the raise path was the only one exercised, and it is the one path on which nothing is bound.