jschoubben
bffd2af40c
Raising a first node hits them in order: the bundle's bus image named for a registry that is gone; the certificate made by openssl in an image that has none; enrolment dialling TLS at a bus that speaks first, then refusing its own token for the empty bus. Each was right until the bus changed and nothing has raised a foundation since. The fourth is not a patch: the installer carries the bus's first user list and the controller composes the rest through the bus being a module, and at genesis there is no module — so the first node cannot be let onto the bus it just raised.
5.4 KiB
Diagnosis
2026-09-29, by raising a first node in the lab over and over and writing down each thing it hit.
Not one fault. Four, stacked, each hidden behind the one before it, and every one of them the same shape: a step that was right while the mesh ran on the previous broker and was never asked a question again after the bus changed. Nothing had raised a foundation since, so nothing said so.
1 — the bundle's bus image is named for a registry that is gone (fixed)
The newer bundle pins <a lab registry>/nats@…, which resolves nowhere outside the lab that
raised that registry. The lab already rewrites the store's and the previous broker's references to
upstream ones for a machine with an uplink; the bus had no such rule because no bed had ever tried
to raise this bundle. Added (mesh-lab test/integration/harness.ts). The digest is the bundle's
own — what the registry served was a copy, so the same digest resolves upstream, and this is a
prefix being removed rather than a reference being replaced.
2 — the bus's certificate was made by a tool the bus does not have (fixed)
failed bus-certificate … docker exited 127
The step ran openssl inside the broker's image. The previous broker's image carried it; the bus's
does not — it is Alpine with a shell and no openssl — and neither does any other image the bundle
names, so there was nothing to substitute. The program that needs the certificate now makes it:
mesh-controller broker certificate --into <dir>, with --check as the step's verify. The
controller is already on the machine at that point (the schema step ran it) and needs nothing from
the image it writes into. Self-signed, as before and on purpose — a host pins this server's exact
certificate (ADR 0004) and at that moment
there is no authority to ask. Idempotent, because a second certificate is one every host that
pinned the first no longer believes. It runs --user 0:0: the volume is root's, and the control
plane's image runs as nobody, which is right for the long-lived server and wrong for a one-shot
writing into a fresh volume.
3 — enrolment dialled TLS at a bus that speaks first (fixed)
mesh-host: cannot reach the broker at …:5671: tls: first record does not look like a TLS handshake
Enrolment opened a raw TLS connection to check the pinned certificate before saying anything. NATS speaks its own protocol and upgrades afterwards, so the handshake met a plaintext greeting. The pin was never the problem: the same pinned configuration is handed to the client that presents the token, and the verification runs inside that handshake — so what ADR 0004 requires still holds, and holds better, because the one-time secret is sent only after the certificate has been checked. The raw dial is gone from the enrolment path and kept only as what its tests always proved: that a wrong certificate is refused before a byte of application data is sent.
Then, immediately behind it:
mesh-host: this token is for the "" bus, and the mesh's bus is nats
The enrolment left the transport empty and meant whatever the mesh runs today, which was true while two buses existed and became a refusal the moment one did. The host knows which bus the mesh runs; it says so now.
4 — a first node cannot be let onto its own bus (open, and this is the real one)
mesh-host: cannot reach the bus at …:5671 as anchor: nats: Authorization Violation
The bus's user list is a file beside its configuration. The installer carries the first one — the controller's own account at a bootstrap password — and the controller composes every user after that (design 25 §6; the controller's own test asserts the carried list matches what it would derive). On the running mesh that composition reaches the bus because the bus is a module, with the list delivered to it the way anything is delivered to a module.
At genesis there is no module. The foundation's bus is raised by the installer, the control plane is given no way to write beside its configuration — it mounts the certificate and nothing else — and so the account a joining node needs cannot come into existence. The first node cannot join the mesh it just raised.
That is not a line to fix in a bundle. It is the open half of the mesh delivering its own components (ADR 0142) and of the bus becoming a module: either the installer's bus is raised as the module the mesh will go on managing, or genesis carries a user list that includes the first node's enrolment and the controller takes over from there. Both are decisions, not patches, and both belong to the genesis step that was deliberately left until last.
Where it belongs
mesh-host (the bundle and the enrolment path) and mesh-controller (the certificate command, and
the composition that cannot reach the bus at genesis). Three of the four are fixed on branches; the
fourth is the genesis work.
What it cost, for the next person
Every lab bed still names foundation-first-node.lock in its own instructions, and that bundle
raises the previous broker with a control plane that refuses to start without MESH_BUS_NATS. Until
the fourth fault is answered and the two bundles become one, a bed runs with MESH_LAB_BUNDLE
pointing at the NATS bundle by hand, and stops at the enrolment.