diff --git a/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md b/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md index f1b2059..ff8c3ce 100644 --- a/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md +++ b/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md @@ -1,7 +1,7 @@ --- -status: open +status: located opened: 2026-09-29 -located-in: [] +located-in: [mesh-host examples + internal/link, mesh-controller internal/broker] --- # 146 — the foundation cannot be raised on the bus the mesh runs on diff --git a/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/01-diagnosis.md b/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/01-diagnosis.md new file mode 100644 index 0000000..bcbd17b --- /dev/null +++ b/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/01-diagnosis.md @@ -0,0 +1,96 @@ +# Diagnosis + +*2026-09-29, by raising a first node in the lab over and over and writing down each thing it hit.* + +Not one fault. **Four, stacked**, each hidden behind the one before it, and every one of them the +same shape: a step that was right while the mesh ran on the previous broker and was never asked a +question again after the bus changed. Nothing had raised a foundation since, so nothing said so. + +## 1 — the bundle's bus image is named for a registry that is gone *(fixed)* + +The newer bundle pins `/nats@…`, which resolves nowhere outside the lab that +raised that registry. The lab already rewrites the store's and the previous broker's references to +upstream ones for a machine with an uplink; the bus had no such rule because no bed had ever tried +to raise this bundle. Added (`mesh-lab test/integration/harness.ts`). The digest is the bundle's +own — what the registry served was a copy, so the same digest resolves upstream, and this is a +prefix being removed rather than a reference being replaced. + +## 2 — the bus's certificate was made by a tool the bus does not have *(fixed)* + +``` +failed bus-certificate … docker exited 127 +``` + +The step ran `openssl` inside the broker's image. The previous broker's image carried it; the bus's +does not — it is Alpine with a shell and no `openssl` — and neither does any other image the bundle +names, so there was nothing to substitute. **The program that needs the certificate now makes it**: +`mesh-controller broker certificate --into `, with `--check` as the step's verify. The +controller is already on the machine at that point (the schema step ran it) and needs nothing from +the image it writes into. Self-signed, as before and on purpose — a host pins this server's exact +certificate ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) and at that moment +there is no authority to ask. Idempotent, because a second certificate is one every host that +pinned the first no longer believes. It runs `--user 0:0`: the volume is root's, and the control +plane's image runs as nobody, which is right for the long-lived server and wrong for a one-shot +writing into a fresh volume. + +## 3 — enrolment dialled TLS at a bus that speaks first *(fixed)* + +``` +mesh-host: cannot reach the broker at …:5671: tls: first record does not look like a TLS handshake +``` + +Enrolment opened a raw TLS connection to check the pinned certificate before saying anything. NATS +speaks its own protocol and upgrades afterwards, so the handshake met a plaintext greeting. The pin +was never the problem: the same pinned configuration is handed to the client that presents the +token, and the verification runs inside *that* handshake — so what +[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) requires still holds, and holds +better, because the one-time secret is sent only after the certificate has been checked. The raw +dial is gone from the enrolment path and kept only as what its tests always proved: that a wrong +certificate is refused before a byte of application data is sent. + +**Then, immediately behind it:** + +``` +mesh-host: this token is for the "" bus, and the mesh's bus is nats +``` + +The enrolment left the transport empty and meant *whatever the mesh runs today*, which was true +while two buses existed and became a refusal the moment one did. The host knows which bus the mesh +runs; it says so now. + +## 4 — a first node cannot be let onto its own bus *(open, and this is the real one)* + +``` +mesh-host: cannot reach the bus at …:5671 as anchor: nats: Authorization Violation +``` + +The bus's user list is a file beside its configuration. The installer carries the first one — the +controller's own account at a bootstrap password — and **the controller composes every user after +that** (design 25 §6; the controller's own test asserts the carried list matches what it would +derive). On the running mesh that composition reaches the bus because the bus is a *module*, with +the list delivered to it the way anything is delivered to a module. + +At genesis there is no module. The foundation's bus is raised by the installer, the control plane is +given no way to write beside its configuration — it mounts the certificate and nothing else — and so +the account a joining node needs cannot come into existence. **The first node cannot join the mesh +it just raised.** + +That is not a line to fix in a bundle. It is the open half of the mesh delivering its own components +([ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md)) and of the +bus becoming a module: either the installer's bus is raised as the module the mesh will go on +managing, or genesis carries a user list that includes the first node's enrolment and the controller +takes over from there. Both are decisions, not patches, and both belong to the genesis step that was +deliberately left until last. + +## Where it belongs + +`mesh-host` (the bundle and the enrolment path) and `mesh-controller` (the certificate command, and +the composition that cannot reach the bus at genesis). Three of the four are fixed on branches; the +fourth is the genesis work. + +## What it cost, for the next person + +Every lab bed still names `foundation-first-node.lock` in its own instructions, and that bundle +raises the previous broker with a control plane that refuses to start without `MESH_BUS_NATS`. Until +the fourth fault is answered and the two bundles become one, a bed runs with `MESH_LAB_BUNDLE` +pointing at the NATS bundle by hand, and stops at the enrolment.