diff --git a/02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md b/02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md new file mode 100644 index 0000000..e0e19a0 --- /dev/null +++ b/02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md @@ -0,0 +1,139 @@ +--- +topic: what runs on it +status: accepted +date: 2026-09-29 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md +--- + +# 147. A module anchors the mesh's authority on a machine, and takes it away again + +## Context + +The mesh runs its own certificate authority and every internal name is served with a certificate +from it. No machine trusts it. On an enrolled, adopted workstation — on the private network, +resolving through the mesh's resolver — every internal HTTPS name fails verification with +*unable to get local issuer certificate* +([issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md)). +The certificates are genuine; nothing on the machine has ever been told what issued them. + +The authority's only consumer today is a proxy, which fetches the root into a directory of its own +and hands it to one program ([ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md)). +That is enough for the proxy and for nothing else: a browser, `git` over HTTPS, `curl`, a package +manager and every module that calls another module by an internal name read the machine's trust +store, which holds the predecessor's authority and a developer tool's local root, and nothing of +the mesh's. + +The predecessor wrote its root into every machine it set up. Removing it was deliberate — an +honest failure beats a name that verifies for the wrong reason — and it leaves the mesh with no +answer at all until this one lands. It is also what keeps the predecessor alive on the machines +that still speak TLS to a mesh name. + +**What makes this a decision rather than a patch** is where the knowledge goes. Two mechanisms in +the mesh already write things onto a machine because it is on the private network: `/etc/hosts` +and the registry's plaintext trust ([ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)). +Following that precedent, the controller would inject an anchor into every such machine's +declaration, and issue 129 proposed exactly that. It would work. It would also put *where this +operating system keeps trust anchors* and *which command refreshes its bundles* into the control +plane, for a fact the control plane does not have (the root does not exist until the authority has +run) and a machine that may have no reason to verify a mesh name at all. + +## Considered Options + +1. **The controller injects the anchor into every machine on the private network**, the + `/etc/hosts` and insecure-registry shape. Rejected: being on the network is what makes the + registry reachable, and that is why network presence is the right trigger *there* — the trust + and the reachability are the same fact. Trusting an authority is not the same fact as being + able to reach it, and the anchor's path and the bundle refresh are a property of the machine's + operating system, which is the host's half of the mesh, not the controller's. +2. **A new host primitive — a `trust-anchor` resource type.** Rejected for now, not on principle. + The host's vocabulary should grow when a shape cannot be said with what exists, and this one + can: a file and a service already express it, as the packet filter proves + ([ADR 0140](0140-the-filter-constrains-what-arrives-from-outside.md), whose module writes a + unit file and a service and nothing else). The primitive becomes right the moment a second + operating system is in play, because the anchor directory and the refresh command are exactly + the difference `internal/system` exists to hold. Until then it would be a vocabulary word with + one speaker. +3. **A module that requires the authority, fetches its root, installs it as an anchor and + refreshes the machine's bundles — and removes both when it is no longer assigned.** Adopted. + +## Decision + +**A machine trusts the mesh's authority because a module put its root there, and stops trusting it +when that module is taken away.** + +1. **The module requires `internal-acme-ca`** and reads the provider's bound address and the path + it serves its root at. It requires nothing else and provides nothing: it is a consumer of the + authority like any other. +2. **It fetches the root over the mesh's own network, without prior trust**, because there is no + prior trust to have — this is the module that establishes it — and the network is what + authenticates the fetch ([ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md), + the same reasoning that lets the proxy fetch it). What it accepts is checked: a body that is + not a certificate fails, and the failure is the module's, not a later handshake's. +3. **It installs the root where this machine's TLS clients look, and refreshes the extracted + bundles** — the command that does the refresh is an ordinary part of the unit that places the + anchor, not a new thing the mesh can be asked to do. +4. **Removal is symmetric and is the same unit's business.** Undeclared, the host stops the unit; + stopping it removes the anchor and refreshes the bundles again. A machine that leaves the mesh + stops trusting the mesh, without anybody remembering to go and look. +5. **It is an ordinary assignment.** No machine is given it automatically. A machine that verifies + a mesh name is assigned it, and a machine that does not is not — which is the same statement + the mesh already makes about every other module, and is why this is not the controller's + business. + +**One operating system, said out loud.** The anchor directory and the refresh command in the +module today are Arch's. On a machine that is not Arch the unit fails, visibly, rather than +writing a file nothing reads. That is the accurate failure, and it is the signal that option 2 +above has become right. + +## How this is checked + +- **The verification that could not succeed before.** On a machine holding the module, a plain + client fetches an internal HTTPS name with no `-k` and no bundle argument and verifies. On a + machine without it, the same fetch fails with *unable to get local issuer certificate*. Both + halves, because only the pair distinguishes "the anchor works" from "something else already + trusted it". +- **The removal half, in the same bed:** unassign the module, refetch, and the failure returns. + Checking only the arrival is how a trust store fills up with authorities nobody can account for. +- **What is deliberately not checked here:** that the authority issues, that a name resolves, that + the proxy serves. Those have their own beds, and this module's bed passing for those reasons is + the failure mode this record is most exposed to — which is why the negative half is not optional. + +**What this bed is dialled at, and why it is the authority itself.** The authority serves its own +API with a certificate it issued, so the handshake under test needs nothing else in the mesh to be +right. A trust bed that reached for a routed name through the proxy would be passing or failing for +the proxy's reasons and the resolver's. + +**Written, and not yet run** *(2026-09-29)*. The bed is `trust-anchor` in the lab, and it cannot +execute: raising a foundation fails before any module is reached, in both bundles that exist +([issue 146](../04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md)). +So what stands behind this record today is the rendering — the script the machine would run names +the authority it was bound to, checked in the control plane's own test suite — and **not** a machine +that verified anything. That is a weaker thing than the paragraph above describes, and it stays +written this way until the bed runs. + +## Consequences + +The predecessor's authority can be retired from a machine once this module is assigned to it, +which is the first time that has been true. `git` over HTTPS to the mesh's forge starts working, +so the ssh-only clone URL stops being a rule. A module on any machine can call another module's +internal name and verify it. + +What got harder: one more module to assign to a machine that needs it, and the machine's trust +store now changes when an assignment changes — which is the point, and is also a thing an operator +can be surprised by. The fetch without prior trust is the same exposure ADR 0098 accepted, now on +every machine that holds the module rather than only where a proxy runs: anything that can stand +in the middle of the mesh's own network at the moment of the fetch can be believed. The mesh +already treats that network as the thing it authenticates. + +## References + +- [issue 129](../04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md) — + the symptom and the evidence. +- [ADR 0098](0098-a-fact-a-provider-makes-at-first-start-is-fetched-from-it.md) — a fact made at + first start is fetched from its provider; this extends it from one program to the machine. +- [ADR 0082](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md) — the precedent + this deliberately does not follow, and why it is right where it is. +- [ADR 0005](0005-the-node-host.md) — the host is where one operating system's difference lives. +- [`03-DESIGN/01-to-be/08-connectivity.md`](../03-DESIGN/01-to-be/08-connectivity.md). diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 3ce3459..3e1cf39 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -225,7 +225,9 @@ python3 00-META/checks/index.py fail if stale - **0141** — [The host delivers its own successor, and versions live side by side](0141-the-host-delivers-its-own-successor.md) - **0143** — [A consumer verifies the grant it is given](0143-a-consumer-verifies-the-grant-it-is-given.md) *(superseded)* - **0144** — [Anything on a machine may call anything on it, and that is the whole of "local"](0144-anything-on-a-machine-may-call-anything-on-it.md) -- **0145** — [A module checks what the mesh claims is reachable, and it checks itself](0145-a-module-checks-what-the-mesh-claims-is-reachable.md) +- **0145** — [A module checks what the mesh claims is reachable, and it checks itself](0145-a-module-checks-what-the-mesh-claims-is-reachable.md) *(superseded)* +- **0146** — [Connectivity is checked by name, per hosting form, with a valid certificate](0146-connectivity-is-checked-by-name-per-hosting-form.md) +- **0147** — [A module anchors the mesh's authority on a machine, and takes it away again](0147-a-module-anchors-the-meshs-authority.md) ### How it is built diff --git a/03-DESIGN/01-to-be/08-connectivity.md b/03-DESIGN/01-to-be/08-connectivity.md index 297aee8..a2c6025 100644 --- a/03-DESIGN/01-to-be/08-connectivity.md +++ b/03-DESIGN/01-to-be/08-connectivity.md @@ -7,8 +7,9 @@ code: - mesh-controller internal/identity/authority.go - mesh-host internal/identity/serving.go - mesh-host internal/apply (the service that reflects a rule set) -updated: 2026-09-28 +updated: 2026-09-29 decisions: + - 02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md - 02-DECISIONS/0138-an-assignment-binds-an-endpoint-and-says-how-far-it-reaches.md - 02-DECISIONS/0140-the-filter-constrains-what-arrives-from-outside.md - 02-DECISIONS/0104-a-provision-may-be-answered-by-an-adapter-to-the-predecessor.md @@ -710,6 +711,22 @@ step, so when the authority moves the root is fetched again and the proxy is rec *How it is checked:* the route-forwarding bed installs the authority, the proxy and a consumer from the catalogue and asserts the routed name is served. +**And a machine trusts that authority because a module put its root in its trust store** +([ADR 0147](../../02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md)). The proxy's fetch +answers for the proxy and for nothing else: a browser, `git` over HTTPS, a package manager and +every module calling another by an internal name read the machine's own trust store, and the mesh +had never written anything there. A module requiring the authority does the whole of it — fetch +the root over the mesh network, place it where this machine's TLS clients look, refresh the +extracted bundles — and stopping it, which is what being unassigned does, takes the anchor away +and refreshes them again. Not the controller's business, because being on the private network is +what makes the authority *reachable* and is not the same fact as having a reason to *verify* a +mesh name; and because where anchors live and which command refreshes them is one operating +system's difference, which is the host's half of the mesh +([ADR 0005](../../02-DECISIONS/0005-the-node-host.md)). +*How it is checked:* on a machine holding the module a plain client verifies an internal HTTPS +name with no bundle argument, and on one without it the same fetch fails to find an issuer — both +halves, because only the pair tells the anchor apart from something that already trusted it. + ### What was built *2026-08-31.* diff --git a/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md b/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md index 322cb2d..e88390a 100644 --- a/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md +++ b/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/00-report.md @@ -1,7 +1,7 @@ --- -status: open +status: located opened: 2026-09-26 -located-in: [mesh-controller, mesh-catalog step-ca] +located-in: [mesh-catalog ca-trust] --- # 129 — nothing makes a machine trust the mesh's own certificate authority diff --git a/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/01-diagnosis.md b/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/01-diagnosis.md new file mode 100644 index 0000000..0071b16 --- /dev/null +++ b/04-ISSUES/129-nothing-makes-a-machine-trust-the-meshs-authority/01-diagnosis.md @@ -0,0 +1,32 @@ +# Diagnosis + +*2026-09-29.* + +## What was ruled out + +**That something already carries the root and it is only misplaced.** It does not. The authority +serves its root at a path beside its ACME directory, and the one thing that fetches it — the route +proxy — puts it in a directory of its own and hands it to one program. Nothing has ever written +into a machine's trust store. Measured on three converged machines: the anchors present are the +predecessor's authority and a developer tool's local root, and on the machines where the +predecessor's was deliberately removed, every internal name fails verification. + +**That the private network could carry it, the way it carries the registry's trust.** That is what +the report proposed, and it was rejected on consideration rather than on difficulty +([ADR 0147](../../02-DECISIONS/0147-a-module-anchors-the-meshs-authority.md), option 1): being on +the network is what makes the registry *reachable* and is therefore the right trigger there, while +trusting an authority is a separate fact from being able to reach it. The anchor's directory and +the command that refreshes the extracted bundles are also one operating system's difference, which +is the host's half of the mesh and not the controller's. + +**That it needs a new host resource type.** It does not, today. A file and a service say the whole +of it, which the packet filter already proves. The primitive becomes the right answer when a second +operating system is in play, and not before. + +## Where it belongs + +A module in the catalogue: it requires `internal-acme-ca`, fetches the root over the mesh's own +network, installs it as a trust anchor, refreshes the machine's bundles, and — because being +unassigned stops its unit, and stopping the unit is what undoes it — takes both away again. + +The owner is therefore `mesh-catalog`, module `ca-trust`, and nothing in the control plane. diff --git a/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md b/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md new file mode 100644 index 0000000..ff8c3ce --- /dev/null +++ b/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/00-report.md @@ -0,0 +1,73 @@ +--- +status: located +opened: 2026-09-29 +located-in: [mesh-host examples + internal/link, mesh-controller internal/broker] +--- + +# 146 — the foundation cannot be raised on the bus the mesh runs on + +## What was observed + +Raising a first node in the lab, to check a module against a real mesh, fails before any module is +reached. Two separate faults, in the two bundles that exist: + +**The older bundle raises a control plane that cannot start.** It brings up the previous broker, +and the control plane it then starts says, once every few seconds, for ever: + +``` +mesh-controller: this control plane has no MESH_BUS_NATS, so it cannot reach the mesh's bus +``` + +That is the control plane being right. The mesh moved to one bus +([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md)) and the +bundle did not. Every bed that raises a foundation raises this one, so every bed is in this state. + +**The newer bundle, written for the new bus, stops one step earlier.** Its certificate step asks a +container to make the broker's certificate: + +``` +docker run --rm --entrypoint sh -v :/tls \ + -c "test -f /tls/tls.crt || (openssl req -x509 ... )" +... +failed bus-certificate: running the action: docker exited 127 +``` + +127 is *command not found*. The bus's image has a shell and no `openssl`; the previous broker's +image had both, which is why the step worked when it was written against that one. Substituting the +store's image — the only other image the bundle carries — does not help: it has no `openssl` +either. So the step as written cannot succeed with anything the bundle names, and the fault is not +one image's: **the bundle asks for a certificate to be made by a tool it never says must be there.** + +Measured 2026-09-29 on a fresh lab machine, both bundles, from bare. + +## Why this is here and not a note in the knowledge base + +The mesh's own foundation is the one thing it cannot raise. Nothing reports that: the bundles are +files in a repository, nothing applies them but a person raising a node, and the last thing that +did was the hand-driven cut-over +([ADR 0131](../../02-DECISIONS/0131-everything-on-the-mesh-speaks-to-the-broker-seat.md), whose +work was done on the machines rather than from a bundle). So the state where the mesh cannot make +another one of itself is reachable, and was reached, without anything saying so. + +It is also load-bearing for everything else: a lab bed proves a claim by raising a mesh, so while +this holds, **no bed can run**, and every "checked in the lab" written from now on is a promise +against a suite nobody can execute. + +## What would have prevented it + +- **Something raising the foundation on a schedule, from the bundle, as it is written** — the + bundle is the mesh's own installer and nothing installs from it. A bed that raises a first node + is exactly that check, and it is the bed that cannot run. +- **A step naming what it needs.** The certificate step names an image and assumes a program inside + it. An action that said which tool it requires would have failed at the declaration rather than + at 127 on a machine. + +## Evidence to carry into diagnosis + +- `mesh-host examples/foundation-first-node.lock` — the previous broker, no `MESH_BUS_NATS`. +- `mesh-host examples/foundation-first-node-nats.lock` — the new bus; `bus-certificate` and its + `verify` both run `openssl` in the bus's image. +- The bus image the bundle pins has `sh` and no `openssl`; the store's image likewise. +- The lab rewrites a bundle's registry-prefixed third-party references to upstream ones for a + machine with an uplink (`test/integration/harness.ts`); the new bundle's bus reference needed + that rule added, which is done and is not this issue. diff --git a/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/01-diagnosis.md b/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/01-diagnosis.md new file mode 100644 index 0000000..bcbd17b --- /dev/null +++ b/04-ISSUES/146-the-foundation-cannot-be-raised-on-the-bus-the-mesh-runs-on/01-diagnosis.md @@ -0,0 +1,96 @@ +# Diagnosis + +*2026-09-29, by raising a first node in the lab over and over and writing down each thing it hit.* + +Not one fault. **Four, stacked**, each hidden behind the one before it, and every one of them the +same shape: a step that was right while the mesh ran on the previous broker and was never asked a +question again after the bus changed. Nothing had raised a foundation since, so nothing said so. + +## 1 — the bundle's bus image is named for a registry that is gone *(fixed)* + +The newer bundle pins `/nats@…`, which resolves nowhere outside the lab that +raised that registry. The lab already rewrites the store's and the previous broker's references to +upstream ones for a machine with an uplink; the bus had no such rule because no bed had ever tried +to raise this bundle. Added (`mesh-lab test/integration/harness.ts`). The digest is the bundle's +own — what the registry served was a copy, so the same digest resolves upstream, and this is a +prefix being removed rather than a reference being replaced. + +## 2 — the bus's certificate was made by a tool the bus does not have *(fixed)* + +``` +failed bus-certificate … docker exited 127 +``` + +The step ran `openssl` inside the broker's image. The previous broker's image carried it; the bus's +does not — it is Alpine with a shell and no `openssl` — and neither does any other image the bundle +names, so there was nothing to substitute. **The program that needs the certificate now makes it**: +`mesh-controller broker certificate --into `, with `--check` as the step's verify. The +controller is already on the machine at that point (the schema step ran it) and needs nothing from +the image it writes into. Self-signed, as before and on purpose — a host pins this server's exact +certificate ([ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md)) and at that moment +there is no authority to ask. Idempotent, because a second certificate is one every host that +pinned the first no longer believes. It runs `--user 0:0`: the volume is root's, and the control +plane's image runs as nobody, which is right for the long-lived server and wrong for a one-shot +writing into a fresh volume. + +## 3 — enrolment dialled TLS at a bus that speaks first *(fixed)* + +``` +mesh-host: cannot reach the broker at …:5671: tls: first record does not look like a TLS handshake +``` + +Enrolment opened a raw TLS connection to check the pinned certificate before saying anything. NATS +speaks its own protocol and upgrades afterwards, so the handshake met a plaintext greeting. The pin +was never the problem: the same pinned configuration is handed to the client that presents the +token, and the verification runs inside *that* handshake — so what +[ADR 0004](../../02-DECISIONS/0004-a-node-and-how-it-joins.md) requires still holds, and holds +better, because the one-time secret is sent only after the certificate has been checked. The raw +dial is gone from the enrolment path and kept only as what its tests always proved: that a wrong +certificate is refused before a byte of application data is sent. + +**Then, immediately behind it:** + +``` +mesh-host: this token is for the "" bus, and the mesh's bus is nats +``` + +The enrolment left the transport empty and meant *whatever the mesh runs today*, which was true +while two buses existed and became a refusal the moment one did. The host knows which bus the mesh +runs; it says so now. + +## 4 — a first node cannot be let onto its own bus *(open, and this is the real one)* + +``` +mesh-host: cannot reach the bus at …:5671 as anchor: nats: Authorization Violation +``` + +The bus's user list is a file beside its configuration. The installer carries the first one — the +controller's own account at a bootstrap password — and **the controller composes every user after +that** (design 25 §6; the controller's own test asserts the carried list matches what it would +derive). On the running mesh that composition reaches the bus because the bus is a *module*, with +the list delivered to it the way anything is delivered to a module. + +At genesis there is no module. The foundation's bus is raised by the installer, the control plane is +given no way to write beside its configuration — it mounts the certificate and nothing else — and so +the account a joining node needs cannot come into existence. **The first node cannot join the mesh +it just raised.** + +That is not a line to fix in a bundle. It is the open half of the mesh delivering its own components +([ADR 0142](../../02-DECISIONS/0142-the-mesh-delivers-its-own-components-as-binaries.md)) and of the +bus becoming a module: either the installer's bus is raised as the module the mesh will go on +managing, or genesis carries a user list that includes the first node's enrolment and the controller +takes over from there. Both are decisions, not patches, and both belong to the genesis step that was +deliberately left until last. + +## Where it belongs + +`mesh-host` (the bundle and the enrolment path) and `mesh-controller` (the certificate command, and +the composition that cannot reach the bus at genesis). Three of the four are fixed on branches; the +fourth is the genesis work. + +## What it cost, for the next person + +Every lab bed still names `foundation-first-node.lock` in its own instructions, and that bundle +raises the previous broker with a control plane that refuses to start without `MESH_BUS_NATS`. Until +the fourth fault is answered and the two bundles become one, a bed runs with `MESH_LAB_BUNDLE` +pointing at the NATS bundle by hand, and stops at the enrolment.