From 8df3469813b18997ee36104825dcef634519ebac Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 17 Sep 2026 22:44:39 +0200 Subject: [PATCH 1/5] =?UTF-8?q?042=20and=20048=20move=20to=20diagnosing=20?= =?UTF-8?q?=E2=80=94=20the=20registry-reach=20work=20begins?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx --- .../00-report.md | 2 +- .../00-report.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md b/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md index 019cf14..e329e3e 100644 --- a/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md +++ b/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md @@ -1,5 +1,5 @@ --- -status: open +status: diagnosing opened: 2026-09-11 located-in: [] fixed-by: diff --git a/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md b/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md index ebc9cf2..c17b928 100644 --- a/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md +++ b/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md @@ -1,5 +1,5 @@ --- -status: open +status: diagnosing opened: 2026-09-14 located-in: [] fixed-by: From 32fa6b40f5b16d62db247d1224ac4b26d0963867 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 17 Sep 2026 23:02:18 +0200 Subject: [PATCH 2/5] ADR 0082: the registry is reached by name, and the overlay is its security https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx --- ...ched-by-name-and-trusted-by-the-overlay.md | 84 +++++++++++++++++++ 02-DECISIONS/README.md | 1 + 2 files changed, 85 insertions(+) create mode 100644 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md diff --git a/02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md b/02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md new file mode 100644 index 0000000..81366a6 --- /dev/null +++ b/02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md @@ -0,0 +1,84 @@ +--- +topic: building it +status: accepted +date: 2026-09-17 +deciders: jochen +reconstructed: false +extends: 0072-two-graphs-and-the-build-chain.md +--- + +# 82. The registry is reached by name, and the overlay is its security + +## Context + +Delivery ends at a node pulling an image, and for every node but the one that built it, that step +has never worked. Three facts conspired, each recorded separately: + +- Artifact references are written under `127.0.0.1:5000` — a deliberate parking (the catalogue + says so in the commit that reverted the mesh-reachable name: *"the runtime refused it: 'http: + server gave HTTP response to HTTPS client' … the binding expression returns then"*), so a joined + node is told to pull from its own loopback + ([issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)). +- A container runtime treats any non-loopback registry as HTTPS, and the mesh's registry serves + plain HTTP; nothing the mesh writes tells any runtime otherwise — the one place that file + existed was the lab's, which is why this never failed in a bed (issue 048). +- Nothing gives a node an account for the registry + ([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)) — + and nothing has ever said whether one is required. + +The tempting fix is TLS from the mesh's own authority. The machinery even exists — the mesh CA +issues leaves for `.internal` names, and a `certificate:` manifest field delivers them. But +the registry is reached **over the overlay**, and the overlay is WireGuard: every byte is already +encrypted and already authenticated to a peer the mesh admitted. The mesh's own code has carried +this position for weeks: *"the registry is reached over the mesh's own private network … a second +layer inside it would be certificates to issue and rotate for no property the first does not +have."* TLS inside the tunnel would also re-order genesis (a certificate needs the network, the +registry precedes it) and put CA handling into three clients (the runtime, the archive fetcher, +the builder) — cost with no new property. + +## Decision + +**The overlay is the registry's transport security, and the mesh writes the trust it means.** + +1. **References name the registry by its mesh name.** The builder publishes to + `.internal:` (the binding's `at` and `port`) — the parked one-line change + lands. Genesis still publishes to loopback on the first machine, before any overlay exists; + references minted at genesis stay loopback and are valid where they matter — on that machine. + A rebuild re-pins to the mesh name. +2. **Every node on the private network is told the mesh's registry speaks plain HTTP.** The + controller injects, into every such node's declaration, a merged `/etc/docker/daemon.json` + naming `.internal:` under `insecure-registries`, and a service resource that + restarts the runtime when that file changes — the `/etc/hosts` pattern for the content, the + nftables pattern for the reload. No module author is involved; being on the network is what + grants the trust, because being on the network is what the trust *is*. +3. **No accounts (issue 042), recorded as the position it always was.** Reading and pushing + require presence on the overlay and nothing else. The boundary is enforced, not assumed: the + registry's `listens` is `from: mesh`, the firewall derives from it, and the overlay admits only + peers holding mesh-issued keys. An operator's *external* registry credential is an ordinary + operator-supplied secret (`secret accept`), owned by whichever module names that registry. + Per-node accounts return as a decision, not a patch, if the boundary assumption ever changes — + the shape would be the existing provision flow. + +## How this is checked + +- The no-fake lab bed: a joined node pulls a mesh-built image by the registry's `.internal` name + over the overlay — an address outside every range the lab's own runtime configuration trusts, + so the mesh-written trust is what makes it work or nothing does. +- The firewall half is generated from `listens` and visible in the node's ruleset (`from: mesh`). +- The plaintext-inside-tunnel position holds exactly as long as the registry is unreachable off + the overlay; the `listens` stanza and the derived ruleset are the check. + +## Consequences + +A runtime restart when the trust first lands on a node — at joining, before workloads, where it +is free. The registry's HTTP is exposed to whatever stands on the overlay, which is the stated +boundary; a mesh that wants defence in depth inside its own tunnel reopens this record rather +than bolting certificates on quietly. Issues 042 and 048 close on this record. + +## References + +- [issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md), + [issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md). +- [ADR 0072](0072-two-graphs-and-the-build-chain.md) — the build chain this completes. +- [ADR 0029](0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md) — the overlay as a + boundary. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 0b0ed56..5a587cb 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -149,6 +149,7 @@ python3 00-META/checks/index.py fail if stale - **0068** — [The lab takes requests, one at a time, and runs each from its own copy](0068-the-lab-takes-requests.md) *(proposed)* - **0069** — [A module is a repository and a path within it](0069-a-module-is-a-repository-and-a-path.md) - **0076** — [The SDK is a published package, and the toolchain resolves it by version](0076-the-sdk-is-a-published-package.md) +- **0082** — [The registry is reached by name, and the overlay is its security](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md) ### How it is checked From 08c814aa0a47dfedc873fc8ae67489ca7f9a01d0 Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 17 Sep 2026 23:02:32 +0200 Subject: [PATCH 3/5] 0082 takes its homes: 042/048 located against it, the delivery design cites it https://claude.ai/code/session_01D6qtiYU3P9jk3pnAXyAFyx --- 03-DESIGN/01-to-be/10-delivery.md | 1 + .../00-report.md | 6 +++--- .../00-report.md | 6 +++--- 3 files changed, 7 insertions(+), 6 deletions(-) diff --git a/03-DESIGN/01-to-be/10-delivery.md b/03-DESIGN/01-to-be/10-delivery.md index 32b04d7..e463161 100644 --- a/03-DESIGN/01-to-be/10-delivery.md +++ b/03-DESIGN/01-to-be/10-delivery.md @@ -7,6 +7,7 @@ code: - mesh-controller internal/inventory/builds.go updated: 2026-08-31 decisions: + - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md - 02-DECISIONS/0010-delivery.md - 02-DECISIONS/0009-modules-and-the-graph.md - 02-DECISIONS/0009-modules-and-the-graph.md diff --git a/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md b/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md index e329e3e..1b99efe 100644 --- a/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md +++ b/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md @@ -1,9 +1,9 @@ --- -status: diagnosing +status: located opened: 2026-09-11 -located-in: [] +located-in: [mesh-controller, mesh-catalog] fixed-by: -amended-design: +amended-design: 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md --- # 042 — Nothing gives a node an account for a registry diff --git a/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md b/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md index c17b928..f677780 100644 --- a/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md +++ b/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md @@ -1,9 +1,9 @@ --- -status: diagnosing +status: located opened: 2026-09-14 -located-in: [] +located-in: [mesh-controller, mesh-catalog] fixed-by: -amended-design: +amended-design: 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md --- # 048 — Nothing makes a machine trust the mesh's own registry From a0acaad86d8d909204f606ebabf78ea20231b6c0 Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 18 Sep 2026 00:23:52 +0200 Subject: [PATCH 4/5] =?UTF-8?q?Issues=20061/062=20=E2=80=94=20two=20silent?= =?UTF-8?q?-success=20defects=20the=20no-fake=20bed=20surfaced?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 061: the broker module's provisioner never ran; its runtime container named no command and the image default is the tool host. 062: a failed artifact-store lookup composed the network without the registry trust, turning a transient error into permanent silent state. Both located, fixes on the 042/048 train branches. --- .../00-report.md | 49 +++++++++++++++++++ .../01-diagnosis.md | 29 +++++++++++ .../00-report.md | 44 +++++++++++++++++ .../01-diagnosis.md | 24 +++++++++ 4 files changed, 146 insertions(+) create mode 100644 04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/00-report.md create mode 100644 04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/01-diagnosis.md create mode 100644 04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/00-report.md create mode 100644 04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/01-diagnosis.md diff --git a/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/00-report.md b/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/00-report.md new file mode 100644 index 0000000..be61b49 --- /dev/null +++ b/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/00-report.md @@ -0,0 +1,49 @@ +--- +status: located +opened: 2026-09-18 +located-in: + - mesh-catalog +fixed-by: +amended-design: +--- + +# 061 — A module container that names no command runs the wrong thing + +## Symptom + +The broker module's provisioner never ran — anywhere, ever. Its runtime container declared no +command, so it ran its image's default entrypoint, which is the tool host. The container came up, +served its tools, logged its audit line, and reported healthy. The provisioner entrypoint compiled +into the same image was never named by anything, so no consumer was ever granted its vhost and +user. + +Observed on the built-store-cross-node bed (2026-09-17): a consumer on the joined node held a +correct binding, its grant sat applied in the provisioner's mounted grants directory, and the +broker's vhost list never grew past the default. Every check between the consumer and the missing +mint passed — the grant was composed, delivered, written; the container it fed was running and +healthy. The bed's vhost assertion was the first thing in the mesh that could notice, and this +bed is the first to reach it without pre-stocked images. + +The defect was present from the module's first commit. It survived conversion, adoption of the +foundation's broker, and every earlier bed, because none of them asserted the provision itself. + +## Why it matters beyond the instance + +The manifest treats a container's command as optional, and the image's default entrypoint makes +the omission *plausible*: the container runs, logs, and stays up. A module can therefore declare a +provisioner-shaped container that provisions nothing, and nothing in validation, build, install, +or health says so. Every module whose runtime image carries more than one entrypoint has this trap +— the sibling module of the same shape names its provisioner explicitly, and only that habit +separated the working provider from the silent one. + +This is the constitution's central failure class — reports success and does nothing — expressed +in one missing manifest field. + +## Open questions + +- Should a manifest be able to omit a container's command at all when the module also declares + `grants` (i.e. claims to be a provider)? A provider whose containers all default their command + provably runs no provisioner. +- Can validation know an image's entrypoints, so "compiled but never named by any container" is + refusable at build time rather than discoverable only by an end-to-end bed? +- Does any other converted module in the catalogue have the same shape today? diff --git a/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/01-diagnosis.md b/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/01-diagnosis.md new file mode 100644 index 0000000..c76aace --- /dev/null +++ b/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/01-diagnosis.md @@ -0,0 +1,29 @@ +# Diagnosis — 2026-09-17/18 + +The trail, from the failing assertion inward: + +1. The bed asserted no vhost existed for the joined node's consumer after four minutes of + polling. The broker's management API listed only the default vhost. +2. On the provider machine, the consumer's grant file was present in the provisioner's mounted + grants directory, applied by the host minutes earlier — composition (including the re-push of + the provider node, issue 057's remedy) and delivery were all correct. +3. The provisioner container's log carried only the tool host's lines: the audit subscription and + the tool listing. No provisioner harness line, no error. The provisioner was not failing — it + was not running. +4. The module's manifest declares two containers: the server (the adopted broker) and the runtime. + The runtime names volumes and environment but **no command**. The image's default entrypoint is + the tool host, so that is what ran. +5. The sibling provider of the same shape (the database module) names its provisioner explicitly + in its runtime container's command — the pattern the broker module was meant to follow. Its own + image documentation says the provisioner "runs separately", named by the declaration; the + declaration never did. +6. History: the command was absent from the module's first commit. No regression — a hole. + +**Located in:** mesh-catalog (the broker module's manifest). The fix names the provisioner +entrypoint as the runtime container's command, mirroring the database module. Verified on the +same bed: with the command named, the vhost and user are minted and the consumer holds its +connection over the overlay (bed run of 2026-09-18). + +The open questions in the report — refusing a provider whose containers all default their +command, and auditing the rest of the catalogue for the same shape — remain open; this record +covers the one module the bed proved. diff --git a/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/00-report.md b/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/00-report.md new file mode 100644 index 0000000..d977ef9 --- /dev/null +++ b/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/00-report.md @@ -0,0 +1,44 @@ +--- +status: located +opened: 2026-09-18 +located-in: + - mesh-controller +fixed-by: +amended-design: +--- + +# 062 — A failed lookup composes the network without the registry trust + +## Symptom + +One machine in three otherwise identical runs joined the mesh with a runtime that never learned to +pull from the mesh's artifact store. Its runtime daemon file was never written at all — not stale, +absent — while the same push on the same commits had carried the trust to the same machine in the +run before. The push reported success both times. + +Observed on the built-store-cross-node bed (2026-09-18): the first fresh run carried the trust to +both machines; the next fresh run carried it to the first machine and composed the second +machine's declaration without it. Nothing recomposes a machine until the next push, so the +omission was permanent for that mesh, and everything reported success: the push, the apply, the +network itself. + +## Why it matters beyond the instance + +The controller answers "where is the artifact store this network reaches?" while composing the +networking module (ADR 0082). The answering code collapsed *the lookup failed* into *the mesh has +no store*: a transient inventory error during one compose produced a valid-looking declaration +that simply lacked the trust. That collapse converts a retryable fault into silent permanent +state — the exact class issues 042/048 name, reappearing as a race after their fix. + +The general rule this instance argues for: **"not found" must be a fact about the mesh, never a +disguise for "the question went unanswered."** Any compose-time lookup that degrades to omission +on error has this bug; the compose should refuse instead, loudly, so the push fails and is retried +rather than delivering a declaration that is quietly less than what the mesh means. + +## Open questions + +- Are there other compose-time lookups that treat an error as an empty answer? The pattern is + easy to write and invisible until a bed asserts the composed fact end to end. +- Should a bed that waits on pushed state always print the push's own output, so a compose that + refused is distinguishable from one that composed short? (The bed now does; the question is + whether the harness should.) diff --git a/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/01-diagnosis.md b/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/01-diagnosis.md new file mode 100644 index 0000000..bda617c --- /dev/null +++ b/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/01-diagnosis.md @@ -0,0 +1,24 @@ +# Diagnosis — 2026-09-18 + +1. The bed's trust wait timed out on the second machine after five minutes; the dump showed the + runtime daemon file still holding the lab base image's content — the trust file resource was + never applied, and the machine's declaration was composed in that window exactly once, by the + one push. +2. The machines were torn down before the declaration itself could be inspected, so the trail + went to the composing code with two candidates: the declaration never named the trust, or it + named it and was never applied. +3. The composer's trust step asks which machine on the network is assigned a module serving the + artifact-store provision. Two silent degradations sat in that path: a failed catalogue read + returned "not found", and a failed per-machine assignment read *skipped that machine* and kept + scanning. Either converts a transient inventory error into a declaration without the trust, + under a push that reports success. +4. The delivery-side candidate could not be positively excluded for the observed run, but the + apply path retries and had applied the same machine's declaration within seconds in the runs + before and after; the silent-omission path needs no second fault to explain the evidence and + matched it exactly (file absent, not stale; push succeeded; one compose, never repeated). + +**Located in:** mesh-controller (the network compose's artifact-store lookup). The fix makes a +lookup failure refuse the compose — the push then fails aloud and is retried — so "no store" can +only ever mean the mesh has none. The bed was also taught to print each push's output and, on a +trust timeout, to dump the host's log and whether the received declaration named the trust, so +the two candidate shapes are distinguishable from the run log if the race ever shows again. From 4279e4914bfcf5c7c1fbf7ff8d98cfd3e292a05b Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 18 Sep 2026 00:26:50 +0200 Subject: [PATCH 5/5] =?UTF-8?q?042/048/061/062=20resolved=20=E2=80=94=20th?= =?UTF-8?q?e=20network=20carries=20the=20registry=20trust?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit One green fresh run of the no-fake two-node bed is the proof: node2's consumers open the store and broker the mesh built and adopted, over the overlay. Fixed by mesh-controller PR 29, mesh-catalog PR 26, gated by mesh-lab PR 34. --- .../00-report.md | 4 ++-- .../00-report.md | 4 ++-- .../00-report.md | 4 ++-- .../00-report.md | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md b/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md index 1b99efe..8412245 100644 --- a/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md +++ b/04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md @@ -1,8 +1,8 @@ --- -status: located +status: resolved opened: 2026-09-11 located-in: [mesh-controller, mesh-catalog] -fixed-by: +fixed-by: mesh-controller PR 29 (98aea8b); mesh-catalog PR 26 (1891b09); gated by mesh-lab PR 34 amended-design: 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md --- diff --git a/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md b/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md index f677780..c3f7c72 100644 --- a/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md +++ b/04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md @@ -1,8 +1,8 @@ --- -status: located +status: resolved opened: 2026-09-14 located-in: [mesh-controller, mesh-catalog] -fixed-by: +fixed-by: mesh-controller PR 29 (98aea8b); mesh-catalog PR 26 (1891b09); gated by mesh-lab PR 34 amended-design: 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md --- diff --git a/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/00-report.md b/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/00-report.md index be61b49..548e480 100644 --- a/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/00-report.md +++ b/04-ISSUES/061-a-module-container-that-names-no-command-runs-the-wrong-thing/00-report.md @@ -1,9 +1,9 @@ --- -status: located +status: resolved opened: 2026-09-18 located-in: - mesh-catalog -fixed-by: +fixed-by: mesh-catalog PR 26 (1891b09); gated by mesh-lab PR 34 amended-design: --- diff --git a/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/00-report.md b/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/00-report.md index d977ef9..527996a 100644 --- a/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/00-report.md +++ b/04-ISSUES/062-a-failed-lookup-composes-the-network-without-the-registry-trust/00-report.md @@ -1,9 +1,9 @@ --- -status: located +status: resolved opened: 2026-09-18 located-in: - mesh-controller -fixed-by: +fixed-by: mesh-controller PR 29 (98aea8b); gated by mesh-lab PR 34 amended-design: ---