Merge pull request 'ADR 0082 and issues 042/048/061/062 — the registry is reached by name and trusted by the overlay' (#52) from issue/042-048-registry-reach into main

2026-09-18 01:00:55 +02:00
parent e7869d27bd 4279e4914b
commit 46efb7ae98
9 changed files with 240 additions and 8 deletions
@@ -0,0 +1,84 @@
+---
+topic: building it
+status: accepted
+date: 2026-09-17
+deciders: jochen
+reconstructed: false
+extends: 0072-two-graphs-and-the-build-chain.md
+---
+
+# 82. The registry is reached by name, and the overlay is its security
+
+## Context
+
+Delivery ends at a node pulling an image, and for every node but the one that built it, that step
+has never worked. Three facts conspired, each recorded separately:
+
+- Artifact references are written under `127.0.0.1:5000` — a deliberate parking (the catalogue
+  says so in the commit that reverted the mesh-reachable name: *"the runtime refused it: 'http:
+  server gave HTTP response to HTTPS client' … the binding expression returns then"*), so a joined
+  node is told to pull from its own loopback
+  ([issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md)).
+- A container runtime treats any non-loopback registry as HTTPS, and the mesh's registry serves
+  plain HTTP; nothing the mesh writes tells any runtime otherwise — the one place that file
+  existed was the lab's, which is why this never failed in a bed (issue 048).
+- Nothing gives a node an account for the registry
+  ([issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md)) —
+  and nothing has ever said whether one is required.
+
+The tempting fix is TLS from the mesh's own authority. The machinery even exists — the mesh CA
+issues leaves for `<node>.internal` names, and a `certificate:` manifest field delivers them. But
+the registry is reached **over the overlay**, and the overlay is WireGuard: every byte is already
+encrypted and already authenticated to a peer the mesh admitted. The mesh's own code has carried
+this position for weeks: *"the registry is reached over the mesh's own private network … a second
+layer inside it would be certificates to issue and rotate for no property the first does not
+have."* TLS inside the tunnel would also re-order genesis (a certificate needs the network, the
+registry precedes it) and put CA handling into three clients (the runtime, the archive fetcher,
+the builder) — cost with no new property.
+
+## Decision
+
+**The overlay is the registry's transport security, and the mesh writes the trust it means.**
+
+1. **References name the registry by its mesh name.** The builder publishes to
+   `<provider>.internal:<port>` (the binding's `at` and `port`) — the parked one-line change
+   lands. Genesis still publishes to loopback on the first machine, before any overlay exists;
+   references minted at genesis stay loopback and are valid where they matter — on that machine.
+   A rebuild re-pins to the mesh name.
+2. **Every node on the private network is told the mesh's registry speaks plain HTTP.** The
+   controller injects, into every such node's declaration, a merged `/etc/docker/daemon.json`
+   naming `<provider>.internal:<port>` under `insecure-registries`, and a service resource that
+   restarts the runtime when that file changes — the `/etc/hosts` pattern for the content, the
+   nftables pattern for the reload. No module author is involved; being on the network is what
+   grants the trust, because being on the network is what the trust *is*.
+3. **No accounts (issue 042), recorded as the position it always was.** Reading and pushing
+   require presence on the overlay and nothing else. The boundary is enforced, not assumed: the
+   registry's `listens` is `from: mesh`, the firewall derives from it, and the overlay admits only
+   peers holding mesh-issued keys. An operator's *external* registry credential is an ordinary
+   operator-supplied secret (`secret accept`), owned by whichever module names that registry.
+   Per-node accounts return as a decision, not a patch, if the boundary assumption ever changes —
+   the shape would be the existing provision flow.
+
+## How this is checked
+
+- The no-fake lab bed: a joined node pulls a mesh-built image by the registry's `.internal` name
+  over the overlay — an address outside every range the lab's own runtime configuration trusts,
+  so the mesh-written trust is what makes it work or nothing does.
+- The firewall half is generated from `listens` and visible in the node's ruleset (`from: mesh`).
+- The plaintext-inside-tunnel position holds exactly as long as the registry is unreachable off
+  the overlay; the `listens` stanza and the derived ruleset are the check.
+
+## Consequences
+
+A runtime restart when the trust first lands on a node — at joining, before workloads, where it
+is free. The registry's HTTP is exposed to whatever stands on the overlay, which is the stated
+boundary; a mesh that wants defence in depth inside its own tunnel reopens this record rather
+than bolting certificates on quietly. Issues 042 and 048 close on this record.
+
+## References
+
+- [issue 042](../04-ISSUES/042-nothing-gives-a-node-an-account-for-a-registry/00-report.md),
+  [issue 048](../04-ISSUES/048-nothing-makes-a-machine-trust-the-mesh-registry/00-report.md).
+- [ADR 0072](0072-two-graphs-and-the-build-chain.md) — the build chain this completes.
+- [ADR 0029](0029-a-network-is-a-shape-because-an-action-cannot-be-undone.md) — the overlay as a
+  boundary.
@@ -149,6 +149,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0068** — [The lab takes requests, one at a time, and runs each from its own copy](0068-the-lab-takes-requests.md)  *(proposed)*
 - **0069** — [A module is a repository and a path within it](0069-a-module-is-a-repository-and-a-path.md)
 - **0076** — [The SDK is a published package, and the toolchain resolves it by version](0076-the-sdk-is-a-published-package.md)
+- **0082** — [The registry is reached by name, and the overlay is its security](0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md)

 ### How it is checked

@@ -7,6 +7,7 @@ code:
  - mesh-controller internal/inventory/builds.go
 updated: 2026-08-31
 decisions:
+  - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
  - 02-DECISIONS/0010-delivery.md
  - 02-DECISIONS/0009-modules-and-the-graph.md
  - 02-DECISIONS/0009-modules-and-the-graph.md
@@ -1,9 +1,9 @@
 ---
-status: open
+status: resolved
 opened: 2026-09-11
-located-in: []
-fixed-by:
-amended-design:
+located-in: [mesh-controller, mesh-catalog]
+fixed-by: mesh-controller PR 29 (98aea8b); mesh-catalog PR 26 (1891b09); gated by mesh-lab PR 34
+amended-design: 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
 ---

 # 042 — Nothing gives a node an account for a registry
@@ -1,9 +1,9 @@
 ---
-status: open
+status: resolved
 opened: 2026-09-14
-located-in: []
-fixed-by:
-amended-design:
+located-in: [mesh-controller, mesh-catalog]
+fixed-by: mesh-controller PR 29 (98aea8b); mesh-catalog PR 26 (1891b09); gated by mesh-lab PR 34
+amended-design: 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
 ---

 # 048 — Nothing makes a machine trust the mesh's own registry
@@ -0,0 +1,49 @@
+---
+status: resolved
+opened: 2026-09-18
+located-in:
+  - mesh-catalog
+fixed-by: mesh-catalog PR 26 (1891b09); gated by mesh-lab PR 34
+amended-design:
+---
+
+# 061 — A module container that names no command runs the wrong thing
+
+## Symptom
+
+The broker module's provisioner never ran — anywhere, ever. Its runtime container declared no
+command, so it ran its image's default entrypoint, which is the tool host. The container came up,
+served its tools, logged its audit line, and reported healthy. The provisioner entrypoint compiled
+into the same image was never named by anything, so no consumer was ever granted its vhost and
+user.
+
+Observed on the built-store-cross-node bed (2026-09-17): a consumer on the joined node held a
+correct binding, its grant sat applied in the provisioner's mounted grants directory, and the
+broker's vhost list never grew past the default. Every check between the consumer and the missing
+mint passed — the grant was composed, delivered, written; the container it fed was running and
+healthy. The bed's vhost assertion was the first thing in the mesh that could notice, and this
+bed is the first to reach it without pre-stocked images.
+
+The defect was present from the module's first commit. It survived conversion, adoption of the
+foundation's broker, and every earlier bed, because none of them asserted the provision itself.
+
+## Why it matters beyond the instance
+
+The manifest treats a container's command as optional, and the image's default entrypoint makes
+the omission *plausible*: the container runs, logs, and stays up. A module can therefore declare a
+provisioner-shaped container that provisions nothing, and nothing in validation, build, install,
+or health says so. Every module whose runtime image carries more than one entrypoint has this trap
+— the sibling module of the same shape names its provisioner explicitly, and only that habit
+separated the working provider from the silent one.
+
+This is the constitution's central failure class — reports success and does nothing — expressed
+in one missing manifest field.
+
+## Open questions
+
+- Should a manifest be able to omit a container's command at all when the module also declares
+  `grants` (i.e. claims to be a provider)? A provider whose containers all default their command
+  provably runs no provisioner.
+- Can validation know an image's entrypoints, so "compiled but never named by any container" is
+  refusable at build time rather than discoverable only by an end-to-end bed?
+- Does any other converted module in the catalogue have the same shape today?
@@ -0,0 +1,29 @@
+# Diagnosis — 2026-09-17/18
+
+The trail, from the failing assertion inward:
+
+1. The bed asserted no vhost existed for the joined node's consumer after four minutes of
+   polling. The broker's management API listed only the default vhost.
+2. On the provider machine, the consumer's grant file was present in the provisioner's mounted
+   grants directory, applied by the host minutes earlier — composition (including the re-push of
+   the provider node, issue 057's remedy) and delivery were all correct.
+3. The provisioner container's log carried only the tool host's lines: the audit subscription and
+   the tool listing. No provisioner harness line, no error. The provisioner was not failing — it
+   was not running.
+4. The module's manifest declares two containers: the server (the adopted broker) and the runtime.
+   The runtime names volumes and environment but **no command**. The image's default entrypoint is
+   the tool host, so that is what ran.
+5. The sibling provider of the same shape (the database module) names its provisioner explicitly
+   in its runtime container's command — the pattern the broker module was meant to follow. Its own
+   image documentation says the provisioner "runs separately", named by the declaration; the
+   declaration never did.
+6. History: the command was absent from the module's first commit. No regression — a hole.
+
+**Located in:** mesh-catalog (the broker module's manifest). The fix names the provisioner
+entrypoint as the runtime container's command, mirroring the database module. Verified on the
+same bed: with the command named, the vhost and user are minted and the consumer holds its
+connection over the overlay (bed run of 2026-09-18).
+
+The open questions in the report — refusing a provider whose containers all default their
+command, and auditing the rest of the catalogue for the same shape — remain open; this record
+covers the one module the bed proved.
@@ -0,0 +1,44 @@
+---
+status: resolved
+opened: 2026-09-18
+located-in:
+  - mesh-controller
+fixed-by: mesh-controller PR 29 (98aea8b); gated by mesh-lab PR 34
+amended-design:
+---
+
+# 062 — A failed lookup composes the network without the registry trust
+
+## Symptom
+
+One machine in three otherwise identical runs joined the mesh with a runtime that never learned to
+pull from the mesh's artifact store. Its runtime daemon file was never written at all — not stale,
+absent — while the same push on the same commits had carried the trust to the same machine in the
+run before. The push reported success both times.
+
+Observed on the built-store-cross-node bed (2026-09-18): the first fresh run carried the trust to
+both machines; the next fresh run carried it to the first machine and composed the second
+machine's declaration without it. Nothing recomposes a machine until the next push, so the
+omission was permanent for that mesh, and everything reported success: the push, the apply, the
+network itself.
+
+## Why it matters beyond the instance
+
+The controller answers "where is the artifact store this network reaches?" while composing the
+networking module (ADR 0082). The answering code collapsed *the lookup failed* into *the mesh has
+no store*: a transient inventory error during one compose produced a valid-looking declaration
+that simply lacked the trust. That collapse converts a retryable fault into silent permanent
+state — the exact class issues 042/048 name, reappearing as a race after their fix.
+
+The general rule this instance argues for: **"not found" must be a fact about the mesh, never a
+disguise for "the question went unanswered."** Any compose-time lookup that degrades to omission
+on error has this bug; the compose should refuse instead, loudly, so the push fails and is retried
+rather than delivering a declaration that is quietly less than what the mesh means.
+
+## Open questions
+
+- Are there other compose-time lookups that treat an error as an empty answer? The pattern is
+  easy to write and invisible until a bed asserts the composed fact end to end.
+- Should a bed that waits on pushed state always print the push's own output, so a compose that
+  refused is distinguishable from one that composed short? (The bed now does; the question is
+  whether the harness should.)
@@ -0,0 +1,24 @@
+# Diagnosis — 2026-09-18
+
+1. The bed's trust wait timed out on the second machine after five minutes; the dump showed the
+   runtime daemon file still holding the lab base image's content — the trust file resource was
+   never applied, and the machine's declaration was composed in that window exactly once, by the
+   one push.
+2. The machines were torn down before the declaration itself could be inspected, so the trail
+   went to the composing code with two candidates: the declaration never named the trust, or it
+   named it and was never applied.
+3. The composer's trust step asks which machine on the network is assigned a module serving the
+   artifact-store provision. Two silent degradations sat in that path: a failed catalogue read
+   returned "not found", and a failed per-machine assignment read *skipped that machine* and kept
+   scanning. Either converts a transient inventory error into a declaration without the trust,
+   under a push that reports success.
+4. The delivery-side candidate could not be positively excluded for the observed run, but the
+   apply path retries and had applied the same machine's declaration within seconds in the runs
+   before and after; the silent-omission path needs no second fault to explain the evidence and
+   matched it exactly (file absent, not stale; push succeeded; one compose, never repeated).
+
+**Located in:** mesh-controller (the network compose's artifact-store lookup). The fix makes a
+lookup failure refuse the compose — the push then fails aloud and is retried — so "no store" can
+only ever mean the mesh has none. The bed was also taught to print each push's output and, on a
+trust timeout, to dump the host's log and whether the received declaration named the trust, so
+the two candidate shapes are distinguishable from the run log if the race ever shows again.