Issues 061/062 — two silent-success defects the no-fake bed surfaced

061: the broker module's provisioner never ran; its runtime container named no command and the image default is the tool host. 062: a failed artifact-store lookup composed the network without the registry trust, turning a transient error into permanent silent state. Both located, fixes on the 042/048 train branches.
2026-09-18 00:23:52 +02:00
parent 08c814aa0a
commit a0acaad86d
4 changed files with 146 additions and 0 deletions
@@ -0,0 +1,49 @@
+---
+status: located
+opened: 2026-09-18
+located-in:
+  - mesh-catalog
+fixed-by:
+amended-design:
+---
+
+# 061 — A module container that names no command runs the wrong thing
+
+## Symptom
+
+The broker module's provisioner never ran — anywhere, ever. Its runtime container declared no
+command, so it ran its image's default entrypoint, which is the tool host. The container came up,
+served its tools, logged its audit line, and reported healthy. The provisioner entrypoint compiled
+into the same image was never named by anything, so no consumer was ever granted its vhost and
+user.
+
+Observed on the built-store-cross-node bed (2026-09-17): a consumer on the joined node held a
+correct binding, its grant sat applied in the provisioner's mounted grants directory, and the
+broker's vhost list never grew past the default. Every check between the consumer and the missing
+mint passed — the grant was composed, delivered, written; the container it fed was running and
+healthy. The bed's vhost assertion was the first thing in the mesh that could notice, and this
+bed is the first to reach it without pre-stocked images.
+
+The defect was present from the module's first commit. It survived conversion, adoption of the
+foundation's broker, and every earlier bed, because none of them asserted the provision itself.
+
+## Why it matters beyond the instance
+
+The manifest treats a container's command as optional, and the image's default entrypoint makes
+the omission *plausible*: the container runs, logs, and stays up. A module can therefore declare a
+provisioner-shaped container that provisions nothing, and nothing in validation, build, install,
+or health says so. Every module whose runtime image carries more than one entrypoint has this trap
+— the sibling module of the same shape names its provisioner explicitly, and only that habit
+separated the working provider from the silent one.
+
+This is the constitution's central failure class — reports success and does nothing — expressed
+in one missing manifest field.
+
+## Open questions
+
+- Should a manifest be able to omit a container's command at all when the module also declares
+  `grants` (i.e. claims to be a provider)? A provider whose containers all default their command
+  provably runs no provisioner.
+- Can validation know an image's entrypoints, so "compiled but never named by any container" is
+  refusable at build time rather than discoverable only by an end-to-end bed?
+- Does any other converted module in the catalogue have the same shape today?
@@ -0,0 +1,29 @@
+# Diagnosis — 2026-09-17/18
+
+The trail, from the failing assertion inward:
+
+1. The bed asserted no vhost existed for the joined node's consumer after four minutes of
+   polling. The broker's management API listed only the default vhost.
+2. On the provider machine, the consumer's grant file was present in the provisioner's mounted
+   grants directory, applied by the host minutes earlier — composition (including the re-push of
+   the provider node, issue 057's remedy) and delivery were all correct.
+3. The provisioner container's log carried only the tool host's lines: the audit subscription and
+   the tool listing. No provisioner harness line, no error. The provisioner was not failing — it
+   was not running.
+4. The module's manifest declares two containers: the server (the adopted broker) and the runtime.
+   The runtime names volumes and environment but **no command**. The image's default entrypoint is
+   the tool host, so that is what ran.
+5. The sibling provider of the same shape (the database module) names its provisioner explicitly
+   in its runtime container's command — the pattern the broker module was meant to follow. Its own
+   image documentation says the provisioner "runs separately", named by the declaration; the
+   declaration never did.
+6. History: the command was absent from the module's first commit. No regression — a hole.
+
+**Located in:** mesh-catalog (the broker module's manifest). The fix names the provisioner
+entrypoint as the runtime container's command, mirroring the database module. Verified on the
+same bed: with the command named, the vhost and user are minted and the consumer holds its
+connection over the overlay (bed run of 2026-09-18).
+
+The open questions in the report — refusing a provider whose containers all default their
+command, and auditing the rest of the catalogue for the same shape — remain open; this record
+covers the one module the bed proved.
@@ -0,0 +1,44 @@
+---
+status: located
+opened: 2026-09-18
+located-in:
+  - mesh-controller
+fixed-by:
+amended-design:
+---
+
+# 062 — A failed lookup composes the network without the registry trust
+
+## Symptom
+
+One machine in three otherwise identical runs joined the mesh with a runtime that never learned to
+pull from the mesh's artifact store. Its runtime daemon file was never written at all — not stale,
+absent — while the same push on the same commits had carried the trust to the same machine in the
+run before. The push reported success both times.
+
+Observed on the built-store-cross-node bed (2026-09-18): the first fresh run carried the trust to
+both machines; the next fresh run carried it to the first machine and composed the second
+machine's declaration without it. Nothing recomposes a machine until the next push, so the
+omission was permanent for that mesh, and everything reported success: the push, the apply, the
+network itself.
+
+## Why it matters beyond the instance
+
+The controller answers "where is the artifact store this network reaches?" while composing the
+networking module (ADR 0082). The answering code collapsed *the lookup failed* into *the mesh has
+no store*: a transient inventory error during one compose produced a valid-looking declaration
+that simply lacked the trust. That collapse converts a retryable fault into silent permanent
+state — the exact class issues 042/048 name, reappearing as a race after their fix.
+
+The general rule this instance argues for: **"not found" must be a fact about the mesh, never a
+disguise for "the question went unanswered."** Any compose-time lookup that degrades to omission
+on error has this bug; the compose should refuse instead, loudly, so the push fails and is retried
+rather than delivering a declaration that is quietly less than what the mesh means.
+
+## Open questions
+
+- Are there other compose-time lookups that treat an error as an empty answer? The pattern is
+  easy to write and invisible until a bed asserts the composed fact end to end.
+- Should a bed that waits on pushed state always print the push's own output, so a compose that
+  refused is distinguishable from one that composed short? (The bed now does; the question is
+  whether the harness should.)
@@ -0,0 +1,24 @@
+# Diagnosis — 2026-09-18
+
+1. The bed's trust wait timed out on the second machine after five minutes; the dump showed the
+   runtime daemon file still holding the lab base image's content — the trust file resource was
+   never applied, and the machine's declaration was composed in that window exactly once, by the
+   one push.
+2. The machines were torn down before the declaration itself could be inspected, so the trail
+   went to the composing code with two candidates: the declaration never named the trust, or it
+   named it and was never applied.
+3. The composer's trust step asks which machine on the network is assigned a module serving the
+   artifact-store provision. Two silent degradations sat in that path: a failed catalogue read
+   returned "not found", and a failed per-machine assignment read *skipped that machine* and kept
+   scanning. Either converts a transient inventory error into a declaration without the trust,
+   under a push that reports success.
+4. The delivery-side candidate could not be positively excluded for the observed run, but the
+   apply path retries and had applied the same machine's declaration within seconds in the runs
+   before and after; the silent-omission path needs no second fault to explain the evidence and
+   matched it exactly (file absent, not stale; push succeeded; one compose, never repeated).
+
+**Located in:** mesh-controller (the network compose's artifact-store lookup). The fix makes a
+lookup failure refuse the compose — the push then fails aloud and is retried — so "no store" can
+only ever mean the mesh has none. The bed was also taught to print each push's output and, on a
+trust timeout, to dump the host's log and whether the received declaration named the trust, so
+the two candidate shapes are distinguishable from the run log if the race ever shows again.