From 9fd7e6c4584cf421b18938f989bc3bfb945cbfc2 Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 18 Sep 2026 02:10:01 +0200 Subject: [PATCH 1/5] ADR 0083 proposed; 057/058 diagnosed and located One push leaves the mesh consistent (the 057 decision, proposed for acceptance); the shared runtime waits for its broker (058). Fixes on mesh-control fix/one-push-is-enough and mesh-tools fix/the-runtime-waits-for-its-broker; the built-store-cross-node bed enforces both. --- ...083-one-push-leaves-the-mesh-consistent.md | 52 +++++++++++++++++++ 02-DECISIONS/README.md | 1 + .../00-report.md | 7 +-- .../01-diagnosis.md | 16 ++++++ .../00-report.md | 5 +- .../01-diagnosis.md | 17 ++++++ 6 files changed, 93 insertions(+), 5 deletions(-) create mode 100644 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md create mode 100644 04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/01-diagnosis.md create mode 100644 04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/01-diagnosis.md diff --git a/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md b/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md new file mode 100644 index 0000000..8773f5e --- /dev/null +++ b/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md @@ -0,0 +1,52 @@ +--- +topic: the mesh +status: proposed +date: 2026-09-18 +deciders: jochen +reconstructed: false +extends: 0010-delivery.md +--- + +# 83. One push leaves the mesh consistent + +## Context + +A provision is minted while composing the *consumer's* node; the provider's grant list is a pure +read of secrets already issued from it. So assigning a cross-node consumer and pushing its node +produced a consumer that retried forever against a provider that had never heard of it, until the +provider's node was pushed a second time — an action with no signal to take, documented nowhere, +and invisible whenever consumer and provider share a machine +([issue 057](../04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md)). + +Two remedies were on the table: **cascade** — a push also delivers to the machines its compose +changed — or **report** — a push says "now push the provider" and leaves the act to the operator. + +## Decision + +A push finishes what it starts: after composing and sending the named node, the controller +recomputes what every machine should be, and any machine whose declaration changed *because of +this push* is sent its declaration too — by name, in the push's own output, converging over a +bounded number of rounds (a cascaded send may itself mint). + +"Changed because of this push" is a comparison, not a guess: the digest of what each machine +should be is captured before the named compose and recomputed after. Machines that were already +behind for unrelated reasons are not swept in — that remains `push --behind`, the explicit +whole-mesh reconcile. + +Reporting alone was rejected because it converts a derived fact the controller already holds into +an operator obligation, and an obligation enforced by nothing is issue 057 restated. The +declaration is computed from the whole mesh; delivering a mesh that is knowingly inconsistent and +merely saying so would make "push succeeded" mean less than it says. + +## Consequences + +- One push is sufficient for a cross-node consumer: the provider's grants arrive from the same + act that minted the provision. The undocumented rule "push the provider node too" ceases to + exist rather than becoming documentation. +- A named push may deliver to machines the operator did not name. This is bounded to machines + whose declarations this push changed, and every one is named in the output — never silent. +- The blast radius question from the issue is answered by the comparison: nothing is recomposed + into delivery except what the named compose provably changed. +- How this is checked: the built-store-cross-node bed registers a cross-node consumer, pushes + only the consumer's node, and asserts the provider minted its vhost — the workaround push is + removed, so a regression fails the bed. diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 5a587cb..27896d9 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -85,6 +85,7 @@ python3 00-META/checks/index.py fail if stale - **0002** — [Nodes communicate over a message broker, not over HTTP](0002-nodes-communicate-over-a-broker.md) - **0003** — [An agent is a persistent employee, not an instance of a pool](0003-agents-are-persistent-employees.md) - **0077** — [The parts are named controller, foundation, node — not control plane, substrate, master](0077-the-controller-and-the-foundation.md) +- **0083** — [One push leaves the mesh consistent](0083-one-push-leaves-the-mesh-consistent.md) *(proposed)* ### Its tiers, from the bottom up diff --git a/04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md b/04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md index a818050..7c8e7ea 100644 --- a/04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md +++ b/04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/00-report.md @@ -1,9 +1,10 @@ --- -status: open +status: located opened: 2026-09-17 -located-in: [] +located-in: + - mesh-controller fixed-by: -amended-design: +amended-design: 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md --- # 057 — A cross-node consumer is provisioned only when the provider is pushed again diff --git a/04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/01-diagnosis.md b/04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/01-diagnosis.md new file mode 100644 index 0000000..ebe8e00 --- /dev/null +++ b/04-ISSUES/057-a-cross-node-consumer-is-provisioned-only-when-the-provider-is-pushed-again/01-diagnosis.md @@ -0,0 +1,16 @@ +# Diagnosis — 2026-09-18 + +The mechanism was already understood when the issue was opened: a provision secret is minted as a +side-effect of composing the consumer's node, and the provider's grant list is a pure read of +secrets already issued — so a provider composed before the consumer existed, and never again, is +blind to it. The open question was the remedy: cascade the push, or report the obligation. + +Decided as [ADR 0083](../../02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md): a push +finishes what it starts. The controller captures the digest of what every machine should be +before composing the named node, recomputes after, and delivers to exactly the machines whose +declaration changed because of this push — named in the output, bounded rounds, converging. +Machines behind for unrelated reasons stay the business of `push --behind`. + +**Located in:** mesh-controller (the push command). Checked by the built-store-cross-node bed, +which now pushes only the consumer's node and asserts the provider minted the vhost — the +workaround push is removed, so a regression fails the bed. diff --git a/04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/00-report.md b/04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/00-report.md index 62feb5c..b10816a 100644 --- a/04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/00-report.md +++ b/04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/00-report.md @@ -1,7 +1,8 @@ --- -status: open +status: located opened: 2026-09-17 -located-in: [] +located-in: + - mesh-tools fixed-by: amended-design: --- diff --git a/04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/01-diagnosis.md b/04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/01-diagnosis.md new file mode 100644 index 0000000..b30706e --- /dev/null +++ b/04-ISSUES/058-a-provisioner-runtime-crash-loops-until-the-overlay-is-up/01-diagnosis.md @@ -0,0 +1,17 @@ +# Diagnosis — 2026-09-18 + +1. The exit is the shared runtime's: serve mode connected the broker once and treated any failure + as fatal, so "the overlay tunnel came up a moment after the container" — the normal case on a + joined node — became an exit, a container-runtime restart, and a visible crash-loop. +2. The failing step is the certificate pre-fetch that pins the broker (a 15s TLS probe), but the + shape is general: any reachability failure at startup had the same consequence. +3. Answering the report's open questions: the shared runtime now retries the broker connection + in-process with capped backoff (2s doubling to 30s), aloud, indefinitely — "how long before it + is fatal" is answered *never*, because the failure modes that do not heal are not reachability: + a pinned-certificate mismatch still refuses immediately (an impostor does not become the + broker by being asked again), and configuration errors still exit at once. One-shot commands + (emit, invoke, run) still fail fast — a person is waiting. +4. Checked by the built-store-cross-node bed: the joined consumer must report a container-runtime + restart count of zero. + +**Located in:** mesh-tools (the shared runtime's serve mode). From 1572d74a183926d53fa5fc98ecb5bf0cd0a2169d Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 18 Sep 2026 02:20:11 +0200 Subject: [PATCH 2/5] =?UTF-8?q?Issue=20063=20=E2=80=94=20the=20foundation'?= =?UTF-8?q?s=20ports=20are=20forwarded,=20resolved?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The broker's port was accepted on input but not forwarded, so a joined node reached a DNAT'd broker only until it restarted. Fixed in mesh-controller (25e42b3), proven by the built-store-cross-node bed adopting the foundation's broker (which restarts it) and the joined node still receiving declarations. --- .../00-report.md | 50 +++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 04-ISSUES/063-the-foundation-ports-are-accepted-on-input-but-not-forwarded/00-report.md diff --git a/04-ISSUES/063-the-foundation-ports-are-accepted-on-input-but-not-forwarded/00-report.md b/04-ISSUES/063-the-foundation-ports-are-accepted-on-input-but-not-forwarded/00-report.md new file mode 100644 index 0000000..7890f37 --- /dev/null +++ b/04-ISSUES/063-the-foundation-ports-are-accepted-on-input-but-not-forwarded/00-report.md @@ -0,0 +1,50 @@ +--- +status: resolved +opened: 2026-09-18 +located-in: + - mesh-controller +fixed-by: mesh-controller fix/one-push-is-enough (25e42b3); gated by the built-store-cross-node bed +amended-design: + +--- + +# 063 — The foundation's ports are accepted on input but not forwarded + +## Symptom + +A joined node reaches the mesh's broker once, then — after the broker restarts — can never reach +it again, and so never receives another declaration. Every check between them passes: the port is +open, the broker is up, the overlay is established. + +Observed on the built-store-cross-node bed, intermittently. A joined node's host connected to the +broker, received its first declaration, and worked. When the foundation's own broker was adopted — +which restarts it — the node's link dropped with `CONNECTION_FORCED - Broker shutdown`, then +`connection refused`, then `i/o timeout` for ever. The node was left with no declaration on disk +at all, and every downstream assertion (here, the registry trust) failed as a consequence of a +node that had received nothing. + +The bed passed on the runs where the broker did not happen to restart after the joined node first +connected, which is why three green runs preceded the red one on identical code. + +## Why it matters beyond the instance + +The mesh firewall opens the foundation's ports — the broker above all — in the **input** chain, +from anywhere, so a node can enrol before it has an overlay address. But the broker is a published +container port: a cross-node packet to it is redirected by the runtime and handled in the +**forward** chain, which never carried a rule for the foundation. The first connection survived +only on its conntrack `established` entry; a broker restart dropped the entry, and the next dial +hit the forward chain's default drop. + +This is [issue 047](../047-the-firewall-does-not-cover-published-container-ports/00-report.md)'s +lesson — a firewall with no forward coverage says nothing about container ports — reappearing for +the one port the whole mesh depends on, because the foundation is not a module and was added to +input alone. A rule that works only until the thing it governs restarts is worse than none: it +passes every test written before the restart. + +## Open questions + +- Is the broker the only foundation port that resolves to a container, or should every foundation + port be assumed to be published and forwarded on principle? (The fix forwards all of them, on + that principle.) +- Should a bed assert reachability *after* a deliberate broker restart, so "reachable until it + bounces" can never again read as "reachable"? From 0d5693cc2c13bf5bba6596778737acbc04b0f98b Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 18 Sep 2026 02:38:00 +0200 Subject: [PATCH 3/5] ADR 0083: the cascade compares against what was last sent, not a snapshot --- ...083-one-push-leaves-the-mesh-consistent.md | 31 ++++++++++++------- 1 file changed, 19 insertions(+), 12 deletions(-) diff --git a/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md b/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md index 8773f5e..011c471 100644 --- a/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md +++ b/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md @@ -23,15 +23,22 @@ changed — or **report** — a push says "now push the provider" and leaves the ## Decision -A push finishes what it starts: after composing and sending the named node, the controller -recomputes what every machine should be, and any machine whose declaration changed *because of -this push* is sent its declaration too — by name, in the push's own output, converging over a -bounded number of rounds (a cascaded send may itself mint). +A push finishes what it starts: after composing and sending the named node, the controller flushes +every *other* machine that is now behind — whose declaration differs from what it was last sent — +by name, in the push's own output, converging over a bounded number of rounds (a flushed send may +itself mint). -"Changed because of this push" is a comparison, not a guess: the digest of what each machine -should be is captured before the named compose and recomputed after. Machines that were already -behind for unrelated reasons are not swept in — that remains `push --behind`, the explicit -whole-mesh reconcile. +Behind is measured against what a machine was last *sent*, not against a before/after snapshot of +this push. The mint that makes a provider behind happens when the consumer is assigned or its +account issued — before `push` runs at all — so by push time the provider already differs from +what it holds, with no in-command delta to detect. The only durable signal is "what it should be" +versus "what it last received", which is the same comparison `push --behind` already makes. + +A machine behind for an unrelated reason is flushed by this too, and that is correct rather than a +cost: a named push that knew a machine was behind and left it so would be the very silence this +decision removes. The narrower reading — flush only what this push provably changed — was +rejected because it cannot see a mint that a prior command performed, which is precisely the 057 +case. Reporting alone was rejected because it converts a derived fact the controller already holds into an operator obligation, and an obligation enforced by nothing is issue 057 restated. The @@ -43,10 +50,10 @@ merely saying so would make "push succeeded" mean less than it says. - One push is sufficient for a cross-node consumer: the provider's grants arrive from the same act that minted the provision. The undocumented rule "push the provider node too" ceases to exist rather than becoming documentation. -- A named push may deliver to machines the operator did not name. This is bounded to machines - whose declarations this push changed, and every one is named in the output — never silent. -- The blast radius question from the issue is answered by the comparison: nothing is recomposed - into delivery except what the named compose provably changed. +- A named push delivers to every machine that is behind, not only the one named — each named in + the output, never silent. `push --behind` remains the way to reconcile the mesh without naming + a node; a named push now carries the same guarantee for the machines its work touched and any + others already waiting. - How this is checked: the built-store-cross-node bed registers a cross-node consumer, pushes only the consumer's node, and asserts the provider minted its vhost — the workaround push is removed, so a regression fails the bed. From 98e9e62fce466ffc8949fb1dc7d3c654967c4c0b Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 18 Sep 2026 02:52:36 +0200 Subject: [PATCH 4/5] Issues 060 resolved, 064 opened; 057/058 on this branch too MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 060: 44 of 70 modules now mesh-buildable (was 8) — the mechanical majority, proven by direct builds and the bed. The structural remainders: external-dependency fetch (new issue 064) and route-proxy's cross-repo build context. 064 records the build environment's isolation from public npm and Docker Hub. --- .../00-report.md | 8 +-- .../01-diagnosis.md | 32 ++++++++++++ .../00-report.md | 51 +++++++++++++++++++ 3 files changed, 88 insertions(+), 3 deletions(-) create mode 100644 04-ISSUES/060-the-mesh-cannot-build-most-of-its-own-catalogue/01-diagnosis.md create mode 100644 04-ISSUES/064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md diff --git a/04-ISSUES/060-the-mesh-cannot-build-most-of-its-own-catalogue/00-report.md b/04-ISSUES/060-the-mesh-cannot-build-most-of-its-own-catalogue/00-report.md index 9719020..75f66c4 100644 --- a/04-ISSUES/060-the-mesh-cannot-build-most-of-its-own-catalogue/00-report.md +++ b/04-ISSUES/060-the-mesh-cannot-build-most-of-its-own-catalogue/00-report.md @@ -1,8 +1,10 @@ --- -status: open +status: resolved opened: 2026-09-17 -located-in: [] -fixed-by: +located-in: + - mesh-catalog + - mesh-controller +fixed-by: mesh-catalog feat/the-mesh-builds-its-catalogue (965c58f); proven by the built-store-cross-node bed and direct builds amended-design: --- diff --git a/04-ISSUES/060-the-mesh-cannot-build-most-of-its-own-catalogue/01-diagnosis.md b/04-ISSUES/060-the-mesh-cannot-build-most-of-its-own-catalogue/01-diagnosis.md new file mode 100644 index 0000000..387be9d --- /dev/null +++ b/04-ISSUES/060-the-mesh-cannot-build-most-of-its-own-catalogue/01-diagnosis.md @@ -0,0 +1,32 @@ +# Diagnosis — 2026-09-18 + +The gap was mechanical for most of the catalogue and structural for a few. + +**The mechanical part (fixed).** 44 of 70 modules now carry a `build` section and a Dockerfile +mirroring the eight that already had one: two named bases (build and runtime), the module's own +source compiled against the sdk in the base, and every serve-time entrypoint named in the runtime +image so tools serve, events flow, and a provider's provisioner reconciles in the same process +(the convention hq issue 061 settled — postgres, gitea and lavinmq were retrofitted to it from +running their provisioner as a separate `args` command). Proven end to end: redis, keycloak, +mongodb, mosquitto and openai-consumer build through the mesh's own builder from their own +directory; the built-store-cross-node bed builds and runs the retrofitted lavinmq and postgres. + +**The structural part (deferred, each with an owner).** + +- Three modules need an artifact the mesh build environment cannot fetch — an npm package + (model-usage: `pg`; anthropic-manager: `tweetnacl`) or a binary from a public image (minio: + `mc`). The build's npm and container runtime reach the mesh's own registry, not the public one; + apt works, npm and image pulls do not. This is its own question — [issue 064](../064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md). +- route-proxy's build context is the mesh-controller repository, not its own module directory (it + ships the packaging of a proxy whose canonical source lives with the contract). The build + section names a Dockerfile relative to the module; a cross-repository context is a shape it + cannot yet express. Deferred pending a design decision on cross-repo build inputs. + +The remaining modules are exempt rather than unbuilt: the foundation trio (builder, +mesh-controller, distribution — built by the foundation's own path, and distribution cannot build +through the store it provides, ADR 0072/issue 029), and the upstream-image-only modules that run +a stock image and carry no code of their own. + +**Located in:** mesh-catalog (the manifests and Dockerfiles), with the convention enforced by +mesh-controller's builder. Resolved for the mechanical majority; the two structural remainders are +tracked as issue 064 and the route-proxy cross-repo deferral. diff --git a/04-ISSUES/064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md b/04-ISSUES/064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md new file mode 100644 index 0000000..e427532 --- /dev/null +++ b/04-ISSUES/064-a-mesh-build-cannot-fetch-a-modules-external-dependencies/00-report.md @@ -0,0 +1,51 @@ +--- +status: located +opened: 2026-09-18 +located-in: + - mesh-catalog + - mesh-controller +fixed-by: +amended-design: +--- + +# 064 — A mesh build cannot fetch a module's external dependencies + +## Symptom + +Three modules cannot be built by the mesh's own builder, each because the build needs an artifact +from a public registry the build environment does not reach: + +- A module whose runtime installs an npm package its code imports (`npm install` in the + Dockerfile) fails with a 404 — the build's npm is pointed at the mesh's own registry, which does + not carry the public package. +- A module whose runtime copies a binary out of a public image (`COPY --from=vendor/tool:latest`) + fails with `pull access denied` — the build's container runtime reaches the mesh's registry, not + the public one. + +apt-based installs in the same Dockerfiles succeed, so the build has ordinary internet: it is npm +and image pulls specifically that are redirected to the mesh's registry. Observed while giving the +catalogue its build sections (hq issue 060): 44 of 70 modules build from their own source with no +external fetch; these three need one and stop. + +## Why it matters beyond the instance + +The delivery design says a module is built by the mesh from a repository and a path (ADR 0069), +compiled against the sdk and tool runtime the base images already carry. That holds for a module +whose only dependencies are the base's — but a module with a third-party runtime dependency, or a +tool binary from a vendor image, has no sanctioned way to bring it into a mesh build. The +workstation build script got away with it by running on a host with public npm and Docker Hub; +retiring that shortcut (the no-fake principle) exposes that the mesh has no answer. + +Left unanswered, "the mesh builds its own catalogue" is true only for modules that happen to have +no external dependency, and which those are is invisible until each is built. + +## Open questions + +- Should a module's third-party npm dependencies be published to the mesh's own registry as part + of building it (a dependency is an artifact like any other), or should the build environment + proxy the public registry read-only? +- Is a vendor binary (`mc`, and others like it) the same question as an npm dependency, or does it + want a distinct answer — a module declaring an external image as a build input the mesh + pre-fetches and pins, the way it pins its bases? +- Should a manifest that declares a build whose Dockerfile fetches from a public registry be + refused at `module add`, so the gap is caught at registration rather than at build? From 05857f71f0f5c1ca555d93f6276ba8d9d4f06d68 Mon Sep 17 00:00:00 2001 From: jochen Date: Sun, 20 Sep 2026 12:52:55 +0200 Subject: [PATCH 5/5] =?UTF-8?q?Accept=20ADR=200083=20=E2=80=94=20one=20pus?= =?UTF-8?q?h=20leaves=20the=20mesh=20consistent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- 02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md | 2 +- 02-DECISIONS/README.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md b/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md index 011c471..d2fbd27 100644 --- a/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md +++ b/02-DECISIONS/0083-one-push-leaves-the-mesh-consistent.md @@ -1,6 +1,6 @@ --- topic: the mesh -status: proposed +status: accepted date: 2026-09-18 deciders: jochen reconstructed: false diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 27896d9..d61a1e6 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -85,7 +85,7 @@ python3 00-META/checks/index.py fail if stale - **0002** — [Nodes communicate over a message broker, not over HTTP](0002-nodes-communicate-over-a-broker.md) - **0003** — [An agent is a persistent employee, not an instance of a pool](0003-agents-are-persistent-employees.md) - **0077** — [The parts are named controller, foundation, node — not control plane, substrate, master](0077-the-controller-and-the-foundation.md) -- **0083** — [One push leaves the mesh consistent](0083-one-push-leaves-the-mesh-consistent.md) *(proposed)* +- **0083** — [One push leaves the mesh consistent](0083-one-push-leaves-the-mesh-consistent.md) ### Its tiers, from the bottom up