Compare commits

...
Author SHA1 Message Date
mesh-admin 9143d0b7c1 Merge pull request 'ADR 0200: genesis pivots to the controller as a container, and the first push hands it to a process' (#346) from decision/0200-genesis-pivots-to-a-container-and-hands-over into main 2026-10-03 23:41:36 +00:00
jochen 24a51a8e53 ADR 0200: genesis pivots to the controller as a container, and the first push hands it to a process 2026-10-04 01:41:30 +02:00
mesh-admin f0d7f91d90 Merge pull request 'Design 38: WP4c complete' (#345) from design/38-wp4c-complete into main 2026-10-03 23:35:31 +00:00
jochen 8578a06ca8 Design 38: WP4c complete, no module's own code runs in a container 2026-10-04 01:35:16 +02:00
mesh-admin 86083f9c9d Merge pull request 'Issues 215, 221, 222 resolved' (#344) from issues/215-221-222-resolved into main 2026-10-03 23:29:38 +00:00
jochen 63d328147e Issues 215, 221 resolved with live proof; 222 diagnosed and resolved 2026-10-04 01:29:32 +02:00
mesh-admin fead0ea440 Merge pull request 'Issues 219-223 and design 38 WP4c built' (#343) from issues/219-223-and-wp4c into main 2026-10-03 23:14:49 +00:00
jochen df503d1cff Issues 219, 220 resolved, 221 located, 222 and 223 opened; WP4c built and proven 2026-10-04 01:14:44 +02:00
mesh-admin c2fc829822 Merge pull request 'Issues 211, 212, 214, 216, 217 resolved; 215's fix recorded' (#342) from issues/211-217-resolved into main 2026-10-03 22:25:07 +00:00
jochen 8d8e5c9a7e Issues 211, 212, 214, 216, 217 resolved with their proofs; 215's fix recorded 2026-10-04 00:24:53 +02:00
mesh-admin affba60b79 Merge pull request 'Issues 219-221: the builder's ordering gaps' (#341) from issues/219-221-the-builder into main 2026-10-03 22:16:13 +00:00
jochen 9ddbc4c68e Issues 219-221: the builder's ordering gaps seen while rolling out issue 218 2026-10-04 00:16:08 +02:00
mesh-admin a972db91f0 Merge pull request 'Issue 218 resolved: the runtime follows its membership, and a refused subscription is not fatal' (#340) from issues/218-rollout into main 2026-10-03 22:14:10 +00:00
jochen 029698fdc8 Issue 218 resolved: the runtime follows its membership, and a refused subscription is not fatal 2026-10-04 00:14:04 +02:00
mesh-admin 895c2afad1 Merge pull request 'Issue 218: a mesh seat answered by a non-holder (located, fixed in mesh-controller#248)' (#339) from issues/218-a-mesh-seat-answered-by-a-non-holder into main 2026-10-03 21:32:24 +00:00
jochen 4b8c5e3b11 Issue 218 located: the controller issued a mesh seat to every claimant; fixed in mesh-controller#248 2026-10-03 23:31:13 +02:00
mesh-admin d362155401 Merge pull request 'Issue 218: a mesh seat is answered by a module on a machine that does not hold it' (#338) from issues/218-a-mesh-seat-answered-by-a-non-holder into main 2026-10-03 21:25:27 +00:00
jochen 557760e537 Issue 218: a mesh seat is answered by a module on a machine that does not hold it 2026-10-03 23:25:17 +02:00
mesh-admin 6b1ebd1d4a Merge pull request 'Issue 217: a refused announcement took down every container's runtime, and the console with it' (#336) from issues/217-a-refused-announcement into main 2026-10-03 21:20:06 +00:00
mesh-admin fa73a17ceb Merge pull request 'Issues 211, 214, 215, 216 diagnosed' (#335) from issues/211-214-216-diagnosed into main 2026-10-03 21:19:03 +00:00
jochen 08cea893bd Issue 217: a refused announcement took down every container's runtime, and the console with it 2026-10-03 22:47:28 +02:00
jochen 04625d3e35 Issues 211, 214, 215, 216 diagnosed: root causes and the branches that fix them 2026-10-03 22:28:40 +02:00
20 changed files with 572 additions and 6 deletions
@@ -0,0 +1,78 @@
---
topic: building it
status: accepted
date: 2026-10-04
deciders: jochen
reconstructed: false
extends: 02-DECISIONS/0067-genesis-is-a-pivot.md
---
# 200. Genesis pivots to the controller as a container, and the first push hands it to a process
## Context
The controller is Go, compiled to one static binary, and is the last of the mesh's own programs a
machine runs from an image ([issue 213](../04-ISSUES/213-the-controller-is-a-go-program-run-in-a-container/00-report.md)).
[ADR 0188](0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md)
§1 says a module's own code is bundles, never an image, and §3 that a service bundle is a `process` the
host runs. The handover exists: a process may name the container it `replaces`, and the host removes
that container only after the process has stayed up across two checks; two controllers are safe
together for that moment, the second standing by on the controller's consumers and every plan held by
one lock.
What stands in the way is genesis ([ADR 0067](0067-genesis-is-a-pivot.md)), which
[issue 223](../04-ISSUES/223-a-new-mesh-installs-its-controller-as-a-container/00-report.md) found
assumes an image and a container at every step from its third: it builds the controller's image,
starts a temporary controller from it, publishes it, finds the controller's container in the pivot
declaration, and from then on talks to the controller through it. A process's bundle is fetched from
the artifact store, and genesis raises the artifact store only after the pivot.
## Considered Options
1. **Raise the artifact store before the pivot**, publish the controller's bundle to it, and talk to
the controller from the host's side. Rejected for now: it reorders genesis around a store that is
itself a module the controller deploys, and rewrites the steps that talk to the controller — a
larger change to the one path that is exercised least, to remove a container that exists for
minutes.
2. **Pivot to the controller as a container, as today, and let the first push hand it over to the
process**, through the handover that already exists. Chosen.
3. **Keep the controller a container.** Rejected: it is the exception to ADR 0188 that every other
module's code has now left, and it costs a container runtime on the control machine and a
container recreation in the middle of a plan.
## Decision
**Genesis raises the controller as a container, under the resource the controller's process
`replaces`, and the first declaration the controller composes for its own machine hands it over.**
The container is genesis's own shape, built from the controller's repository, and is recorded on the
control machine exactly as the manifest's `replaces` names it, so the first apply after the pivot
finds a replacement for it and removes it once the process is up. The controller's manifest declares
only the process; the image form exists for genesis alone and is not a second way to run the
controller on a live mesh.
This is the one bounded exception to ADR 0188 §1: a module's own code in an image, for the minutes
between the pivot and the first push, on a mesh being created.
## Consequences
- A new mesh ends where a running one is: the controller a process, no controller container.
- Genesis keeps its steps; what changes is that it no longer reads the controller's container from the
manifest, and that it records the container under the name the handover expects.
- The controller's repository keeps its image build for genesis and the lab.
- The handover is now on genesis's path too: a process that fails to stay up leaves the genesis
container serving, and the apply says so — the same rule as on a live mesh.
## How it is checked
The installer's test raises a mesh whose controller manifest is the process form, and asserts that the
container genesis recorded is exactly what the process `replaces`, so the first apply hands over and
leaves one controller. Live, on the running mesh: after the manifest change is pushed, the control
machine runs the controller as a process and no controller container, and the controller's seat
answers throughout.
## References
- [Issue 213](../04-ISSUES/213-the-controller-is-a-go-program-run-in-a-container/00-report.md),
[issue 223](../04-ISSUES/223-a-new-mesh-installs-its-controller-as-a-container/00-report.md)
- mesh-host#86 (the handover), mesh-controller#252 (two controllers safe together),
mesh-controller#253 (the controller's manifest as a process)
+1
View File
@@ -320,6 +320,7 @@ python3 00-META/checks/index.py fail if stale
- **0111** — [A build source is on the mesh's git seat, or it is an external repository](0111-a-build-source-is-on-the-git-seat-or-external.md)
- **0149** — [The live mesh is the test bed](0149-the-live-mesh-is-the-test-bed.md)
- **0174** — [A node varies a module through settings and kept regions, never through an edit](0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)
- **0200** — [Genesis pivots to the controller as a container, and the first push hands it to a process](0200-genesis-pivots-to-the-controller-as-a-container-and-the-first-push-hands-it-to-a-process.md)
### How it is checked
@@ -2,7 +2,7 @@
layer: to-be
status: in-progress
code: [mesh-tools, mesh-controller, mesh-host, mesh-catalog]
updated: 2026-10-03
updated: 2026-10-04
decisions:
- 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
- 02-DECISIONS/0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md
@@ -309,6 +309,25 @@ nextcloud, minio), the two whose clients exist on no system (mongodb, mssql: a d
and last the mesh's own (mesh-catalog, mesh-vault, records, gitea, mailu, audit-logger, lab, and the
three mains).
*Built 2026-10-04.* Thirty-four modules no longer run their own code in a container: the first wave
(mesh-catalog#245), the mesh's own and the media modules (mesh-catalog#248, mesh-media-catalog#13),
with a step run where and as it is declared (mesh-host#85) and a process's words filled like a
container's (mesh-controller#250). **Proven live** on every machine that runs them: each moved
module's tools answer from the node's runtime, the steps run as their oneshot units, and the forge's
merge events reach the build pipeline from the runtime — the merge after the move started its own
plan. **Two corrections the machines taught:** a tool that called a broker's command-line client now
runs it inside the broker's own container, because a host package may be uninstallable on a machine
whose package index is stale (mesh-catalog#249); and a module reading its application's own key reads
it through the application's container, because that directory belongs to the account the application
runs as there, which is not the runtime's (mesh-media-catalog#14). *Completed 2026-10-04:* the last three — mesh-catalog, mongodb and mssql, whose code imports npm
packages of its own — moved once the builder installs a bundle's own dependencies before compiling,
keeping the toolchain's SDK authoritative (mesh-controller#255, mesh-catalog#250). Their database
clients are now drivers inlined into the bundle, not command-line clients fetched by a container; one
more correction the machines taught: a driver reaching its server on loopback must give TLS a host
name, since the runtime's Node refuses an address (mesh-catalog#252). **No module's own code runs in
a container any more;** every module with tools answers from its node's runtime, proven by calling a
tool of each.
## WP5 — The shell, on a server first
*mesh-catalog #224, already written. Half a day to assign and prove.*
@@ -1,9 +1,10 @@
---
status: located
status: resolved
opened: 2026-10-03
located-in:
- mesh-controller
fixed-by:
- mesh-controller#246
amended-design:
---
@@ -45,3 +46,7 @@ have to write — a bundle of language L depends on the module that publishes L'
bundle lands in the tier after it. A test: a merge touching the toolchain module and a TypeScript
bundle plans the bundle one tier later. Worked around on the day by building the bundle again once
the toolchain was built.
## Resolved
Every mesh-tools plan since the fix ran in two rounds: the toolchain and runtime images first, what is built in or on them after. Before it, a merge moving both tiered them together. The planner's test proves the order: a merge moving a bundle and its toolchain plans two rounds.
@@ -0,0 +1,11 @@
# 211 — Diagnosis
*2026-10-03.* The planner orders a merge's modules by `inventory.Dependencies`, whose edges come from a
manifest's `build.on`, from what a build recorded it stood on, from the repositories it read, and from
the build machine. A bundle names its toolchain by `language`; the builder takes the toolchain image
(`ToolchainFor(language)`) from what the mesh holds and records nothing of it as stood on. So no edge
ran from a bundle to the module publishing its toolchain, and a merge moving both (mesh-tools: the
images and node-tools) tiered them together. **Fix (mesh-controller, branch
`fix/issue-211-a-bundle-stands-on-its-toolchain`, commit c72f6ca):** `dependenciesOf` adds a `stands-on`
edge from every bundle artifact to its toolchain's module, read from the manifest. Tested: TypeScript
bundle → mesh-tools, Go bundle → mesh-tools-go, image → none; a merge moving both plans two tiers.
@@ -1,10 +1,11 @@
---
status: located
status: resolved
opened: 2026-10-03
located-in:
- mesh-tools
- mesh-controller
fixed-by:
- mesh-tools#44
amended-design:
---
@@ -42,3 +43,7 @@ install from a lockfile that a release updates, or build the install layer witho
the build should record which SDK version the image carries, so a bundle's record says what it was
compiled against. A check: after an SDK release and a toolchain rebuild, a bundle built on it reports
the released version.
## Resolved
The toolchain now installs the exact SDK version the mesh last published, passed in as a build argument from the SDK module's published package, and the planner orders the toolchain after the SDK. Proven 2026-10-04: the toolchain built with the published SDK, and the six TypeScript bundles built in it serve their tools and seat verbs.
@@ -50,3 +50,14 @@ of a plan today.
Located only by owner; the move is a change of the controller's module and its deployment, not of
its code.
## Fix prepared (2026-10-04)
Three changes. mesh-host#86, merged: a process may name the container it `replaces`, and the host
removes that container only once the process has stayed up across two checks. mesh-controller#252,
awaiting the operator's merge: the controller's composition for a service process, and two
controllers safe together for the handover — the second stands by on the controller's consumers
until the first lets go, and all plan work holds one advisory lock. mesh-controller#253, held: the
controller's manifest as a Go bundle and a process. It waits on
[issue 223](../223-a-new-mesh-installs-its-controller-as-a-container/00-report.md), because with it a
new mesh cannot be installed.
@@ -1,9 +1,10 @@
---
status: open
status: resolved
opened: 2026-10-03
located-in:
- mesh-controller
fixed-by:
- mesh-controller#246
amended-design:
---
@@ -34,3 +35,7 @@ Owner mesh-controller (the planner). **Fix direction:** on start, and whenever a
build, the plan settles an `asked` build against the build records — a build recorded as built from
the plan's commit is that tier's outcome — so a plan resumes after the controller replaced itself.
A test: a plan whose build outcome was recorded while no controller followed it resumes on start.
## Resolved
A plan settles an asked build from the build records, whoever heard the outcome. Proven 2026-10-03 and 2026-10-04: both controller merge plans since the fix finished all three rounds, including the round that replaced the controller, without being stopped by hand.
@@ -0,0 +1,10 @@
# 214 — Diagnosis
*2026-10-03.* A plan learns a tier's outcome only through `planBuilt`, called when a controller takes
in a build result off the bus. A merge to the controller's repository replaces the controller in tier
0; the build that produced the new controller was recorded, but the plan state the new controller
loaded still read `asked`, and no path ever revisited it. **Fix (branch
`fix/issue-214-a-plan-settles-from-the-build-records`, commit d86baeb):** `advanceOnce` settles every
still-asked module from its build records — a build recorded after the ask is that ask's outcome,
built or failed — on every advance and on the 30-second ticker. Tested with a pure helper. Live
proof: the next merge to mesh-controller passes tier 0 on its own.
@@ -1,9 +1,10 @@
---
status: open
status: resolved
opened: 2026-10-03
located-in:
- mesh-controller
fixed-by:
- mesh-controller#246
amended-design:
---
@@ -33,3 +34,9 @@ Owner mesh-controller. **Fix direction:** a build asked at a commit does not cha
module follows; or, if pinning is meant, the pin is said — in `module list`, in `status`, and by a
merge's plan naming the module it leaves out and why. A test: building a module at a commit and then
merging a change to it plans it.
## Resolved
Proven 2026-10-04: the catalogue merges since the fix rebuilt the module that had been pinned at an
old commit, at the merge's commit, by an ordinary plan — the same as every other module of the
catalogue. Nothing was asked for it by hand.
@@ -0,0 +1,10 @@
# 215 — Diagnosis
*2026-10-03.* `takeIn` registers a build's `Ref` as the branch the module follows. unifi was once
built with `ref=9c97a8a`, which became its followed ref. `sourceIs` matches a merge only to modules
whose ref is empty or the merged base — so every merge into main left unifi out — and `askTier`
re-asks `Source.Ref`, so every plan that rebuilt unifi built the same old commit again (its build
records all read "at 9c97a8a"). **Fix (branch `fix/issue-215-a-commit-is-never-a-branch-to-follow`,
commit 6784efa):** registration keeps the followed branch when a build names a commit; matching and
re-asking read a recorded commit as the default branch, healing existing records; a merge names the
modules of its repository it leaves out. Store-backed test fails without the fix.
@@ -1,9 +1,10 @@
---
status: open
status: resolved
opened: 2026-10-03
located-in:
- mesh-controller
fixed-by:
- mesh-controller#246
amended-design:
---
@@ -28,3 +29,7 @@ Owner mesh-controller (the catalogue's registration check). **Fix direction:** a
loads, runs or unpacks — no `loads`, no `tools` list on its module, no resource naming it — is refused
at registration, naming the field that would deliver it. A test: such a manifest is refused; adding
`loads` admits it.
## Resolved
A bundle nothing would deliver is refused at registration, by name. Proven by the controller's tests; the catalogue's bundles all name what the runtime loads (mesh-catalog#243).
@@ -0,0 +1,9 @@
# 216 — Diagnosis
*2026-10-03.* The composer delivers a bundle as an archive only when its `Loads` is non-empty, and
`Loads` derives from the artifact's `loads` or, failing that, from the module's `tools` list. The
seven modules had neither, so their bundles were recorded and never composed into any declaration;
nothing checked it. **Fix (branch `fix/issue-216-a-bundle-nothing-delivers-is-refused`, commit
cf2bb3b):** registration refuses a bundle that nothing loads, runs or unpacks — no `loads`, no `tools`,
no resource naming it, and not the runtime — naming the field that would deliver it. The current
catalogue passes the check.
@@ -0,0 +1,58 @@
---
status: resolved
opened: 2026-10-03
located-in:
- mesh-tools
fixed-by:
- mesh-tools#42
- mesh-tools#43
amended-design:
---
# 217 — A refused announcement took down every container's runtime, and the console with it
## What was observed
2026-10-03, rolling out [ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md).
The tool runtimes announced themselves by subscribing `$SRV.<verb>.>`; the grants composed for them
allowed only `$SRV.<verb>` and the service's own name and instance. The bus refused the wildcard:
```
Subscription Violation - User "novox.gitea", Subject "$SRV.PING.>"
NatsError: 'Permissions Violation for Subscription to "$SRV.PING.>"'
```
The TypeScript runtime the per-module containers run treats a refused subscription as fatal, so on
the one machine that had received the new images nine containers crash-looped — gitea's runtime,
postgres's, the catalogue, the vault, mongodb, mssql, keycloak, mailu, nextcloud. Gitea's tools went
with them, which closed the usual path for merging the fix.
Then the console stopped answering the controller's verbs, though the controller held every
subscription: the console builds the list that tells a seat's verb from a module's tool by asking the
controller *and* the catalogue, and gives up on both when the catalogue does not answer — so
`mesh-controller.status` was asked of a module subject nobody serves.
## Why it matters beyond this instance
Two properties, each worse than the mistake that exposed it:
- **A runtime dies for an optional subscription.** Announcing is discovery; serving tools and running
provisioners is the work. A refusal of the first should never stop the second.
- **The console's view of the mesh's own verbs depended on a module.** The controller's verbs are how
the operator repairs the mesh; they must not become unreachable because the catalogue is down.
## Diagnosis
Owner mesh-tools. The wildcard is fixed on branch `fix/announce-only-what-the-grants-allow`: both
runtimes subscribe exactly what the grants allow. The console that discovers from what announces itself
(ADR 0195, 0197, on main) asks the bus and the controller, not the catalogue, which removes the second
property once it is deployed. **Still to do:** the TypeScript runtime treats a refused announcement
subscription as a logged warning, not as fatal; a test against a bus with real grants proves the
announcement subscriptions are allowed for every principal kind.
Recovered on the day without the forge's API: the toolchain images built by the controller straight
from the fix branch, every module image rebuilt on them, and the machine pushed.
## Resolved
A refused announcement is logged and the runtime serves on, and every runtime subscribes only the discovery subjects its grants allow. Proven 2026-10-04: after rolling out to every machine, no container restarts anywhere, and the discovery console lists no runtime as not answering. A refused tool subscription was made non-fatal the same way afterwards, under issue 218 (mesh-tools#46).
@@ -0,0 +1,92 @@
---
status: resolved
opened: 2026-10-03
located-in:
- mesh-controller
- mesh-tools
fixed-by:
- mesh-controller#248
- mesh-tools#45
- mesh-tools#46
amended-design:
---
# 218 — A seat held once for the mesh is answered by a module on a machine that does not hold it
## What was observed
2026-10-03. Asked which databases the mesh's store holds, `mesh-store.databases` answered from the
postgres on one machine with that machine's application databases; the controller's own database
lives on the postgres of the other machine, which the controller's records name as the seat's one
holder:
```
mesh-store scope: mesh delivers: postgres-database holders: [ {node: <the control machine>, module: postgres} ]
```
The discovery console, which reads what answers on the bus ([ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)),
shows the same seat announced from **both** machines running postgres.
## Why it matters beyond this instance
A seat held once for the mesh promises one answerer: the role's holder. A module that implements a
seat's verbs on every machine it runs on, and is let serve them on each, turns "the mesh's store" into
"whichever postgres replied first" — a read against the wrong database that looks like a right one,
and a write would be worse. Every mesh-scoped seat whose implementing module runs on more than one
machine has this shape.
## Where to look
Whether the runtime serves a seat's verbs where its module merely *claims* the seat rather than where
the mesh made it the holder ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md),
[ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)):
the membership the controller issues each assignment, and what the runtime admits from it. A check:
a mesh-scoped seat's verbs are served by exactly the holder the records name, on every machine.
## Root cause
The controller composed each assignment's held seats from what its module *claims*, once per module
and not once per machine. Every machine running postgres was therefore given the store seat's grants
and issued its subjects, and each runtime served the seat's verbs because it serves what it is issued
([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
The runtime behaved as designed. The fault was in what it was issued.
The seat's verbs are not the module's tools. The store's `databases` and `query` are a separate
implementation registered under the seat's name ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md)).
Only that implementation should be withdrawn where the module does not hold the seat. postgres's own
tools stay served on every machine it runs on.
## Fix
The controller now reads the recorded seat holdings when it composes grants and memberships. A seat
held once for the mesh is issued only to the machine and module the records name as its holder. A
seat held once per machine, and a mesh seat with no holder on record, are issued as before. Grants
and memberships come from the same list, so they cannot disagree.
**How it is checked.** A controller test asserts that a claimant on another machine keeps its node
seats and loses the recorded mesh seat. Live, the discovery console's overview must show each
mesh-scoped seat announced from exactly the holder the records name. Status moves to `resolved` once
that holds after the fix is rolled out.
## A second cause, and what the rollout broke (2026-10-04)
With the grants corrected, calls to the store reached only the holder, yet the console still showed
the seat announced from both machines. The module's runtime added every seat its start-up credential
claims, even after the mesh issued a membership that left the seat out. Once a membership exists,
it now alone decides which seat verbs a runtime serves (mesh-tools#45).
The rollout then exposed a third fault. The module on the machine that does not hold the seat was
still running an image built before #45, so it subscribed to the seat's subject. The corrected grants
refused that subscription, and the refusal ended the process. Its runtime crash-looped until the
module was rebuilt on the new runtime image. The database itself kept running. A refused tool
subscription is now logged and costs only that subject (mesh-tools#46), as a refused announcement
already did ([issue 217](../217-a-refused-announcement-took-down-every-containers-runtime/00-report.md)).
The module was not rebuilt by the plan that rebuilt the runtime image. This is the ordering gap of
[issue 211](../211-a-bundle-is-built-before-the-toolchain-it-is-compiled-in/00-report.md) seen
from a container module.
**Proven 2026-10-04.** The discovery console's overview shows the store seat announced from the
recorded holder only. Repeated calls to the store are answered by that machine, and the answers
include the controller's own database. The non-holder still answers its own module tools. No
container restarts on any machine.
@@ -0,0 +1,48 @@
---
status: resolved
opened: 2026-10-04
located-in:
- mesh-controller
fixed-by:
- mesh-controller#249
amended-design:
---
# 219 — An older build that finishes later replaces a newer one
## What was observed
2026-10-03. Two merges to the runtime module came minutes apart. Each plan asked for every module
built on the runtime image to be rebuilt. One module's two builds were of the same source and
differed only in the runtime image they stood on:
| Build | Requested | Finished | Stood on |
|---|---|---|---|
| asked by the first plan | 21:33 | 22:04 | the runtime image before the fix |
| asked by the second plan | 21:49 | 21:56 | the runtime image with the fix |
The older request finished last, and its image became the module's current artifact. The next push
deployed it, and the module's runtime crash-looped on a fault the newer image had already fixed
([issue 218](../218-a-mesh-seat-is-answered-by-a-module-that-does-not-hold-it/00-report.md)).
## Why it matters beyond this instance
Which build is current should follow what it was built from, not which build machine was slowest.
Whenever two plans overlap, which happens on any busy evening, a fix can be silently reverted by a
build that started before it existed. Every passing check still passes.
## Where to look
How a finished build is recorded and how a module's current artifact is chosen. **How it is checked:**
a test in which an older request completes after a newer one for the same artifact, and the newer
stays current.
## Resolved
A build is ordered by when it was requested, read from the id the controller gives it, and a
module's registered manifest is replaced only by a build requested at or after the one it came from.
An older request finishing later is recorded and changes nothing; a plan settles only from builds it
asked for itself. **How it is checked:** store-backed tests replay the incident — the newer request
stays what the module is — and fail without the fix. Live since 2026-10-04: the rebuilds of every
runtime-image module and the three waves of module code moves since then each registered the build
they asked for.
@@ -0,0 +1,41 @@
---
status: resolved
opened: 2026-10-04
located-in:
- mesh-host
fixed-by:
- mesh-host#84
amended-design:
---
# 220 — A delivered bundle keeps the files of the one before
## What was observed
2026-10-04. A tools bundle was rebuilt as one self-contained file per entrypoint
([ADR 0193](../../02-DECISIONS/0193-every-bundle-the-runtime-serves-is-launched-and-the-runtime-knows-no-language.md)),
so it no longer carries a package directory. On the machine it was delivered to, its directory still
held the package directory and a compiled file from the earlier delivery, dated hours before the
new files. The new files were written over the old directory, and nothing removed what the new
bundle no longer contains.
## Why it matters beyond this instance
A bundle on disk should be exactly the artifact that was built. Leftover files can be imported by
code that should no longer find them. A fix that removes a file then works on a fresh machine and
fails on every machine that ran an earlier version. It also makes "what runs here" impossible to
read from the artifact.
## Where to look
How the host unpacks a bundle into its directory. **How it is checked:** deliver a bundle, then a
version without one of its files, and the file is gone.
## Resolved
An archive is unpacked into a fresh directory beside the old one and swapped in by rename; a refused
or failed unpack leaves the old tree whole. **How it is checked:** a second delivery without a file
removes it, and nothing is left beside the directory; both tests fail without the fix. Proven
2026-10-04: a bundle rebuilt and delivered after the fix holds exactly the new build and nothing
beside it. A bundle that has not changed since keeps its old leftovers until its next version, by
design: an unchanged archive is not unpacked again.
@@ -0,0 +1,52 @@
---
status: resolved
opened: 2026-10-04
located-in:
- mesh-controller
fixed-by:
- policy: `upgrade build-agent roll-out` (2026-10-04)
amended-design:
---
# 221 — A build machine learns a new builder only from a push
## What was observed
2026-10-04. A controller merge changed how TypeScript bundles are built: one file per entrypoint
instead of a package directory. Its plan finished, and six bundles were rebuilt right after. They
came out in the old shape, because the build machines still ran the previous builder. They got the
new one only from the next push. Rebuilt after that push, the same six came out right.
The same order showed in the bus grants the same night. A push sent while the controller was still
the previous build composed grants with the previous code. A second push was needed after the new
controller had started.
## Why it matters beyond this instance
A merge to the controller is not in effect when its plan says done. Anything built or pushed in the
gap uses the old code and looks current. Today only the operator knows to push first and build
second, and even the operator forgot.
## Where to look
Whether a controller plan should end by delivering itself to the build machines and the control
machine, or whether a build should refuse a builder older than the controller that asked for it.
**How it is checked:** after a controller merge, a build asked right after its plan finishes runs
the new builder.
## Located
The mechanism existed: a module whose upgrade policy is `roll-out` is sent to its machines when its
tier is built, and the plan's next tier waits until it is applied. The build agent's policy was
`record` — built, never sent — as was that of 76 other modules, which is why every rollout on
2026-10-03 needed a push by hand. The build agent was set to `roll-out`, one machine at a time,
stopping at the first failure. **How it is checked:** the next controller merge's plan sends the
build agent to the build machines before its last tier, and a build asked right after the plan
finishes runs the new builder; then this moves to resolved. Whether the other modules roll out is
the operator's policy, not this issue's.
## Resolved
Proven 2026-10-04: the next controller merge's plan logged that its first tier was built and the
build agent sent to all four build machines, and only then asked its next tier. The builder change
in that merge reached the build machines without a hand push.
@@ -0,0 +1,51 @@
---
status: resolved
opened: 2026-10-04
located-in:
- mesh-tools
fixed-by:
- mesh-tools#47
amended-design:
---
# 222 — An assignment is refused on the bus until the bus's machine is pushed
## What was observed
2026-10-04. A module was assigned to the laptop, and the laptop alone was pushed. The module's two
bundles arrived and the node's runtime launched both, and then the bus refused every one of the
module's subjects:
```
nats: permissions violation: Permissions Violation for Subscription to "mesh.mod.<module>.tool.<tool>.<node>"
```
The tools were unreachable until a later push that included the machine running the bus. Then the
runtime served them without a restart, because a new membership arrived and it re-subscribed.
## Why it matters beyond this instance
What an account may answer lives in the bus's user list, and the controller writes that list only
into the declaration of the machine that runs the bus. Assigning anything to any machine changes
that list, so `push <machine>` after `assign <machine> <module>`, which is what the controller itself
tells the operator to run, leaves the module running and unreachable, with nothing reporting a fault.
## Where to look
Whether a push to one machine should also send the bus's machine when the user list it would
compose differs from the one that machine holds. **How it is checked:** assign a module with tools
to a machine that does not run the bus, push only that machine, and its tools answer.
## Diagnosed and resolved
The push did send the bus's machine: the controller's log shows both machines applying in the same
second. The fault was the order within that second. The node's runtime subscribed before the bus
had reloaded its user list, the bus refused, and the bus client marks a refused subscription dead.
Nothing asked again until a later membership happened to re-serve the module.
A subject the runtime answers on is now asked for again when the bus refuses it, after waits from
two seconds to two minutes, and given up and said after about five minutes. **How it is checked:**
a test refuses a subject and finds it asked for again and answering, given up past its attempts,
and not asked again once stopped; and by hand against a bus whose permissions were reloaded while
connected, the runtime answered two seconds after the grant arrived, where the runtime before the
fix never answered.
@@ -0,0 +1,48 @@
---
status: located
opened: 2026-10-04
located-in:
- mesh-host
fixed-by:
amended-design:
---
# 223 — A new mesh installs its controller as a container
## What was observed
2026-10-04. Preparing [issue 213](../213-the-controller-is-a-go-program-run-in-a-container/00-report.md),
the controller's own manifest was changed to a Go bundle run as a process. The installer that raises a
new mesh assumes the controller is an image and a container at every step from its third on:
- it requires the controller's build to produce exactly one image;
- it starts a temporary controller from that image, and publishes the image to the registry;
- at the pivot, it finds the controller's container in the declaration, reads its environment and
volumes, waits for it, and from then on talks to the controller only through the container.
## Why it matters beyond this instance
With the controller's manifest changed, a new mesh cannot be installed: the pivot fails. The deeper
constraint is ordering. A process's bundle is fetched from the artifact store, and the installer
raises the artifact store only after the pivot, so the controller's first declaration names a bundle
nothing can serve yet.
## What a fix has to settle
One of two shapes, and it is a decision, not a repair:
1. raise the artifact store before the pivot, publish the controller's bundle to it, and talk to the
controller from the host's side rather than through a container; or
2. pivot to the image form as today, and let the first push hand over to the process, which
requires the controller's manifest to carry both forms.
Until it is settled, the change of the controller's manifest (mesh-controller#253) is held. The
handover itself is built and merged (mesh-host#86); the controller's half (mesh-controller#252) waits
on the operator. **How it is checked:** the installer's test raises a mesh whose controller manifest
is the process form, and the controller answers its seat's verbs at the end.
## Decided (2026-10-04)
Option 2, [ADR 0200](../../02-DECISIONS/0200-genesis-pivots-to-the-controller-as-a-container-and-the-first-push-hands-it-to-a-process.md):
genesis pivots to the controller as a container recorded under the name the process `replaces`, and
the first push hands it over.