Compare commits
22
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9143d0b7c1 | ||
|
|
24a51a8e53 | ||
|
|
f0d7f91d90 | ||
|
|
8578a06ca8 | ||
|
|
86083f9c9d | ||
|
|
63d328147e | ||
|
|
fead0ea440 | ||
|
|
df503d1cff | ||
|
|
c2fc829822 | ||
|
|
8d8e5c9a7e | ||
|
|
affba60b79 | ||
|
|
9ddbc4c68e | ||
|
|
a972db91f0 | ||
|
|
029698fdc8 | ||
|
|
895c2afad1 | ||
|
|
4b8c5e3b11 | ||
|
|
d362155401 | ||
|
|
557760e537 | ||
|
|
6b1ebd1d4a | ||
|
|
fa73a17ceb | ||
|
|
08cea893bd | ||
|
|
04625d3e35 |
+78
@@ -0,0 +1,78 @@
|
||||
---
|
||||
topic: building it
|
||||
status: accepted
|
||||
date: 2026-10-04
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
extends: 02-DECISIONS/0067-genesis-is-a-pivot.md
|
||||
---
|
||||
|
||||
# 200. Genesis pivots to the controller as a container, and the first push hands it to a process
|
||||
|
||||
## Context
|
||||
|
||||
The controller is Go, compiled to one static binary, and is the last of the mesh's own programs a
|
||||
machine runs from an image ([issue 213](../04-ISSUES/213-the-controller-is-a-go-program-run-in-a-container/00-report.md)).
|
||||
[ADR 0188](0188-a-modules-own-code-is-bundles-in-any-language-and-a-tools-bundle-speaks-mcp-to-the-runtime.md)
|
||||
§1 says a module's own code is bundles, never an image, and §3 that a service bundle is a `process` the
|
||||
host runs. The handover exists: a process may name the container it `replaces`, and the host removes
|
||||
that container only after the process has stayed up across two checks; two controllers are safe
|
||||
together for that moment, the second standing by on the controller's consumers and every plan held by
|
||||
one lock.
|
||||
|
||||
What stands in the way is genesis ([ADR 0067](0067-genesis-is-a-pivot.md)), which
|
||||
[issue 223](../04-ISSUES/223-a-new-mesh-installs-its-controller-as-a-container/00-report.md) found
|
||||
assumes an image and a container at every step from its third: it builds the controller's image,
|
||||
starts a temporary controller from it, publishes it, finds the controller's container in the pivot
|
||||
declaration, and from then on talks to the controller through it. A process's bundle is fetched from
|
||||
the artifact store, and genesis raises the artifact store only after the pivot.
|
||||
|
||||
## Considered Options
|
||||
|
||||
1. **Raise the artifact store before the pivot**, publish the controller's bundle to it, and talk to
|
||||
the controller from the host's side. Rejected for now: it reorders genesis around a store that is
|
||||
itself a module the controller deploys, and rewrites the steps that talk to the controller — a
|
||||
larger change to the one path that is exercised least, to remove a container that exists for
|
||||
minutes.
|
||||
2. **Pivot to the controller as a container, as today, and let the first push hand it over to the
|
||||
process**, through the handover that already exists. Chosen.
|
||||
3. **Keep the controller a container.** Rejected: it is the exception to ADR 0188 that every other
|
||||
module's code has now left, and it costs a container runtime on the control machine and a
|
||||
container recreation in the middle of a plan.
|
||||
|
||||
## Decision
|
||||
|
||||
**Genesis raises the controller as a container, under the resource the controller's process
|
||||
`replaces`, and the first declaration the controller composes for its own machine hands it over.**
|
||||
The container is genesis's own shape, built from the controller's repository, and is recorded on the
|
||||
control machine exactly as the manifest's `replaces` names it, so the first apply after the pivot
|
||||
finds a replacement for it and removes it once the process is up. The controller's manifest declares
|
||||
only the process; the image form exists for genesis alone and is not a second way to run the
|
||||
controller on a live mesh.
|
||||
|
||||
This is the one bounded exception to ADR 0188 §1: a module's own code in an image, for the minutes
|
||||
between the pivot and the first push, on a mesh being created.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A new mesh ends where a running one is: the controller a process, no controller container.
|
||||
- Genesis keeps its steps; what changes is that it no longer reads the controller's container from the
|
||||
manifest, and that it records the container under the name the handover expects.
|
||||
- The controller's repository keeps its image build for genesis and the lab.
|
||||
- The handover is now on genesis's path too: a process that fails to stay up leaves the genesis
|
||||
container serving, and the apply says so — the same rule as on a live mesh.
|
||||
|
||||
## How it is checked
|
||||
|
||||
The installer's test raises a mesh whose controller manifest is the process form, and asserts that the
|
||||
container genesis recorded is exactly what the process `replaces`, so the first apply hands over and
|
||||
leaves one controller. Live, on the running mesh: after the manifest change is pushed, the control
|
||||
machine runs the controller as a process and no controller container, and the controller's seat
|
||||
answers throughout.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 213](../04-ISSUES/213-the-controller-is-a-go-program-run-in-a-container/00-report.md),
|
||||
[issue 223](../04-ISSUES/223-a-new-mesh-installs-its-controller-as-a-container/00-report.md)
|
||||
- mesh-host#86 (the handover), mesh-controller#252 (two controllers safe together),
|
||||
mesh-controller#253 (the controller's manifest as a process)
|
||||
@@ -320,6 +320,7 @@ python3 00-META/checks/index.py fail if stale
|
||||
- **0111** — [A build source is on the mesh's git seat, or it is an external repository](0111-a-build-source-is-on-the-git-seat-or-external.md)
|
||||
- **0149** — [The live mesh is the test bed](0149-the-live-mesh-is-the-test-bed.md)
|
||||
- **0174** — [A node varies a module through settings and kept regions, never through an edit](0174-a-node-varies-a-module-through-settings-and-kept-regions-never-an-edit.md)
|
||||
- **0200** — [Genesis pivots to the controller as a container, and the first push hands it to a process](0200-genesis-pivots-to-the-controller-as-a-container-and-the-first-push-hands-it-to-a-process.md)
|
||||
|
||||
### How it is checked
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: to-be
|
||||
status: in-progress
|
||||
code: [mesh-tools, mesh-controller, mesh-host, mesh-catalog]
|
||||
updated: 2026-10-03
|
||||
updated: 2026-10-04
|
||||
decisions:
|
||||
- 02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md
|
||||
- 02-DECISIONS/0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md
|
||||
@@ -309,6 +309,25 @@ nextcloud, minio), the two whose clients exist on no system (mongodb, mssql: a d
|
||||
and last the mesh's own (mesh-catalog, mesh-vault, records, gitea, mailu, audit-logger, lab, and the
|
||||
three mains).
|
||||
|
||||
*Built 2026-10-04.* Thirty-four modules no longer run their own code in a container: the first wave
|
||||
(mesh-catalog#245), the mesh's own and the media modules (mesh-catalog#248, mesh-media-catalog#13),
|
||||
with a step run where and as it is declared (mesh-host#85) and a process's words filled like a
|
||||
container's (mesh-controller#250). **Proven live** on every machine that runs them: each moved
|
||||
module's tools answer from the node's runtime, the steps run as their oneshot units, and the forge's
|
||||
merge events reach the build pipeline from the runtime — the merge after the move started its own
|
||||
plan. **Two corrections the machines taught:** a tool that called a broker's command-line client now
|
||||
runs it inside the broker's own container, because a host package may be uninstallable on a machine
|
||||
whose package index is stale (mesh-catalog#249); and a module reading its application's own key reads
|
||||
it through the application's container, because that directory belongs to the account the application
|
||||
runs as there, which is not the runtime's (mesh-media-catalog#14). *Completed 2026-10-04:* the last three — mesh-catalog, mongodb and mssql, whose code imports npm
|
||||
packages of its own — moved once the builder installs a bundle's own dependencies before compiling,
|
||||
keeping the toolchain's SDK authoritative (mesh-controller#255, mesh-catalog#250). Their database
|
||||
clients are now drivers inlined into the bundle, not command-line clients fetched by a container; one
|
||||
more correction the machines taught: a driver reaching its server on loopback must give TLS a host
|
||||
name, since the runtime's Node refuses an address (mesh-catalog#252). **No module's own code runs in
|
||||
a container any more;** every module with tools answers from its node's runtime, proven by calling a
|
||||
tool of each.
|
||||
|
||||
## WP5 — The shell, on a server first
|
||||
|
||||
*mesh-catalog #224, already written. Half a day to assign and prove.*
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
opened: 2026-10-03
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by:
|
||||
- mesh-controller#246
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -45,3 +46,7 @@ have to write — a bundle of language L depends on the module that publishes L'
|
||||
bundle lands in the tier after it. A test: a merge touching the toolchain module and a TypeScript
|
||||
bundle plans the bundle one tier later. Worked around on the day by building the bundle again once
|
||||
the toolchain was built.
|
||||
|
||||
## Resolved
|
||||
|
||||
Every mesh-tools plan since the fix ran in two rounds: the toolchain and runtime images first, what is built in or on them after. Before it, a merge moving both tiered them together. The planner's test proves the order: a merge moving a bundle and its toolchain plans two rounds.
|
||||
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
# 211 — Diagnosis
|
||||
|
||||
*2026-10-03.* The planner orders a merge's modules by `inventory.Dependencies`, whose edges come from a
|
||||
manifest's `build.on`, from what a build recorded it stood on, from the repositories it read, and from
|
||||
the build machine. A bundle names its toolchain by `language`; the builder takes the toolchain image
|
||||
(`ToolchainFor(language)`) from what the mesh holds and records nothing of it as stood on. So no edge
|
||||
ran from a bundle to the module publishing its toolchain, and a merge moving both (mesh-tools: the
|
||||
images and node-tools) tiered them together. **Fix (mesh-controller, branch
|
||||
`fix/issue-211-a-bundle-stands-on-its-toolchain`, commit c72f6ca):** `dependenciesOf` adds a `stands-on`
|
||||
edge from every bundle artifact to its toolchain's module, read from the manifest. Tested: TypeScript
|
||||
bundle → mesh-tools, Go bundle → mesh-tools-go, image → none; a merge moving both plans two tiers.
|
||||
@@ -1,10 +1,11 @@
|
||||
---
|
||||
status: located
|
||||
status: resolved
|
||||
opened: 2026-10-03
|
||||
located-in:
|
||||
- mesh-tools
|
||||
- mesh-controller
|
||||
fixed-by:
|
||||
- mesh-tools#44
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -42,3 +43,7 @@ install from a lockfile that a release updates, or build the install layer witho
|
||||
the build should record which SDK version the image carries, so a bundle's record says what it was
|
||||
compiled against. A check: after an SDK release and a toolchain rebuild, a bundle built on it reports
|
||||
the released version.
|
||||
|
||||
## Resolved
|
||||
|
||||
The toolchain now installs the exact SDK version the mesh last published, passed in as a build argument from the SDK module's published package, and the planner orders the toolchain after the SDK. Proven 2026-10-04: the toolchain built with the published SDK, and the six TypeScript bundles built in it serve their tools and seat verbs.
|
||||
|
||||
@@ -50,3 +50,14 @@ of a plan today.
|
||||
|
||||
Located only by owner; the move is a change of the controller's module and its deployment, not of
|
||||
its code.
|
||||
|
||||
## Fix prepared (2026-10-04)
|
||||
|
||||
Three changes. mesh-host#86, merged: a process may name the container it `replaces`, and the host
|
||||
removes that container only once the process has stayed up across two checks. mesh-controller#252,
|
||||
awaiting the operator's merge: the controller's composition for a service process, and two
|
||||
controllers safe together for the handover — the second stands by on the controller's consumers
|
||||
until the first lets go, and all plan work holds one advisory lock. mesh-controller#253, held: the
|
||||
controller's manifest as a Go bundle and a process. It waits on
|
||||
[issue 223](../223-a-new-mesh-installs-its-controller-as-a-container/00-report.md), because with it a
|
||||
new mesh cannot be installed.
|
||||
|
||||
@@ -1,9 +1,10 @@
|
||||
---
|
||||
status: open
|
||||
status: resolved
|
||||
opened: 2026-10-03
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by:
|
||||
- mesh-controller#246
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -34,3 +35,7 @@ Owner mesh-controller (the planner). **Fix direction:** on start, and whenever a
|
||||
build, the plan settles an `asked` build against the build records — a build recorded as built from
|
||||
the plan's commit is that tier's outcome — so a plan resumes after the controller replaced itself.
|
||||
A test: a plan whose build outcome was recorded while no controller followed it resumes on start.
|
||||
|
||||
## Resolved
|
||||
|
||||
A plan settles an asked build from the build records, whoever heard the outcome. Proven 2026-10-03 and 2026-10-04: both controller merge plans since the fix finished all three rounds, including the round that replaced the controller, without being stopped by hand.
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
# 214 — Diagnosis
|
||||
|
||||
*2026-10-03.* A plan learns a tier's outcome only through `planBuilt`, called when a controller takes
|
||||
in a build result off the bus. A merge to the controller's repository replaces the controller in tier
|
||||
0; the build that produced the new controller was recorded, but the plan state the new controller
|
||||
loaded still read `asked`, and no path ever revisited it. **Fix (branch
|
||||
`fix/issue-214-a-plan-settles-from-the-build-records`, commit d86baeb):** `advanceOnce` settles every
|
||||
still-asked module from its build records — a build recorded after the ask is that ask's outcome,
|
||||
built or failed — on every advance and on the 30-second ticker. Tested with a pure helper. Live
|
||||
proof: the next merge to mesh-controller passes tier 0 on its own.
|
||||
@@ -1,9 +1,10 @@
|
||||
---
|
||||
status: open
|
||||
status: resolved
|
||||
opened: 2026-10-03
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by:
|
||||
- mesh-controller#246
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -33,3 +34,9 @@ Owner mesh-controller. **Fix direction:** a build asked at a commit does not cha
|
||||
module follows; or, if pinning is meant, the pin is said — in `module list`, in `status`, and by a
|
||||
merge's plan naming the module it leaves out and why. A test: building a module at a commit and then
|
||||
merging a change to it plans it.
|
||||
|
||||
## Resolved
|
||||
|
||||
Proven 2026-10-04: the catalogue merges since the fix rebuilt the module that had been pinned at an
|
||||
old commit, at the merge's commit, by an ordinary plan — the same as every other module of the
|
||||
catalogue. Nothing was asked for it by hand.
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
# 215 — Diagnosis
|
||||
|
||||
*2026-10-03.* `takeIn` registers a build's `Ref` as the branch the module follows. unifi was once
|
||||
built with `ref=9c97a8a`, which became its followed ref. `sourceIs` matches a merge only to modules
|
||||
whose ref is empty or the merged base — so every merge into main left unifi out — and `askTier`
|
||||
re-asks `Source.Ref`, so every plan that rebuilt unifi built the same old commit again (its build
|
||||
records all read "at 9c97a8a"). **Fix (branch `fix/issue-215-a-commit-is-never-a-branch-to-follow`,
|
||||
commit 6784efa):** registration keeps the followed branch when a build names a commit; matching and
|
||||
re-asking read a recorded commit as the default branch, healing existing records; a merge names the
|
||||
modules of its repository it leaves out. Store-backed test fails without the fix.
|
||||
@@ -1,9 +1,10 @@
|
||||
---
|
||||
status: open
|
||||
status: resolved
|
||||
opened: 2026-10-03
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by:
|
||||
- mesh-controller#246
|
||||
amended-design:
|
||||
---
|
||||
|
||||
@@ -28,3 +29,7 @@ Owner mesh-controller (the catalogue's registration check). **Fix direction:** a
|
||||
loads, runs or unpacks — no `loads`, no `tools` list on its module, no resource naming it — is refused
|
||||
at registration, naming the field that would deliver it. A test: such a manifest is refused; adding
|
||||
`loads` admits it.
|
||||
|
||||
## Resolved
|
||||
|
||||
A bundle nothing would deliver is refused at registration, by name. Proven by the controller's tests; the catalogue's bundles all name what the runtime loads (mesh-catalog#243).
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
# 216 — Diagnosis
|
||||
|
||||
*2026-10-03.* The composer delivers a bundle as an archive only when its `Loads` is non-empty, and
|
||||
`Loads` derives from the artifact's `loads` or, failing that, from the module's `tools` list. The
|
||||
seven modules had neither, so their bundles were recorded and never composed into any declaration;
|
||||
nothing checked it. **Fix (branch `fix/issue-216-a-bundle-nothing-delivers-is-refused`, commit
|
||||
cf2bb3b):** registration refuses a bundle that nothing loads, runs or unpacks — no `loads`, no `tools`,
|
||||
no resource naming it, and not the runtime — naming the field that would deliver it. The current
|
||||
catalogue passes the check.
|
||||
@@ -0,0 +1,58 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-10-03
|
||||
located-in:
|
||||
- mesh-tools
|
||||
fixed-by:
|
||||
- mesh-tools#42
|
||||
- mesh-tools#43
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 217 — A refused announcement took down every container's runtime, and the console with it
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-03, rolling out [ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md).
|
||||
The tool runtimes announced themselves by subscribing `$SRV.<verb>.>`; the grants composed for them
|
||||
allowed only `$SRV.<verb>` and the service's own name and instance. The bus refused the wildcard:
|
||||
|
||||
```
|
||||
Subscription Violation - User "novox.gitea", Subject "$SRV.PING.>"
|
||||
NatsError: 'Permissions Violation for Subscription to "$SRV.PING.>"'
|
||||
```
|
||||
|
||||
The TypeScript runtime the per-module containers run treats a refused subscription as fatal, so on
|
||||
the one machine that had received the new images nine containers crash-looped — gitea's runtime,
|
||||
postgres's, the catalogue, the vault, mongodb, mssql, keycloak, mailu, nextcloud. Gitea's tools went
|
||||
with them, which closed the usual path for merging the fix.
|
||||
|
||||
Then the console stopped answering the controller's verbs, though the controller held every
|
||||
subscription: the console builds the list that tells a seat's verb from a module's tool by asking the
|
||||
controller *and* the catalogue, and gives up on both when the catalogue does not answer — so
|
||||
`mesh-controller.status` was asked of a module subject nobody serves.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
Two properties, each worse than the mistake that exposed it:
|
||||
|
||||
- **A runtime dies for an optional subscription.** Announcing is discovery; serving tools and running
|
||||
provisioners is the work. A refusal of the first should never stop the second.
|
||||
- **The console's view of the mesh's own verbs depended on a module.** The controller's verbs are how
|
||||
the operator repairs the mesh; they must not become unreachable because the catalogue is down.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
Owner mesh-tools. The wildcard is fixed on branch `fix/announce-only-what-the-grants-allow`: both
|
||||
runtimes subscribe exactly what the grants allow. The console that discovers from what announces itself
|
||||
(ADR 0195, 0197, on main) asks the bus and the controller, not the catalogue, which removes the second
|
||||
property once it is deployed. **Still to do:** the TypeScript runtime treats a refused announcement
|
||||
subscription as a logged warning, not as fatal; a test against a bus with real grants proves the
|
||||
announcement subscriptions are allowed for every principal kind.
|
||||
|
||||
Recovered on the day without the forge's API: the toolchain images built by the controller straight
|
||||
from the fix branch, every module image rebuilt on them, and the machine pushed.
|
||||
|
||||
## Resolved
|
||||
|
||||
A refused announcement is logged and the runtime serves on, and every runtime subscribes only the discovery subjects its grants allow. Proven 2026-10-04: after rolling out to every machine, no container restarts anywhere, and the discovery console lists no runtime as not answering. A refused tool subscription was made non-fatal the same way afterwards, under issue 218 (mesh-tools#46).
|
||||
@@ -0,0 +1,92 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-10-03
|
||||
located-in:
|
||||
- mesh-controller
|
||||
- mesh-tools
|
||||
fixed-by:
|
||||
- mesh-controller#248
|
||||
- mesh-tools#45
|
||||
- mesh-tools#46
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 218 — A seat held once for the mesh is answered by a module on a machine that does not hold it
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-03. Asked which databases the mesh's store holds, `mesh-store.databases` answered from the
|
||||
postgres on one machine with that machine's application databases; the controller's own database
|
||||
lives on the postgres of the other machine, which the controller's records name as the seat's one
|
||||
holder:
|
||||
|
||||
```
|
||||
mesh-store scope: mesh delivers: postgres-database holders: [ {node: <the control machine>, module: postgres} ]
|
||||
```
|
||||
|
||||
The discovery console, which reads what answers on the bus ([ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)),
|
||||
shows the same seat announced from **both** machines running postgres.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
A seat held once for the mesh promises one answerer: the role's holder. A module that implements a
|
||||
seat's verbs on every machine it runs on, and is let serve them on each, turns "the mesh's store" into
|
||||
"whichever postgres replied first" — a read against the wrong database that looks like a right one,
|
||||
and a write would be worse. Every mesh-scoped seat whose implementing module runs on more than one
|
||||
machine has this shape.
|
||||
|
||||
## Where to look
|
||||
|
||||
Whether the runtime serves a seat's verbs where its module merely *claims* the seat rather than where
|
||||
the mesh made it the holder ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md),
|
||||
[ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)):
|
||||
the membership the controller issues each assignment, and what the runtime admits from it. A check:
|
||||
a mesh-scoped seat's verbs are served by exactly the holder the records name, on every machine.
|
||||
|
||||
## Root cause
|
||||
|
||||
The controller composed each assignment's held seats from what its module *claims*, once per module
|
||||
and not once per machine. Every machine running postgres was therefore given the store seat's grants
|
||||
and issued its subjects, and each runtime served the seat's verbs because it serves what it is issued
|
||||
([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
|
||||
The runtime behaved as designed. The fault was in what it was issued.
|
||||
|
||||
The seat's verbs are not the module's tools. The store's `databases` and `query` are a separate
|
||||
implementation registered under the seat's name ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md)).
|
||||
Only that implementation should be withdrawn where the module does not hold the seat. postgres's own
|
||||
tools stay served on every machine it runs on.
|
||||
|
||||
## Fix
|
||||
|
||||
The controller now reads the recorded seat holdings when it composes grants and memberships. A seat
|
||||
held once for the mesh is issued only to the machine and module the records name as its holder. A
|
||||
seat held once per machine, and a mesh seat with no holder on record, are issued as before. Grants
|
||||
and memberships come from the same list, so they cannot disagree.
|
||||
|
||||
**How it is checked.** A controller test asserts that a claimant on another machine keeps its node
|
||||
seats and loses the recorded mesh seat. Live, the discovery console's overview must show each
|
||||
mesh-scoped seat announced from exactly the holder the records name. Status moves to `resolved` once
|
||||
that holds after the fix is rolled out.
|
||||
|
||||
## A second cause, and what the rollout broke (2026-10-04)
|
||||
|
||||
With the grants corrected, calls to the store reached only the holder, yet the console still showed
|
||||
the seat announced from both machines. The module's runtime added every seat its start-up credential
|
||||
claims, even after the mesh issued a membership that left the seat out. Once a membership exists,
|
||||
it now alone decides which seat verbs a runtime serves (mesh-tools#45).
|
||||
|
||||
The rollout then exposed a third fault. The module on the machine that does not hold the seat was
|
||||
still running an image built before #45, so it subscribed to the seat's subject. The corrected grants
|
||||
refused that subscription, and the refusal ended the process. Its runtime crash-looped until the
|
||||
module was rebuilt on the new runtime image. The database itself kept running. A refused tool
|
||||
subscription is now logged and costs only that subject (mesh-tools#46), as a refused announcement
|
||||
already did ([issue 217](../217-a-refused-announcement-took-down-every-containers-runtime/00-report.md)).
|
||||
|
||||
The module was not rebuilt by the plan that rebuilt the runtime image. This is the ordering gap of
|
||||
[issue 211](../211-a-bundle-is-built-before-the-toolchain-it-is-compiled-in/00-report.md) seen
|
||||
from a container module.
|
||||
|
||||
**Proven 2026-10-04.** The discovery console's overview shows the store seat announced from the
|
||||
recorded holder only. Repeated calls to the store are answered by that machine, and the answers
|
||||
include the controller's own database. The non-holder still answers its own module tools. No
|
||||
container restarts on any machine.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-10-04
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by:
|
||||
- mesh-controller#249
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 219 — An older build that finishes later replaces a newer one
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-03. Two merges to the runtime module came minutes apart. Each plan asked for every module
|
||||
built on the runtime image to be rebuilt. One module's two builds were of the same source and
|
||||
differed only in the runtime image they stood on:
|
||||
|
||||
| Build | Requested | Finished | Stood on |
|
||||
|---|---|---|---|
|
||||
| asked by the first plan | 21:33 | 22:04 | the runtime image before the fix |
|
||||
| asked by the second plan | 21:49 | 21:56 | the runtime image with the fix |
|
||||
|
||||
The older request finished last, and its image became the module's current artifact. The next push
|
||||
deployed it, and the module's runtime crash-looped on a fault the newer image had already fixed
|
||||
([issue 218](../218-a-mesh-seat-is-answered-by-a-module-that-does-not-hold-it/00-report.md)).
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
Which build is current should follow what it was built from, not which build machine was slowest.
|
||||
Whenever two plans overlap, which happens on any busy evening, a fix can be silently reverted by a
|
||||
build that started before it existed. Every passing check still passes.
|
||||
|
||||
## Where to look
|
||||
|
||||
How a finished build is recorded and how a module's current artifact is chosen. **How it is checked:**
|
||||
a test in which an older request completes after a newer one for the same artifact, and the newer
|
||||
stays current.
|
||||
|
||||
## Resolved
|
||||
|
||||
A build is ordered by when it was requested, read from the id the controller gives it, and a
|
||||
module's registered manifest is replaced only by a build requested at or after the one it came from.
|
||||
An older request finishing later is recorded and changes nothing; a plan settles only from builds it
|
||||
asked for itself. **How it is checked:** store-backed tests replay the incident — the newer request
|
||||
stays what the module is — and fail without the fix. Live since 2026-10-04: the rebuilds of every
|
||||
runtime-image module and the three waves of module code moves since then each registered the build
|
||||
they asked for.
|
||||
@@ -0,0 +1,41 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-10-04
|
||||
located-in:
|
||||
- mesh-host
|
||||
fixed-by:
|
||||
- mesh-host#84
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 220 — A delivered bundle keeps the files of the one before
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-04. A tools bundle was rebuilt as one self-contained file per entrypoint
|
||||
([ADR 0193](../../02-DECISIONS/0193-every-bundle-the-runtime-serves-is-launched-and-the-runtime-knows-no-language.md)),
|
||||
so it no longer carries a package directory. On the machine it was delivered to, its directory still
|
||||
held the package directory and a compiled file from the earlier delivery, dated hours before the
|
||||
new files. The new files were written over the old directory, and nothing removed what the new
|
||||
bundle no longer contains.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
A bundle on disk should be exactly the artifact that was built. Leftover files can be imported by
|
||||
code that should no longer find them. A fix that removes a file then works on a fresh machine and
|
||||
fails on every machine that ran an earlier version. It also makes "what runs here" impossible to
|
||||
read from the artifact.
|
||||
|
||||
## Where to look
|
||||
|
||||
How the host unpacks a bundle into its directory. **How it is checked:** deliver a bundle, then a
|
||||
version without one of its files, and the file is gone.
|
||||
|
||||
## Resolved
|
||||
|
||||
An archive is unpacked into a fresh directory beside the old one and swapped in by rename; a refused
|
||||
or failed unpack leaves the old tree whole. **How it is checked:** a second delivery without a file
|
||||
removes it, and nothing is left beside the directory; both tests fail without the fix. Proven
|
||||
2026-10-04: a bundle rebuilt and delivered after the fix holds exactly the new build and nothing
|
||||
beside it. A bundle that has not changed since keeps its old leftovers until its next version, by
|
||||
design: an unchanged archive is not unpacked again.
|
||||
@@ -0,0 +1,52 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-10-04
|
||||
located-in:
|
||||
- mesh-controller
|
||||
fixed-by:
|
||||
- policy: `upgrade build-agent roll-out` (2026-10-04)
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 221 — A build machine learns a new builder only from a push
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-04. A controller merge changed how TypeScript bundles are built: one file per entrypoint
|
||||
instead of a package directory. Its plan finished, and six bundles were rebuilt right after. They
|
||||
came out in the old shape, because the build machines still ran the previous builder. They got the
|
||||
new one only from the next push. Rebuilt after that push, the same six came out right.
|
||||
|
||||
The same order showed in the bus grants the same night. A push sent while the controller was still
|
||||
the previous build composed grants with the previous code. A second push was needed after the new
|
||||
controller had started.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
A merge to the controller is not in effect when its plan says done. Anything built or pushed in the
|
||||
gap uses the old code and looks current. Today only the operator knows to push first and build
|
||||
second, and even the operator forgot.
|
||||
|
||||
## Where to look
|
||||
|
||||
Whether a controller plan should end by delivering itself to the build machines and the control
|
||||
machine, or whether a build should refuse a builder older than the controller that asked for it.
|
||||
**How it is checked:** after a controller merge, a build asked right after its plan finishes runs
|
||||
the new builder.
|
||||
|
||||
## Located
|
||||
|
||||
The mechanism existed: a module whose upgrade policy is `roll-out` is sent to its machines when its
|
||||
tier is built, and the plan's next tier waits until it is applied. The build agent's policy was
|
||||
`record` — built, never sent — as was that of 76 other modules, which is why every rollout on
|
||||
2026-10-03 needed a push by hand. The build agent was set to `roll-out`, one machine at a time,
|
||||
stopping at the first failure. **How it is checked:** the next controller merge's plan sends the
|
||||
build agent to the build machines before its last tier, and a build asked right after the plan
|
||||
finishes runs the new builder; then this moves to resolved. Whether the other modules roll out is
|
||||
the operator's policy, not this issue's.
|
||||
|
||||
## Resolved
|
||||
|
||||
Proven 2026-10-04: the next controller merge's plan logged that its first tier was built and the
|
||||
build agent sent to all four build machines, and only then asked its next tier. The builder change
|
||||
in that merge reached the build machines without a hand push.
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
---
|
||||
status: resolved
|
||||
opened: 2026-10-04
|
||||
located-in:
|
||||
- mesh-tools
|
||||
fixed-by:
|
||||
- mesh-tools#47
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 222 — An assignment is refused on the bus until the bus's machine is pushed
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-04. A module was assigned to the laptop, and the laptop alone was pushed. The module's two
|
||||
bundles arrived and the node's runtime launched both, and then the bus refused every one of the
|
||||
module's subjects:
|
||||
|
||||
```
|
||||
nats: permissions violation: Permissions Violation for Subscription to "mesh.mod.<module>.tool.<tool>.<node>"
|
||||
```
|
||||
|
||||
The tools were unreachable until a later push that included the machine running the bus. Then the
|
||||
runtime served them without a restart, because a new membership arrived and it re-subscribed.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
What an account may answer lives in the bus's user list, and the controller writes that list only
|
||||
into the declaration of the machine that runs the bus. Assigning anything to any machine changes
|
||||
that list, so `push <machine>` after `assign <machine> <module>`, which is what the controller itself
|
||||
tells the operator to run, leaves the module running and unreachable, with nothing reporting a fault.
|
||||
|
||||
## Where to look
|
||||
|
||||
Whether a push to one machine should also send the bus's machine when the user list it would
|
||||
compose differs from the one that machine holds. **How it is checked:** assign a module with tools
|
||||
to a machine that does not run the bus, push only that machine, and its tools answer.
|
||||
|
||||
## Diagnosed and resolved
|
||||
|
||||
The push did send the bus's machine: the controller's log shows both machines applying in the same
|
||||
second. The fault was the order within that second. The node's runtime subscribed before the bus
|
||||
had reloaded its user list, the bus refused, and the bus client marks a refused subscription dead.
|
||||
Nothing asked again until a later membership happened to re-serve the module.
|
||||
|
||||
A subject the runtime answers on is now asked for again when the bus refuses it, after waits from
|
||||
two seconds to two minutes, and given up and said after about five minutes. **How it is checked:**
|
||||
a test refuses a subject and finds it asked for again and answering, given up past its attempts,
|
||||
and not asked again once stopped; and by hand against a bus whose permissions were reloaded while
|
||||
connected, the runtime answered two seconds after the grant arrived, where the runtime before the
|
||||
fix never answered.
|
||||
@@ -0,0 +1,48 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-04
|
||||
located-in:
|
||||
- mesh-host
|
||||
fixed-by:
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 223 — A new mesh installs its controller as a container
|
||||
|
||||
## What was observed
|
||||
|
||||
2026-10-04. Preparing [issue 213](../213-the-controller-is-a-go-program-run-in-a-container/00-report.md),
|
||||
the controller's own manifest was changed to a Go bundle run as a process. The installer that raises a
|
||||
new mesh assumes the controller is an image and a container at every step from its third on:
|
||||
|
||||
- it requires the controller's build to produce exactly one image;
|
||||
- it starts a temporary controller from that image, and publishes the image to the registry;
|
||||
- at the pivot, it finds the controller's container in the declaration, reads its environment and
|
||||
volumes, waits for it, and from then on talks to the controller only through the container.
|
||||
|
||||
## Why it matters beyond this instance
|
||||
|
||||
With the controller's manifest changed, a new mesh cannot be installed: the pivot fails. The deeper
|
||||
constraint is ordering. A process's bundle is fetched from the artifact store, and the installer
|
||||
raises the artifact store only after the pivot, so the controller's first declaration names a bundle
|
||||
nothing can serve yet.
|
||||
|
||||
## What a fix has to settle
|
||||
|
||||
One of two shapes, and it is a decision, not a repair:
|
||||
|
||||
1. raise the artifact store before the pivot, publish the controller's bundle to it, and talk to the
|
||||
controller from the host's side rather than through a container; or
|
||||
2. pivot to the image form as today, and let the first push hand over to the process, which
|
||||
requires the controller's manifest to carry both forms.
|
||||
|
||||
Until it is settled, the change of the controller's manifest (mesh-controller#253) is held. The
|
||||
handover itself is built and merged (mesh-host#86); the controller's half (mesh-controller#252) waits
|
||||
on the operator. **How it is checked:** the installer's test raises a mesh whose controller manifest
|
||||
is the process form, and the controller answers its seat's verbs at the end.
|
||||
|
||||
## Decided (2026-10-04)
|
||||
|
||||
Option 2, [ADR 0200](../../02-DECISIONS/0200-genesis-pivots-to-the-controller-as-a-container-and-the-first-push-hands-it-to-a-process.md):
|
||||
genesis pivots to the controller as a container recorded under the name the process `replaces`, and
|
||||
the first push hands it over.
|
||||
Reference in New Issue
Block a user