Compare commits

...
9 changed files with 160 additions and 3 deletions
@@ -0,0 +1,11 @@
# 211 — Diagnosis
*2026-10-03.* The planner orders a merge's modules by `inventory.Dependencies`, whose edges come from a
manifest's `build.on`, from what a build recorded it stood on, from the repositories it read, and from
the build machine. A bundle names its toolchain by `language`; the builder takes the toolchain image
(`ToolchainFor(language)`) from what the mesh holds and records nothing of it as stood on. So no edge
ran from a bundle to the module publishing its toolchain, and a merge moving both (mesh-tools: the
images and node-tools) tiered them together. **Fix (mesh-controller, branch
`fix/issue-211-a-bundle-stands-on-its-toolchain`, commit c72f6ca):** `dependenciesOf` adds a `stands-on`
edge from every bundle artifact to its toolchain's module, read from the manifest. Tested: TypeScript
bundle → mesh-tools, Go bundle → mesh-tools-go, image → none; a merge moving both plans two tiers.
@@ -1,5 +1,5 @@
--- ---
status: open status: located
opened: 2026-10-03 opened: 2026-10-03
located-in: located-in:
- mesh-controller - mesh-controller
@@ -0,0 +1,10 @@
# 214 — Diagnosis
*2026-10-03.* A plan learns a tier's outcome only through `planBuilt`, called when a controller takes
in a build result off the bus. A merge to the controller's repository replaces the controller in tier
0; the build that produced the new controller was recorded, but the plan state the new controller
loaded still read `asked`, and no path ever revisited it. **Fix (branch
`fix/issue-214-a-plan-settles-from-the-build-records`, commit d86baeb):** `advanceOnce` settles every
still-asked module from its build records — a build recorded after the ask is that ask's outcome,
built or failed — on every advance and on the 30-second ticker. Tested with a pure helper. Live
proof: the next merge to mesh-controller passes tier 0 on its own.
@@ -1,5 +1,5 @@
--- ---
status: open status: located
opened: 2026-10-03 opened: 2026-10-03
located-in: located-in:
- mesh-controller - mesh-controller
@@ -0,0 +1,10 @@
# 215 — Diagnosis
*2026-10-03.* `takeIn` registers a build's `Ref` as the branch the module follows. unifi was once
built with `ref=9c97a8a`, which became its followed ref. `sourceIs` matches a merge only to modules
whose ref is empty or the merged base — so every merge into main left unifi out — and `askTier`
re-asks `Source.Ref`, so every plan that rebuilt unifi built the same old commit again (its build
records all read "at 9c97a8a"). **Fix (branch `fix/issue-215-a-commit-is-never-a-branch-to-follow`,
commit 6784efa):** registration keeps the followed branch when a build names a commit; matching and
re-asking read a recorded commit as the default branch, healing existing records; a merge names the
modules of its repository it leaves out. Store-backed test fails without the fix.
@@ -1,5 +1,5 @@
--- ---
status: open status: located
opened: 2026-10-03 opened: 2026-10-03
located-in: located-in:
- mesh-controller - mesh-controller
@@ -0,0 +1,9 @@
# 216 — Diagnosis
*2026-10-03.* The composer delivers a bundle as an archive only when its `Loads` is non-empty, and
`Loads` derives from the artifact's `loads` or, failing that, from the module's `tools` list. The
seven modules had neither, so their bundles were recorded and never composed into any declaration;
nothing checked it. **Fix (branch `fix/issue-216-a-bundle-nothing-delivers-is-refused`, commit
cf2bb3b):** registration refuses a bundle that nothing loads, runs or unpacks — no `loads`, no `tools`,
no resource naming it, and not the runtime — naming the field that would deliver it. The current
catalogue passes the check.
@@ -0,0 +1,52 @@
---
status: located
opened: 2026-10-03
located-in:
- mesh-tools
fixed-by:
amended-design:
---
# 217 — A refused announcement took down every container's runtime, and the console with it
## What was observed
2026-10-03, rolling out [ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md).
The tool runtimes announced themselves by subscribing `$SRV.<verb>.>`; the grants composed for them
allowed only `$SRV.<verb>` and the service's own name and instance. The bus refused the wildcard:
```
Subscription Violation - User "novox.gitea", Subject "$SRV.PING.>"
NatsError: 'Permissions Violation for Subscription to "$SRV.PING.>"'
```
The TypeScript runtime the per-module containers run treats a refused subscription as fatal, so on
the one machine that had received the new images nine containers crash-looped — gitea's runtime,
postgres's, the catalogue, the vault, mongodb, mssql, keycloak, mailu, nextcloud. Gitea's tools went
with them, which closed the usual path for merging the fix.
Then the console stopped answering the controller's verbs, though the controller held every
subscription: the console builds the list that tells a seat's verb from a module's tool by asking the
controller *and* the catalogue, and gives up on both when the catalogue does not answer — so
`mesh-controller.status` was asked of a module subject nobody serves.
## Why it matters beyond this instance
Two properties, each worse than the mistake that exposed it:
- **A runtime dies for an optional subscription.** Announcing is discovery; serving tools and running
provisioners is the work. A refusal of the first should never stop the second.
- **The console's view of the mesh's own verbs depended on a module.** The controller's verbs are how
the operator repairs the mesh; they must not become unreachable because the catalogue is down.
## Diagnosis
Owner mesh-tools. The wildcard is fixed on branch `fix/announce-only-what-the-grants-allow`: both
runtimes subscribe exactly what the grants allow. The console that discovers from what announces itself
(ADR 0195, 0197, on main) asks the bus and the controller, not the catalogue, which removes the second
property once it is deployed. **Still to do:** the TypeScript runtime treats a refused announcement
subscription as a logged warning, not as fatal; a test against a bus with real grants proves the
announcement subscriptions are allowed for every principal kind.
Recovered on the day without the forge's API: the toolchain images built by the controller straight
from the fix branch, every module image rebuilt on them, and the machine pushed.
@@ -0,0 +1,65 @@
---
status: located
opened: 2026-10-03
located-in:
- mesh-controller
fixed-by: mesh-controller#248
amended-design:
---
# 218 — A seat held once for the mesh is answered by a module on a machine that does not hold it
## What was observed
2026-10-03. Asked which databases the mesh's store holds, `mesh-store.databases` answered from the
postgres on one machine with that machine's application databases; the controller's own database
lives on the postgres of the other machine, which the controller's records name as the seat's one
holder:
```
mesh-store scope: mesh delivers: postgres-database holders: [ {node: <the control machine>, module: postgres} ]
```
The discovery console, which reads what answers on the bus ([ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)),
shows the same seat announced from **both** machines running postgres.
## Why it matters beyond this instance
A seat held once for the mesh promises one answerer: the role's holder. A module that implements a
seat's verbs on every machine it runs on, and is let serve them on each, turns "the mesh's store" into
"whichever postgres replied first" — a read against the wrong database that looks like a right one,
and a write would be worse. Every mesh-scoped seat whose implementing module runs on more than one
machine has this shape.
## Where to look
Whether the runtime serves a seat's verbs where its module merely *claims* the seat rather than where
the mesh made it the holder ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md),
[ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)):
the membership the controller issues each assignment, and what the runtime admits from it. A check:
a mesh-scoped seat's verbs are served by exactly the holder the records name, on every machine.
## Root cause
The controller composed each assignment's held seats from what its module *claims*, once per module
and not once per machine. Every machine running postgres was therefore given the store seat's grants
and issued its subjects, and each runtime served the seat's verbs because it serves what it is issued
([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
The runtime behaved as designed. The fault was in what it was issued.
The seat's verbs are not the module's tools. The store's `databases` and `query` are a separate
implementation registered under the seat's name ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md)).
Only that implementation should be withdrawn where the module does not hold the seat. postgres's own
tools stay served on every machine it runs on.
## Fix
The controller now reads the recorded seat holdings when it composes grants and memberships. A seat
held once for the mesh is issued only to the machine and module the records name as its holder. A
seat held once per machine, and a mesh seat with no holder on record, are issued as before. Grants
and memberships come from the same list, so they cannot disagree.
**How it is checked.** A controller test asserts that a claimant on another machine keeps its node
seats and loses the recorded mesh seat. Live, the discovery console's overview must show each
mesh-scoped seat announced from exactly the holder the records name. Status moves to `resolved` once
that holds after the fix is rolled out.