Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4b8c5e3b11 | ||
|
|
557760e537 | ||
|
|
6b1ebd1d4a | ||
|
|
fa73a17ceb | ||
|
|
08cea893bd | ||
|
|
04625d3e35 |
+11
@@ -0,0 +1,11 @@
|
|||||||
|
# 211 — Diagnosis
|
||||||
|
|
||||||
|
*2026-10-03.* The planner orders a merge's modules by `inventory.Dependencies`, whose edges come from a
|
||||||
|
manifest's `build.on`, from what a build recorded it stood on, from the repositories it read, and from
|
||||||
|
the build machine. A bundle names its toolchain by `language`; the builder takes the toolchain image
|
||||||
|
(`ToolchainFor(language)`) from what the mesh holds and records nothing of it as stood on. So no edge
|
||||||
|
ran from a bundle to the module publishing its toolchain, and a merge moving both (mesh-tools: the
|
||||||
|
images and node-tools) tiered them together. **Fix (mesh-controller, branch
|
||||||
|
`fix/issue-211-a-bundle-stands-on-its-toolchain`, commit c72f6ca):** `dependenciesOf` adds a `stands-on`
|
||||||
|
edge from every bundle artifact to its toolchain's module, read from the manifest. Tested: TypeScript
|
||||||
|
bundle → mesh-tools, Go bundle → mesh-tools-go, image → none; a merge moving both plans two tiers.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
---
|
---
|
||||||
status: open
|
status: located
|
||||||
opened: 2026-10-03
|
opened: 2026-10-03
|
||||||
located-in:
|
located-in:
|
||||||
- mesh-controller
|
- mesh-controller
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# 214 — Diagnosis
|
||||||
|
|
||||||
|
*2026-10-03.* A plan learns a tier's outcome only through `planBuilt`, called when a controller takes
|
||||||
|
in a build result off the bus. A merge to the controller's repository replaces the controller in tier
|
||||||
|
0; the build that produced the new controller was recorded, but the plan state the new controller
|
||||||
|
loaded still read `asked`, and no path ever revisited it. **Fix (branch
|
||||||
|
`fix/issue-214-a-plan-settles-from-the-build-records`, commit d86baeb):** `advanceOnce` settles every
|
||||||
|
still-asked module from its build records — a build recorded after the ask is that ask's outcome,
|
||||||
|
built or failed — on every advance and on the 30-second ticker. Tested with a pure helper. Live
|
||||||
|
proof: the next merge to mesh-controller passes tier 0 on its own.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
---
|
---
|
||||||
status: open
|
status: located
|
||||||
opened: 2026-10-03
|
opened: 2026-10-03
|
||||||
located-in:
|
located-in:
|
||||||
- mesh-controller
|
- mesh-controller
|
||||||
|
|||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# 215 — Diagnosis
|
||||||
|
|
||||||
|
*2026-10-03.* `takeIn` registers a build's `Ref` as the branch the module follows. unifi was once
|
||||||
|
built with `ref=9c97a8a`, which became its followed ref. `sourceIs` matches a merge only to modules
|
||||||
|
whose ref is empty or the merged base — so every merge into main left unifi out — and `askTier`
|
||||||
|
re-asks `Source.Ref`, so every plan that rebuilt unifi built the same old commit again (its build
|
||||||
|
records all read "at 9c97a8a"). **Fix (branch `fix/issue-215-a-commit-is-never-a-branch-to-follow`,
|
||||||
|
commit 6784efa):** registration keeps the followed branch when a build names a commit; matching and
|
||||||
|
re-asking read a recorded commit as the default branch, healing existing records; a merge names the
|
||||||
|
modules of its repository it leaves out. Store-backed test fails without the fix.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
---
|
---
|
||||||
status: open
|
status: located
|
||||||
opened: 2026-10-03
|
opened: 2026-10-03
|
||||||
located-in:
|
located-in:
|
||||||
- mesh-controller
|
- mesh-controller
|
||||||
|
|||||||
@@ -0,0 +1,9 @@
|
|||||||
|
# 216 — Diagnosis
|
||||||
|
|
||||||
|
*2026-10-03.* The composer delivers a bundle as an archive only when its `Loads` is non-empty, and
|
||||||
|
`Loads` derives from the artifact's `loads` or, failing that, from the module's `tools` list. The
|
||||||
|
seven modules had neither, so their bundles were recorded and never composed into any declaration;
|
||||||
|
nothing checked it. **Fix (branch `fix/issue-216-a-bundle-nothing-delivers-is-refused`, commit
|
||||||
|
cf2bb3b):** registration refuses a bundle that nothing loads, runs or unpacks — no `loads`, no `tools`,
|
||||||
|
no resource naming it, and not the runtime — naming the field that would deliver it. The current
|
||||||
|
catalogue passes the check.
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
---
|
||||||
|
status: located
|
||||||
|
opened: 2026-10-03
|
||||||
|
located-in:
|
||||||
|
- mesh-tools
|
||||||
|
fixed-by:
|
||||||
|
amended-design:
|
||||||
|
---
|
||||||
|
|
||||||
|
# 217 — A refused announcement took down every container's runtime, and the console with it
|
||||||
|
|
||||||
|
## What was observed
|
||||||
|
|
||||||
|
2026-10-03, rolling out [ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md).
|
||||||
|
The tool runtimes announced themselves by subscribing `$SRV.<verb>.>`; the grants composed for them
|
||||||
|
allowed only `$SRV.<verb>` and the service's own name and instance. The bus refused the wildcard:
|
||||||
|
|
||||||
|
```
|
||||||
|
Subscription Violation - User "novox.gitea", Subject "$SRV.PING.>"
|
||||||
|
NatsError: 'Permissions Violation for Subscription to "$SRV.PING.>"'
|
||||||
|
```
|
||||||
|
|
||||||
|
The TypeScript runtime the per-module containers run treats a refused subscription as fatal, so on
|
||||||
|
the one machine that had received the new images nine containers crash-looped — gitea's runtime,
|
||||||
|
postgres's, the catalogue, the vault, mongodb, mssql, keycloak, mailu, nextcloud. Gitea's tools went
|
||||||
|
with them, which closed the usual path for merging the fix.
|
||||||
|
|
||||||
|
Then the console stopped answering the controller's verbs, though the controller held every
|
||||||
|
subscription: the console builds the list that tells a seat's verb from a module's tool by asking the
|
||||||
|
controller *and* the catalogue, and gives up on both when the catalogue does not answer — so
|
||||||
|
`mesh-controller.status` was asked of a module subject nobody serves.
|
||||||
|
|
||||||
|
## Why it matters beyond this instance
|
||||||
|
|
||||||
|
Two properties, each worse than the mistake that exposed it:
|
||||||
|
|
||||||
|
- **A runtime dies for an optional subscription.** Announcing is discovery; serving tools and running
|
||||||
|
provisioners is the work. A refusal of the first should never stop the second.
|
||||||
|
- **The console's view of the mesh's own verbs depended on a module.** The controller's verbs are how
|
||||||
|
the operator repairs the mesh; they must not become unreachable because the catalogue is down.
|
||||||
|
|
||||||
|
## Diagnosis
|
||||||
|
|
||||||
|
Owner mesh-tools. The wildcard is fixed on branch `fix/announce-only-what-the-grants-allow`: both
|
||||||
|
runtimes subscribe exactly what the grants allow. The console that discovers from what announces itself
|
||||||
|
(ADR 0195, 0197, on main) asks the bus and the controller, not the catalogue, which removes the second
|
||||||
|
property once it is deployed. **Still to do:** the TypeScript runtime treats a refused announcement
|
||||||
|
subscription as a logged warning, not as fatal; a test against a bus with real grants proves the
|
||||||
|
announcement subscriptions are allowed for every principal kind.
|
||||||
|
|
||||||
|
Recovered on the day without the forge's API: the toolchain images built by the controller straight
|
||||||
|
from the fix branch, every module image rebuilt on them, and the machine pushed.
|
||||||
@@ -0,0 +1,65 @@
|
|||||||
|
---
|
||||||
|
status: located
|
||||||
|
opened: 2026-10-03
|
||||||
|
located-in:
|
||||||
|
- mesh-controller
|
||||||
|
fixed-by: mesh-controller#248
|
||||||
|
amended-design:
|
||||||
|
---
|
||||||
|
|
||||||
|
# 218 — A seat held once for the mesh is answered by a module on a machine that does not hold it
|
||||||
|
|
||||||
|
## What was observed
|
||||||
|
|
||||||
|
2026-10-03. Asked which databases the mesh's store holds, `mesh-store.databases` answered from the
|
||||||
|
postgres on one machine with that machine's application databases; the controller's own database
|
||||||
|
lives on the postgres of the other machine, which the controller's records name as the seat's one
|
||||||
|
holder:
|
||||||
|
|
||||||
|
```
|
||||||
|
mesh-store scope: mesh delivers: postgres-database holders: [ {node: <the control machine>, module: postgres} ]
|
||||||
|
```
|
||||||
|
|
||||||
|
The discovery console, which reads what answers on the bus ([ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md)),
|
||||||
|
shows the same seat announced from **both** machines running postgres.
|
||||||
|
|
||||||
|
## Why it matters beyond this instance
|
||||||
|
|
||||||
|
A seat held once for the mesh promises one answerer: the role's holder. A module that implements a
|
||||||
|
seat's verbs on every machine it runs on, and is let serve them on each, turns "the mesh's store" into
|
||||||
|
"whichever postgres replied first" — a read against the wrong database that looks like a right one,
|
||||||
|
and a write would be worse. Every mesh-scoped seat whose implementing module runs on more than one
|
||||||
|
machine has this shape.
|
||||||
|
|
||||||
|
## Where to look
|
||||||
|
|
||||||
|
Whether the runtime serves a seat's verbs where its module merely *claims* the seat rather than where
|
||||||
|
the mesh made it the holder ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md),
|
||||||
|
[ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)):
|
||||||
|
the membership the controller issues each assignment, and what the runtime admits from it. A check:
|
||||||
|
a mesh-scoped seat's verbs are served by exactly the holder the records name, on every machine.
|
||||||
|
|
||||||
|
## Root cause
|
||||||
|
|
||||||
|
The controller composed each assignment's held seats from what its module *claims*, once per module
|
||||||
|
and not once per machine. Every machine running postgres was therefore given the store seat's grants
|
||||||
|
and issued its subjects, and each runtime served the seat's verbs because it serves what it is issued
|
||||||
|
([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
|
||||||
|
The runtime behaved as designed. The fault was in what it was issued.
|
||||||
|
|
||||||
|
The seat's verbs are not the module's tools. The store's `databases` and `query` are a separate
|
||||||
|
implementation registered under the seat's name ([ADR 0159](../../02-DECISIONS/0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md)).
|
||||||
|
Only that implementation should be withdrawn where the module does not hold the seat. postgres's own
|
||||||
|
tools stay served on every machine it runs on.
|
||||||
|
|
||||||
|
## Fix
|
||||||
|
|
||||||
|
The controller now reads the recorded seat holdings when it composes grants and memberships. A seat
|
||||||
|
held once for the mesh is issued only to the machine and module the records name as its holder. A
|
||||||
|
seat held once per machine, and a mesh seat with no holder on record, are issued as before. Grants
|
||||||
|
and memberships come from the same list, so they cannot disagree.
|
||||||
|
|
||||||
|
**How it is checked.** A controller test asserts that a claimant on another machine keeps its node
|
||||||
|
seats and loses the recorded mesh seat. Live, the discovery console's overview must show each
|
||||||
|
mesh-scoped seat announced from exactly the holder the records name. Status moves to `resolved` once
|
||||||
|
that holds after the fix is rolled out.
|
||||||
Reference in New Issue
Block a user