Compare commits

..
Author SHA1 Message Date
jschoubben ba7c9e7f17 Try a small fix's branch before its pull request, since local runs miss the check's toolchain (playbook 08)
mesh/delivery checking
mesh/merge-gate out of date: judged against trunk 9bfd57174fab, the trunk is now f8c0676b3a1d; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 9bfd57174fab, the trunk is now f8c0676b3a1d; judged again on a recheck or when it is merged
2026-10-15 23:33:20 +02:00
mesh-tools 9bfd57174f Merge pull request 'Playbook 08: a small fix may cover two or three related issues in one repository' (#368) from playbooks/08-related-issues into main 2026-10-15 20:38:39 +00:00
jschoubben a227f98bdd Let a small fix cover two or three related issues in one repository (playbook 08)
mesh/merge-gate head a227f98bddd4 merged into trunk 409fa9bbc979 — pass: the change touches no module of the mesh's graph
mesh/repo-check head a227f98bddd4 merged into trunk 409fa9bbc979 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 22:37:12 +02:00
mesh-tools 409fa9bbc9 Merge pull request 'Playbook 08: a small fix gets three review rounds, as a feature does' (#367) from playbooks/08-three-rounds into main 2026-10-15 19:29:07 +00:00
jschoubben 747fc546c1 Allow a small fix three review rounds, as a feature has (playbook 08)
mesh/merge-gate head 747fc546c1df merged into trunk 046bbd3b308d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 747fc546c1df merged into trunk 046bbd3b308d — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 21:28:29 +02:00
mesh-tools 046bbd3b30 Merge pull request 'Playbooks 01, 02 and 04: one cycle from idea to delivered milestones' (#366) from playbooks/01-02-04-the-cycle into main 2026-10-15 19:23:08 +00:00
jschoubben 99243ec47e Confirm milestones before opening their issues, name issue 845 for the check, keep 01's withdrawal and history lines
mesh/merge-gate head 99243ec47efa merged into trunk 5ff97ec3aa02 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 99243ec47efa merged into trunk 5ff97ec3aa02 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 21:19:01 +02:00
jschoubben 0d66edb24c Make playbooks 01, 02 and 04 one cycle from idea to delivered milestones
The operator found 04 odd on 2026-10-15: it went from a settled design straight to building,
with no work breakdown, and contradicted playbook 07 on pull requests. 01 now starts from the
operator's idea with setup questions, findings and a direction the operator confirms; 02
graduates that direction into one record per reversible choice and hands the design on; 04
breaks it into milestones usable on their own, one issue per repository part, each milestone
one feature step of 07. 01 notes the knowledge base's insight-only rule for an active effort
until issue 844 is delivered.
2026-10-15 21:18:15 +02:00
mesh-tools 5ff97ec3aa Merge pull request 'Say rollout, not walk (issue 646)' (#365) from fix/646-rollout-not-walk into main 2026-10-15 17:59:22 +00:00
jschoubben 75ac87c72b Merge remote-tracking branch 'origin/main' into fix/646-rollout-not-walk
mesh/merge-gate head 75ac87c72b6a merged into trunk 058d76ee8a4e — pass: the change touches no module of the mesh's graph
mesh/repo-check head 75ac87c72b6a merged into trunk 058d76ee8a4e — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 19:57:54 +02:00
jschoubben 437523554e Say rollout, not walk, for the sending of a batch across nodes (issue 646) 2026-10-15 19:57:20 +02:00
mesh-tools 058d76ee8a Merge pull request 'Playbook 07 as five bounded phases with reused context; playbook 08 for small fixes' (#364) from playbooks/07-08-fast-flow into main 2026-10-15 17:54:22 +00:00
jschoubben db6430ccbf Show the report's issues as a tree, and keep the agent's brief, the red-check rule and the forge-commit check, in playbook 07
mesh/merge-gate head db6430ccbf39 merged into trunk 99a9da095010 — pass: the change touches no module of the mesh's graph
mesh/repo-check head db6430ccbf39 merged into trunk 99a9da095010 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 19:52:48 +02:00
jschoubben 458f248c05 Give the lead's ten-minute report a full and a short layout, in playbook 07 2026-10-15 19:52:31 +02:00
jschoubben 464833d0ef Rewrite playbook 07 as five bounded phases with reused context, and add playbook 08 for small fixes
mesh/merge-gate head 464833d0ef3f merged into trunk 99a9da095010 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 464833d0ef3f merged into trunk 99a9da095010 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A feature waited a day for its last issue, each small issue paid for its context again, and
waits nobody saw cost the most time. The operator set the flow on 2026-10-15: prepared context
per repository, one to two hours of local development on unit tests, one review of at most three
rounds back to the same agents, one push for the integration checks, then the delivery plan;
steps for a larger feature; a separate playbook for small fixes; models per step; a wait list
with bounds; a prepared clear at each phase end.
2026-10-15 19:26:02 +02:00
mesh-tools 99a9da0950 Merge pull request 'Add three lessons from feature mail-hardening to playbook 07' (#363) from playbook-07-lessons into main 2026-10-15 15:14:16 +00:00
jochen de8a7176ab Add three lessons from feature mail-hardening to playbook 07
mesh/merge-gate head de8a7176abb8 merged into trunk a7d4c3953599 — pass: the change touches no module of the mesh's graph
mesh/repo-check head de8a7176abb8 merged into trunk a7d4c3953599 — pass: its merge-check.sh passed
mesh/delivery delivered
A temporary-directory quota failure, a delivery group's manifest-check
ordering, and an agent that stops or finishes without committing each cost
time during that feature because the playbook did not yet say what to do.
Writing them down so the next feature avoids the same detours.
2026-10-15 17:08:43 +02:00
mesh-tools a7d4c39535 Merge pull request 'Playbook 07: an issue is one repository and five to ten minutes; a long run says where it stands' (#362) from pb07-issue-size into main 2026-10-15 12:40:48 +00:00
mesh-toolsandClaude Opus 5.5 048f14540a Size an issue for one repository and five to ten minutes, and have a long run say where it stands, in playbook 07
mesh/merge-gate head 048f14540a50 merged into trunk c2ee44255e5b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 048f14540a50 merged into trunk c2ee44255e5b — pass: its merge-check.sh passed
mesh/delivery delivered
On 2026-10-15 issues spanning one repository finished in two to four minutes, while those spanning two or
three ran for thirty to sixty, mostly merging the feature head and checking again as other agents landed.
The operator asked for agents that finish in five to ten minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:37:33 +00:00
mesh-tools c2ee44255e Merge pull request 'Playbook 07: the ready note and the brief a fixing agent follows' (#361) from pb07-agent-brief into main 2026-10-15 12:33:46 +00:00
mesh-toolsandClaude Opus 5.5 ddf23a7861 Put the ready note and the brief a fixing agent follows into playbook 07
mesh/merge-gate head ddf23a786101 merged into trunk f3bbfd4bf628 — pass: the change touches no module of the mesh's graph
mesh/repo-check head ddf23a786101 merged into trunk f3bbfd4bf628 — pass: its merge-check.sh passed
mesh/delivery delivered
They lived in a file on one machine that only one session's agents read. They are how every feature's
agents work, so they belong with the method; a feature keeps only its local facts (paths, branch, locks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:33:00 +00:00
mesh-tools f3bbfd4bf6 Merge pull request 'Playbook 07: up to five Sonnet agents at once, at most two on the larger model' (#360) from pb07-five-agents into main 2026-10-15 12:02:56 +00:00
mesh-toolsandClaude Opus 5.5 893d90fb41 Allow up to five Sonnet agents at once and keep the larger model to two, in playbook 07
mesh/merge-gate head 893d90fb417e merged into trunk 1e577168bf0c — pass: the change touches no module of the mesh's graph
mesh/repo-check head 893d90fb417e merged into trunk 1e577168bf0c — pass: its merge-check.sh passed
mesh/delivery delivered
The operator, 2026-10-15: the smaller model is as good as free against the weekly allowance, so the
limit that matters is how many agents the lead can settle and follow (three to five); subagents on the
larger model stay at two at most and as rare as possible.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:00:09 +00:00
mesh-tools 1e577168bf Merge pull request 'Playbook 07: three to five review rounds, the reviewer told its round, and the allowance in every report' (#359) from pb07-review-rounds into main 2026-10-15 11:22:56 +00:00
mesh-toolsandClaude Opus 5.5 f349c940f0 Cap review rounds at three to five and report the allowance used, in playbook 07
mesh/merge-gate head f349c940f0fe merged into trunk 1699375476c1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head f349c940f0fe merged into trunk 1699375476c1 — pass: its merge-check.sh passed
mesh/delivery delivered
A reviewer that does not know its round keeps finding the next smaller thing: one decision record went
through nine rounds on 2026-10-15, each spending the operator's shared weekly allowance. The operator
asked for three to five rounds, the round told to the reviewer, and the allowance in every report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:21:37 +00:00
mesh-tools 1699375476 Merge pull request 'Playbook 07: the lead settles an issue's questions before a smaller-model agent builds it, two at a time' (#358) from pb07-builders into main 2026-10-14 23:26:01 +00:00
mesh-toolsandClaude Opus 5.5 912fd428fb Have the lead settle an issue's questions before a smaller-model agent builds it, two at a time
mesh/merge-gate head 912fd428fb89 merged into trunk a75faad88240 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 912fd428fb89 merged into trunk a75faad88240 — pass: its merge-check.sh passed
mesh/delivery delivered
Five agents at once, each stopping on an undecided design question and restarting, spent a
week's allowance in a night. Also write down the lessons of feature 57: never land on a red
check, a new manifest word lands with the controller that reads it, and review the tree that
is pushed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:24:14 +00:00
mesh-admin a75faad882 Merge pull request 'To-be 42: the kernel module holds the kernel's run-time settings, huge pages first (ADR 0369, issue 746)' (#357) from design/42-kernel-module-holds-runtime-settings into main 2026-10-14 21:52:48 +00:00
jschoubben 9f4757a8e4 Let the kernel module hold the kernel's run-time settings, huge pages first (ADR 0369, issue 746)
mesh/merge-gate head 9f4757a8e49d merged into trunk 93f8bbfc8eaf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 9f4757a8e49d merged into trunk 93f8bbfc8eaf — pass: its merge-check.sh passed
mesh/delivery delivered
Huge pages at always flooded the control-node's spinning disks with journal
write-back and cost the controller its lease; the setting needs an owner,
and the kernel module is the one to-be 42 already names for the kernel.
2026-10-14 23:48:35 +02:00
mesh-admin 93f8bbfc8e Merge pull request 'To-be 48: cite ADR 0368, the resolver restarts itself for that module alone' (#356) from design/48-cites-adr-0368 into main 2026-10-14 21:29:04 +00:00
jschoubben 58edc4b4fb Cite ADR 0368 in design 48: the resolver restarts itself, for that module alone
mesh/merge-gate head 58edc4b4fb90 merged into trunk 010a34cc33e8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 58edc4b4fb90 merged into trunk 010a34cc33e8 — pass: its merge-check.sh passed
mesh/delivery delivered
Redoes the design-48 edit of the unmerged pull request 284, which cited the
record under its file number 0280; the record now lives in the knowledge base
as ADR 0368.
2026-10-14 23:18:38 +02:00
mesh-admin 010a34cc33 Merge pull request 'To-be 08: the five connectivity contexts walked through the network layers' (#275) from design/connectivity-through-the-layers into main 2026-10-14 21:14:49 +00:00
mesh-admin 35d5b2814f Merge pull request 'Playbook 07: review the local commits before anything is pushed' (#355) from playbook-07-review-before-push into main 2026-10-14 21:12:20 +00:00
jschoubben 3c5edc3879 Cite decisions by number now that they live in the knowledge base, and date the design today
mesh/merge-gate head 3c5edc387955 merged into trunk 35d5b2814f09 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 3c5edc387955 merged into trunk 35d5b2814f09 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 23:12:18 +02:00
jochen ab5025d508 Connectivity: walk the five contexts through the network layers, so a reader can see which layer each one acts at and read a failure from the bottom up 2026-10-14 23:11:53 +02:00
mesh-adminandClaude Opus 5.5 6c8532ce70 Review the local commits before anything is pushed
mesh/merge-gate head 6c8532ce7090 merged into trunk c032cae10030 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 6c8532ce7090 merged into trunk c032cae10030 — pass: its merge-check.sh passed
mesh/delivery delivered
A push starts the forge's checks, which build the modules and run every
test for up to a quarter of an hour, so a review finding fixed after the
push costs another full run (the operator, 2026-10-14).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 20:03:11 +00:00
mesh-admin c032cae100 Merge pull request 'Playbook 07: how several agents work one feature on one branch' (#354) from playbook-07-parallel-issues into main 2026-10-14 19:55:30 +00:00
mesh-adminandClaude Opus 5.5 071fc1c29d Name the lead, and have it take over a finished issue
mesh/merge-gate head 071fc1c29dce merged into trunk 21b79a04a283 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 071fc1c29dce merged into trunk 21b79a04a283 — pass: its merge-check.sh passed
mesh/delivery delivered
A finished issue's agent is gone, so left in fixing it goes stale; the lead
takes it over until the feature is delivered. One repository is one
delivery, not a group; readiness comes from depends-on links (review
comment 12466).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:53:20 +00:00
mesh-adminandClaude Opus 5.5 c2d5323dea Say how several agents work one feature on one branch
mesh/merge-gate head c2d5323dea5a merged into trunk 21b79a04a283 — pass: the change touches no module of the mesh's graph
mesh/repo-check head c2d5323dea5a merged into trunk 21b79a04a283 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Feature 57 was first worked as a pull request per issue into protected
branches and lost most of a day waiting on merge checks. Record the issue
as the unit, a review and checks once at the end, landing by fast-forward
under a lock, and the lessons met on the way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:50:14 +00:00
mesh-admin 21b79a04a2 Merge pull request 'Fit design 57 to walks with no first-machine gate (ADR 0360, ADR 0362)' (#353) from design/57-adr-0360 into main 2026-10-14 18:15:22 +00:00
mesh-adminandClaude Opus 5.5 bafc7b7d05 A silent machine is caught up by the backlog walk, not fixed forward
mesh/merge-gate head bafc7b7d05dd merged into trunk dcb5d5ce3883 — pass: the change touches no module of the mesh's graph
mesh/repo-check head bafc7b7d05dd merged into trunk dcb5d5ce3883 — pass: its merge-check.sh passed
mesh/delivery delivered
A machine that says nothing is often asleep; its builds wait for the
backlog walk (ADR 0360 clause 10), so no new pull request is needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:12:52 +00:00
mesh-adminandClaude Opus 5.5 5e493934a5 Say the rehearsal is the only live check, and that a fix goes into main
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 5e493934a583 merged into trunk dcb5d5ce3883 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5e493934a583 merged into trunk dcb5d5ce3883 — pass: its merge-check.sh passed
The pull requests' checks still run before the walk, and by then the
release's branches have reached main. Name the silent machine too, and
list ADR 0360, which the passages rest on (review comment 12348).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:11:52 +00:00
mesh-adminandClaude Opus 5.5 c957fcc6d7 Fit design 57 to walks with no first-machine gate
mesh/merge-gate head c957fcc6d751 merged into trunk dcb5d5ce3883 — pass: the change touches no module of the mesh's graph
mesh/repo-check head c957fcc6d751 merged into trunk dcb5d5ce3883 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
ADR 0360 sends a merge's builds to every machine at once, so release 57
has no first node to hold it and P6 can no longer wait at a gate. ADR 0362
moves the wait for the provisioner to the walk's end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:09:02 +00:00
mesh-admin dcb5d5ce38 Merge pull request 'To-be 45: the bus module says every user's slow consumer, and S9 raises it (issue 270, ADR 0357)' (#352) from design/270-the-bus-module-says-every-users-slow-consumer into main 2026-10-14 07:00:14 +00:00
jochen bb8075b37b Give ADR 0357 a home in design 45: the bus module says every user's slow consumer, and S9 raises it (issue 270)
mesh/merge-gate head bb8075b37bc1 merged into trunk 40a39c860278 — pass: the change touches no module of the mesh's graph
mesh/repo-check head bb8075b37bc1 merged into trunk 40a39c860278 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 08:42:54 +02:00
mesh-admin 40a39c8602 Merge pull request 'To-be 57: building config and provisioning, as 37 packages each an issue (ADR 0348, ADR 0349)' (#350) from design/57-building-config-and-provisioning into main 2026-10-13 23:57:18 +00:00
mesh-adminandClaude Opus 5.5 e5b98e3b0a Design 57: answer the second review of hq#350
mesh/merge-gate head e5b98e3b0a42 merged into trunk fd416cef91bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head e5b98e3b0a42 merged into trunk fd416cef91bf — pass: its merge-check.sh passed
mesh/delivery delivered
The word setting waits for R2, P3 waits for F3, the group's order is declared with after: lines, and a stop's fix is a new merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:53:59 +00:00
mesh-adminandClaude Opus 5.5 2f3e6e6641 Design 57: answer the review of hq#350
mesh/merge-gate head 2f3e6e6641d0 merged into trunk fd416cef91bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 2f3e6e6641d0 merged into trunk fd416cef91bf — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Role names instead of node names, the failure rule as ADR 0276 and 0299 have it, readiness and adoption of existing credentials said exactly, and the counts and refusals corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:47:32 +00:00
mesh-adminandClaude Opus 5.5 1405dd39e8 Design 57: ship the packages as one release, rehearsed in the lab
mesh/merge-gate head 1405dd39e8bb merged into trunk fd416cef91bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 1405dd39e8bb merged into trunk fd416cef91bf — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The operator chose one delivery over one per package; the packages meet on one branch per repository and reach main as one delivery group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:40:15 +00:00
mesh-adminandClaude Opus 5.5 11294372b7 Design 57: the work of ADR 0348 and ADR 0349 as packages, each an issue, in dependency order
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 11294372b769 merged into trunk fd416cef91bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 11294372b769 merged into trunk fd416cef91bf — pass: its merge-check.sh passed
So config and provisioning are built in small, provable steps with at most five worked at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:12:23 +00:00
91 changed files with 1501 additions and 798 deletions
+41 -115
View File
@@ -69,8 +69,7 @@ is listed under [Homonyms](#homonyms), with the qualified form each domain uses.
**Recorded in** the catalogue, one manifest per module. **Upstream of** every other domain; its words
are a published vocabulary, fixed by the manifest's schema. **Decided by** ADR 0040, 0126, 0174, 0188,
0207, 0236, 0262.
**Uses:** module, seat, config value (Config), provision (Provisioning), data class (Data), health (Health
and repair).
**Uses:** module, seat, provision (Provisioning), data class (Data), health (Health and repair).
- **manifest** — the file a module is defined by (`module.json`): what it is, what it provides and
requires, the seats it claims, the resources it declares, its tools, its data and its health. Say
@@ -84,9 +83,18 @@ and repair).
it guards: the `mesh-controller`, `postgres` and `nats` modules claim the `mesh-controller`,
`mesh-store` and `mesh-broker` seats (ADR 0079,
ADR 0116).
- **setting** — a value a manifest declares and an assignment gives, one of the two ways a node varies
a module (ADR 0174).
What a "flavor" once varied is a setting or a separate module.
*Not:* ~~flavor~~
- **preference** — a setting whose manifest gives it a default, because its value is true of the
software and only tuned per node: a font size, a width
(ADR 0262).
A setting with no default is the operator's own value: a domain, an identity.
*Not:* ~~tunable~~ (hq)
- **kept region** — a marked block in a managed file the mesh writes *into*, where the operator's own
lines survive every send and are given back when the module goes (ADR 0174). The other way a node
varies a module, besides a config value; a **managed** file's marked block is one (Provisioning).
varies a module.
- **bundle** — the artifact a module's own code is built into — its tools, a seat's implementation, a
daemon — in any language the mesh has a toolchain for; never an image. A **tools bundle** speaks MCP to
the tool runner. One module may declare several
@@ -103,55 +111,6 @@ and repair).
(ADR 0236).
The controller's verb `upgrade` shows it.
## Config — the values someone may change
**Purpose.** To hold, check and resolve every value a module reads that someone may change, and to say
where each resolved value came from. **Recorded by** the `mesh-config` module, in its buckets on the bus,
with its store as the history behind them. **Upstream of** Provisioning, which places the values;
downstream of Module and Placement. **Decided by** ADR 0262, 0277, 0348.
**Uses:** module, node, manifest (Module), assignment (Placement), provision (Provisioning), proposal
(Operator and conversation).
- **config value** — anything a module reads that someone may change: a font size, a domain, an address.
A module declares every one it reads under `config` in its manifest, with a type, a description,
validation, whether it is trusted, optionally the predefined key it falls back to, and a default or
`required`; it lands in a template file as `${config:<key>}` or is read by the module's code at runtime.
It resolves from four sources, the most specific winning: the default in the manifest; **mesh config**,
the value of the predefined key it falls back to for the whole mesh; **node config**, that key's value
on one node; and **module-assignment config**, the module's own value on one node. There is no module
value for the whole mesh. *Config* is the concept, the verb (`config`) and the manifest field
(ADR 0348). Mesh config is a source level, the value set for the whole mesh; `mesh-config` is the module
that keeps every level. The controller holds, mirrors and resolves no config value (ADR 0348 §9).
Older records and documents say "setting" for a config value and "settings" for the concept; new writing
says config value and config. The word goes on this entry's *Not:* line once the documents in force that
still use it in running prose are reworded, which is issue 683; until then it is checked by review. The
manifest field and the verb "settings" are kept as aliases until every module has moved (ADR 0348 §1,
issue 630), and are named on the line below with the rest once issue 683 is done.
*Identifier until renamed:* `takes-settings`, `${setting:}` — manifest forms kept as aliases until every
module has moved to their config names (ADR 0348 §1, issue 630).
What a "flavor" once varied is a config value or a separate module.
*Not:* ~~flavor~~
- **predefined key** — a key of the catalogue's `predefined.json`, the fixed list of keys for which mesh
config and node config hold values, each with its type, description, validation and whether it is
trusted, such as `font-size`. A config value may name the predefined key it falls back to, of its own
type; one that names none resolves from its default and the module's own value only. The list is the
catalogue build's: a change to it is validated with every manifest naming a key, and removing a key still
named or still holding a value is refused (ADR 0348).
- **trusted** — said of a config value that decides what runs or who can reach what. At every level it is
set only at the controller's terminal or through `config` with propose, on the operator's Approve; a
module value that falls back to a trusted predefined key is itself trusted (ADR 0277, ADR 0348 §8). Not
the trusted sender of an operator message (see [Homonyms](#homonyms)).
- **preference** — a config value whose manifest gives it a default, because its value is true of the
software and only tuned per node: a font size, a width
(ADR 0262).
A config value with no default is `required`: the operator's own value, such as a domain or an
identity, and while nothing sets it the module's files are not written (ADR 0348).
*Not:* ~~tunable~~ (hq)
- **`mesh-config`** — the module that owns every config value, and not the source level mesh config: it
holds the seat `mesh-config` once for the mesh, validates, takes a trusted value only from the
controller's terminal or the operator's Approve, keeps the history with who set it and why, and resolves
the four sources (ADR 0348).
## Core — the mesh's own machinery
**Purpose.** To keep the controller, the bus, the store and every node's engine running and agreed, so
@@ -219,7 +178,7 @@ every other domain has something to run on. **Recorded by** the controller and t
(ADR 0229).
- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its
records (nodes, assignments, walks, builds), after each change commits in its store and only
records (nodes, assignments, rollouts, builds, settings), after each change commits in its store and only
when the value differs, with one event per write. The store stays the record; the self-check compares
the two and raises `mirror-drifted` on any difference
(ADR 0289). Shortened to *mirror* where the
@@ -231,10 +190,9 @@ every other domain has something to run on. **Recorded by** the controller and t
**Purpose.** To decide, send and apply what each node runs, and to say why.
**Recorded by** the controller's `inventory` database. **Upstream of** Provisioning, Connectivity, Data
and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 0252, 0261.
**Uses:** node, machine, module, seat, manifest (Module), config value (Config).
**Uses:** node, machine, module, seat, setting (Module), manifest (Module).
- **assignment** — a module put on a node. Its config values on that node (module-assignment config,
Config) are `mesh-config`'s, never part of what the controller records (ADR 0348 §9).
- **assignment** — a module put on a node, with the settings that node gives it.
- **pending assignment** — an assignment of a module whose build is still running, kept, and made by the
controller when that build registers the module, or ended with why when it does not
(ADR 0261).
@@ -269,7 +227,7 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
(ADR 0181).
Not a name a manifest carries.
- **agent account** — the login every coding agent on a node runs as, when the node names one of its own;
stated on the node record at the controller's terminal, never by a verb or a config value. Empty means agents
stated on the node record at the controller's terminal, never by a verb or a setting. Empty means agents
run as the operator account. It cannot become root without a person, and the node-engine judges that on
every look (ADR 0266).
- **account group** — a group of the machine's group database that a module puts an account in, by
@@ -280,21 +238,15 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
## Provisioning — one module serving another
**Purpose.** To resolve which module serves a provision for which consumer, to wire the two, and to
place every value a module needs into its files on its node. **Recorded by** the controller and
`mesh-provisioner`. **Upstream of** Data; downstream of Module, Placement, Config and Identity.
**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263, 0349.
**Uses:** module, seat, node-engine (Core), declaration (Placement), config value (Config), credential
and vault (Identity and access).
**Purpose.** To resolve which module serves a provision for which consumer, and to wire the two.
**Recorded by** the controller. **Upstream of** Data; downstream of Module, Placement and Identity.
**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263.
**Uses:** module, seat, credential (Identity and access).
- **provision** — a service one module `provides` and others `require`; the mesh resolves a provider
and wires the two with an endpoint and a credential. A provision is a service you offer, a seat is a
role you occupy, and the two meet where a seat delivers a provision: occupying the seat is what makes
a module *the* provider of it.
Every value a module needs is a provision from the domain that owns it: placement from the controller,
config values from `mesh-config`, shared secrets from the vault, a consumer's values from the module
providing it, an item of a collection from its contributor. A module says only what it needs and where
it lands (ADR 0349).
- **provider** and **consumer** — the module that provides a provision, and the module that requires
it. Which of several providers serves a consumer is resolved
(ADR 0084).
@@ -312,28 +264,6 @@ and vault (Identity and access).
node seat `node-mounts`, as a mount unit and an automount unit, mounted on access (ADR 0263).
- **occasional source** — a share on a device that is not a node and comes and goes, such as a USB disk;
its absence is a state, *not present*, never a fault (ADR 0263).
- **mesh-provisioner** — the module that places provisions on the nodes: it holds the seat
`mesh-provisioner` once for the mesh, keeping which domain answers which kind of placeholder, what each
module on each node needs, what is applied where and the items of every collection, and the seat
`node-provisioner` on every node, which its provisioner client serves (ADR 0349).
- **provisioner client** — mesh-provisioner's part on one node. For each module there it reads the
placeholders of its template files, records the needs, watches each owner's values for the node, fills a
template only once every value is there, has the node-engine write the file through a narrow local verb,
makes the program take the change as the file declares (`restart-on`, `reload-on`, `read-on-use`), and
reports `prov.applied`. It runs as a service with a bus account of its own, never in the tool runner
(ADR 0349).
- **collection** — a provision with many providers. The **collector**, a module that takes additions from
others (a shell's snippets, a bar's blocks), requires it; each **contributor** ships its **item** as a
template in its own repository and provides it. The provisioner client fills the collector's template
from the items of the contributors placed on its node, and an item goes when its contributor leaves the
node (ADR 0349).
- **owned** — a placed file that is the mesh's whole: the provisioner writes it back whenever it differs.
It is one of three answers a manifest gives for each file it places (`owner-at-runtime`): **owned**,
**seed** or **managed**. Whichever it is, the file has one writer: the node-engine writes it only on the
provisioner client's word (ADR 0349).
- **seed** — a placed file the provisioner writes once; from then on it is the program's.
- **managed** — a placed file of which only its named keys or a marked block (a kept region, Module) are
the mesh's; the program keeps the rest.
## Identity and access — who may do what
@@ -348,7 +278,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
- **secret** — a credential or other value the vault keeps sealed and the node-engine unseals into a
file at apply (ADR 0085).
- **secret family** — a module's own secrets declared as one name ending in `-*`; each member is the prefix
and a name (`smb-password-<source>`, one per source the module's config values name), given by its full name
and a name (`smb-password-<source>`, one per source the module's settings name), given by its full name
and never made by the mesh
(ADR 0283).
- **vault** — the module that owns every secret (`mesh-vault`,
@@ -376,7 +306,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
**Purpose.** To take one commit from its pull request to every node that should run it, and to put it
back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the
planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
planner, the rollout). **Downstream of** Module, Core, Health and repair and Data.
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323.
**Uses:** module, node, declaration and send (Placement), health (Health and repair), person.
@@ -418,7 +348,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
- **delivery** — one commit in one repository on its way to the nodes, from its pull request's head
being announced to one of its final states: one pull request, one status, one note
(ADR 0239).
From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk
From its merge on it is delivered by the rollout of the batch that answers it, and ends with that rollout
(ADR 0276).
Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`,
`held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`;
@@ -427,9 +357,9 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
mechanism, which the as-is designs describe; it is never a word for a delivery.
*Not:* ~~deployment~~ (hq)
- **covered** — the end of a delivery that a later merge containing it delivered: every module it moves was
sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own
sent to every machine running it by the rollout of a later merge into the same trunk, delivered on its own
passing check, whose merge commit the forge says contains this one. Not *superseded*, which is a delivery
whose walk a newer delivery took over before it ended; nor *delivered*, which its own walk or an empty plan
whose rollout a newer delivery took over before it ended; nor *delivered*, which its own rollout or an empty plan
ends in (ADR 0270).
- **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's
repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One
@@ -442,18 +372,18 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
- **merge window** — the rolling wait in which the controller collects merges before it plans them: each
merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest
10 minutes after its first merge (ADR 0276).
- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more
than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*,
- **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more
than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*,
which is a delivery group.
- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat
of the same name. It records and decides; the controller sends, judges and rolls back when it is asked.
- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
- **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236,
ADR 0276). One walk is open at a time; it names every merge it answers. A primitive the delivering stage
asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules
built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own:
a module *rolls out* by its `roll` policy.
*Not:* ~~rollout~~ (hq)
ADR 0276). One rollout is open at a time; it names every merge it answers. A primitive the delivering stage
asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules
built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll`
policy.
*Not:* ~~walk~~ (hq)
- **first-node gate** — the judgement on a delivery's first node: healthy three times, no witness put it
back, no condition raised since the send (ADR 0236 §2). Not *the gate* bare (see [Homonyms](#homonyms)).
- **core module** and **leaf module** — a core module is one on the controller's own path (the controller,
@@ -464,7 +394,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
the core modules add mesh-delivery and the resolver.
- **delivery time** — from a delivery's merge to every node of its deploy plan that is heard from running its
modules' builds and reporting them healthy. Its **delivery budget** is what it is held to: three minutes for
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the walk's time, which ends at the send. A
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the rollout's time, which ends at the send. A
measure inside the fix budget, not the operator's target (ADR 0313).
- **fix time** — from an issue's first move to `fixing` to the moment the last of its fixes is delivered: the
operator's measure of a fix, the agent's work, the review and the checks included, the delivery time its last
@@ -492,7 +422,7 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283.
running session, started before, does not have that group yet. Said by the node-engine as the module's
resource of kind `account`; it clears at the first look after a new login (ADR 0252).
- **waiting** — a resource's health when its module's tool check says it waits for the operator: a named own
secret or config value of the module, not given yet, without which a part cannot work. Not a fault: the
secret or setting of the module, not given yet, without which a part cannot work. Not a fault: the
controller checks it and says it as the condition `needs-operator`, naming the act, and the first-node gate
reads it as *waits for a person*
(ADR 0283).
@@ -616,7 +546,7 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
published for that asker alone. The only thing an asker acts on (ADR 0259).
- **proposal** — a change an agent, or anyone the bus admits, puts forward through a verb and that takes
effect only on the operator's warrant, bound to the exact thing shown: a shared instruction section (ADR
0274), a trusted config value (ADR 0277, ADR 0348).
0274), a trusted setting's layer (ADR 0277).
A proposal changes nothing by itself; declined, expired or refused, it is discarded.
- **approval mark** — a tool's or verb's declaration in its manifest (`needs-approval`) that each call to it runs
only on the operator's warrant for that one call, with its exact arguments; the tool runner that serves it, or
@@ -633,7 +563,7 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
build seat's holder, whose seat name `node-build-agent` is an identifier to rename.
- **instruction section** — one named section of the managed instruction file, which every agent session on the
nodes it is for reads first, as the mesh's word: set at the controller's terminal in the agent module's
config, or **proposed** by an agent and made active on each node only by the operator's warrant, which that node
settings, or **proposed** by an agent and made active on each node only by the operator's warrant, which that node
proves itself (ADR 0274). A hand-over between sessions is one.
- **personal rule** — one short text in the operator's own words about how the operator wants their agent
to work with them, registered for the operator (the **operator scope** of the agent module) and rendered
@@ -648,10 +578,10 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
## Media — what a person watches, listens to and reads
**Purpose.** To find, fetch and file copies of the works a person watches, listens to and reads, each in
the library its kind names. **Recorded by** the `media-importer` module's config values and its memory of what
the library its kind names. **Recorded by** the `media-importer` module's settings and its memory of what
it was told, and the servarr apps' own libraries. **Upstream of** nothing. **Decided by** ADR 0295, 0296,
0302.
**Uses:** module, seat, config value (Config).
**Uses:** module, seat, setting (Module).
- **content** — a work a person watches, listens to or reads: a film, an album and its tracks, an
episode, a book, a session of a sport event. It is known by an identity from a catalogue outside the
@@ -662,7 +592,7 @@ it was told, and the servarr apps' own libraries. **Upstream of** nothing. **Dec
- **media release** — one copy of a work an engine can fetch, such as an NZB or a video. Downloaded, it
sits in the downloads folder, not yet filed. ADR 0302 says *release*.
- **library** — one folder of one kind, filed by one importer: the media-importer's, named in its
config, or a servarr app's. Several libraries may hold one kind, as TV Shows and Anime both hold
settings, or a servarr app's. Several libraries may hold one kind, as TV Shows and Anime both hold
series.
- **media file** — a file on disk together with its metadata: its kind, the fields that identify the
work, and what part of the work the file is. It is what the media-importer takes.
@@ -742,7 +672,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|---|---|---|---|
| plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement |
| | what a delivery would do to the mesh | **delivery plan** | Change and delivery |
| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery |
| | the sending of one batch of commits across nodes | **rollout** (verb `plans`) | Change and delivery |
| | a step a person starts, like the bus's | **planned step** | Core |
| membership | the subjects a bus account is issued | **membership** | Provisioning |
| | an account in a group of the machine | **account group** | Placement |
@@ -756,7 +686,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
| | the pull request status | **merge gate** | Change and delivery |
| | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement |
| tier | a level of the mesh | **layer** | Core |
| | a step of a walk | **tier** | Change and delivery |
| | a step of a rollout | **tier** | Change and delivery |
| | how much proof an ask needs | **assurance level** | Identity and access |
| ask | a request for the operator's input | **ask** | Operator and conversation |
| | an entry in the build queue | **build request** | Change and delivery |
@@ -782,10 +712,6 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
| | a question someone must answer | **issue** | The record |
| deploy | the per-node part of a delivery plan | **deploy plan** | Change and delivery |
| | a wrapped program's own deploy | its own word | — |
| managed | a placed file of which only named keys or a marked block are the mesh's | **managed** (`owner-at-runtime`) | Provisioning |
| | a file the mesh writes, such as the agent's instruction file or `managed_settings` | the file's own name, or *a file the mesh places* | — |
| trusted | a config value that decides what runs or who can reach what | **trusted** config value | Config |
| | a sender on the operator's own list | **trusted** sender, its message an **operator message** | Operator and conversation |
## How this page is kept
+5 -4
View File
@@ -50,13 +50,14 @@ the expensive half.
| # | Playbook | Trigger |
|---|---|---|
| [01](01-research.md) | Research | An idea worth investigating before committing to design |
| [02](02-graduation.md) | Graduation & design change | Research concludes, or a design must change |
| [01](01-research.md) | Research | An idea from the operator, or one worth investigating before committing to design: setup questions, findings, a direction the operator confirms |
| [02](02-graduation.md) | Graduation & design change | Research concludes (its direction confirmed), or a design must change: decision records, the to-be design |
| [03](03-issues.md) | Issues | Something is wrong — often with the owner unknown |
| [04](04-build-handoff.md) | Build handoff | A design is ready to be built |
| [04](04-build-handoff.md) | Work breakdown and handoff | A design is ready to be built: milestones usable on their own, issues per repository, each milestone one feature step |
| [05](05-constitution-sync.md) | Constitution sync | `how-we-build.md` changed a rule the mesh enforces |
| [06](06-writing-a-module.md) | Writing a module | Something that runs today must run on the mesh |
| [07](07-feature-branches.md) | Feature branches across repos | Work that changes code, in one repo or several at once |
| [07](07-feature-branches.md) | Features | Work that changes code across issues or repositories: prepared context, local development, one review, one check, one delivery |
| [08](08-small-fixes.md) | Small fixes | One issue in one repository, about an hour of work: one review, one push, one delivery |
## The cycle is checked
+77 -38
View File
@@ -1,66 +1,105 @@
# Playbook 01 — Research
**Trigger.** An idea, technology or approach worth investigating before it is committed to
design. Also: an as-is document that raises a question nobody can answer.
**Trigger.** An idea the operator expresses, or an idea, technology or approach worth investigating before
it is committed to design. Also: an as-is document that raises a question nobody can answer. A symptom (a
bug, something broken) is not research: it follows playbook [03](03-issues.md).
**Who runs it.** Anyone.
**Who runs it.** Anyone; in an agent session, the session that heard the idea leads it.
**Where it lives.** A research effort lives whole in the knowledge base, the seat `knowledge-base`, reached
through the mesh MCP server from any node and any folder (ADR 0290). This repository holds none: an effort
is written with the knowledge base's verbs, never as a folder here.
through the mesh MCP server from any node and any folder (ADR 0290). This repository holds none.
## The cycle it starts
```
idea ─► research (01) ─► decisions and design (02) ─► work breakdown in milestones (04)
─► each milestone built and delivered as one feature step (07) ─► what is reported afterwards (03, 08)
```
An idea arrives in one of two forms, and the first step differs:
- **A new idea**, where only the mesh's own context matters: read what the knowledge base holds on it.
- **An idea for an existing application**: read that application's to-be and as-is designs, its decision
records and its open issues first; the effort `touches` them.
## Steps
1. **Read what is known first.** `knowledge-base.ask` with the question in plain words returns the decision
records and research efforts that bear on it; `knowledge-base.get` with `kind: research` and a number
reads one effort, and `reasoning: true` (or `section` with a document's name) reads its documents.
2. **Take a number.** `knowledge-base.reserve` with `kind: research` and `for`: the effort's title. The
number comes from the issue tracker's counter and is spent whether or not the effort is ever proposed;
one never proposed is withdrawn after a week. Never pick a number by hand.
3. **Propose it.** `knowledge-base.propose` with `kind: research`, the `number` reserve gave, a one-line
`title`, `status: active`, `date` (the day it began), and `sections`: its documents in order, each a
`name` and its markdown `text`. The first is `00-overview.md`: **what** is being investigated, **why**,
and **what it touches**. Name what it touches as `links` of kind `touches` (`"0276"`, `"research 045"`,
`"issue 365"`, or a design's path), not only in prose, and what it names for another reason as `cites`.
4. **Do the research** in further documents of the same effort: notes, option analyses, evidence, draft
designs. Each change is `knowledge-base.propose` again with the effort whole and `why`, one line saying
why this version. Every version is kept; `knowledge-base.history` lists them and gives the difference
between two.
1. **Read what is known.** `knowledge-base.ask` with the idea in plain words returns the decision records and
research efforts that bear on it; `knowledge-base.get` reads one (`reasoning: true`, or `section`, for its
documents). Read the issue tracker for open issues in the same area (`issue-tracker.list`).
2. **Ask the operator to set it up.** A few questions, one at a time, each answerable with a choice and
carrying a recommendation: what comes first, what is out of scope, which options must be tried, what
done looks like. Three to five questions is usual; the answers are the effort's first findings.
3. **Take a number.** `knowledge-base.reserve` with `kind: research` and `for`: the effort's title. The number
is spent whether or not the effort is ever proposed; one never proposed is withdrawn after a week. Never
pick one by hand.
4. **Propose it.** `knowledge-base.propose` with `kind: research`, the `number`, a one-line `title`,
`status: active`, `date` (the day it began), and `sections`, each a `name` and its markdown `text`:
- `00-overview.md`: the idea in the operator's words, what exists, the operator's answers (dated), and the
words the effort uses that the glossary lacks, each defined in one line;
- `01-questions.md`: the questions to answer, the first ones first;
- `02-method.md`: what will be tried and measured, and on what.
`propose` refuses an effort that breaks a rule that judges one record (an unknown status, a link to
nothing, a retired word of the glossary, a daemon's line), naming every reason at once and writing nothing.
The knowledge base then reviews the proposal itself, through a judgement that sees only the record, the
records it links and the rules; on `changes` the effort stays proposed with the findings, and its author
proposes it again. An effort that needs the operator's word is proposed with `needs-operator: true` and
waits for the operator's Approve on a channel that proves who answered.
Name what it touches as `links` of kind `touches` (`"0276"`, `"research 045"`, `"issue 365"`, or a
design's path), and what it names for another reason as `cites`.
5. **Do the research.** Try, measure and read, smallest question first. Each finding goes into
`04-findings.md`: what was tried, what it showed (with numbers), what it means. Each change of the effort
is `knowledge-base.propose` again, whole, with `why`; every version is kept, and `knowledge-base.history`
lists them and gives the difference between two. Until issue 844 is delivered, the knowledge base
takes a new version of an accepted, active effort only as a progressive insight: give `insight`, saying
what the effort held before the new finding, and mark the finding in the text with its date.
6. **Settle the direction with the operator.** When the questions are answered, write `05-direction.md`:
the proposed decisions in a few lines each, the options rejected and why, what is left open. Ask the
operator to confirm or change it, one point at a time. Then the effort graduates through playbook
[02](02-graduation.md).
## Bounds and models
| Step | Bound | Model |
|---|---|---|
| reading and the setup questions (1, 2) | 30 minutes | the session's own, larger |
| proposing (3, 4) | 15 minutes | larger |
| each experiment or reading task (5) | 30 minutes | smaller, one agent at a time unless the operator allows more |
| the direction (6) | 30 minutes | larger |
A step past its bound is said to the operator at once, with what is being done about it. The lead reports
as playbook 07 does: every ten minutes and at once when something is found, stops or needs the operator,
with the allowance used.
## What makes research worth reading
State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"*
outranks *"the dependency rule is not followed"*. An effort that measured nothing has not
finished.
State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"* outranks *"the
dependency rule is not followed"*. An effort that measured nothing has not finished.
Research describes real observations but never identifies the mesh it observed. The shape of a
finding survives anonymisation intact — *a node publicly named but behind a household NAT*
carries the whole lesson without naming anything.
Research describes real observations but never identifies the mesh it observed. The shape of a finding
survives anonymisation intact — *a node publicly named but behind a household NAT* carries the whole
lesson without naming anything.
`propose` refuses an effort that breaks a rule that judges one record (an unknown status, a link to
nothing, a retired word, a daemon's line), naming every reason at once and writing nothing. The knowledge
base then reviews the proposal itself; on `changes` the effort stays proposed with the findings and its
author proposes it again, at most three times before the findings go to the operator with a
recommendation. An effort that needs the operator's word is proposed with `needs-operator: true` and waits
for the operator's Approve on a channel that proves who answered.
## Do not
- Do not put status in prose. It is the effort's `status`, and the prose must not restate it.
- Do not give a `became` link while the effort is open — playbook 02 gives it at closure.
- Do not write into `03-DESIGN` from an open effort.
- Do not write an effort, or a copy of one, into this repository: the knowledge base is the only place one
is read, and `records.py` refuses a new path under `01-RESEARCH/`.
- Do not write an effort, or a copy of one, into this repository: `records.py` refuses a new path under
`01-RESEARCH/`.
## Closing
An effort never just stops. It closes through playbook [02](02-graduation.md) as `graduated`
or `abandoned`, always with a `became` link to what it turned into. Nothing is deleted —
what was rejected, and why, is the more expensive half to rediscover.
An effort never just stops. It closes through playbook [02](02-graduation.md) as `graduated` or
`abandoned`, always with a `became` link to what it turned into. Nothing is deleted — what was rejected,
and why, is the more expensive half to rediscover.
## How it is checked
The rules on an effort are the knowledge base's: `propose` refuses one that breaks a rule that judges one
record (the module's table test holds one case per rule), and the rules that judge the whole record set run
after every acceptance and raise a condition when one fails. That this repository holds no effort is
`records.py`'s `no-records` check, on every pull request.
`records.py`'s `no-records` check, on every pull request. An active effort with no new version for seven
days is the smell of research left behind: its lead closes or continues it.
+11 -3
View File
@@ -13,11 +13,16 @@ clauses in force for a topic or a record).
## Graduating research
The effort's `05-direction.md`, confirmed by the operator (playbook [01](01-research.md), step 6), is what
graduates. The lead writes the records and the design on the larger model; it is one sitting, bounded at
about an hour, and said to the operator if it runs past.
1. **Check it against GENESIS.** An effort graduates only if its conclusion is traceable to
[`mission.md`](../mission.md), [`context.md`](../context.md) and
[`effect.md`](../effect.md). If it conflicts, either the effort is wrong or GENESIS is —
say which, in writing, before proceeding.
2. **Record the decision** (below, *Writing a decision record*).
2. **Record the decisions** (below, *Writing a decision record*): one record per choice that could be
reversed on its own, not one record for the whole effort.
3. **Write the design.** Create the document under `03-DESIGN/01-to-be/` with frontmatter:
```yaml
@@ -31,7 +36,9 @@ clauses in force for a topic or a record).
---
```
4. **Close the effort.** `knowledge-base.propose` the effort again, whole, with `status: graduated` and a
4. **Hand it to the work breakdown.** The design goes on through playbook [04](04-build-handoff.md):
milestones, issues, and each milestone built as one feature step (playbook [07](07-feature-branches.md)).
5. **Close the effort.** `knowledge-base.propose` the effort again, whole, with `status: graduated` and a
`became` link to the decision record (`"NNNN"`) and to the design document (its path). An effort that
stops without a decision is proposed with `status: abandoned` and a `became` link to what replaced it.
@@ -66,7 +73,8 @@ clauses in force for a topic or a record).
superseded record cited alone, an unknown topic, a retired word of the glossary, a daemon's line, a
deferral no issue tracks. The knowledge base then reviews the proposal itself, through a judgement that
sees only the record, the records it links and the rules, and answers `approve` or `changes`. On
`changes` the record stays proposed with the findings: revise it by proposing it again, whole, with `why`.
`changes` the record stays proposed with the findings: revise it by proposing it again, whole, with `why`;
at most three rounds, then the findings go to the operator with a recommendation.
On `approve` it is accepted, or, when it says `needs-operator`, it waits for the operator's Approve on a
channel that proves who answered. Nothing proposed is in force. `knowledge-base.get` shows a record's
reviews and the operator's ask when it waits for one.
+55 -25
View File
@@ -1,35 +1,65 @@
# Playbook 04 — Build handoff
# Playbook 04 — Work breakdown and handoff
**Trigger.** A to-be design is settled and work is about to start in a code repository.
**Trigger.** A to-be design is settled (playbook [02](02-graduation.md)) and its building is about to start.
**Who runs it.** Whoever starts the build.
**Who runs it.** The lead who will run the feature, on the larger model. It is one sitting, bounded at about
an hour, and said to the operator if it runs past.
## What it produces
A design becomes **milestones**, and each milestone becomes **issues**:
- **A milestone is usable on its own.** Once its delivery plan is done, the operator or the mesh can do
something new, said in one sentence ("a Claude Code session is mirrored into a thread on the bus").
A milestone that changes nothing anyone can use is not a milestone; fold it into the next.
- **A milestone is one feature step** of playbook [07](07-feature-branches.md): prepared, developed in one to
two hours, reviewed, checked and delivered as one. A milestone larger than that is split.
- **An issue is one repository's part of a milestone,** sized so a fixing agent finishes it inside the
milestone's development time. Issues that must land in order are linked `depends-on`.
## Steps
1. **Confirm the design is settled.** Its frontmatter reads `status: designed`, and every
claim in it traces to a decision record in the knowledge base (`knowledge-base.get` for each entry of
its `decisions:`). An open question in the text is a
reason to run playbook [01](01-research.md), not to start building around it.
2. **Name the owner.** Set `code:` in the design's frontmatter to the repositories from
[`repos.md`](../repos.md). If the repository does not exist yet, add it to `repos.md` in
the same change.
3. **Check the as-is.** Read the matching `03-DESIGN/00-as-is/` document. What is being
replaced is stated there; if it is not, write it before changing it. Building against an
undocumented as-is is how a shipped behaviour gets lost.
4. **Flip the status.** `status: in-progress`, `updated:` today.
5. **Build in the code repository.** HQ is not a code repository and never carries
implementation.
6. **On completion**, run the "when something ships" section of playbook
1. **Confirm the design is settled.** Its frontmatter reads `status: designed`, and every claim in it traces
to a decision record in the knowledge base (`knowledge-base.get` for each entry of its `decisions:`). An
open question in the text is a reason to go back to playbook [01](01-research.md), not to build around it.
2. **Name the owners.** Set `code:` in the design's frontmatter to the repositories from
[`repos.md`](../repos.md). A repository that does not exist yet is added to `repos.md` in the same change.
3. **Check the as-is.** Read the matching `03-DESIGN/00-as-is/` document. What is being replaced is stated
there; if it is not, write it before changing it. Building against an undocumented as-is is how a shipped
behaviour gets lost.
4. **Break it into milestones.** Write them into the design, in a section *Milestones*: for each, its number,
its one-sentence outcome, the repositories it touches, and what it depends on. Order them so each is
usable as soon as it is delivered, the smallest useful one first.
5. **Ask the operator to confirm the milestones**, in a short table: number, outcome, repositories, the
parts each repository builds. Change what they change.
6. **Open the issues.** For each confirmed milestone, `issue-tracker.open` one issue per repository part, its
title the symptom or capability in plain words, its report naming the design, the milestone and the
decisions it builds with; link the order with `depends-on`. Write the issue numbers into the milestone's
entry in the design.
7. **Flip the status.** `status: in-progress`, `updated:` today; the design change goes through playbook
[08](08-small-fixes.md) (one hq pull request).
8. **Hand each milestone to playbook 07, one at a time.** The milestone is the feature step; its issues are the
step's issues; the next milestone starts when the previous one's delivery plan is done.
9. **On completion of the last milestone**, run the "when something ships" section of playbook
[02](02-graduation.md).
## Rules
- Every merge is a human checkpoint, without exception.
- What the handoff leaves for later is an issue with an owner, named in the design or the decision that
leaves it — never a sentence saying "later". The design's `code:` names who owns what is built; the issue
names who owns what is not. the knowledge base's rule `deferral` checks the decision record when it is proposed
- What the handoff leaves for later is an issue with an owner, named in the design or the decision that leaves
it — never a sentence saying "later". The design's `code:` names who owns what is built; the issue names who
owns what is not. The knowledge base's rule `deferral` checks the decision record when it is proposed
([issue 354](../../04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/00-report.md)).
- Never open a pull request unprompted. A permissions list saying it is allowed is not a
request.
- Work in an isolated worktree, never a shared checkout. A failed `cd` in a shared checkout
commits to the wrong branch, and the error scrolls past.
- HQ is not a code repository and never carries implementation.
- How the building is done — worktrees, agents, review, checks, the ready mark and the delivery — is playbook
07's, not this one's.
## How it is checked
Read by the lead and the reviewer today; a check in hq's merge-check is tracked by issue 845.
- A design in `status: in-progress` has a *Milestones* section, each milestone with its outcome and its
issues; an in-progress design without one is the smell of building without a breakdown.
- Every issue a milestone names exists in the tracker (`issue-tracker.get`), and an issue in `fixing` names
its milestone in its report.
- A milestone whose issues are all `resolved` has had its delivery plan done; its outcome sentence is proven on
the mesh before the next milestone starts.
+199 -55
View File
@@ -1,69 +1,213 @@
# Playbook 07 — Feature branches across repos
# Playbook 07 — Features
**Trigger.** Work that changes code — in one code repo or in several at once (`mesh-sdk`,
`mesh-controller`, `mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a decision rides along).
**Trigger.** A feature: work that changes code in one repository or several (`mesh-sdk`, `mesh-controller`,
`mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a design rides along), too large to be one small fix.
A small fix (one repository, one issue, about an hour of work) follows
[playbook 08](08-small-fixes.md) instead.
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly —
it is the guard against the failure it was written for.
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly.
## The failure it prevents
## The failures it prevents
A feature was worked as a branch-and-MR per *unit of thought* — one per decision, one per
stacked increment — and each MR was treated as finished when it was *opened*, not when it was
*merged*. Across repos the same feature took a different branch name in each. The MRs piled up
unmerged: one session left **sixteen** stacked intermediate MRs that had to be consolidated and
closed by hand. An MR is a review checkpoint, not a scratchpad.
- **Pull requests as scratchpads.** A feature worked as one pull request per unit of thought left sixteen
stacked pull requests unmerged, consolidated by hand afterwards. A pull request is the end of a feature,
not a step in it.
- **Waiting on checks mid-feature.** A feature worked as one pull request per issue into protected
branches spent most of a day waiting for merge checks while the code was ready.
- **Context bought again and again.** A fresh agent per small issue spends most of its run reading the
playbook, the issue and the code before it writes a line. On 2026-10-14 an issue sized for five to ten
minutes took forty-one.
- **Finished work that never ships.** A feature that kept taking new issues held thirteen landed issues
undelivered for a day: an issue done at 21:09 was still not on the machines at 15:00 the next day.
- **Work that waits and nobody notices.** Most lost time was not work at all, but a check, a delivery plan
or an agent waiting on something that had stopped, unseen.
## The rule
## The shape
One feature is **one branch name**, **one worktree per repo**, **one MR per repo**, opened
**once, at the end**.
One feature is **one branch name**, **one worktree per repository**, **one pull request per repository**,
opened **once**, after the review. It runs in five phases, and its time is bounded:
1. **Name the feature once.** `feat/<slug>`. The *same* branch name in every repo the feature
touches — never a different name per repo, never a fresh branch per increment within the
feature.
2. **Isolate each repo.** One git worktree per touched repo under `.work/<slug>/<repo>`, branched
off `main`:
```
git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main
```
Parallel features never collide, and no shared checkout is edited.
| Phase | What happens | Bound |
|---|---|---|
| 0. Prepare | the lead settles the open questions and writes one context file per repository | 15 minutes |
| 1. Develop | one fixing agent per repository builds every issue locally, unit tests only | 1 to 2 hours |
| 2. Review | one reviewer reads all of it; reports go both ways; at most 3 rounds | 15 minutes |
| 3. Check | push; the forge's merge checks run the integration tests; no further review | one check run |
| 4. Deliver | merge; the delivery plan builds, publishes and sends the whole feature | its delivery plan |
**Add the untouched siblings the lab reads.** The lab beds find the other repositories by
sibling path from the lab checkout (`../mesh-tools/module.json`, `../mesh-sdk`, …), the way
the main layout has them. A `.work/<slug>/` directory holding only the touched repos fails a
bed at once with `no manifest for mesh-tools at …/.work/<slug>/mesh-tools/module.json`, after
genesis has already passed. Give the directory those repos as **detached worktrees on `main`**
— never symlinks:
```
git worktree add --detach .work/<slug>/mesh-tools main
```
3. **Commit as you go — locally.** Increments land on the one branch. Nothing is pushed and no
MR is opened mid-feature.
4. **Finish, then publish.** When the whole feature is done — every repo, tests green — push
every branch and open **one MR per touched repo**, together.
5. **Merge promptly, once approved.** Every merge into `main` is notified and approved
(ADR 0023); once it is, merge —
do not leave it sitting. The branch is deleted on merge.
6. **Leave nothing behind.** After the MRs merge, no `feat/<slug>` branch and no `.work/<slug>`
worktree survive.
A feature that cannot be developed in about two hours is split into **steps**. Each step runs all five
phases on its own; the next step starts only when the previous step's delivery plan is done. A step is the
unit that ships, so nothing finished waits for work that is not.
## What this is not
**Scope is fixed at phase 0.** An issue found during a feature goes to the next step or the next feature,
never into the running one, unless it blocks the running one.
- **Not a licence to batch unbounded work.** A feature is a *bounded* unit; if it sprawls for
days, end-of-feature bloat merely replaces per-increment bloat. Split it into features, each
its own branch and MR.
- **Not a second trunk.** Every repo branches off `main`. There is no longer an `initialization`
trunk.
## Phase 0 — Prepare (the lead, larger model)
1. **Name the feature once:** `feat/<slug>`, the same branch name in every repository it touches. The
branch is never protected.
2. **Make the worktrees:** one per touched repository under `.work/<slug>/<repo>`, branched off `main`
(`git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main`). The lab beds find the other
repositories by sibling path, so add the untouched siblings the lab reads as detached worktrees on
`main` (`git worktree add --detach .work/<slug>/mesh-tools main`), never symlinks.
3. **Settle every open question** in the feature's issues before any agent starts: on the issue, through
the knowledge base, or by asking the operator. A fixing agent that meets an undecided design question
stops, and its run is spent again.
4. **Write one context file per repository**, `.work/<slug>/<repo>.context.md`. It is what a fixing agent
reads instead of rediscovering the code, and it is reused by every agent and every round:
- the issues it builds, in order, each with: the files and functions it changes, the decisions it builds
with (quoted, with where each was decided), the test to write first, done-when in one sentence;
- the repository's map for this feature: the packages and files involved, with one line each on what they
hold, and the patterns to copy (file:line);
- what not to touch, and what lands beside it in other repositories;
- the local facts: the worktree, the branch, the check commands, a temporary directory for tests outside
the home (`TMPDIR`), a throwaway database server if the store tests need one.
When the reading is large, a read-only agent on the smaller model answers the lead's exact questions with
file:line, and the lead writes the file from its answers.
5. **Move the issues to `fixing`**, `by` naming the feature's agent for that repository.
## Phase 1 — Develop (one fixing agent per repository, smaller model)
- **One agent per repository, kept for the whole feature.** It reads its context file once and builds the
issues one after another, committing each locally with a message naming the issue ("Keep the assignments
on the bus (issue 652)"). It is not replaced per issue: the lead continues the same agent (by its id) with
the next instruction, so its context is paid for once.
- **Unit tests only.** Each issue's first test fails before its change and passes after; `gofmt`, `go vet`
and the repository's unit tests stay green. The integration tests (`merge-check.sh` with the store and the
bus) wait for phase 3.
- **Nothing is pushed** and no pull request is opened in this phase. Never land on a red check: run the
checks, read the result, then land in a separate step.
- **The fixing agent's brief** (the lead gives it with the context file): name your issue in every step and
message ("issue 805: landing in mesh-catalog"); reach the mesh only through the mesh MCP server, never ssh;
never push, open, merge or close a pull request; never write to a store's database by hand, a schema change
is a numbered migration and the consolidated schema; Go only unless the context file says otherwise; never
move another issue. Report back in under 150 words: commits, tests added and that they failed first, the
check results, anything you stopped on.
- **An agent that finds the code differs from its context file stops and reports** (file:line); the lead
settles it, updates the context file, and continues the agent.
- **Two repositories that depend on each other** keep their order in the context files: the reader before the
writer, the controller's new manifest word before the catalogue manifest that uses it.
- **Several agents in one repository** (only for a large repository with independent areas): each works on a
local branch of its own from the feature branch's head and lands by fast-forward under a lock
(`flock <feature worktree>.lock git -C <feature worktree> merge --ff-only <its branch>`), after its unit
tests pass; if another landed first, it merges again and repeats. In a shared clone, fetch the feature
branch by name before branching from it. Never resolve a conflict with a blind `git merge-file --union`.
## Phase 2 — Review (one reviewer, larger model; at most 3 rounds, about 15 minutes)
1. **Bring every feature branch up to the current trunk** (fetch by name, then merge), so the reviewed tree is
the tree that is pushed.
2. **One reviewer reads every repository's local commits** together, against the context files and the
decisions. Its report is per repository: each finding as file:line, the failure it causes, and the fix.
It says which round it is in ("round 1 of at most 3").
3. **The report goes to the same fixing agents** that wrote the code, continued by their ids, so they fix
with the context they already hold. Each answers the reviewer: what it changed, per finding, or why not.
4. **The reviewer reads the answers and the new commits**, and approves or reports again. From round 2 it
asks changes only for a finding that makes the feature wrong; smaller points become issues of their own.
After round 3 the lead stops and brings the operator what is left, with a recommendation.
## Phase 3 — Check (no review)
1. **Push every feature branch and open one pull request per repository, together.** Pull requests in two or
more repositories that share the branch name are one delivery group (ADR 0239 §2).
Where a session pushes through the forge tool `gitea_commit` (one commit on the forge's current base), the
reviewer first confirms that commit's tree equals the reviewed one.
2. **The forge's merge checks run the integration tests once** (`mesh/merge-gate`, `mesh/repo-check`). No
second review: phase 2 reviewed this tree.
3. **A check that fails** goes back to the reviewer with the failure, for **one more cycle** of phase 2 with
the same fixing agents; then push once again.
4. **Mark each pull request ready** (`mesh/ready` and a comment `ready at <full sha>`) once the operator's word
for the feature is given; mesh-delivery merges it when its checks pass (ADR 0364).
**One exception: a catalogue change that uses a new manifest word.** The catalogue's merge check reads
manifests with the controller the mesh runs, so the controller's pull request is merged and delivered first;
then `mesh-delivery.recheck` the catalogue's and merge it on the new verdict. Plan such a feature as two steps.
## Phase 4 — Deliver
The merge starts the delivery plan: build, publish, send. The lead follows it to done
(`mesh-controller.plans` with `wait` and `until: done`), then resolves the feature's issues with `fixed-by`
naming its pull requests, and removes what is left: no `feat/<slug>` branch and no `.work/<slug>` worktree
survive.
## Models
| Who | Model | Why |
|---|---|---|
| The lead (phase 0 decisions, the context files, following and reporting) | larger | judgement |
| Read-only questions about the code for the context files | smaller | reading, answered with file:line |
| Fixing agents (phase 1, and the fixing in phases 2 and 3) | smaller | building to a written context |
| The reviewer (phase 2, and the extra cycle in phase 3) | larger | judgement, once per round |
| Decision records | larger | judgement |
Every agent the mesh runs draws on one weekly allowance shared by every session, and the larger model
spends it many times faster. Up to five agents on the smaller model at once, at most two subagents on the
larger one (the lead not counted).
## Following and reporting
Most lost time is waiting nobody saw. The lead therefore keeps a **wait list**: everything the feature waits
on, with when it started and its bound.
| Waiting on | Bound | Past the bound |
|---|---|---|
| a fixing agent's issue | 30 minutes | ask it where it stands; split or take over |
| the review | 15 minutes a round | ask the reviewer; stop at round 3 |
| a merge check | 15 minutes | read the build queue (`mesh-controller.queue`) and the check's log |
| a merge after ready | 5 minutes after the checks pass | read `mesh-delivery.show` for the delivery |
| a delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
**Every ten minutes, and at once when something lands, stops or needs the operator,** the lead reports, in
one of the two layouts below. Times are local and absolute; every number has its unit and its bound or
baseline; an issue is named by its number and a few words, never by its number alone.
**The full layout** (in the session), the issues as a tree: grouped by repository, each issue under the one it
waits for, marked ✓ landed, ▶ fixing, ⏸ waiting, ✗ stopped:
```
Feature <slug> · step <n> of <m> · phase <name>, <minutes used> of <bound> min
Issues: <landed> of <total> landed · <fixing> fixing · <waiting> waiting
mesh-controller (agent: fixing, 2 of 3 landed)
├── ✓ 652 assignments on the bus landed 14:02
├── ▶ 736 placement words leave the layers fixing since 13:51, 23 of 30 min
│ └── ⏸ 751 config keys named (mesh-catalog) waits for 736
└── ✓ 654 a file names its template landed 13:20
mesh-catalog (agent: waiting)
└── ⏸ 751 (above, under 736)
Past its bound: <the wait, how long, what is being done> — or "none"
Needs you: <one line per question, with my recommendation> — or "nothing"
Allowance: week <n> % (<+d> since the last report), session <n> %
Next: <the next action, one line>
```
**The short layout** (on a channel such as Telegram, or when nothing changed but time):
```
<slug> step <n>/<m> · <phase> <used>/<bound> min · <landed>/<total> landed · past bound: <none | what> · needs you: <nothing | what> · week <n> %
```
A report says what changed since the last one first; an unchanged table is not repeated, the short layout
says it instead.
## The context of the lead's own session
At the end of each phase, and whenever the session's context grows large or spread over unrelated work, the
lead prepares a `/clear` as the mesh's instruction "prepare before clear" says: a save point with the
feature's phase, its branches and heads, the context files' paths, the agents' ids, the wait list and the
next action; then the prompt the operator pastes. The context files make the feature resumable after a clear.
## How it is checked
The end state is visible, and its absence is the smell:
- A feature's issues show their commits on the feature branch (`git log --grep 'issue <n>'`), and no pull
request into a feature branch exists. A pull request per issue, or an agent waiting on a merge check in
phase 1, is the failure this playbook prevents.
- A context file exists per touched repository under `.work/<slug>/` while the feature runs.
- After a feature's delivery, `git branch -r | grep feat/<slug>` and `git worktree list` return nothing for it.
- More than one open pull request in a repository that share no feature name, or a feature step whose issues
stay in `fixing` for more than a working day, is the smell: stop and bring it to the operator.
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it cannot find is a
missing sibling worktree, not a mesh fault.
- After a feature merges, `git branch -r | grep feat/<slug>` and `git worktree list` return
nothing for it. A surviving branch or worktree means step 6 was skipped.
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it
cannot find is a missing sibling worktree, not a mesh fault.
- More than one open MR in a repo that share no feature name, or a stack of MRs none of which is
merged, is the failure this playbook exists to prevent — stop and consolidate before opening
more.
When this playbook meets a problem, the lesson is added here, so the next feature avoids it.
+68
View File
@@ -0,0 +1,68 @@
# Playbook 08 — Small fixes
**Trigger.** One issue in one repository, small enough to build in about an hour: a bug with a known cause,
a missing check, a wording, a playbook or design edit. Anything larger, or anything that needs a change in a
second repository first, is a feature and follows [playbook 07](07-feature-branches.md).
**Related issues in one fix.** A small fix may cover two or three related issues when they are in the same
repository, mostly in the same module, and together still fit in about an hour: they touch the same files, or
one would be wrong or unsafe without the other (switching a clipboard's history on before its history tool stops
handing out what was copied). Each issue is named in the commit and the pull request and moved to `fixing`,
and each is resolved on its own once the fix is delivered. More issues, a second repository, or more than an
hour is a feature.
**Who runs it.** Anyone who writes code, engineers and agents alike.
## The aim
A small fix spends its time on the fix, and runs the long part once: **one review, one push, one check run,
one delivery**. Everything before the push is local and fast.
## Steps
1. **Read the issue and the code it names** (the session's own model, or one read-only agent on the smaller
model answering exact questions with file:line). If a design question is open, it is settled before any
code is written: on the issue, through the knowledge base, or by the operator. A fix that turns out to
need a decision or a second repository stops here and becomes a feature.
2. **Move the issue to `fixing`** with `by` as who@node.
3. **Build on a local branch** `fix/<issue>-<slug>`, in a worktree of its own branched from `main`. Write the
test that fails today first; make the change; keep `gofmt`, `go vet` and the repository's unit tests green.
Commit, naming the issue. For a playbook or design edit, run the repository's own checks (in hq,
`sh merge-check.sh`).
4. **One review of the local commit, before anything is pushed** (a reviewer that did not write it, larger
model; at most 3 rounds, as playbook 07; from round 2 only what is wrong). The reviewer's findings go back to the same agent that wrote the fix, continued by
its id, so the fix keeps its context. A playbook or design edit of a few lines may skip the reviewer: the
operator's word on the pull request is its review.
5. **Push the branch, try it, then open the pull request.** `mesh-controller.try` with the repository and the
branch runs the merge check in the check's own toolchain before any status is set; open the pull request
only once it passes. A local run never stands in for it: the check's containers carry other tools than the
nodes (a test that called util-linux `flock -E` passed on a node and failed on BusyBox's in the check). The
forge's merge checks run the integration tests. A failed check
goes back to the same agent with the failure, and the reviewer reads only the change it makes.
6. **Mark it ready** (`mesh/ready` and `ready at <full sha>`) once the operator's word is given; mesh-delivery
merges it when its checks pass.
7. **Follow its delivery plan to done** (`mesh-controller.plans` with `wait` and `until: done`), prove the fix on
the mesh (the symptom gone, measured), and resolve the issue with `located-in` and `fixed-by` the same hour.
## Bounds
| Waiting on | Bound | Past the bound |
|---|---|---|
| building the fix | 1 hour | it is a feature: stop and plan it under playbook 07 |
| the review | 15 minutes | ask the reviewer where it stands |
| the merge check | 15 minutes | read `mesh-controller.queue` and the check's log |
| the merge after ready | 5 minutes after the checks pass | read `mesh-delivery.show` for the delivery |
| the delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
A wait past its bound is said to the operator at once, with what is being done about it.
## Models
The fixing agent runs on the smaller model; the reviewer on the larger. A session fixing alone does the
reading itself and uses one subagent only for the review.
## How it is checked
- A small fix has one pull request with one or few commits naming its issue, opened after its review: a pull
request with many heads is a fix reviewed after its push.
- Its issue moves to `resolved` within the hour its delivery plan ends.
+1 -1
View File
@@ -340,7 +340,7 @@ once, and maintain for a year. The modules are the input; a person reads what on
writes what it declares tomorrow.
**It also changes what "safe" means for the system being retired.** A fix to it has to be safe on
its own, because there is no careful walk to sequence it into: the thing is being switched off
its own, because there is no careful rollout to sequence it into: the thing is being switched off
by hand, not managed into retirement. A change needing three steps in the right order is a change
that will be half-applied.
+110 -1
View File
@@ -22,8 +22,10 @@ code:
- mesh-host internal/apply (the service that reflects a rule set; a whole file handed to its new owner)
- mesh-controller cmd/mesh-controller/nodes.go (a token issued for a machine's tunnel key, ADR 0169)
- mesh-host cmd/mesh-host/join_tunnel.go (a machine joins through the tunnel, ADR 0169)
updated: 2026-10-13
updated: 2026-10-14
decisions:
- "ADR 0241"
- "ADR 0117"
- "ADR 0340"
- "ADR 0247"
- "ADR 0226"
@@ -1119,6 +1121,113 @@ against today's behaviour, where both are always composed; an unrouted endpoint
named; an assignment naming an endpoint the module does not declare refused where it is said; and a
silent assignment rendering byte-identically to today.
## The five through the network layers
*2026-10-09. A reading aid: it decides nothing and states no rule. Each claim names the section or the
record that decided it.*
The OSI model is the standard seven-layer picture of a network: every layer wraps what the layer above
hands it, and the receiving side unwraps in reverse. The five contexts of this document act at
different layers, and seeing which explains why they fail independently. A resolver that stopped
answering leaves the private network working, and a tunnel that cannot handshake leaves every name
resolving to an address nothing can reach.
| Layer | What it carries | Where the mesh acts there |
|---|---|---|
| 7 Application | the message itself | the bus (ADR 0106); resolution (§2); exposure, where the proxy chooses a route by the name a request asks for (§3) |
| 6 Presentation | how what is said is encoded and encrypted | certificates (§5): TLS, which is usually drawn between 4 and 7 and placed here by the book |
| 5 Session | a conversation kept open | a node's link to the bus, opened once and reopened after the machine sleeps |
| 4 Transport | ports, TCP and UDP | filtering (§4), which opens a port to a source; the private network itself travels as UDP |
| 3 Network | addresses and the path between them | the private network (§1): every node's address, the peer graph, the hub relaying |
| 2 Data link | frames on one network segment | the machine's own uplink (ADR 0117), on the underlay |
| 1 Physical | cable, fibre, radio | nothing of the mesh's |
### Two networks, one inside the other
**The private network is a layer 3 network carried at layer 4 of the underlay.** A packet for another
node's private address is encrypted whole and becomes the payload of a UDP datagram addressed to a
peer's underlay address. Every hop on the underlay reads only the outer envelope, up to its layer 3,
and never sees inside.
```
underlay │ L1 radio │ L2 frame │ L3 to the hub's public address │ L4 UDP │ ▓▓ encrypted ▓▓ │
│
decrypted at the hub │
┌────────────────────────────────────────────────┘
▼
private network │ L3 to a private address │ L4 TCP │ L5–L7 a message on the bus │
```
### The shape the layers are drawn on
The shape is §1's: a hub, direct peering between nodes at the same site, and a roaming node that goes
through the hub only. In a small mesh one node is both the anchor and the hub, and holds the bus.
```
┌──────────────────────────────────────────┐
│ hub and anchor — reachable from outside │
│ holds the bus │
└─────▲──────────────────▲─────────────▲───┘
│ │ │
│ each tunnel is dialled out by the node below it
│ │ │
┌─────┴─────┐ ┌──────┴──────┐ ┌────┴────────┐
│ laptop │ │ home-server │ │ home-server │
│ no site: │ └──────▲──────┘ └──────▲──────┘
│ hub only │ └──── direct ───┘
└───────────┘ same site, peered
```
**Every arrow points at the hub.** A node with no public address still has a tunnel, because it dials
out and the hub answers; only the hub's tunnel must be dialable, at a stable address
(ADR 0169).
### One tool call, from the laptop to a home-server
The laptop wraps the call one layer at a time:
| Layer | On the laptop |
|---|---|
| 7 | a message on the bus naming the tool and its arguments |
| 6 | the message encoded for the wire |
| 5 | the laptop's link to the bus, already open |
| 4 | TCP to the bus at its private address |
| 3 | a packet from the laptop's private address to the bus's |
| 3 inside 4 | the tunnel: that packet encrypted, in a UDP datagram to the hub's public address |
| 2 | a frame to the gateway of whatever network the laptop is on |
| 1 | radio |
**At the hub the outer envelope comes off** — frame, packet, datagram — and the tunnel decrypts what was
inside. The inner packet is addressed to the bus, which is on this node, so the hub keeps unwrapping up
to layer 7. The bus finds the home-server's subscription and sends the message down the home-server's
own link, which that node opened earlier through its own tunnel. Neither spoke ever accepted a
connection; both dialled the hub.
**How far the hub unwraps depends on what the packet is for.** A packet addressed to a home-server's
private address, not to the bus, is opened only to layer 3: the hub relays it, encrypted again for the
home-server's tunnel. That is the whole of the glossary's *a hub relays for nodes that cannot reach
each other directly*. Two home-servers at one site skip the hub: their tunnel crosses the home network
at layer 2, and nothing of theirs reaches the hub at all.
**Why a roaming node has one path is a layer 3 fact.** A more specific route to an endpoint that is
gone takes the packets and drops them; it does not fall back (§1). A laptop that moves between networks
therefore has exactly one route, the hub's.
### Reading a failure by layer
The machine's own network checks
(ADR 0241)
each look at one layer:
| Check | Layer | Failing alone means |
|---|---|---|
| **route** — a default route | 3, underlay | the machine has no way out; nothing above can work |
| **tunnel** — a handshake with the hub within five minutes | 3 carried at 4 | the underlay works and the private network does not: a key, the hub's address, or a filter on the way |
| **bus** — the link is open | 5 to 7 | the private network works and the bus, or the node's account on it, does not |
| **resolv-conf** and **names** | 7 | addresses work and names do not, as when a VPN client rewrites the machine's resolver file |
Read them from the bottom. A layer that fails explains every failure above it, and never one below it.
## What this removes
The list is worth having in one place, because it is most of the argument:
+3 -3
View File
@@ -54,7 +54,7 @@ rather than two kinds of thing
node** — it is a machine with a program on it that has not been told which mesh it belongs to.
There is no state for *the first node*. That is the point of
ADR 0004: the first node walks the
ADR 0004: the first node rollouts the
same path, in an unusual order.
---
@@ -239,7 +239,7 @@ carries, to the machine it is already on.
**After step 3 the first node is not special in any way**, which is the property `adopt.sh` and
the bootstrap script never had. Its specialness lasted two commands.
**And enrolment is exercised on node one.** The path every other node depends on is walked
**And enrolment is tried on node one.** The path every other node depends on is exercised
immediately, against a controller on the same machine, rather than being written and first
used months later on node two.
@@ -524,7 +524,7 @@ remains locally authoritative for *operating*; the copy exists only for this.
The node-engine is delivered like anything else
(ADR 0010), and this is worth
walking through because tier 0 looks like it should be special and is not.
going through step by step because tier 0 looks like it should be special and is not.
```
push to mesh-host
+5 -5
View File
@@ -277,15 +277,15 @@ A transition the table does not hold is refused. Two or more deliveries that sha
together as one future state of the mesh.
**This answers the first open question above.** A fit artifact does not declare itself. Off the trunk it is
only checked. On the trunk it is published, and its walk across the machines starts when its delivery says
only checked. On the trunk it is published, and its rollout across the machines starts when its delivery says
so: by itself for the core, which cannot wait for anything to be repaired, and by `mesh-delivery` for
everything else while that module is held. A person can always deliver by hand.
The controller keeps the comparison, the planner, the gate, sending and rollback. `mesh-delivery` keeps the
record and the order, and asks.
**Merges are walked in batches** (2026-10-10, by ADR 0276). The controller collects
**Merges are rolled out in batches** (2026-10-10, by ADR 0276). The controller collects
merges in a rolling merge window, 90 seconds by default and at most 10 minutes, and plans each batch once, as
one walk with at most one commit per repository: the latest of its branch, which contains the earlier ones.
One walk is open at a time, and a walk that started is never superseded. `plans` shows a batch while it
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's walk.
one rollout with at most one commit per repository: the latest of its branch, which contains the earlier ones.
One rollout is open at a time, and a rollout that started is never superseded. `plans` shows a batch while it
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's rollout.
+1 -1
View File
@@ -215,7 +215,7 @@ and recreate a container naming a step after the step ran.
*Written 2026-10-02, from ADR 0189
and [issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
Some work cannot be done underneath a running service: an artifact store's collector walks the
Some work cannot be done underneath a running service: an artifact store's collector rollouts the
storage and requires every writer stopped. A `run-once` step runs *beside* containers and a
scheduled one is the same container again, so until this a module had no way to say it — and the
mesh inherited a store that has never collected anything, because the predecessor said it with a
+1 -1
View File
@@ -163,7 +163,7 @@ again restarts nothing. The provider asks when it starts, when the software's va
consumer that binds in between holds a value the mesh made, which the software refuses, for at most the hour until the next ask. A change is recorded in the
hand-act log as a decision, not a repair, and `rotate` of such a provision is refused. *How it is checked:*
the tests ADR 0305 §13 names, in both repositories; the live row is `hand-acts` showing `secret issued` for
the media server's provision once both walks are done.
the media server's provision once both rollouts are done.
## Beyond generate and hold
+1 -1
View File
@@ -75,7 +75,7 @@ and uses, and the subjects, streams, consumers and permissions all follow from t
(ADR 0039).
**Added 2026-10-10** (ADR 0289): the controller's own records join
the bus as **record mirrors** — nodes, assignments, walks, builds and settings, each a bucket written from
the bus as **record mirrors** — nodes, assignments, rollouts, builds and settings, each a bucket written from
the controller's store only when the record changed, with one event per write, and audited against the
store on every self-check run. Designed in [to-be 54](54-the-controllers-records-on-the-bus.md). The
rule it states for every bucket, that a write is made only when what the bucket describes changed, holds
+1 -1
View File
@@ -709,7 +709,7 @@ healthy while reacting to nothing.
Found on the way: **no build had ever recorded what it stood on.** A recipe reads its base from
a build argument, so the digest was never in the file the builder derived edges from, and every
order that says *bases first* — `build --on`, `build --behind`, the merge follow-up of
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — walked a
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — traversed a
graph with no edges. The builder now reports the bases it was handed, the controller records
them by artifact path, and the graph is read from the newest build of each module — a recorded
manifest carries no `build.on`, so the edge is derived from the build or it does not exist.
@@ -141,7 +141,7 @@ Revision, ADR 0218. Three rules on how a plan delivers what it built.
A membership that could not be issued fails the send.
- **One machine first.** Unless a module's upgrade policy says *together*, a plan sends it to one machine,
the first by name, and to the rest only once that machine reports the new declaration applied and
current. A first machine that fails stops the module's walk there, with the reason in the plan.
current. A first machine that fails stops the module's rollout there, with the reason in the plan.
- **A newer merge takes over.** A merge's plan supersedes every older open plan for the same repository
and branch, and takes in the modules they had not yet built. A plan that waits on nothing can be closed
by hand, by its id.
@@ -182,7 +182,7 @@ policy or a plan is holding back
builds are not known is left too. The push names the machine, the module, the builds and the reason,
and says `push <node>` sends it.
- The machine a push names, a push of every machine, and `push --behind` send held builds as before.
Walking a change through the mesh under `record` is one `push <node>` per machine.
Rolling out a change through the mesh under `record` is one `push <node>` per machine.
## Why now, and why not yet
@@ -253,7 +253,7 @@ the controller creates from the catalogue, mesh-wide, existing from registration
watch before the owner runs anywhere ([design 25](25-the-bus-on-nats.md) §3). Every instance of the
owner writes; a reader reads and watches. A key may name a machine by the module's own convention;
the mesh keeps one bucket per name, not one per machine, because "every server, for every machine"
is then one list rather than a walk.
is then one list rather than a rollout.
What a module sees is what it named. Its assignment's membership lists its buckets by those names,
with whether it may write, and the runtime answers `get`, `put`, `delete`, `keys` and `watch` for
@@ -338,7 +338,7 @@ module, version, digest. An event again: a fact about the builder.
**The build cascade is that event fanning out through a graph the mesh already has.** A module
whose image is built *on* another's artifact declares that in `build.on`. So `built` reaches the
controller, which walks the declared graph and submits rebuild jobs for everything downstream. A
controller, which traverses the declared graph and submits rebuild jobs for everything downstream. A
dependency cascade is not special machinery — it is one event, one derived graph, and the same job
queue.
@@ -351,7 +351,7 @@ entrypoint that brings its state to the shape that version needs — the same vo
tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it
runs the module's own code, to completion, in the module's own context, and a version whose preparation
did not succeed does not run: the step gates that module and nothing else on the machine
(ADR 0136), and the walk stops
(ADR 0136), and the rollout stops
at the first machine that did not take it
(ADR 0135, superseding
ADR 0133).
@@ -2,8 +2,9 @@
layer: to-be
status: in-progress
code: [mesh-catalog, mesh-controller, mesh-host]
updated: 2026-10-12
updated: 2026-10-14
decisions:
- "ADR 0369"
- "ADR 0303"
- "ADR 0215"
- "ADR 0173"
@@ -75,7 +76,7 @@ In order:
| 8 | `docker` | the runtime's packages, base configuration, group | four configurations, one owner on one machine |
| 9 | `ssh-client` | everything under `~/.ssh` (research 027/03) | a predecessor's entries winning over the mesh's; stale keys |
| 10 | scripts | the operator's own scripts, shared and per role (research 027/03) | under no version control, copied by hand |
| 11 | `kernel` | kernel, microcode, boot entries | two machines without microcode |
| 11 | `kernel` | kernel, microcode, boot entries, and the kernel's run-time settings per machine (transparent huge pages first) | two machines without microcode; huge pages left at the distribution's `always` on a machine whose slow disks they flood (issue 746) |
`systemd`, `pacman` and `docker` hold the three seats that apply resources
(ADR 0207):
@@ -83,7 +84,19 @@ every module that declares a service, a package or a container depends on them b
They go on every machine before the rest, and once they have, an unmet dependency is refused rather
than reported.
`kernel` is last because a mistake in it costs a boot. `docker` holds the runtime seat
`kernel` is last because a mistake in it costs a boot. Its run-time settings are the exception and come
first (ADR 0369, issue 746): they change nothing at boot that the kernel does not already accept at run time, and
each is a per-machine setting whose default is what the kernel boots with, so assigning the module changes
no machine until that machine is given another value. The first is transparent huge pages. On the
control-node, with `always`, the journal's memory-mapped files were held in 2 MiB pages, so each small journal
write cost a 2 MiB write-back; the journal daemon and a log-reading intrusion-prevention service wrote
gigabytes in minutes onto a mirrored pair of spinning disks that also holds the bus's store, and the
controller lost its lease. That machine is set to `madvise`. The module writes the mode at boot and, on a
change, the new value directly; it reads it back and fails its unit if it did not take, gives back the mode
it found when unassigned, and on a running machine rotates the journal once so the open journal files are
mapped anew in small pages. How it is checked: the module's tests run its script against
a stand-in directory, its unit's health is the unit's state, and its tool `kernel_huge_pages` shows the
mode in force and the file pages each unit holds in huge pages. `docker` holds the runtime seat
(ADR 0166), with its settings declared as
ADR 0164 decides and the capability detected as
ADR 0165 decides; what of the three is not built yet is tracked by issue 398.
@@ -53,9 +53,9 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214).
*Added 2026-10-06, ADR 0233.*
The mesh composes a second file for the holder: every data item every module on the machine declares,
with its class, its path, the path that covers it and its protection. Once an hour and after every
night the holder measures each item that is not a cache, by the item's `measure`: a bounded walk at
night the holder measures each item that is not a cache, by the item's `measure`: a bounded rollout at
most daily for a small item (stopped after ten minutes or two million files and said as partial), a ZFS
dataset's own counters hourly, or the top-level entries only — never an hourly walk of anything large —
dataset's own counters hourly, or the top-level entries only — never an hourly rollout of anything large —
and reads the redundant storage it is on (a ZFS pool's health, an md array's
members, a btrfs filesystem's error counters). `backed-up` says each item with that measurement and its
newest good backup. The holder judges nothing: the controller's self-check (to-be 45, D13) keeps the
@@ -2,8 +2,9 @@
layer: to-be
status: in-progress
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
updated: 2026-10-12
updated: 2026-10-14
decisions:
- "ADR 0357"
- "ADR 0325"
- "ADR 0323"
- "ADR 0318"
@@ -100,7 +101,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
| the record mirrors: nodes, assignments, walks, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
| the record mirrors: nodes, assignments, rollouts, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) |
A bus subject two components may publish on is refused when the controller composes grants, unless
@@ -175,7 +176,7 @@ process of its own reads them back from the store.
*Added 2026-10-08, ADR 0253:*
**a condition says itself to the operator in plain words, beside its summary, and says whether the operator
is needed.** The headline, needs, explanation, resolved line and actions are made where the condition is
made: by the producer when it knows more than the subject (a walk's modules, a failed unit's name, a
made: by the producer when it knows more than the subject (a rollout's modules, a failed unit's name, a
hand-act's cause), otherwise by the wording registered for the kind, otherwise from the scope, which is
reported. The explanation opens with its verdict. All of it is held to the plain rule when the controller
takes an observation: no hash, numbered id, dotted name, duration in code, clock time, flag, markup or line
@@ -259,25 +260,39 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — |
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted; every user's connection the server closed as a slow consumer or for breaking the bus's rules (ADR 0357) | bus server's advisory subjects; the controller's connection; the bus module's watch of its server (`bus.slow-consumer`, `bus.connection-closed`) | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `connection-closed` (one per bus user), `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
| S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — |
| S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | `facts-stale` | warning | — |
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — |
| S16 | a walk waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose walk waits | 30 min; urgent after 4 h | `waiting` (names the walk and `plans go`) | warning | — |
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a walk out | warning | — |
| S18 | a batch of merges is cut into its walk (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no walk is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
| S19 | a batch waiting behind an open walk is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a walk is open | that walk's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the walk) | warning | — |
| S16 | a rollout waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose rollout waits | 30 min; urgent after 4 h | `waiting` (names the rollout and `plans go`) | warning | — |
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a rollout out | warning | — |
| S18 | a batch of merges is cut into its rollout (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no rollout is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
| S19 | a batch waiting behind an open rollout is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a rollout is open | that rollout's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the rollout) | warning | — |
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
controller translates each into a condition naming the thing in the mesh's words, as the refused reply
of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer
is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a
system account the controller hears a slow consumer and a refused subject for its own connection only
([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md),
open question 2).
is said only for one the mesh names and expects, and clears when it exists again.
**Every user's slow consumer, from the bus module (ADR 0357).** The server closes every client it judges
a slow consumer and keeps its user and reason in its record of closed connections; it publishes neither
in the mesh's account. So the `nats` module's tool bundle reads that record on its own machine every 30
seconds and publishes `bus.slow-consumer` for each connection closed as a slow consumer, and
`bus.connection-closed` for each the server closed for something its client did against the bus's
rules or limits — naming the bus user, the connection and the reason, never a credential. A closure by
the client, by the network or by the server's own shutdown is not said: D15 says a user dropped in a
loop. A closure — its user, connection id and opening time — is said at least once: again when publishing
it failed, and again for the last hour when the bundle starts. The controller's watchdogs read the last
hour of both events back from the events stream on every tick, count each closure once, and raise one
condition per bus user and kind, naming the module and its machine; it clears an hour after the newest
closure. A closure is a condition at most about a minute after the server closed it.
**A refused subject is still heard for the controller's own connection only.** The server tells the
client and writes its log, and says it nowhere else — not in its monitoring endpoint and not to a system
account (issue 702, open).
## 4. The self-check: `doctor` (rule 6)
@@ -525,7 +540,7 @@ except an act recorded by a verb that is a person's decision by design, which is
counts, whatever cause it gives. The controller keeps one table of the verbs that write the log, and
each says whether it records a repair or a decision, and why: approving or rejecting a retirement and
deleting what was retired (ADR 0230); `bus upgrade`, because the bus is never rolled by the mesh, and
`upgrade release-backlog`, because after a failed walk the next opens only on a person's word
`upgrade release-backlog`, because after a failed rollout the next opens only on a person's word
(ADR 0236); and `secret rotate` when its cause is a leak (`leaked-in-logs`) — a person judges what was
disclosed, one leak rotates several values, and a leak that recurs is a defect of the module that
prints them, an issue against it, not a healer that rotates. A rotation for any other cause counts. A
@@ -604,7 +619,7 @@ the other machines follow. "Reported applied" is not enough.
**As built** (ADR 0236):
- **Every walk is gated, not only the core's.** A module is judged by its own health: it reported
- **Every rollout is gated, not only the core's.** A module is judged by its own health: it reported
applied, no witness put it back, no condition was raised since its send about the machine or about the
module there, and its tools are served on that machine where it has tools. Three healthy judgings at
least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of
@@ -627,7 +642,7 @@ the other machines follow. "Reported applied" is not enough.
- **No build reaches a machine without a gate**: a gated send carries and judges everything waiting on
its machine; every other send — a plan's rest, a cascade, a healer's, a whole-mesh push — is refused
or leaves the machine while a build no gate has seen waits there; a rebuild with the same artifacts and
manifest is no move. What waits is moved by a **walk**, one machine at a time, the control
manifest is no move. What waits is moved by a **rollout**, one machine at a time, the control
node last, each judged; one that fails holds the next until `upgrade release-backlog --why`.
- **A wait for a person passes, and each module of a send keeps its own pass**
(ADR 0254): a module whose only fault is a new login the operator owes
@@ -787,10 +802,10 @@ has a week of entries; the durations are recorded for every machine.
| Repository | Delivers |
|---|---|
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9), and every user's slow consumer and closed connection read back from the bus module's events (ADR 0357); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
| `mesh-host` | the heartbeat carries its interval; D1's validator published as a package the controller imports |
| mesh-tools | the tool runner's heartbeat (S11) |
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) |
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5); the `nats` module's watch of its server's closed connections, publishing `bus.slow-consumer` and `bus.connection-closed` (S9, ADR 0357) |
| mesh-lab | R9: each signal suppressed in turn |
**Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a
@@ -923,7 +938,7 @@ and "the commit is the build at hand"), on branches, not yet merged:
| hq | its own `merge-check.sh`, running `records.py`, `index.py` and `cycle.py` |
**Not yet:** the delivery plan kept (id, inputs, result) and the release comparing its plan with it; the
state machine replacing the walks' states, with its table, its test, its conditions and healer H2;
state machine replacing the rollouts' states, with its table, its test, its conditions and healer H2;
the status's link to the kept plan; the notes under `refs/notes/mesh-plan`; check builds in a scratch
namespace (the check builds no module today, so there is none to keep apart yet); the statuses made
required — prepared as calls of the forge module's tool, applied by the operator.
@@ -327,36 +327,36 @@ Presence facts are events on the bus. The router keeps them as **current state o
machine and per intake kind, overwritten — never as a history, and never in a message's words. A
consumer that wants a timeline of the operator's day is a decision of its own.
### While a delivery walks
### While a delivery rollouts
*2026-10-12, ADR 0308; the build is tracked by issue 532.*
A delivery restarts what it moves, and those restarts raise conditions that clear once it is done; a router
that restarted used to say again what it had been told to keep quiet (issue 531). So the router holds what a
walk touches and tells the walk's result once.
rollout touches and tells the rollout's result once.
- **A walk names what it touches.** The controller's walk events say, when a walk starts and again when it
- **A rollout names what it touches.** The controller's rollout events say, when a rollout starts and again when it
ends with its outcome (delivered, failed or stopped), the modules it moves and the machines it sends them to.
What it touches is those modules on those machines, and every module a moved tool runner or node-engine
carries there.
- **The router holds what an open walk touches.** A message whose subject is a touched module or machine is
- **The router holds what an open rollout touches.** A message whose subject is a touched module or machine is
held instead of sent, kept with when it was held, and `history` says it was held. A message several open
walks touch is held until the last of them ends or its bound passes.
rollouts touch is held until the last of them ends or its bound passes.
- **Some messages are never held:** an ask (§6), a secret given or replaced, a message about anything no open
walk touches, and a walk's own failure.
- **A walk's end is told once:** one message with its outcome, the modules and machines it reached and how
rollout touches, and a rollout's own failure.
- **A rollout's end is told once:** one message with its outcome, the modules and machines it reached and how
long it took, then each held message whose condition is still open, one line each. One whose condition
cleared during the walk is only counted. A walk delivered with nothing held and nothing open is one line, or
cleared during the rollout is only counted. A rollout delivered with nothing held and nothing open is one line, or
nothing on a channel set to say only what needs the operator.
- **A hold has a bound:** twice the largest delivery budget in force among the walk's deliveries
- **A hold has a bound:** twice the largest delivery budget in force among the rollout's deliveries
([to-be 47](47-delivery-from-commit-to-delivered.md)): 20 minutes for a core module, 10 for leaf modules at
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a walk none of whose
deliveries has a budget. Past it, every held message still open is sent with a line saying the walk still
runs and since when, and the cleared ones are counted. A walk that hangs never silences the mesh.
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a rollout none of whose
deliveries has a budget. Past it, every held message still open is sent with a line saying the rollout still
runs and since when, and the cleared ones are counted. A rollout that hangs never silences the mesh.
- **A restart says nothing it has said:** a router that restarts says again only what is open and was never
answered, never a silenced message or one withheld for its content.
*How it is checked:* the controller's tests of the walk events, and the router's tests of holding, never
*How it is checked:* the controller's tests of the rollout events, and the router's tests of holding, never
holding, the end message, the bound and the restart, as ADR 0308 lists them; live, the next delivery of the
tool runner gives the operator one message at its end, and the channel's `history` shows what was held and
released.
@@ -467,13 +467,13 @@ the factors are the router's state.
### An approved settings change takes effect (ADR 0344)
*2026-10-13, issue 641.* Approving a settings proposal is the act that makes it take effect. The
serving controller sets the layer and, in the same act, opens a **settings walk**: every machine
serving controller sets the layer and, in the same act, opens a **settings rollout**: every machine
that runs the module and that the layer reaches, the control node last, each sent and its module
judged at the first-node gate before the next. A settings walk sends the layer and no build: every
judged at the first-node gate before the next. A settings rollout sends the layer and no build: every
module on the machine stays at the build it was last sent, so a build held for a person, or waiting
for a first-node gate, stays where it is. A send that would move a running module's data stops the
walk and names the `push --move` that sends it. The walk says its start and end like every walk, so
its end is told as every walk's is. A failure stops the walk and leaves the layer as approved; the
rollout and names the `push --move` that sends it. The rollout says its start and end like every rollout, so
its end is told as every rollout's is. A failure stops the rollout and leaves the layer as approved; the
previous layer is in the settings history. The ask says all this before the Approve, and no longer
names a push. A layer set by hand still waits for a push.
@@ -481,9 +481,9 @@ A delivered file is not yet an applied one: a module whose program reads its set
starts applies them itself (a service's `restart-on`, or a tool of its own). Betterbird is such a
module; issue 643 makes it restart Betterbird at a quiet moment.
How it is checked: the controller's tests open the walk on the warrant, send one machine at a time
How it is checked: the controller's tests open the rollout on the warrant, send one machine at a time
with every build kept and the module judged, stop before a data move and on a failure without
holding other walks back, and read the ask's words.
holding other rollouts back, and read the ask's words.
### Proofs and tiers
@@ -30,9 +30,9 @@ delivery group is two or more deliveries that share a head branch name, delivere
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
(ADR 0239.)
**From its merge on, a delivery is delivered by the walk of its batch.** The controller collects merges in a
rolling merge window and plans each batch once, as one walk with at most one commit per repository, one walk
at a time. Every delivery the walk answers ends with it
**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a
rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout
at a time. Every delivery the rollout answers ends with it
(ADR 0276).
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
@@ -53,7 +53,7 @@ commit's journey and in what order the steps are allowed.
- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants,
registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk
rule, a gated send, the rollback and the walk of one trunk commit's tiers across machines. It still answers
rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers
every pull request's head with its check itself, so a check never depends on mesh-delivery.
- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the
group, the state table and their state, and nothing else.
@@ -61,9 +61,9 @@ commit's journey and in what order the steps are allowed.
request, and appends to the commit's note under `refs/notes/mesh-plan`.
- **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350):
a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a
reuse key (the source fingerprint and the builder's version), and the walk of the merge commit copies a build
of the same key, which the check's verdict names, by digest instead of building. Otherwise the walk builds and
says why, per module, in its record. Check builds go when the walk answering the merge ends, when the pull
reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build
of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and
says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull
request closes unmerged, and after seven days.
## A delivery
@@ -76,7 +76,7 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
**machine steps** while delivering;
- once merged, **the commit it landed on the trunk as**, and the walk that delivers it: the walk of the batch
- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
window and the batch*).
@@ -93,9 +93,9 @@ allows.
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
| ready | it passed; it waits to be merged | none | — |
| rejected | it failed its check | none | — |
| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a walk that does not answer its merge took over its walk; `go` when its walk started |
| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started |
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
| delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says |
| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says |
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
**A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head
@@ -105,7 +105,7 @@ commits in the transition's reason; a `rejected` one stays `rejected`, its statu
of date. Nothing is asked for it: a recheck, a new head or the merge tool asks the check again. The row from
`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for
an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own walk.
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout.
What this leaves as it is, said so nobody reads more into it:
@@ -125,7 +125,7 @@ What this leaves as it is, said so nobody reads more into it:
The transitions are those of ADR 0239 decision 2, ADR 0270 and ADR 0323. A delivery that is
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
delivery merged with no walk answering it within ten minutes is held too, with the true reason, and listed by
delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by
`stalled` at once. Nothing waits silently.
**What needs no person ends by itself** (ADR 0270):
@@ -133,37 +133,37 @@ delivery merged with no walk answering it within ten minutes is held too, with t
- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and
sends nowhere) is `delivered` as soon as it is merged, whatever its check said;
- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine
running it by the walk of a later merge into the same trunk, delivered on its own passing check, and the
running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the
forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge
order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery`
says `covered: a later merge that contains it was delivered` as a success. A module that runs on one
machine only is covered too: its walk records the empty rest as sent once its first machine passes
machine only is covered too: its rollout records the empty rest as sent once its first machine passes
(ADR 0270, as corrected 2026-10-12);
- *a merge whose walk said its modules unchanged* moves nothing too, and is `delivered` when that walk ends,
whichever commit the walk carried (issue 611). A delivery plan made while another walk built a module reads
that module whole and names it; the walk, cut later, reads the build source that build recorded (ADR 0267) and
may find none of the changed files in it. The walk's record then says so in `delivery.unchanged`: each such
- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends,
whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads
that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and
may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such
module with why, "its build source holds none of the changed files", and not in its modules, which keep
meaning what the walk builds. A delivery whose every module the walk built or said unchanged is delivered,
saying which matched what runs. One with a module the walk neither built nor said unchanged is read as below.
meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered,
saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below.
Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's
`TestAMergeWhoseModulesMatchWhatRunsIsDelivered`;
- *a stop whose walk ended without building its modules* (issue 474) stays `stopped`, a final state, and
- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and
is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after
its stop` with the covering merge, once, as a success. No state or transition is added (issue 480);
- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes.
**History is retired, not held** (ADR 0250).
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no walk
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout
of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes
to `retired` at adoption, before any check, hold or walk, and its pull request is not told. A held delivery that
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its walk
to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout
starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a
build registered before its turn would be carried by somebody else's send.
**The machine steps.** While a delivery is `delivering`, each machine its walk reaches has a row with the
**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the
module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy
readings so far), `passed`, `failed`, `rolled-back`. They are read from the walk's own record in the
readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the
controller's `plan-moved` event, never inferred from time.
## The merge window and the batch
@@ -174,95 +174,95 @@ controller's `plan-moved` event, never inferred from time.
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
merges and its times are kept in the controller's store and survive a restart.
- **The cut.** When the window closes and no walk is open, the batch becomes one walk: one delivery plan for
- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for
every commit in it, tiered along the module graph across repositories, with at most one commit per
repository, the latest of its branch. The walk names every merge it answers: its own commits, and the
repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the
earlier merges of a repository that its later commit contains.
- **One walk at a time.** Merges that arrive while a walk is open join the next batch. A walk that started
is never superseded. A walk still waiting for its word when the next batch is cut is folded into the new
walk, which answers every merge the folded one answered.
- **The controller's own path.** A merge that moves a module whose walk waits for nobody's word (the controller,
- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started
is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new
rollout, which answers every merge the folded one answered.
- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller,
the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for
mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its
walk like any other. An own-path walk folds no waiting catalogue walk; that walk's merges are batched again
rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again
behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10).
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a walk is
answered by that walk: the assembling batch, the open walk, or, when that walk was delivered, it ends
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is
answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends
`covered` at once.
- **The ends.** A walk done ends every delivery it answers: `delivered` for its own commits, `covered` for
an earlier merge of a repository it carried later, naming that merge. A walk that failed fails its own
commits' deliveries and marks none delivered; each earlier merge it answered is then walked on its own
commit, the newest first, one walk each, until one is delivered. A walk stopped stops every delivery whose
merge it walks again nowhere (below).
- **A group inside a walk.** The pairs of a delivery group's order (ADR 0249) are edges of the walk's
tiers beside the module graph. mesh-delivery gives a walk its word only when every delivery it carries is
- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for
an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own
commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own
commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose
merge it rollouts again nowhere (below).
- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's
tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is
at its turn.
- **The bookkeeping.** mesh-delivery ties a walk to every live delivery whose merge it answers, follows a
folded walk to the walk that took it over, and never ties a walk to a delivery that has ended. A waiting
walk no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a
folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting
rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
and a batch waiting behind an open walk names that walk. The controller says the batch on the bus as
and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
delivery it holds.
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no walk open, and a batch
waiting behind an open walk past that walk's bound, each raise a condition. A merged delivery that no walk
answers within ten minutes is held with the true reason (the open walks of its repository and that none
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch
waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout
answers within ten minutes is held with the true reason (the open rollouts of its repository and that none
names its merge, or that the controller holds none) and listed by `stalled` at once.
## A stop on a shared walk
## A stop on a shared rollout
*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.*
One walk carries every merge of its batch, so ending a walk for one delivery ends it for every other
One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other
delivery on it. A stop therefore ends only the change it is for:
- **A group's stop.** When a member after a failed one is stopped and its walk is open, mesh-delivery asks
`delivery-stop` for that walk with the stopped members' merges to leave out, every member the same pass
stops on that walk in one ask. The controller ends the walk, said as stopped, and walks every other merge
- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks
`delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass
stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge
it answered again.
- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A
module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too.
Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped
merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is
walked again.
- **Where the others go.** They join the open batch of their kind, where merges heard while the walk ran
already wait, so an older commit never walks after a newer one; only with none open do they form a new
rolled out again.
- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran
already wait, so an older commit never rollouts after a newer one; only with none open do they form a new
batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop
before it is cut, are cut without the rest of their window. Its walk withholds the left-out merges' modules
before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules
where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes
to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every
merge moved are written in one transaction.
- **The walk's record.** The stopped walk names the batch that walks its merges again, every merge it left
out, and each module it already sent that nothing walks again, with the machines it reached. A delivery
whose merge is walked again goes from `delivering` back to `published` (the row `rewalk`), or stays
`published`, and follows the fresh walk once it is cut. A delivery whose merge was left out, and every
delivery on a walk stopped whole, ends `stopped`, naming the walk and the stop.
- **A stopped merge never ships by a walk.** mesh-delivery asks every open walk or batch that names a stopped
delivery's merge to leave it out, and never gives such a walk its word. When the controller refuses, it
stops the whole walk.
- **A person's stop.** `stop` of a delivery whose open walk carries other deliveries does nothing at first: it
- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left
out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery
whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays
`published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every
delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop.
- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped
delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it
stops the whole rollout.
- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it
answers with each of them, which of them contain this merge and end either way, and two ways given with
`way`: `again` (the walk ended and the others walked again without this merge) or `walk` (the whole walk
stopped, every delivery on it with it). A delivery alone on its walk is stopped at once. A delivery is
marked stopped only once its walk is; a stop the controller refuses leaves it as it was.
- **What it does not do.** What the stopped walk already sent stays on the machines it reached, and its
`way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout
stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is
marked stopped only once its rollout is; a stop the controller refuses leaves it as it was.
- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its
asked builds still register, so a later send may carry them further; the stop names the machines, and
putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the
stopped merge's modules, contains the stopped change and delivers it when walked; the stop names each it
stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it
can see, and holding a merged change back for good is a revert's job.
## The delivery budget
*Decided 2026-10-10 by ADR 0282.
Where it differs, this section replaces the window for every merge and the one walk at a time above, slice by
Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by
slice as each slice below is built; until a slice is built, the text above is what runs.*
**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard
from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes;
a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is
shown and held to no budget. Measured on 51 walks on 2026-10-09 and 10, a leaf walk took a median of 6.0 minutes
(7 of 20 within five), a core walk of one tier 5.6 (worst 8.8), and a controller walk dragging two catalogue tiers
shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes
(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers
14.2 (issue 338).
*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget
@@ -273,56 +273,56 @@ phases — implementing, review, checks, and merge to running (the delivery time
budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget.
The delivery budget stays as written: a measure inside the fix budget, not replaced by it.
**What changes in the walk.**
**What changes in the rollout.**
- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged.
- **Independent walks run at once**: no shared repository, module or module-graph edge with any open walk. One
node judges one walk at a time; a send to a node under another walk's judgement waits for its verdict; no leaf
- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One
node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf
build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged.
- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf
module with a long-running resource; one for a leaf module whose every resource is a tools bundle.
- **A tier's first nodes are judged at once.**
- **The bus's grants arrive before the gate** (ADR 0298): when a walk's builds change the bus's user list (a new
- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new
tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before
anything is judged; a step that would carry anything else is refused unsent and said on the gate line.
- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy
report; the plan shows an expected landing time from the last seven days' medians.
*Walks beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
a change shares).* Building the walks at once found what those bullets left open, and ADR 0306 decides it:
*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it:
- **A walk's builds are its own, never another's backlog.** A send composes a module whose build another open
walk owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
judged only by its own walk, so a failed walk puts back only its own builds. What waits on a node with no open
walk to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
for builds an open walk owns).
- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open
rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open
rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
for builds an open rollout owns).
- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the
merge window whether or not a batch is assembling, so a group's members walk in one batch, in their order.
- **Related means everything the walks would move.** A module in common is any module a walk plans to build or
merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order.
- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or
send, the dependents of what its merges moved included; a change whose modules are not known yet is related to
every change. Two related walks are neither cut beside each other nor judged beside each other on one node.
every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node.
- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a
warning naming the walk and the core walks that set it aside.
warning naming the rollout and the core rollouts that set it aside.
- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an
unhealthy network or engine) fails every judgement open on it, leaf and core, each walk putting back only its
unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its
own builds.
- **A rollback beside another walk's judgement** behaves as it does today; whether it should wait is tracked by
- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by
issue 528.
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of walk A left out
of walk B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
overlapping dependents keeping two walks apart; the warning on the third set-aside; a failed apply failing every
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out
of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every
open judgement on its node.
*A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`,
a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge
into the controller's repository no longer throws away the checks in flight. When a walk's send would replace
into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace
such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once
it says idle, or after 15 minutes at the latest. Only that node's walk waits: the walk says it waits for a busy
it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy
container there, its first-node gate starts at the report that says the container replaced, and the wait is a
phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget
is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a
walk whose module's container waits as busy on its first node says so and starts that node's gate only once the
rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the
node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the
judgement.
@@ -330,11 +330,11 @@ judgement.
| Part | Owner | What it adds |
|---|---|---|
| the walk's phases and readings | mesh-controller | each phase's start and end and each reading in the walk and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` |
| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery.<class>.over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` |
| the lone merge | mesh-controller | the cut at once when nothing waits to join |
| walks at once | mesh-controller | the independence test, one judgement per node, a core walk's ground, a repository's next batch |
| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch |
| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once |
| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` |
@@ -342,11 +342,11 @@ judgement.
| Slice | Delivers | Done when |
|---|---|---|
| 1 — the measure | the walk's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its walk's phases |
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its walk |
| 4 — walks at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
| 5 — the landing time | the expected landing time | `plans` shows it for an open walk, and the week's deliveries land within a minute of it at the median |
| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases |
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout |
| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median |
Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges
only once it is built.
@@ -374,9 +374,9 @@ only once it is built.
- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`.
A member published while its group's composed check is asked and not answered waits there for the answer;
one held because that check had not passed is *lifted* back to `published` once it passes or warns for the
heads that merged and its own walk is still open (ADR 0270).
heads that merged and its own rollout is still open (ADR 0270).
- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its
walk while the walk's other merges are walked again (ADR 0299). The members before it stay delivered.
rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered.
Its own builds were put back at its gate.
A group's state is derived from its members and its composed check, and is shown, never stored apart from
@@ -408,22 +408,22 @@ In this order, each by its owner:
| Verb | Answers or does |
|---|---|
| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group |
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the walk |
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout |
| `groups` | every group with its members in order and its derived state |
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting walk no live delivery holds |
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds |
| `recheck` | a rejected or ready delivery back to proposed, with why |
| `release` | a held delivery to delivering, with why: a person's word |
| `stop` | any delivery not final to stopped, with why; its walk is ended through the controller. On a walk that carries other deliveries it first answers what ending the walk would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise |
| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing |
**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`,
`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which
takes the merges to leave out and walks the walk's other merges again without them, ADR 0299),
takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299),
`delivery-walks`. A person has `plans go <plan> --why`, which does what `deliver` does under their name and
is recorded as a hand act. The controller says every walk it keeps as `plan-moved`, from a verb's command
is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command
too.
**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It
@@ -438,12 +438,12 @@ up.
## The bootstrap
- A walk that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
controller's own, started by the merge and witnessed on the machine. mesh-delivery records it.
- Any other walk waits for `deliver` only while the seat has a holder on record. With none on record, the
- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the
controller starts it as it always did.
- A waiting walk is said by the controller whatever mesh-delivery says of itself: S16 raises
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the walk answers and
- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and
`plans go <walk> --why` as the way on.
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
@@ -453,22 +453,22 @@ up.
merged and rolls out as the core does. With no holder on record it behaves exactly as before.
2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built
and published.
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Walks that start
from then on wait for its word. Walks already open finish on their own and are adopted as deliveries in
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start
from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in
`delivering`.
4. To step back, unassign it. Waiting walks are then started by the controller at its next pass.
4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass.
## Phases
| Phase | Delivers | Done when |
|---|---|---|
| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` |
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a walk never let go is raised by S16 |
| C — the walk itself | the per-tier walk moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 |
| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
## What is not decided here
- A web view of deliveries beyond the pull request's comment.
- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge
module's protection tool.
- Moving the per-tier walk out of the controller (Phase C).
- Moving the per-tier rollout out of the controller (Phase C).
@@ -2,7 +2,7 @@
layer: to-be
status: in-progress
code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog, mesh-sdk]
updated: 2026-10-11
updated: 2026-10-14
decisions:
- "ADR 0292"
- "ADR 0283"
@@ -13,6 +13,7 @@ decisions:
- "ADR 0236"
- "ADR 0227"
- "ADR 0256"
- "ADR 0368"
---
# 48 — A module says how it is healthy
@@ -200,7 +201,7 @@ build alike, so any build of the module is excused while the wait checks out.
**Each module of a send keeps its own pass** (ADR 0254). A send is still judged as one, with one verdict.
The gate also counts, per module, the judgings in a row that found it healthy or waiting for a person;
when the send fails, a module whose count reached the passes asked keeps a pass, said "on its own", and is
neither put back nor left as a move no gate has seen for every other walk to wait on. A module found broken
neither put back nor left as a move no gate has seen for every other rollout to wait on. A module found broken
is marked failed and put back at once, and the send's verdict waits until the modules beside it have their
own, or the bound, so every module of a failed send leaves it with a verdict: its own pass, or put back.
@@ -246,6 +247,11 @@ The runtime restarts what exits, as now. The node-engine never restarts, recreat
being unhealthy. The condition reaches the operator through the conversation (to-be 46), or a healer once a
record gives one that act.
One module repairs itself, by its own record: the dnsmasq module's watch restarts the resolver's unit when
it stops answering, at most once every two minutes and three times an hour, and says through its declared
health whether the restart helped (ADR 0368). The node-engine still restarts nothing; any other restart on
health needs a record of its own.
## 8. A declaration is proved before it is trusted
- **`module check`** refuses a `health` field naming an endpoint the module does not declare under
@@ -520,7 +526,8 @@ controller.
## What is not decided here
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231).
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231). The resolver's own
restart is decided, for that module alone, by ADR 0368.
- Health of scheduled work, and of what a module's events do (issue 276).
- A slow starter that needs more than five minutes to be ready — a change to the bound, recorded, when one
exists.
+1 -1
View File
@@ -31,7 +31,7 @@ shape — what each is for, where its records live, and how they depend on one a
| **Placement** | to decide, send and apply what each node runs, and say why | the controller's `inventory` database |
| **Provisioning** | to resolve which module serves a provision for which consumer, and wire the two | the controller |
| **Identity and access** | to issue, hold, rotate and check credentials and grants | the controller's `identity` and `licences` databases, and the vault |
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and walk |
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and rollout |
| **Health and repair** | to notice what is wrong, say it once, repair what may be repaired unattended, and record what was done by hand | the controller's condition store |
| **Data** | to know every item of data a module holds, how precious it is, and keep it recoverable | the manifests' data declarations, and every node's `node-backup` seat |
| **Connectivity** | to make every node and module reachable by name where it should be, and unreachable where it should not | the controller |
@@ -12,7 +12,7 @@ decisions:
# 54 — The controller's records on the bus
**What each node runs, how far each walk has got, what each build is doing and what each setting holds
**What each node runs, how far each rollout has got, what each build is doing and what each setting holds
can be watched on the bus, and what is watched never says something the controller's store does not**
(ADR 0289,
from gap G2 of research 036).
@@ -51,7 +51,7 @@ do not have: a check that the bucket says what the store says.
Each value is built from a named list of fields. A field not on the list is never in the value. A value
holds only what is built from its record: no epoch, no time of writing and no time of the last look,
because any lease holder must build the same bytes from the same record, and the audit must too. Times
that belong to the record itself, such as when a walk was opened or a resource's state began, are fields
that belong to the record itself, such as when a rollout was opened or a resource's state began, are fields
like any other. The epoch and the time of the write travel in the event.
### nodes — `mesh-controller_nodes`, key: the node's name
@@ -74,15 +74,15 @@ like any other. The epoch and the time of the write travel in the event.
| the build the node was last sent for it and the build its node reports running | — |
| the seats it holds there, and the provisions it is bound to | the bindings' credentials, which are secrets |
### walks — `mesh-controller_walks`, key: the walk's id
### rollouts — `mesh-controller_walks`, key: the rollout's id
| Holds | Leaves out, and why |
|---|---|
| the repository and commit, or the batch of merges it walks (ADR 0276) | — |
| the repository and commit, or the batch of merges it rollouts (ADR 0276) | — |
| its state (building, rolling, done, failed), its current tier, and each tier's modules and nodes with each step's state | the build and apply logs, which the `plans` and `builds` verbs answer |
| when it was opened, when it last moved, when it ended, and its note | — |
A walk is removed from the mirror 14 days after it ended.
A rollout is removed from the mirror 14 days after it ended.
### builds — `mesh-controller_builds`, key: the build's id
@@ -137,8 +137,8 @@ called).
caller today): a numbered migration of `inventory` and the consolidated schema, in the same change.
- **A removal is a purge** of the key, said as a `removed` event. A purge leaves one marker for the key.
Each purge gives its marker a time to live of one day (a per-message time to live, which the bus server
has offered since release 2.11 and the bucket enables), so the markers of removed walks and builds do
not pile up in a bucket that has no age. The controller's one-minute tick removes walks and builds 14
has offered since release 2.11 and the bucket enables), so the markers of removed rollouts and builds do
not pile up in a bucket that has no age. The controller's one-minute tick removes rollouts and builds 14
days after they ended.
- **A write that fails** sends no event, is logged, and is left to the audit. The writer does not retry
in a loop: a retry loop against a refusing bus is the reconnect loop of issue 327 in another place.
@@ -169,7 +169,7 @@ counts in §6 shows the gap.
**D17, the mirrors match their store**, is a probe in the self-check's registry (to-be 45 §4). Every run
does this for each mirror:
1. It reads every record the mirror should hold from the store (for walks and builds, those open or ended
1. It reads every record the mirror should hold from the store (for rollouts and builds, those open or ended
within 14 days) and builds each value as the writer would.
2. It lists the bucket's keys and reads their values.
3. It compares. A key missing, a value that differs, and a key with no record are each a candidate. A
@@ -225,10 +225,10 @@ remain open there.
| The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run; and a test that a change of lease alone raises nothing |
| Quiet means no writes | live: over ten minutes in which `plans`, `nodes`, `builds` and `settings` show no change, no mirror's write count rises |
| A watcher sees a change | live: a setting changed through `settings` is seen by a watch on `mesh-controller_settings` and as one `setting-changed` within a second |
| A value is under its cap, or cut and marked | the controller's test, which builds the largest walk the mesh could make (every module on every node) |
| A value is under its cap, or cut and marked | the controller's test, which builds the largest rollout the mesh could make (every module on every node) |
| Only the audit reads a mirror | a test over the controller's packages, which fails when any package but the audit's reads a mirror bucket |
| A write follows the commit | the controller's test that a rolled-back transaction makes no write and sends no event |
| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals |
| Ended rollouts and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the rollouts bucket's keys including deleted ones at most its live keys plus a day's removals |
| A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run |
| A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set |
| A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised |
@@ -236,7 +236,7 @@ remain open there.
## What it leaves out
- **History.** A mirror is current state. What happened beyond the 14 days a walk or build is kept, and
- **History.** A mirror is current state. What happened beyond the 14 days a rollout or build is kept, and
beyond EVENTS' week, is gap G8 of research 036, which remains open there.
- **People reaching the mirrors**, through a WebSocket and an auth callout, is gap G1, open in the same
effort.
+1 -1
View File
@@ -130,7 +130,7 @@ rules that judge the whole record set run (an accepted decision cited by nothing
undo the acceptance. It is a fault of the records, which their authors repair, not of the module: it is told to the
operator once, as a warning naming the rule and the records, and cleared once the rule holds again. The module's
health names it and stays healthy, because health says only whether the module works, and a finding never holds a
walk at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
rollout at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
reviewed).
The rule `cited` is also judged ahead, in the review, before the judgement: a decision that names no home and that no
@@ -0,0 +1,348 @@
---
layer: to-be
status: designed
code: [mesh-controller, mesh-host, mesh-sdk, mesh-catalog]
updated: 2026-10-14
decisions:
- "ADR 0348"
- "ADR 0349"
- "ADR 0201"
- "ADR 0113"
- "ADR 0158"
- "ADR 0360"
- "ADR 0362"
---
# 57. Building config and provisioning
**The work of ADR 0348 (config is a domain of its own) and ADR 0349 (provisioning), broken into
packages small enough that each ends at something that can be seen to work, in the order their
dependencies allow.** The two decision records are the authority on *what* is built; this document holds
only the packages, their order, their sizes and their proofs, and is wrong the moment it disagrees with
them. It is the shape [design 38](38-building-the-operators-machine.md) gave the operator's machine.
## How this is built
**Each package is one issue in the issue tracker, and one pull request per repository it touches.** The
issue carries the package's number here in its title, links the packages it waits for with `depends-on`,
and links ADR 0348 or ADR 0349. At most five packages are worked at once, each by its own agent in its
own worktree; a package starts once every package it depends on is merged. Every pull request is
reviewed independently at its head and merged only with its checks green.
**It ships as one release.** The operator's decision of 2026-10-14: one delivery, not one per package.
Each of mesh-controller, `mesh-host`, mesh-sdk, mesh-catalog and hq has the branch
`feat/57-config-and-provisioning`, and every package's pull request merges into that branch, never into
main. When the last package is merged, the whole is rehearsed in the lab: the rollout, the moves of data in
their order, and every proof that is run live. Then each repository's branch reaches main in one pull
request, and the five, sharing their branch's name, are one delivery group (ADR 0239 §2), judged as one
future state. The five are merged within one merge window, so they are one batch and one rollout (ADR 0276
§1, §2), in an order declared with `after:` lines in each pull request (ADR 0239 §11): mesh-sdk, then
`mesh-host`, then mesh-controller, then mesh-catalog, then hq. The release holds every package but T6, R1 and R2:
the media catalogue and the applications live in repositories of their own and move after it, and only
then can the old forms be refused (R1) and the record say what runs (R2).
**The moves of data run after the delivery, in this order, each a verb that says what it did:**
placement out of the settings layers (F2), the settings layers into mesh-config (K5), the secrets into
the vault, nothing rotated (V3), then the providers one by one (B2, B3). Each is rehearsed in the lab
first, and each is checked by reading back through the module that owns the data before anything
points at it.
**What a release costs.** A proof that is run live (dunst on a workstation, a secret reaching its file, a
database consumer connecting) is first seen in the lab rehearsal, and a fault found then holds the
release, not one package. A merge's builds go to every machine at once (ADR 0360), so the release has no
first node to hold it: after the pull requests' checks, the lab rehearsal is the only live check before it
reaches every node. A machine that fails to apply it, or whose provisioner reports a module's files failed
(ADR 0362), is named in the rollout's end and raises apply-failing, and is fixed forward with a new pull
request into main; one that says nothing within fifteen minutes is named silent and gets its builds from
the backlog rollout once it is heard from (ADR 0360 §10).
**Nothing breaks on the way.** ADR 0349 clause 17: each file is filled the old way or the new way, never
both, by the form of its manifest, and the old forms stay accepted until the last module has moved.
Inside the release this still holds: the media catalogue and the applications kept in their own
repositories move after it, so the old forms stay accepted until they have (R1).
## What exists already, measured
Counted 2026-10-14 at mesh-controller `1601cb1`, the node-engine (`mesh-host`) `1a26778`, mesh-sdk `a2d2af0`, mesh-catalog
`ebf7f37`.
| Piece | Today | Becomes |
|---|---|---|
| the controller's composer | `catalogue.Resolution.compose` fills every namespace but `${secret:}` in one loop (`declaration.go` ~1000–1085); the namespaces and their fields are tabled in `fillers` | fills placement only, for a module that has moved |
| settings | tables `settings`, `settings_history`; the `settings` verb (`modules.go` 407–760); proposals and asks (`proposals.go`, 975 lines); trusted files; `takes-settings` merging; about 4 100 lines and 2 500 of tests | the `mesh-config` module and its buckets; the verb `config` |
| placement kept as settings | `places`, `accesses` and the kept networks are read from settings layers | facts of an assignment, the controller's own |
| secrets | the controller mints, seals and accepts every secret (`internal/secrets`, `inventory/secrets.go` 1 125 lines; tables `secret`, `module_secret`, `secret_ask`, `operator_key`); the vault (`mesh-vault`, TypeScript) is read-only and never asked | the vault, in Go, holding and minting; handing out on request and reply |
| bindings | grant files and binding files composed by the controller (`declaration.go` 750–818); `${bound:}` 117 uses | each provider answers its consumers; the vault mints the pair's credential |
| contributions | `${contribution:}` (4 collectors), `${shell:}` (3), `${environment:}` (1); 22 contributors | collections, filled by the provisioner client |
| file text | 225 file resources in 88 modules, all inline (about 193 KB); 12 modules keep a second copy in 34 files with a test that the two agree (ADR 0349's context counted eleven; the twelfth is lemurs) | template files under `files/`, packed by the module's build |
| packing a directory | the builder's `archive` artifact packs a tree and publishes it by digest; the node-engine fetches and unpacks it | the same mechanism carries a module's templates |
| the node-engine's file writing | `applyFile`, `applyBlock`, `applyInto`, create-once, drift written back, `Keep` backups, restart-on and reload-on, container reads | the same code, reached also through a narrow local write verb |
| the node-engine's local verbs | one socket (`cli.sock`), peer-checked by uid, forwarding to the controller; no local write | a second, narrow socket for the provisioner client |
| bus identities | one principal per node for the tool runner, one per module with a process of its own | the provisioner client gets one of its own |
| the SDK | KV, watches, asks, caller proof; no sealing helper; the TypeScript `provisioner` is a database-provider harness | a seal box, provider helpers, and readers for config, bound values and secrets |
**Two measurements decide the order.** The builder can already pack a module's directory, and the
controller already fills every placeholder at registration time from text; so template files can ship
first, the controller reading the text back from the packed `files/`, and nothing on a node changes. And
the node-engine's writing code needs no change in kind, only a second way in; so the provisioner client
reuses it rather than writing as root itself.
## The order the work allows
```
package waits for package waits for
F1 — P1 —
F2 — P2 P1
F3 — P3 F3
F4 — P4 P3
T1 — P5 P2, P4, T1, K3
T2 T1 P6 P5
T3 T4 T5 T2 P7 P6
T6 T1 V1 F1
K1 T1 (same file) V2 V1
K2 K1 V3 V2
K3 F3, K2 V4 V3, P5
K4 K3 V5 V3, V4
K5 F2, K4 B1 P5, V5
K6 K1 B2 B1
K7 K5, P6 B3 B2
L1 P6 L2 L1
S1 P6 R1 T5, T6, K6, K7, P7, V4, B3, L2, S1
R2 R1
```
The issue tracker carries these as `depends-on` links (thirteen issues also link what they wait for
through another package), so the packages ready to start are those whose linked packages are all merged
into the release's branches. An issue's "Waits for: … delivered" means merged into the release's
branches: nothing is delivered before the release. The first wave touches different repositories and different files:
**T1, F1, F3, P1 and F4**. K1 waits for T1 because both change `internal/catalogue/manifest.go`; P3
waits for F3 because both change `internal/broker`; F2 joins as soon as a slot is free.
## 0 — Foundations
**F1 (issue 650) — A seal box in the SDK.** *mesh-sdk. Half a day.* The ADR 0206 box (X25519, HKDF-SHA256,
AES-256-GCM) moves from the licence manager into `mesh-sdk/go/seal`, with a TypeScript twin, and the
licence manager uses it. **Proof:** a byte-for-byte test against the licence manager's existing sealed
values; the licence manager's tests pass unchanged.
**F2 (issue 651) — Placement leaves the settings layers.** *mesh-controller. Two days.* `places`, `accesses` and the
kept networks become facts of an assignment, held by the controller beside the assignment, with a
migration moving each existing layer's keys; `Settle` stops letting a setting replace a provider's
served keys. **Proof:** composition of every node before and after the migration is byte-identical;
the settings layers hold no placement key afterwards.
**F3 (issue 652) — The assignments and the modules' config declarations on the bus.** *mesh-controller. Two days.*
The controller keeps the bucket of assignments design 54 decides (which module on which node), and a
bucket of each registered module's declared config values, written when they change, with one event per
write. mesh-config needs both to resolve. **Proof:** a test against a real bus: an assignment and a
registration each change exactly one key and raise one event.
**F4 (issue 653) — The words.** *hq. Half a day.* The glossary gains *config value*, *predefined key*, *provision*
and *provisioner client*, *collection*, and the ownership words *owned*, *seed* and *managed*. The word "setting" goes on *config
value*'s *Not:* line only in R2, once the 67 documents still using it are reworded; until then the
glossary says it is the older word. **Proof:** `words.py` passes.
## 1 — Template files (ADR 0349 clause 2)
**T1 (issue 654) — A file resource may name a template.** *mesh-controller. Three days; the largest package of the
stage.* A file resource takes `"template": "files/<name>"` in place of `content`; the module's build packs
`files/` into an archive artifact of its own (the existing `archive` packing); registration and `Resolve`
read each template's text back from the packed tree into `content`, so composition and the node-engine
are unchanged; `module check` and the merge gate read templates from the repository tree beside the
manifest; a file naming both `template` and `content` is refused. **Proof:** composition tests: a module
with its text in `files/` composes byte-identically to the same module with it inline; the merge gate
judges a change to a template alone.
**T2 (issue 655) — The twelve modules with copies.** *mesh-catalog. A day.* adwaita, clipmenu, dunst, feh,
gnome-keyring, i3, i3status-rust, lemurs, picom, rofi, screen-lock and xdg move to their templates; the
34 inline copies and the `sameAsSource` tests go. **Proof:** each module's composed declaration on its
nodes is unchanged (the merge gate's plan says no file changes).
**T3 (issue 656) — The desktop modules.** *mesh-catalog. A day.* Every remaining module assigned only to
workstations. **Proof:** as T2.
**T4 (issue 657) — The servers' modules.** *mesh-catalog. Two days.* Every remaining module with a build. **Proof:**
as T2.
**T5 (issue 658) — The ten modules without a build.** *mesh-catalog. A day.* ca-trust, de-spiegel, invoicing,
local-model-consumer, matrix, node-env, only-office, sshd, step-ca and supabase gain an archive build for
their templates. **Proof:** as T2; afterwards no manifest in the catalogue carries file text.
**T6 (issue 659) — The media catalogue and the applications.** *mesh-media-catalog and each application's
repository. Two days.* **Proof:** as T2, per repository.
## 2 — Config (ADR 0348)
**K1 (issue 660) — A module declares its config.** *mesh-controller. Two days.* The manifest takes `config` (type,
description, validation, trusted, `predefined`, `default` or `required`), `takes-config` and
`${config:<key>}`, with `settings`, `takes-settings` and `${setting:}` read as aliases; registration
refuses everything ADR 0348's "How it is checked" lists for registration: a value without type or
description; a value with neither a default nor `required`; a default failing its validation; an
object or list not trusted; a predefined key not on the list or of another type (with K2); an untrusted
value falling back to a trusted key; a value neither in a template nor read at runtime; a placeholder
naming an undeclared value; a `reload-on` or `restart-on` naming an undeclared file; a moved module's
mergeable file not naming its keys; a mesh word or `puid`/`pgid` in `takes-config`. **Proof:** the registration refusal tests, one per
rule; every manifest in the catalogue still registers.
**K2 (issue 661) — The list of predefined keys.** *mesh-controller and mesh-catalog. A day.* `predefined.json` at the
catalogue's root, with `font-size` and `agent-managed-settings` first; the build validates it and every
manifest naming a key; the merge check refuses removing a key still named or holding a value; the newest
trunk build's list is published. **Proof:** the merge check refuses a planted removal; an older build's
list does not replace a newer one.
**K3 (issue 662) — The mesh-config module.** *mesh-catalog. Four days.* A Go module holding the seat `mesh-config`:
the buckets `mesh-config_values`, `_resolved`, `_predefined`, `_proposals`; the four sources resolved from
F3's assignments and declarations; the verb `config` (set, clear, list, history, narrowed by module and
node); validation; the hand-act log; `config-changed`; only its account writes its buckets. Terminal-only
for trusted values until K4. **Proof:** its tests resolve each source in turn; a test against a real bus
refuses a put from another account.
**K4 (issue 663) — Trusted values are proposed and approved.** *mesh-catalog. Two days.* mesh-config asks under its
own name, with its own grant and cap of open asks, binds each option to the level, key and value, records
the approver beside the value, and checks the proposal on Approve (ADR 0277's rules, ADR 0348 clause 8).
**Proof:** a proposal test end to end with a stand-in channel; an Approve against a changed value is
refused.
**K5 (issue 664) — The values move, and `settings` forwards.** *mesh-controller and mesh-catalog. Two days.* Each
existing layer moves into mesh-config (a node layer to `module.<node>.<module>`; a whole-mesh layer to
mesh config, a manifest default or each node, as ADR 0348's consequences say; claude-code's managed
settings to `agent-managed-settings`), recorded with why; for a module not yet handed to the provisioner
the controller reads `mesh-config_resolved` where it read its layers (ADR 0349 clause 17's old way,
an exception to ADR 0348 §9 that R1 removes), and `settings` answers by forwarding to `config`. **Proof:** composition of
every node before and after is byte-identical; `settings` and `config` list the same values.
**K6 (issue 665) — The catalogue says config.** *mesh-catalog. A day.* The 15 modules using `${setting:}`, the 9
declaring settings and the 4 with `takes-settings` move to the new names and the full declaration.
**Proof:** the catalogue's check finds no old name in a moved module.
**K7 (issue 666) — The operator's acceptance on dunst.** *live. An hour.* With every level cleared the first
workstation's dunstrc says 10; a mesh font size, then that node's font size, then dunst's own value on
that node each appear in its dunstrc within the call's answer, and the history shows each change with who and why. Needs P6.
## 3 — The provisioner (ADR 0349 clauses 3–5, 11, 13)
**P1 (issue 667) — The node-engine knows who owns a placed file.** *`mesh-host`. A day.* `declaration.File` gains
`owner-at-runtime` (`owned`, `seed`, `managed`), mapped onto today's paths (drift written back,
create-once, `into`), and the declaration gains the list of files the provisioner fills. **Proof:** the
apply tests: each ownership behaves as its old form did.
**P2 (issue 668) — The narrow local write verb.** *`mesh-host`. Two days.* A second unix socket accepts a write only from
the provisioner client's account (`SO_PEERCRED`) and only for a path the declaration names as filled,
writes through `applyFile`, `applyBlock` or `applyInto` with `Keep`, and fires restart-on, reload-on and
container reads as an apply does. **Proof:** tests: an undeclared path and another uid are refused; a
write under `reload-on` reloads its program once.
**P3 (issue 669) — The provisioner client's own bus account.** *mesh-controller. A day.* A principal of its own per
node for `node-provisioner`, granted to read its node's keys in each domain's bucket, publish
`prov.applied.<node>.>` and ask the vault; nothing else. **Proof:** the grants test, and a test against a
real bus refusing a read of another node's keys.
**P4 (issue 670) — The provisioner's server.** *mesh-catalog. Two days; after P3.* A Go module holding `mesh-provisioner` once:
the registry of which domain answers which kind, the buckets `_registry`, `_needs`, `_applied` and
`_collect`. **Proof:** its tests; the buckets listed live.
**P5 (issue 671) — The provisioner's client.** *mesh-catalog. Four days.* A Go service on every node, outside the tool
runner: fetches a module's templates by digest, records needs, watches each owner's keys for its node,
fills only when every value is there, writes through P2, reports `prov.applied`, writes an `owned` file
back on drift, removes a module's files and needs when it leaves, fills from current values when its node
comes back. Its first owners are placement (from the declaration) and config (from
`mesh-config_resolved`). **Proof:** the client's test list in ADR 0349, against a real bus.
**P6 (issue 672) — The controller hands a module to the provisioner, and the rollout waits for it.** *mesh-controller.
Two days.* For a module marked moved, the declaration carries placement, the templates' digest and the
filled-files list, and the controller runs no other filling pass for it; a rollout counts the module applied
on a machine only after the node-engine's report and that node's `prov.applied` for the new build, and a
failed `prov.applied` is a failed apply (ADR 0362). dunst moves first. **Proof:** composition and rollout
tests; dunst on the first workstation filled by the client, live.
**P7 (issue 673) — The node-engine stops writing placed files from the declaration.** *`mesh-host`. A day.* For a moved
module the engine writes its filled files only through P2, and removes as an orphan only what the verb
placed. **Proof:** the apply tests; no placed file written twice in a day's journal on the
first workstation.
## 4 — Secrets (ADR 0349 clauses 6–11, 16; issue 647)
**V1 (issue 674) — The vault in Go, holding secrets.** *mesh-catalog. Four days.* `mesh-vault` is rewritten in Go:
its own store, a copy of each secret sealed to the operator key, `get` on request and reply sealed to the
asker's key (F1's box; the node-engine's key format is settled here and written down), `secret-changed`
without a value, no secret in any bucket. **Proof:** the vault's test list in ADR 0349.
**V2 (issue 675) — The vault's verbs.** *mesh-catalog and mesh-controller. Three days.* `secret accept`, `secret-ask`
(with ADR 0277 §24–§27's record and bounds in the vault's store), `issued` (ADR 0305, the provider's tool
answering the vault alone) and `rotate` become the vault's; the controller's verbs forward. **Proof:** the
`secret-ask` and issuing provider tests in ADR 0349.
**V3 (issue 676) — The secrets move to the vault.** *mesh-controller and mesh-catalog. Two days.* Every row of
`secret`, `module_secret`, `secret_ask` and the operator key's public half are copied into the vault as they are,
nothing rotated, and read back through the vault before anything points at it. **Proof:** for every
secret, the vault's sealed answer opens to the same value the controller's does.
**V4 (issue 647) — A node's secrets come from the vault.** *mesh-catalog, mesh-controller, `mesh-host`.
Three days.* The client fills `${secret:}` from the vault's `get`; for a moved module the controller seals
nothing; the core's credentials travel sealed by the vault to their node and the node-engine opens them.
**Proof:** a secret rotated in the vault reaches its file without a new declaration, live.
**V5 (issue 677) — The vault mints pair credentials.** *mesh-catalog. Two days.* One value per provider assignment and
provision, remade for all holders when the vault sees a consumer's need recorded or deleted (ADR 0158
§4–§6). A need recorded for a bind the vault already holds from V3 adopts that value and remakes
nothing, so the move itself rotates no credential. **Proof:** the provider test in ADR 0349, with a stand-in provider.
## 5 — Bindings (ADR 0349 clause 12)
**B1 (issue 678) — A provider's helper in the SDK.** *mesh-sdk. Two days.* Go: watch the needs for a kind, write one
entry per consumer, refuse a derived value for a multi-holder consumer, answer from the vault's pair
credential. The TypeScript `provisioner` harness is renamed so the word means one thing. **Proof:** its
tests with a stand-in bus.
**B2 (issue 679) — PostgreSQL answers its consumers.** *mesh-catalog and mesh-controller. Three days.* The first
provider moves; its grant files and binding files are retired. **Proof:** every database consumer on
every node connects after the move with the credential it had, adopted by the vault as V5 says, live.
**B3 (issue 680) — The other providers.** *mesh-catalog. A week, one pull request per provider.* **Proof:** as B2, per
provider; afterwards the controller composes no grant file.
## 6 — Collections (ADR 0349 clause 13)
**L1 (issue 684) — A collection is declared and filled by the client.** *mesh-controller and mesh-catalog. Three days.*
A collector requires a collection; a contributor provides an item from a template; the client fills the
collector's template from `_collect`. `${contribution:}`, `${shell:}` and `${environment:}` read as
collections. **Proof:** zsh's file holds every placed contributor's item and loses one whose contributor
leaves.
**L2 (issue 681) — The collectors and contributors move.** *mesh-catalog. Two days.* zsh first, then the bar, the
display session, the hotkeys, xorg, power and node-env, with their 22 contributors. **Proof:** each
collector's composed file is unchanged at the move.
## 7 — A module's code asks at runtime (ADR 0349 clause 14)
**S1 (issue 685) — The SDK's readers.** *mesh-sdk and mesh-catalog. Two days.* `config`, `bound` and their watches,
answered by the node's client for what the manifest declares. `secrets` refuses until the module runs
under a bus account of its own, which waits on issue 569 and its decision. **Proof:** the SDK's test in
ADR 0349; claude-code reads its managed settings through it.
## 8 — The old forms go (ADR 0349 clause 17)
**R1 (issue 682) — Refused and removed.** *mesh-controller, `mesh-host`, mesh-catalog. Three days.* Once every module of
every catalogue has moved: the catalogue's check refuses inline text, `settings`, `${setting:}` and the
old contribution forms; the controller's filling passes but placement, its settings, secret and binding
code and tables (a new migration) go; the node-engine keeps opening sealed values for the core's files
only; the controller stops reading `mesh-config_resolved`. **Proof:** the controller's test that its
code opens no secret outside genesis and reads no config value; the absence of the
removed code.
**R2 (issue 683) — The record says what runs.** *hq. Two days.* The as-is designs are written, designs 27 and 32 are
amended, the 67 documents that still say "setting" in running prose are reworded so the glossary can
retire the word (F4 left it off the *Not:* line for them), design 49 lists the Config domain, ADR 0348's and 0349's built status is said, and this document's status becomes `implemented`.
## What is deliberately not here
- **Secrets in a module's own code** wait on issue 569 (a module's tools run as the tool runner's
account) and its decision; S1 ships without them.
- **Release separated from delivery** stays issue 644's, and **the bus's replicas** issue 645's.
- **Renaming "walk"** is issue 646's.
## How this list is kept true
Each package's proof that runs in its tests is run when it is merged into the release's branch, and its
line here gains the date and the commit; a proof that runs live is run in the lab rehearsal and again
after the release's delivery. A package whose proof fails is not reworded; the failure is recorded under it and the package
stays open. The issue tracker says which packages are worked now; this document says only the order.
+3 -2
View File
@@ -47,15 +47,16 @@ document is written and this one's status becomes `implemented`.
| [`42-the-machines-modules-in-order.md`](42-the-machines-modules-in-order.md) | **In progress.** The order the machines' modules of research 026 and 027 are built and rolled out: every machine's first (sudo, localization, time sync, pacman, logrotate, avahi, systemd, docker, `~/.ssh`, scripts, kernel), then both workstations', then one machine model's; each proven on one workstation before the rest | ADR 0173, ADR 0182, ADR 0205 |
| [`45-a-core-that-cannot-fail-silently.md`](45-a-core-that-cannot-fail-silently.md) | **Designed.** The core says when it is wrong, refuses what is stale or unreadable, heals what it knows, upgrades one machine at a time with a witness that rolls it back, and is checked against the real mesh before merge: the writers and signals tables, the condition store, `doctor`, the minimal output channel, healers, the lease and report order, staged upgrades, the facts snapshot and replays, in six phases | ADR 0227, ADR 0224, ADR 0218 |
| [`46-the-conversation-with-the-operator.md`](46-the-conversation-with-the-operator.md) | **Designed.** The mesh tells and asks its operator over channels that are holders of two kinded benches, `channel` and `intake`, declaring capabilities from a fixed vocabulary; the router orders them by work context and never lowers the bar; an answer that performs an action is checked and performed by the controller, on a TOTP code or a verified Telegram sender, never on a desk click alone; operator messages addressed or answered by the mesh's responder, untrusted input kept as data, references for what words may not carry, and a recoverable factor; Telegram first, in six phases | ADR 0234, ADR 0227 |
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the walk, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the rollout, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
| [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of eleven domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | ADR 0244, ADR 0006, ADR 0008, ADR 0302 |
| [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | ADR 0247, ADR 0223 |
| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | ADR 0251, ADR 0189 |
| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | ADR 0263, ADR 0051, ADR 0092 |
| [`53-the-issue-tracker.md`](53-the-issue-tracker.md) | **Designed.** The mesh keeps its issues in one module, `issues`, on the control-node: every number from one counter advanced by compare-and-set, the cycle's issue rules refused at the write, an event per change; an issue lives whole in the tracker, hq keeping no issue content (ADR 0291) | ADR 0275, ADR 0291 |
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, walks, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, rollouts, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
| [`55-the-knowledge-base.md`](55-the-knowledge-base.md) | **Designed.** The mesh keeps its decision records and research efforts whole in one module, `mesh-knowledge`, on the control-node: number, status, topic, typed links both ways, each decision's clauses with their own status, and the reasoning, with every version; a record is proposed, checked at the write, reviewed by the knowledge base's own judgement and accepted there, with the operator's Approve when it needs it; agents ask what holds in one bounded call | ADR 0290 |
| [`56-the-mail-client-set-up-by-the-mesh.md`](56-the-mail-client-set-up-by-the-mesh.md) | **Designed.** On every node with a display server, the `betterbird` module installs Betterbird from a pinned upstream release mirrored into the artifact store, ships the Owl add-on pinned and installed by Betterbird's enterprise policy, and declares the operator's mail accounts in a trusted setting without credentials; each account signs in once at a person's first use, and one the profile already holds is adopted by its address | ADR 0316, ADR 0317 |
| [`57-building-config-and-provisioning.md`](57-building-config-and-provisioning.md) | **Designed.** The work of ADR 0348 and ADR 0349 as 37 packages, each an issue with its proof: foundations, template files, config in its own module, the provisioner and its client, secrets in the vault, bindings from providers, collections, a module's runtime readers, and removing the old forms; at most five worked at once | ADR 0348, ADR 0349 |
## Not yet written
@@ -28,7 +28,7 @@ for _, n := range nodes {
```
So a provider on the same machine is never a `Provider` in `world.Offered`, never becomes a
`Needed`, and the credential loop — which walks `resolved.Needs` — has nothing to walk. Every step
`Needed`, and the credential loop — which traverses `resolved.Needs` — has nothing to traverse. Every step
is individually reasonable and the sum is a silent gap.
## Why it was not noticed
@@ -92,7 +92,7 @@ node that looks like a real one.
## Fixed
Needs fan out per consuming module in one place, after the resolution walk. The credential's key
Needs fan out per consuming module in one place, after the resolution rollout. The credential's key
gains the consuming module, the grant file is named after both halves of the consumer, needs are
matched by provision *and* module, and the provisioners name the role and the access key after the
module rather than the machine. The refusal is gone because there is nothing left to refuse.
@@ -87,7 +87,7 @@ too new to have anybody watching it.
## Fixed
**The registry module names its image and is added, not built.** The lab's artifact-store test now
walks the only path open to a real first mesh — the manifest goes in directly, no builder and no
rollouts the only path open to a real first mesh — the manifest goes in directly, no builder and no
store involved — and passes.
**And the cycle is refused where it is written.** A manifest that provides `artifact-store` and
@@ -45,7 +45,7 @@ field it does not know — whole. That was
mesh now records which host each machine reports and `status` names every machine running an older one
than another does.
So the flag day is visible before it is walked into, which it was not when this was filed. It does not
So the flag day is visible before the mesh runs into it, which it was not when this was filed. It does not
make the field free: **the oldest host in the mesh is still what the mesh may send**, and one machine of
four is behind today. A sequence that an old host refuses takes that machine out of the mesh's reach
entirely — worse than the replay it prevents, which has never been observed.
@@ -18,7 +18,7 @@ half-understood declaration off a machine, and exactly what makes this expensive
[Issue 087](../087-the-controller-cannot-tell-a-host-is-too-old/00-report.md) is resolved: the mesh now
records the host version each machine reports and `status` names every machine running an older host
than another does. The flag day is visible before it is walked into, which it was not on 2026-09-23.
than another does. The flag day is visible before the mesh runs into it, which it was not on 2026-09-23.
That makes the cost measurable rather than hypothetical, and the measurement is the reason this is not
being built today.
@@ -72,7 +72,7 @@ what the sidecar is has nowhere in the design layer to look, which is how this w
resource type no decision records and contradicting an accepted one. The rule is "no design
without a decision"; the check is "no design without *a* decision." An unenforced rule is
indistinguishable from a wrong one, and these two documents are what that gap looks like when
something walks through it.
something rollouts through it.
## Open questions
@@ -92,7 +92,7 @@ built from a repository elsewhere, or a mesh whose forge module is down — and
event performs, so the two cannot disagree about what "moved" means.
**Bases first needed edges, and there were none.** The order this report asked for was written and
walked a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
traversed a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
digest was never in the file the builder read edges from. A build now reports what it was handed, the
control plane records it by artifact path, and the order is read from each module's newest build.
@@ -38,7 +38,7 @@ Every node's resolution is read first, and the names are attributed across them
terminus serves the name**: among the providers a name reaches, the one that is not itself published
under a labelled name through another provider. An identity provider is routed through the proxy and
so is a consumer of names, not their end; the proxy contributes no label to anyone and is. The rule
knows nothing of what "route" means — it reads the graph the modules declared — and it walks nodes,
knows nothing of what "route" means — it reads the graph the modules declared — and it rollouts nodes,
requirements and contributions in order, so one mesh yields one region. Every shape of contribution
is counted, so a module routed under several names has every one of them resolved.
@@ -73,5 +73,5 @@ give.
All three faults are closed on the controller's main: one ask in flight (8d4e940), a merge rebuilds what
stands on it (7e701c0), and a merge produces a tiered plan the mesh keeps, read with `plans` (d2ed6d5,
76f2756, ADR 0162). Plans were since carried into walks and deliveries (ADR 0236, ADR 0239); the order a
76f2756, ADR 0162). Plans were since carried into rollouts and deliveries (ADR 0236, ADR 0239); the order a
person kept is the order the tiers give.
@@ -86,6 +86,6 @@ machine. That is the path the console exists to replace, so it is a workaround a
Both pull requests named above are merged and live. Read through the console on 2026-10-08:
`mesh_describe mesh-controller.plan` declares `node` (required) and `files`; `mesh-controller.push`
declares `node`, `behind`, `cause` and `why`, and its description says that a push naming no machine
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` walks every verb it serves:
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` exercises every verb it serves:
an argument a verb does not declare is refused, and every flag of a verb's command is in its schema
or accounted for.
@@ -46,7 +46,7 @@ Pull requests: novox/mesh-tools#14 and novox/mesh-controller#71 — open, not me
first line of its answer, that it is a push of the whole mesh and which machines it is sending, and
its last line names the machines told. `plan` gains `files`, `push` gains `behind` (the whole mesh
said outright, refused beside `node`), and the listing verbs a `limit`.
- **Checked by tests that walk every verb the controller announces:** for every combination of a
- **Checked by tests that exercise every verb the controller announces:** for every combination of a
verb's declared arguments, the verb either refuses or every argument given changes the command it
runs; every verb refuses an argument it does not declare, and `node` where it takes none; and every
flag of the command a verb runs — read from the command's own source — is an argument of the verb's
@@ -26,7 +26,7 @@ list alone. The flush has no such narrowing.
## What it costs
`record` is the policy for a change that must be walked through the mesh by hand. A named push cannot
`record` is the policy for a change that must be rolled out through the mesh by hand. A named push cannot
do that, so the policy does not hold the change back from anything but the merge. The command's name
says one machine, and the command's output is the only place that says otherwise.
@@ -61,7 +61,7 @@ The mesh-delivery tests `checkask_test.go` cover:
- An announcement that carries a decided gate status uses it.
- H2 re-asks once, a second re-ask is refused by the table, and a late verdict still settles the delivery.
The table tests walk the new row, and the pairs the table does not hold are refused by name. Live: once
The table tests exercise the new row, and the pairs the table does not hold are refused by name. Live: once
the fix is rolled out, the stalled delivery has its check asked at the owner's first tick. The controller
journal then shows its verdict, and `show` moves from proposed to checked to ready without a person
acting.
@@ -39,7 +39,7 @@ The mesh's own record shows it:
- **What each machine reports.** `postgres.server`, `mongodb.server` and nine `mailu.*` containers
have run only since 18:57 on the control node, and `postgres.server` since 18:56 on the anchor.
No walk held back for later was involved. The sends were the plan's own sends to its first machines.
No rollout held back for later was involved. The sends were the plan's own sends to its first machines.
## Why it happened
@@ -53,7 +53,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
push". So the gate did not judge the recorded builds, the refusal of ungated sends did not see
them, and the gated send carried them. `record` held only against the end of a person's push to
another machine (issue 259's `heldBack`) and against the bus (the bus step). It did not hold
against a plan's walk, a rollback, a healer or a rotation.
against a plan's rollout, a rollback, a healer or a rotation.
3. **Nothing tells a person, before or during a send, that it will recreate every container of a
module at once.** A change that only alters how containers are declared reads as harmless. A
health check, an environment variable or a label each recreates the container all the same,
@@ -64,7 +64,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
## What is at stake
`record` is how the mesh keeps the providers, the network path and the irreplaceable data out of
unattended walks. Tonight showed that it holds only until some other module on the same machine
unattended rollouts. Tonight showed that it holds only until some other module on the same machine
rolls out. This was still true after the incident: plex (`record`, irreplaceable data) was rebuilt
by a media-catalogue merge at 19:01 and sent nowhere. The next gated send to the anchor, for any
module, would recreate it.
@@ -13,8 +13,8 @@ the tier durations measured for the repository: three times their ninetieth perc
than 30 minutes. So the watchdog and the line a person reads used different start times and could use
different bounds. The line was the one that was wrong.
The line for a walk waiting on its delivery's word had the same flaw. It said "for" the time since the
last save. The watchdog for that wait (S16) counts from when the walk was opened.
The line for a rollout waiting on its delivery's word had the same flaw. It said "for" the time since the
last save. The watchdog for that wait (S16) counts from when the rollout was opened.
Ruled out: a clock difference between the machine running the verb and the store. The count was small
and kept growing again from zero, which only a column being rewritten explains.
@@ -44,8 +44,8 @@ nothing.
## Fix (mesh-controller)
- The line, the `since` and `late` fields of `status --json`, and the count of late plans all start from
when the plan entered its tier. If the delivery let the walk go after that, they start from the go. A
walk still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
when the plan entered its tier. If the delivery let the rollout go after that, they start from the go. A
rollout still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
called late. A plan read without the tier stamp falls back to its last save.
- LATE uses the bound the watchdog uses: the measured bound for the plan's repository, or the 30-minute
floor when nothing has been measured. The watchdog's start time and bound now come from the same two
@@ -53,4 +53,4 @@ nothing.
- The loop compares the plan before and after each step and saves only when they differ. The fields the
store stamps itself are left out of that comparison.
- A refused step is logged and saved only when its note is new. The refusal's wording called the gated
walk a "release plan", a name the glossary retires; it now says "walk".
rollout a "release plan", a name the glossary retires; it now says "walk" (since renamed rollout, issue 646).
@@ -23,7 +23,7 @@ another, with nothing on the mesh noting the difference.
ADR 0238 makes the delivery plan the planner's one answer for a pull request's check, the merge gate and
the merge. The plan counts a new module's directory as something the merge builds. The merge handler
read the same answer (`touchedBy`'s `added`) and dropped it: it only walks modules the catalogue already
read the same answer (`touchedBy`'s `added`) and dropped it: it only rollouts modules the catalogue already
holds, and a new one is not held until it is built. The code said so outright: "a new module, which a
check judges before it merges and a merge does not build". The take-in refuses a build off the trunk
with "merge it, and the merge builds it". So one part of the controller promised the build and another
@@ -29,7 +29,7 @@ already built from it are, and without waiting (`askNewModules`). The build's ta
module, as it does for a hand `build`. Nothing is sent: no machine is assigned a new module, and
assigning one stays a person's act.
The new module is outside the merge's plan. A plan walks modules the catalogue holds, tier by tier, and
The new module is outside the merge's plan. A plan rollouts modules the catalogue holds, tier by tier, and
sends each built one to the machines running it. A new module has no machine and nothing standing on it,
and the plan's ask for a module the catalogue does not hold fails the plan.
@@ -7,16 +7,16 @@ opened: 2026-10-08
## Symptom
On 2026-10-08 an agent proving the tunnel join (novox/mesh-lab PR #61) on the laptop could not run the
two-node walk, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
two-node rollout, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
reported by any check before someone tried to run a bed:
1. **The walk's scenario stocks a packet-filter runtime image that cannot be built.**
1. **The rollout's scenario stocks a packet-filter runtime image that cannot be built.**
`scripts/build-module-runtime.sh` copied the tool runner's compiled TypeScript and its `node_modules`
out of mesh-tools. mesh-tools has had no package at its root since the runtime became `node-tools`, a
Go binary that launches every bundle (ADR 0175,
ADR 0193),
so the script failed at its first step.
2. **The walk's builder dialled the old broker.** It was started with the old bus's guest account on
2. **The rollout's builder dialled the old broker.** It was started with the old bus's guest account on
loopback. The mesh moved its bus to NATS (ADR 0106,
ADR 0127,
ADR 0131), and the builder
@@ -27,8 +27,8 @@ reported by any check before someone tried to run a bed:
4. **The replays prover could not run a test in mesh-tools**, because that repository's Go module lives
in a subdirectory (`node-tools/`), and the prover ran `go test` only at a repository's root.
The same day, while trying to run the walk again, two more gaps surfaced: the object store's
provisioner image, which the walk also stocks, cannot be built, because the client image it copies
The same day, while trying to run the rollout again, two more gaps surfaced: the object store's
provisioner image, which the rollout also stocks, cannot be built, because the client image it copies
from is gone upstream; and the laptop could not reach the public container registry, which the lab's
egress check needs (an outside condition, not the lab's fault).
@@ -56,9 +56,9 @@ In novox/mesh-lab PR #63, stacked on #61:
builds `node-tools` from mesh-tools' Go module, and runs it as the image's entrypoint. The image serves
on the node's runtime credential: the Go runtime refuses to serve a module on that module's own
credential.
- **The walk no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
- **The rollout no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
names an image (its verbs are a tools bundle the node's runtime launches), so the image was never run.
- **The walk's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
- **The rollout's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
an own `broker` secret, as the catalogue's build agent does. Assigning it to the anchor issues the
account ([issue 203](../203-a-fresh-assignment-is-pushed-before-its-credential-exists/00-report.md)).
The bus's user list is placed, the push writes the sealed credential, and the builder starts on it.
@@ -71,12 +71,12 @@ In novox/mesh-lab PR #63, stacked on #61:
Still open:
- `builds.test.ts` still dials the old broker. Its bed enrols no machine, so the mesh has nowhere to
deliver a builder's credential. It needs the walk's enrolment, or it should be retired into the walk.
deliver a builder's credential. It needs the rollout's enrolment, or it should be retired into the rollout.
- Beds that run a per-module runtime container on the module's own credential (the tools beds for the
wiki and the code forge, and the audit bed) run into the runtime's refusal. They need the node's
runtime credential instead.
- The object store provisioner's image needs a client source that still exists.
- The walk has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
- The rollout has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
public container registry.
## How it is checked
@@ -14,25 +14,25 @@ script now does those steps by hand. Verified: the image for the packet filter b
answers `initialize` and `tools/list`; `node-tools` in it, started against a real bus with a plain URL,
launches the bundle and serves nothing until the mesh issues a membership, which is correct.
**Why the walk stocked that image at all.** The tunnel branch passed the held images to
**Why the rollout stocked that image at all.** The tunnel branch passed the held images to
`deriveTheFilterOn`, which turns the catalogue's manifest into the lab's form: build section gone, each
container's artifact replaced by a stocked image. The catalogue's nftables manifest has no container,
so no image is referenced. Removing the build section is what the bed needed; the stocked image was
not.
**The builder.** mesh-controller's `cmd/mesh-builder` reads `MESH_BROKER_FILE` and nothing else for the
bus. Without it, it says it has no credential. The walk set the old broker's variable, which the builder
bus. Without it, it says it has no credential. The rollout set the old broker's variable, which the builder
no longer reads. A builder's account is a module's account (`builder issue` and `module issue` both mint
through the inventory), composed into the bus's user list and sealed to the machine as the module's
`broker` own secret. Assigning a module that declares that secret now issues it in the same act (issue
203). The catalogue's build agent is that module, but it runs the builder as a container from an
artifact the mesh builds, and it needs the artifact store and package registry bound. The walk has
artifact the mesh builds, and it needs the artifact store and package registry bound. The rollout has
neither before it can build, so it declares a lab module with the same claim and the same secret. In
this bed the bus comes from the bundle, so its user list is placed by hand, the step the walk already
this bed the bus comes from the bundle, so its user list is placed by hand, the step the rollout already
takes at enrolment.
**The uplink and the prover** were each fixed by the agent that met them (mesh-lab #61 and #62), and are
listed here only so the lab's state on that day is in one place.
**Ruled out.** That the walk failed because of the tunnel changes. The builder and runtime faults are in
**Ruled out.** That the rollout failed because of the tunnel changes. The builder and runtime faults are in
code that #61 does not touch, and they reproduce on main.
@@ -21,7 +21,7 @@
the planner's sort stalled. Every two-way pair was a cycle, whatever the rules.
- The person's order, the controller first, was right for this change. The new node-engine calls a verb
the new controller adds, which is ADR 0246's case: a verb is promised before it is required.
*Engine before controller* comes from ADR 0236's order for its walks, where the node-engine had to
*Engine before controller* comes from ADR 0236's order for its rollouts, where the node-engine had to
accept something new first. The rule reads only module names, so it cannot tell the two cases apart. *Built
by* is read from the graph.
@@ -59,7 +59,7 @@ and fails only when it cannot read.
**Merged.** mesh-controller#135 (the build agent judges a pull request by the base branch's
`merge-check.sh`) merged on 2026-10-08 at 09:10 UTC, and mesh-lab#65 (R310) at 09:10 UTC. Every later
controller delivery carries it; the last, of mesh-controller#170, finished its walk on 2026-10-09 at
controller delivery carries it; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at
00:11 UTC, and `mesh-controller.status` named no machine behind.
**The open half is closed.** `gitea_branch_protection_get` for the media catalogue's main, read on
@@ -32,11 +32,11 @@ was asked of a new, unrelated pull request.
## Why it is a design issue
ADR 0239
decision 10, *the switch*, adopts the walks open when mesh-delivery is assigned. It says nothing of the merges the
decision 10, *the switch*, adopts the rollouts open when mesh-delivery is assigned. It says nothing of the merges the
bus still holds from before: the forge's `pull.merged` is kept on the bus and heard again by a new consumer.
Each was taken as a merge just made whose head was never heard, so the table's row *merged unchecked → held*
held each for a person. The table had no state for "this already happened", and `held`'s only ways on were
a release, which would walk it, and a stop, which says a person stopped a delivery that was never running.
a release, which would roll it out, and a stop, which says a person stopped a delivery that was never running.
Healer H2's `close` has no transition from `held`, rightly: a held delivery is the operator's.
Decision 5 says what a transition owes outside is "retried until done". An effect the forge answers can never
@@ -45,14 +45,14 @@ be done was retried without end, every ten seconds, and nothing said so.
## Fix
- **A new final state, `retired`: history.** A delivery made from the forge's word at its merge, whose merge
was heard fifteen minutes or more after it happened and whose walk, if the controller keeps one, is not
was heard fifteen minutes or more after it happened and whose rollout, if the controller keeps one, is not
open, is history. At adoption it goes straight to `retired` by the observed row *history*, before any check,
hold or walk. Its pull request is not told: the forge may no longer hold it, or holds another under its
hold or rollout. Its pull request is not told: the forge may no longer hold it, or holds another under its
number. Its transitions are still said on the bus and kept on the commit's note.
- **A person's verb for the ones already held: `retire-history`**, with why. One delivery by id, refused when
the rule does not take it, or every held one (of one repository when named), answering how many it retired
and naming each one it left held. `dry` says what it would do and changes nothing. It is the act
*held → retired* of the table, never a release: nothing is walked, delivered or put back.
*held → retired* of the table, never a release: nothing is rolled out, delivered or put back.
- **Owed work ends.** The forge's answer that what an effect is for does not exist (a 404, or "does not
exist") is final: the effect is given up with one line, kept among the owner's refusals. Any other failure is
tried again later each time, from ten seconds doubling to an hour, and given up after sixty tries, saying so.
@@ -5,20 +5,20 @@
- 699 deliveries were `held`. 688 say "merged without a passing check", all made between 23:05:27 and
23:05:29 on 2026-10-06, the owner's first start. A sample across repositories shows merges from
2026-10-02 to two hours before that start, each with the transition "its head was not heard by this owner,
so it is made from the forge's word", then *merged unchecked → held*. None has a walk.
- The other 11 are held for other reasons: a merge with no walk opened within ten minutes (10), and a group
so it is made from the forge's word", then *merged unchecked → held*. None has a rollout.
- The other 11 are held for other reasons: a merge with no rollout opened within ten minutes (10), and a group
whose composed check did not pass (1). They are not history, and this issue leaves them alone. One of them
was past its bound, which makes the 689th finding.
**Where it comes from.** The holder subscribes to the forge's `pull.merged` on the bus after reading back its
state and the controller's walks. The bus hands a new consumer the merges it still holds. For each, the holder
state and the controller's rollouts. The bus hands a new consumer the merges it still holds. For each, the holder
finds no delivery for the head, so it makes one from the forge's word. The pull request's statuses at its
merge do not include a passing `mesh/merge-gate` (the check did not exist for most of them), so the table's
*merged unchecked* row holds it.
**What would close them, and why not.**
- `release` walks a held delivery: wrong, these reached the machines already.
- `release` rollouts a held delivery: wrong, these reached the machines already.
- `stop` takes it to `stopped`, which says a person stopped a delivery and sets `mesh/delivery` to error on
the commit. Wrong in meaning, and it would ask the forge for a view on a pull request that is gone.
- Healer H2's `close` reads `held`'s bound, which names no H2 transition, and refuses: "the state is the
@@ -43,6 +43,6 @@ mesh-catalog#121 (the delivery's owner retires history and ends owed work the fo
under ADR 0250.
**Seen live.** `mesh-delivery.deliveries` on 2026-10-09 lists the held history as `retired`, all at
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The walk of
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The rollout of
mesh-catalog#121's own delivery reads `stopped`; the holder that answers today serves the fix all the
same, and `mesh-controller.status` names no machine behind.
@@ -53,6 +53,6 @@ lift.
every other state in the table moves on its facts?
3. Should `mesh/delivery-group` be required on the trunks, so a group cannot merge before its composed
check passed? To-be 47 leaves this to the operator; this issue is the case that setting decides.
4. The group's order put the catalogue's member first, and the node-engine's member walked at once on the
controller's own path. Is a group's order meant to hold for a member whose walk never waits for
4. The group's order put the catalogue's member first, and the node-engine's member rolled out at once on the
controller's own path. Is a group's order meant to hold for a member whose rollout never waits for
mesh-delivery?
@@ -3,7 +3,7 @@
**2026-10-08.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`),
from `mesh-delivery.show` of both members, and from the holder's journal on the control node.
**Where the hold is set.** `groupTurns` (holder.go) walks a closed group's members in order. For a member
**Where the hold is set.** `groupTurns` (holder.go) rolls out a closed group's members in order. For a member
in `published` it computes `composed`: the group's check exists, its verdict is `pass` or `warning`, and it
names the members' current heads. When `composed` is false it writes the member's `HeldWhy` ("its group's
composed check did not pass for the heads that merged") and settles it, and the table's `hold` row takes it
@@ -20,7 +20,7 @@ to `held`. `composed` is false in three different cases, and the code does not t
look at the members.
- `groupTurns` skips a member in `held` (`default: return nil`), so the tick after the verdict sees the
passing check and does nothing with it.
- The table has two rows out of `held`: `go` (the walk started on a word that was not this owner's) and
- The table has two rows out of `held`: `go` (the rollout started on a word that was not this owner's) and
`release` (a person's act). There is no observed row that takes a held delivery back when the reason it
was held is gone, and `HeldWhy` is never cleared once set.
@@ -41,7 +41,7 @@ check was asked. To-be 47 lists whether to require it as not decided.
- A lost verdict. The holder logged it at 10:10:21 and the group's record holds it.
- The order. The group's order was asked and kept before the check; it plays no part in the hold.
**Not this issue, noted.** The node-engine's member walked at once (`published` → `delivering` at the
**Not this issue, noted.** The node-engine's member rolled out at once (`published` → `delivering` at the
same instant), because the controller's own path does not wait for mesh-delivery (ADR 0239 decision 8).
The group's order put it second. Open question 4 in the report asks whether that is intended.
@@ -57,9 +57,9 @@ ADR 0270
its group's composed check is asked for the current heads and has no verdict.
- **A hold whose reason is gone lifts, by observation.** A member held because its group's composed check had not
passed goes back to `published` (the event *lifted*) once the check passes or warns for the heads that merged,
while its own walk is still open. It then takes its turn in order.
- A held member whose own walk was since taken over by a later walk is not lifted, since its walk can no longer
deliver it. It is `covered` when that later walk was delivered and sent its module (issue 319).
while its own rollout is still open. It then takes its turn in order.
- A held member whose own rollout was since taken over by a later rollout is not lifted, since its rollout can no longer
deliver it. It is `covered` when that later rollout was delivered and sent its module (issue 319).
Open question 3, whether `mesh/delivery-group` is required on the trunks, stays the operator's setting. Open
question 4 is not touched.
@@ -2,7 +2,7 @@
opened: 2026-10-08
---
# 317. A held delivery's module was walked by a later merge's plan
# 317. A held delivery's module was rolled out by a later merge's plan
## Symptom
@@ -22,7 +22,7 @@ took `openrazer` along. `mesh-controller.plans`:
| c78b5fc9 (#124's merge) | 11:07:02 | superseded at tier 0 by the plan at 86077780; docker, gitea, openrazer planned there again |
| 86077780 (#123's merge) | 11:07:03 | tier 0: distribution, docker, gitea and openrazer built from c78b5fc9; openrazer's gate on the laptop judging |
The controller's walk of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
The controller's rollout of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
c78b5fc9, beside `docker` and `node-tools`. From 11:10:58 UTC the laptop's condition for openrazer said
its account was *relogin needed*. That verdict exists only in the build #125 added, so the held change was
running on the laptop while its delivery said it waited for a person.
@@ -41,10 +41,10 @@ gone out. A person who releases it later releases nothing; a person who stops it
## Open questions
1. When a newer merge supersedes an older plan, does a module the older plan's walk was holding for its
1. When a newer merge supersedes an older plan, does a module the older plan's rollout was holding for its
delivery's word come along, or stay held with that delivery?
2. If it comes along, what does the held delivery become: superseded by the newer delivery, with the hold
carried over to it, or something else? Today the table has no row from `held` for a superseded walk.
carried over to it, or something else? Today the table has no row from `held` for a superseded rollout.
3. A trunk commit holds every earlier commit, so a later build of a module carries the held change in
any case. Is a hold on one delivery meaningful once a later merge of the same repository is walked,
any case. Is a hold on one delivery meaningful once a later merge of the same repository is rolled out,
or does a hold have to hold the module on that trunk until it is released?
@@ -3,7 +3,7 @@
**2026-10-08.** Read from `mesh-controller.plans` (the list, and each plan by id), `mesh-delivery.show`
and `deliveries`, and the code on the main branches of the controller and the catalogue.
**The held delivery's walk.** #125's merge made the plan at a6385479. Its walk waited for mesh-delivery's
**The held delivery's rollout.** #125's merge made the plan at a6385479. Its rollout waited for mesh-delivery's
word (`waited_for: mesh-delivery`, `waits: true`) and was never let go, because its delivery was held. Its
one module, `openrazer`, was "not yet asked": nothing was built.
@@ -11,32 +11,32 @@ one module, `openrazer`, was "not yet asked": nothing was built.
plan. For every open plan of the same repository and branch made before it, it takes each module the
older plan had not built (not asked, asked and not answered, or built and not yet sent), adds it to the
newer plan, and closes the older plan as superseded with the note "… planned there again". It does not
ask why the older plan had not built them. A walk waiting for a held delivery's word is open, so it is
ask why the older plan had not built them. A rollout waiting for a held delivery's word is open, so it is
superseded like any slow plan. This happened three times over (3f552274, c78b5fc9, 86077780), each newer
plan folding in what the one before it held.
The newest plan's own delivery (#123) was not held, so mesh-delivery let its walk go, and `openrazer` was
built from c78b5fc9. The controller's walk of the builds waiting for a gate then carried it to the laptop,
The newest plan's own delivery (#123) was not held, so mesh-delivery let its rollout go, and `openrazer` was
built from c78b5fc9. The controller's rollout of the builds waiting for a gate then carried it to the laptop,
since a send carries every move waiting on its machine (ADR 0236).
**Why the held delivery did not notice.** mesh-delivery's table has `superseded` rows only from
`published` and `delivering` ("a newer delivery to the same trunk took over its walk"). From `held` there
are only `go` and `release`. When the walk record said `superseded`, no row applied, so the delivery stayed
`held` with `held_why` unchanged, and its walk shows "superseded … openrazer planned there again". Two of
are only `go` and `release`. When the rollout record said `superseded`, no row applied, so the delivery stayed
`held` with `held_why` unchanged, and its rollout shows "superseded … openrazer planned there again". Two of
the three later deliveries (#122, #124) went to `superseded` correctly, through the `published` row.
**What the design says.** ADR 0218 decision 3 makes a newer merge take over an older plan's unfinished
work, and was written before deliveries could be held. ADR 0239 makes a held delivery wait for a person,
and to-be 47 says that nothing of a published delivery is registered before its walk starts, so that a
and to-be 47 says that nothing of a published delivery is registered before its rollout starts, so that a
send cannot carry it early. Neither says what a supersession does to a module a hold is keeping back.
**Ruled out.**
- A person's act. #125's transitions end at `hold`, with no `release`, and its own walk was superseded
- A person's act. #125's transitions end at `hold`, with no `release`, and its own rollout was superseded
while still waiting (`let_go` empty), so no `plans go` started it.
- Issue 295's path, a recorded build carried by a send: here the build was asked by a plan, not left
recorded.
**Located** in the controller's `supersededBy`, which folds a held walk's modules into a newer plan, and in
**Located** in the controller's `supersededBy`, which folds a held rollout's modules into a newer plan, and in
mesh-delivery's table, which has no way out of `held` when that happens, so the delivery's state stops
being true.
@@ -2,11 +2,11 @@
opened: 2026-10-08
---
# 318. One module's wait for a new login held every other module's walk
# 318. One module's wait for a new login held every other module's rollout
## Symptom
On 2026-10-08 at 11:08 UTC the controller opened its walk of the builds waiting for a gate: the laptop
On 2026-10-08 at 11:08 UTC the controller opened its rollout of the builds waiting for a gate: the laptop
first, then the workstation, then the control node. On the laptop it sent three moves in one send and
judged them together:
@@ -23,14 +23,14 @@ account was *relogin needed* from 11:10:58 UTC, which
ADR 0252
expects until the operator logs out and in again.
Meanwhile three other walks could not send to the rest of the mesh. `mesh-controller.plans` said of the
Meanwhile three other rollouts could not send to the rest of the mesh. `mesh-controller.plans` said of the
tool runner's (`novox/mesh-tools` d4845f7c, `node-tools`), the catalogue's (86077780, `docker`) and the
controller's (950562af, `build-agent`): "a build waits there for its gate: … the workstation (…, openrazer
88135ad0 → c78b5fc9) — a walk sends them, one machine at a time, each judged".
88135ad0 → c78b5fc9) — a rollout sends them, one machine at a time, each judged".
At 11:21:06 UTC the laptop's gate failed: "not healthy within 10m0s of its apply: its unit
openrazer.daemon … is unhealthy", and `openrazer` was put back to 88135ad0 on the laptop
(`build.openrazer.<laptop>.rolled-back`). The walk ended `failed`. After it, all three walks still said
(`build.openrazer.<laptop>.rolled-back`). The rollout ended `failed`. After it, all three rollouts still said
the same: `openrazer` c78b5fc9 waits on the workstation for a gate, so nothing is sent there or to the
machines after it.
@@ -40,14 +40,14 @@ The wait was known, and it was a person's: ADR 0252 says a new login holds a del
that machine. It held every module on every machine after it instead, for changes that had nothing to do
with it: the tool runner, the container engine and the build agent. The end was worse than the wait. The
gate's bound turned "waiting for a person" into "failed", the build that added the account group was put
back, and the walk that carries the other modules failed with it. A failed walk of the waiting builds
back, and the rollout that carries the other modules failed with it. A failed rollout of the waiting builds
does not open again on its own: a person releases it.
Three things meet here, each sound alone:
- a send carries every move waiting on its machine, and the gate judges them all at once
(ADR 0236);
- no other send may go where a move waits for a gate, so every walk waits for the one that judges it;
- no other send may go where a move waits for a gate, so every rollout waits for the one that judges it;
- a module's health that waits for a person reads as *not yet healthy*, which fails at the gate's bound.
## Open questions
@@ -61,5 +61,5 @@ Three things meet here, each sound alone:
back the account group it added, so the login the gate waited for can no longer help.
4. The daemon's unit had failed since 10:13 UTC, almost an hour before the new build reached the laptop.
Should a gate fail a build for a failure that was there before it was sent?
5. While a move waits for a gate on one machine, should walks of unrelated modules still be refused there,
5. While a move waits for a gate on one machine, should rollouts of unrelated modules still be refused there,
or only those whose send would carry it?
@@ -1,6 +1,6 @@
# 318 — diagnosis
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the walk of the waiting builds by id) between
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the rollout of the waiting builds by id) between
11:17 and 11:22 UTC, `mesh-controller.conditions` and `status`, and the controller's code on its main
branch (`cmd/mesh-controller`).
@@ -9,15 +9,15 @@ gate all of them as carried moves. `judgeMoves` (gate.go) reads each module on e
worst reading. Any module *not yet healthy* resets the passes to zero for the whole send: `docker` and
`node-tools` were healthy, and the gate still said "0 of 3 passes" because `openrazer` was not. Past
`gateBound` (ten minutes after the apply) a *not yet healthy* gate fails. `g.Failing` names only the
modules found wanting, so only `openrazer` was put back, but the walk itself ended `failed`.
modules found wanting, so only `openrazer` was put back, but the rollout itself ended `failed`.
**Everything else waits for that verdict.** `ungatedIn` (release.go) refuses any send that is not a person's
and would carry a move no gate has seen, outside the send's own scope. A plan's sends to the rest of its
machines carry the machine's whole declaration, so while `openrazer` c78b5fc9 waited for a gate on the
workstation, the tool runner's, the catalogue's and the controller's walks were each refused there, and
they say so on every retry. They do not fail; they wait for the walk of the waiting builds, which goes one
workstation, the tool runner's, the catalogue's and the controller's rollouts were each refused there, and
they say so on every retry. They do not fail; they wait for the rollout of the waiting builds, which goes one
machine at a time and had stopped on the laptop. After it failed, the release code's rule applies: a
failed walk of the waiting builds stops the next one opening on its own until a person releases it.
failed rollout of the waiting builds stops the next one opening on its own until a person releases it.
**How *relogin needed* reached the gate.** ADR 0252 decision 5 says the node-engine states the account's
health as the module's resource of kind `account`, unhealthy with *relogin needed*, and that "the
@@ -33,8 +33,8 @@ running session lacks the group the daemon checks for. That is a guess, not chec
readings existed before or because of the person's step, and neither is a fault the build brought.
**What the design says.** ADR 0252's consequences say "a new login holds a delivery of that module on that
machine" and accept that a walk that starts on a workstation waits for the person. They do not consider the
other modules carried in the same send, the walks refused behind it, or the gate's ten-minute bound turning
machine" and accept that a rollout that starts on a workstation waits for the person. They do not consider the
other modules carried in the same send, the rollouts refused behind it, or the gate's ten-minute bound turning
the wait into a rollback. ADR 0236 makes one send judge everything waiting on a machine, to stop unjudged
builds from moving; it does not distinguish a module waiting for a person from one that is broken.
@@ -50,15 +50,15 @@ Whether the account actually left the group was not checked.
- A machine-wide fault on the laptop: its node-engine reported `applied` and current, and the gate's
wanting named one module's unit.
**Located** in the controller's walk and first-node gate: one verdict per send (`judgeMoves`), sends refused
**Located** in the controller's rollout and first-node gate: one verdict per send (`judgeMoves`), sends refused
on any machine where a move waits (`ungatedIn`), and no reading for a health that waits for a person, so a
known wait fails at the bound and stops every walk behind it.
known wait fails at the bound and stops every rollout behind it.
**Decided and fixed, 2026-10-08**, by ADR 0254.
Open questions 1 to 3: *relogin needed*, and a unit failing in that same account's own manager, read *waits
for a person*, which passes with the wait carried and is said as the module's condition `relogin-needed`;
nothing is put back for it. Question 2's other half: each module of a send counts its own passes, and, after
the settle time, in the walk path, keeps a pass when another fails beside it. Question 5: other walks are
the settle time, in the rollout path, keeps a pass when another fails beside it. Question 5: other rollouts are
still refused while a move waits for a gate (ADR 0236 §4a stands). With the wait a pass, 318's case leaves
nothing waiting for them. A fault decided before the settle time, and a plan's own tier path, still leave
the healthy modules of a failed send without a verdict (ADR 0254, consequences). Question 4, a unit that
@@ -71,4 +71,4 @@ between 12:41 and 13:03 UTC; the node-engine's and the controller's deliveries r
**Seen live.** On 2026-10-09 `mesh-controller.conditions` holds `module.openrazer.<machine>.relogin-needed`
for the laptop and the workstation, raised at 21:02 UTC: "Its update is in place and nothing was undone",
and no walk waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
and no rollout waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
@@ -15,8 +15,8 @@ What they are, read with `mesh-delivery.deliveries` and `show`:
| how many | repository | why held | what it builds |
|---|---|---|---|
| 4 | hq | merged without a passing check | nothing: "the change touches no module of the mesh's graph" |
| 2 | catalogue | no walk opened within ten minutes of the merge | a new module, sent nowhere |
| 8 | controller | no walk opened within ten minutes of the merge | the controller, the build agent, the route proxy |
| 2 | catalogue | no rollout opened within ten minutes of the merge | a new module, sent nowhere |
| 8 | controller | no rollout opened within ten minutes of the merge | the controller, the build agent, the route proxy |
| 2 | catalogue | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | one module each |
Two examples:
@@ -26,7 +26,7 @@ Two examples:
`proposed` to `held`: "merged without a passing check". Its plan builds nothing.
- `novox/mesh-controller@ec3769a8a669`, controller #105. Ready at 2026-10-07 00:26 UTC, merged at 00:27,
held at 00:37: "merged, and the controller opened no walk for it within 10m0s". The controller's modules
have been walked since from later commits of the same main, for example the group
have been rolled out since from later commits of the same main, for example the group
`feat/delivery-checks-verb`, whose controller member is `delivered`.
Five of them (the four hq merges and controller #105) are past the held bound of one day. The self-check's
@@ -39,7 +39,7 @@ A held delivery asks a person "does this go on?". For these sixteen the question
a later commit of the same trunk has already delivered what they held, or they hold nothing to deliver. The
state table offers a person four ways out of `held`, and none fits:
- `release` starts a walk of the old commit: wrong for a commit a later one superseded, and nothing for one
- `release` starts a rollout of the old commit: wrong for a commit a later one superseded, and nothing for one
that builds nothing.
- `retire-history` is for history made from the forge's word (ADR 0250),
and refuses every one of these because their heads were heard.
@@ -57,7 +57,7 @@ The issue states the need; the decision is for graduation.
- **An end for a held delivery that was superseded**, as `published` and `delivering` already have
(`superseded`, "a newer delivery to the same trunk took over its walk"): either because a later commit's
delivered walk carried the same modules on the same trunk, or because it builds nothing at all.
delivered rollout carried the same modules on the same trunk, or because it builds nothing at all.
- **Automatic where nothing is lost**: a held delivery whose plan builds nothing has nothing for a person to
decide.
- **A person's act, with why, otherwise**: a held delivery whose modules a later commit delivered is ended
@@ -70,5 +70,5 @@ The issue states the need; the decision is for graduation.
2. The four hq merges were heard at the owner's first start from the bus's history, as announcements rather
than merges. Are they history in the sense of ADR 0250, missed by its rule because their head was
replayed too?
3. Why did the controller open no walk for eight of its own merges within ten minutes? The controller's own
path never waits, so its walk may have been missed rather than absent. Not diagnosed here.
3. Why did the controller open no rollout for eight of its own merges within ten minutes? The controller's own
path never waits, so its rollout may have been missed rather than absent. Not diagnosed here.
@@ -7,14 +7,14 @@
| event | kind | needs |
|---|---|---|
| `go` | observed | its walk started on a word not this owner's (the controller's own path, or a person's `plans go`) |
| `release` | a person's act | why and who; it walks the delivery's own commit |
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no walk open |
| `go` | observed | its rollout started on a word not this owner's (the controller's own path, or a person's `plans go`) |
| `release` | a person's act | why and who; it rolls out the delivery's own commit |
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no rollout open |
| `stop` | a person's act | why and who; from any state not final |
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the walk's
record saying `superseded`. A held delivery whose walk never opened has no walk record to read, and one
whose walk was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the rollout's
record saying `superseded`. A held delivery whose rollout never opened has no rollout record to read, and one
whose rollout was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
does not match it either.
**Why `retire-history` refuses them.** `isHistory` requires `FromForgesWord()`: the delivery was made from
@@ -41,7 +41,7 @@ Neither considers a held delivery whose content a later commit delivered.
**Ruled out.**
- A missing release. Releasing any of these would walk an old commit over a newer one, or walk nothing.
- A missing release. Releasing any of these would roll out an old commit over a newer one, or roll out nothing.
- D14 itself. It reports the state truthfully; the state has no way to change.
**Located** in mesh-delivery's state table (no end for a superseded or empty held delivery, and history
@@ -52,14 +52,14 @@ read by how a delivery was made) and in to-be 47, whose state table carries the
**Eleven held deliveries, read again.** On 2026-10-09 eleven deliveries were `held`, each raising D14's
`delivery.<id>.stalled` and held for 24 to 48 hours. For each, `mesh-delivery.show` gave its state, why it was
held, its merge commit and its plan. A fresh clone of the repository then showed whether the merge commit is an
ancestor of the commit a finished walk delivered. `mesh-controller.status` listed none of the modules involved
ancestor of the commit a finished rollout delivered. `mesh-controller.status` listed none of the modules involved
as behind. All eleven were covered: none held a change missing where it should run.
**Why the controller opened no walk (open question 3).** This part comes from reading the code, and part of it
**Why the controller opened no rollout (open question 3).** This part comes from reading the code, and part of it
is still a guess.
- *A new module.* When a merge only adds a module, the controller's `SourceMoved`
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no walk ("built,
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no rollout ("built,
registered when the build lands, and sent nowhere", issue 300). mesh-delivery's `MovesNothing()`
(`model.go`) counts `Plan.New` as something to move, though. So a delivery whose plan is "builds new: …, sent
nowhere" is not let through to `published` → `delivered`. Instead the unwalked rule in `holder.go` holds it
@@ -67,25 +67,25 @@ is still a guess.
read from the code, and it matches `novox/mesh-catalog@60641176abaf`.
- *The controller's own merges (a guess, not proved).* `SourceMoved` also returns without a plan in two cases:
when the modules the merge touches are "already built from it" (`already > 0`), and when the merge is read as
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no walk
of its own when another walk, or a person's push, had already built from a later commit. Which of the two hit
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no rollout
of its own when another rollout, or a person's push, had already built from a later commit. Which of the two hit
these merges is not yet known. The controller's journal for the minutes after each merge would say.
- *The link from a walk to a delivery.* `deliveryWalking()` (`holder.go`) links a walk to the delivery whose
`merged_as` equals the walk's commit, and to no other delivery. A later walk that carried an earlier merge's
change therefore never reaches that earlier delivery, and nothing reads "this walk's commit contains your
- *The link from a rollout to a delivery.* `deliveryWalking()` (`holder.go`) links a rollout to the delivery whose
`merged_as` equals the rollout's commit, and to no other delivery. A later rollout that carried an earlier merge's
change therefore never reaches that earlier delivery, and nothing reads "this rollout's commit contains your
merge". That is the missing input the first open question asks about. mesh-delivery could answer it from the
forge (is the merge commit an ancestor of the delivered walk's commit) together with the walk's moved modules,
forge (is the merge commit an ancestor of the delivered rollout's commit) together with the rollout's moved modules,
without asking the controller what each machine runs.
**Four were closed with `stop`, before this issue was found.** On 2026-10-09 the operator agreed to close the
held deliveries. Four were stopped with why "covered: its merge … is contained in … which is delivered (walk …)":
| delivery | merge | contained in | walk |
| delivery | merge | contained in | rollout |
|---|---|---|---|
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier walk of the controller |
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same walk |
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same walk |
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue walk |
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier rollout of the controller |
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same rollout |
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same rollout |
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue rollout |
Their stalled conditions cleared at the next D14 observation. Open D14 conditions went from 11 to 7. As this
issue already said, `stop` is the wrong record for them: their merged pull requests now carry `mesh/delivery`
@@ -101,12 +101,12 @@ remaining seven were left `held` for this issue's fix.
| `novox/hq@2221be11a6ef` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/hq@897bcce50252` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/hq@ef6ab814b053` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/mesh-controller@b98fd0f396c8` | no walk opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
| `novox/mesh-controller@ea92af2a7d77` | no walk opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release walk that is done; the step looks stale but is not final |
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished walk of the same module | its group is still `delivering`, and its own walk is `superseded` |
| `novox/mesh-controller@b98fd0f396c8` | no rollout opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
| `novox/mesh-controller@ea92af2a7d77` | no rollout opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release rollout that is done; the step looks stale but is not final |
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished rollout of the same module | its group is still `delivering`, and its own rollout is `superseded` |
The seven show the two ends this issue asks for: four hold nothing to deliver, and three are covered by a
later walk.
later rollout.
## 2026-10-09 — the fix, proposed
@@ -115,12 +115,12 @@ ADR 0270
mesh-delivery:
- **A merge that moves nothing is delivered at once.** `MovesNothing()` no longer counts `Plan.New`: a module
the controller builds and sends nowhere opens no walk, so there is nothing to wait for. A merge whose plan moves
the controller builds and sends nowhere opens no rollout, so there is nothing to wait for. A merge whose plan moves
nothing goes from `proposed`, `checked`, `rejected` or `held` to `delivered`, before the row *merged unchecked
→ held*.
- **`covered`, a new final state: a later merge that contains it was delivered.** For every module its plan
moves, a delivered delivery of the same repository and trunk, merged after it, delivered on its own passing
check by the walk of its own merge commit, whose machine steps say the module reached every machine running
check by the rollout of its own merge commit, whose machine steps say the module reached every machine running
it. The forge then says, through a new read-only tool `gitea_contains` (its compare of the two commits),
whether the later merge commit contains this one; merge order is only the first filter. A forge that cannot
answer leaves it held. `mesh/delivery` says `covered: a later merge that contains it was delivered: …` as a
@@ -128,7 +128,7 @@ mesh-delivery:
- **Read on every tick**, so what is held now ends without an event.
This answers open question 1: mesh-delivery reads it from its own records and asks the forge for ancestry; the
controller is not asked. Open question 3, why the controller's own merges opened no walk, is not answered by this fix; it no
controller is not asked. Open question 3, why the controller's own merges opened no rollout, is not answered by this fix; it no
longer leaves anything held.
**How to verify after mesh-delivery is delivered with it.** Within a minute of its start:
@@ -137,13 +137,13 @@ longer leaves anything held.
merges (`builds new: …, sent nowhere`), and none of the five controller merges held for "no walk opened".
The hq and module-adding ones are `delivered`; the controller ones are `covered`.
2. `mesh-delivery.show` of one of them, for example `novox/mesh-controller@b98fd0f396c8`, ends with a
transition `held → covered (covered)` whose why names the later delivery and its walk.
transition `held → covered (covered)` whose why names the later delivery and its rollout.
3. `mesh-delivery.checks` for its pull request shows `mesh/delivery` as `success covered: …`.
4. The forge module is delivered in the same pull request. Until its `gitea_contains` answers, nothing is
covered, and the held controller merges stay held.
5. At the self-check's next reading, `mesh-controller.conditions` no longer lists their
`delivery.<id>.stalled`.
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered walk sent
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered rollout sent
their module. One that stays held is a real question: its change may not run anywhere.
**Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed.
@@ -52,7 +52,7 @@ not addresses, paths or secrets — so they reached the operator untouched.
**Nothing said whether the operator was needed, or let them answer.** No field of a condition said whether
the operator was needed, so every warning was shown, and an explanation could imply an instruction. The
verbs that answer the common cases existed (release or stop a delivery, start a waiting walk, restart a
verbs that answer the common cases existed (release or stop a delivery, start a waiting rollout, restart a
service, silence a condition), but the desk could not take an answer: `node-notifier.send` had no actions,
and the conversation design that adds them (to-be 46) was not built. The one way to act was a session.
@@ -70,7 +70,7 @@ nothing is not sent. The desk offers the actions, and the operator channel's hol
through its seat verb. Every channel says the time in the operator's zone; a clearance is one line.
- novox/mesh-controller: the condition's three new fields, the plain rule, a wording for every kind the
controller raises, the richer words at the producers that know more (a walk's modules, a unit's name, a
controller raises, the richer words at the producers that know more (a rollout's modules, a unit's name, a
hand-act's cause, a delivery's repository), and a lint over every condition the suite raises.
- novox/mesh-catalog: the messenger shows the words, keeps quiet what needs nothing, offers and performs
the answers, says since when in the operator's zone (the new setting `time-zone`), says a clearance in
@@ -88,5 +88,5 @@ To-be 45 §2 is amended: what `status` lists, and the new condition.
## Resolved, 2026-10-09
mesh-controller#150 and mesh-lab#68 (R325) merged on 2026-10-08 at 14:58 UTC. Every later controller
delivery carries #150; the last, of mesh-controller#170, finished its walk on 2026-10-09 at 00:11 UTC, and
delivery carries #150; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at 00:11 UTC, and
`mesh-controller.status` named no machine behind.
@@ -20,7 +20,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
1.4 a second, or about 43 every 30 seconds.
- Over the server's whole uptime (39 hours 49 minutes at 14:13 UTC) the average is 39.7 writes a second.
In that window one walk was in progress, so a few of those writes were real changes. The diagnosis
In that window one rollout was in progress, so a few of those writes were real changes. The diagnosis
shows why most were not.
## Why it is an issue
@@ -40,7 +40,7 @@ watcher, and the write rate cannot be used to see that deliveries are moving.
## How it is checked
- **In mesh-delivery's tests:** a reconcile pass over walks that did not change performs no write. The
- **In mesh-delivery's tests:** a reconcile pass over rollouts that did not change performs no write. The
test's fake store counts writes, and the test fails on any write.
- **Live:** over ten minutes in which no delivery changes state, the write count on
`$KV.mesh-delivery_deliveries.>` (`nats.nats_subscriptions`) rises by no more than the number of
@@ -56,6 +56,6 @@ build agent. So it resolves without a replay. The test above is still the proof.
- `keep` writes only when the delivery differs from the value it last wrote. Either compare the encoded
value with the last one written, or set a changed flag wherever the delivery is changed.
- `Reconcile` skips a walk whose revision the delivery already holds.
- `Reconcile` skips a rollout whose revision the delivery already holds.
- A retry of an owed effect that fails again changes only its try count and next time. Whether that
needs a write at every try is a choice for the fix: the count may stay in memory between writes.
@@ -12,16 +12,16 @@ runtime. `keep` does not compare the delivery with what it wrote before. Every c
of transitions is empty.
- `settleAndKeep` calls `settle` and passes the result to `moved`. A delivery with nothing to move is
still written.
- `walkMoved` takes one walk from the controller, records it on its delivery (`linkWalk`) and calls
`settleAndKeep`. A walk that has not changed since the last pass leaves the delivery as it was, and
- `walkMoved` takes one rollout from the controller, records it on its delivery (`linkWalk`) and calls
`settleAndKeep`. A rollout that has not changed since the last pass leaves the delivery as it was, and
the delivery is written anyway.
- `Reconcile` asks the controller's `delivery-walks` for the walks it keeps and calls `walkMoved` for each.
The answer holds every open walk and the last 50 that ended. The ticker in main.go runs `Reconcile`
- `Reconcile` asks the controller's `delivery-walks` for the rollouts it keeps and calls `walkMoved` for each.
The answer holds every open rollout and the last 50 that ended. The ticker in main.go runs `Reconcile`
every 30 seconds.
So each pass writes about one delivery per walk the controller returns. On 2026-10-08 that was 50 walks
(32 done, 14 superseded, 3 failed, 1 walking), against about 43 writes per pass measured. *I think* the
difference is walks that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
So each pass writes about one delivery per rollout the controller returns. On 2026-10-08 that was 50 rollouts
(32 done, 14 superseded, 3 failed, 1 `walking`), against about 43 writes per pass measured. *I think* the
difference is rollouts that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
**What made it 40 a second before today.** Before mesh-catalog #121 (the fix of
[issue 313](../313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md), merged
@@ -39,5 +39,5 @@ doubling to an hour) and ends them after 60 tries. A retry still writes its deli
- *The groups bucket.* `mesh-delivery_groups` has 26 keys, and its subscription counted 719 writes in the
same 39 hours.
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every walk the
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every rollout the
controller keeps through `settleAndKeep` every 30 seconds.
@@ -24,7 +24,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
The long-lived connections were young too. At 14:15 UTC the controller's two were 4 minutes old, the
build agents' 2 to 19 minutes, the route proxies' 13 to 17 minutes and the tool runners' 10 minutes to 3
hours. Two walks were sending new builds of the controller, the build agent and catalogue modules
hours. Two rollouts were sending new builds of the controller, the build agent and catalogue modules
between 14:00 and 14:20 UTC, and the build agent's container on the control node started at 14:19 UTC.
*I think* those young connections are restarts by deliveries; they are tens, not thousands.
@@ -17,7 +17,7 @@ controller's, so each place that needs the bus opens one, uses it and closes it:
command.
- `sayingOnTheBus` (delivery.go): a delivery's `check`, `go` and `stop`, which mesh-delivery asks for.
The serving controller dials too. Its watchdogs run every 30 seconds, and while a walk waits for a
The serving controller dials too. Its watchdogs run every 30 seconds, and while a rollout waits for a
build, `buildSeatPause` (plan_retry.go) opens a fresh connection to read whether the build seat's
holders paused, and closes it. `status` does the same.
@@ -41,7 +41,7 @@ its long-lived connections are named `mesh-controller` too and are told apart on
minute. The research's guess came from their age alone.
**Located** in the controller: one connection per command process that touches the bus, and one per
watchdog tick while a walk waits for a build.
watchdog tick while a rollout waits for a build.
## 2026-10-09, resolved
@@ -17,7 +17,7 @@ the fix is the attribution tool; with it, this diagnosis continues.
- **A holder that never said it paused.** `PausedSaid` (controller, internal/link/queue.go) asks for the
last message on each build holder's pause subject. A holder that never said one gets a not-found
answer, which the server counts as an API error. The serving controller's watchdog reads it every 30
seconds while a walk waits for a build (issue 327).
seconds while a rollout waits for a build (issue 327).
- **Short-lived consumers.** The controller heard 495 "consumer deleted" advisories in the first 8
minutes of its current connection, about one a second. They are short-lived consumers: the controller's
advisory reader (internal/link/advisories.go) ignores them and says why, "every watch of a bucket and
@@ -7,12 +7,12 @@ opened: 2026-10-08
## What was observed
On 2026-10-08 a person released a delivery of the catalogue that the group check (issue 316) had held. Its
walk had one tier with one module, `dunst`, which runs on the laptop and the workstation. The walk sent the
rollout had one tier with one module, `dunst`, which runs on the laptop and the workstation. The rollout sent the
new build to the laptop first, and the gate passed it there:
> dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent
Twenty-eight minutes later the same walk said the opposite, about the same gate:
Twenty-eight minutes later the same rollout said the opposite, about the same gate:
> dunst failed its gate on the laptop in tier 0: healthy 3 times over 2m5s; put back dunst to ea462666 on
> the laptop, in one send; the workstation left as it was
@@ -31,22 +31,22 @@ build that worked, recorded it as failed, and so blocked its delivery until a ne
## Fix
The controller judged the first machine's report again on every step of the walk, also after the gate had
The controller judged the first machine's report again on every step of the rollout, also after the gate had
passed (see `01-diagnosis.md`). The fix merged in mesh-controller pull request 165 (8170fc5), with its
replay in mesh-lab pull request 72.
- Once a module's gate has passed, the first machine's later reports no longer decide anything about it:
the walk sends the rest.
the rollout sends the rest.
- A module carried by another module's send takes over that pass before its step is read.
- A guard, not reached with the two changes above: should a later path still stop a walk at a module whose
gate passed, that module is not put back and its build is not marked failed. The walk stops with the
- A guard, not reached with the two changes above: should a later path still stop a rollout at a module whose
gate passed, that module is not put back and its build is not marked failed. The rollout stops with the
reason, and the module is left sent to its first node only, which `plans retry` takes up again.
## How it is checked
`TestReplay335` in the controller replays the live case with only what the controller had before the fix:
a passed gate on the laptop, the laptop's later report not about this build (or a failure of another
send), and the walk 30 minutes and 9 seconds after the first send. A second test drives the walk over a
send), and the rollout 30 minutes and 9 seconds after the first send. A second test drives the rollout over a
tier of two modules, the second carried by the first's send, and fails if the carried module's step is read
before it takes over the pass. On the commit before the fix it fails
with "laptop did not report it applied within 30m0s", and the passed build reaches the put-back. It is R335
@@ -6,15 +6,15 @@
found ADR 0236 (what a rollback is), issue 318 (a wait for a login failing a gate) and issue 279, none of
them about a gate that passed and was then failed.
**The walk's own lines.** The controller's journal on the control-node said, for this walk (times UTC):
**The rollout's own lines.** The controller's journal on the control-node said, for this rollout (times UTC):
| time | what the controller said |
|---|---|
| 16:44:45 | tier 0 built; sent dunst to the laptop first, the rest once its gate passes |
| 16:44:55 | the laptop applied what it was sent |
| 16:46:52 | dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent |
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another walk sends |
| 16:49:12 | the other walk (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another rollout sends |
| 16:49:12 | the other rollout (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
| 17:14:54 | dunst failed its gate on the laptop: healthy 3 times over 2m5s; put back on the laptop; the workstation left as it was |
The laptop's node-engine logged `applied` at 16:44:55 and nothing more until 18:01:58. It did not report
@@ -25,14 +25,14 @@ on the 16:49:12 send for over an hour. Why it did not is not yet known; the lapt
waits 30 minutes (`planWaitBound`) for a first machine to report what it was sent. The failure came at that
bound and not at the gate's.
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a walk reads
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a rollout reads
`nextRollout(state, …)` before anything else, for every module in the tier not yet sent to the rest. For a
module sent first, `nextRollout` looks at the first machine's last report. If that report is not about the
declaration the machine was last sent (`Current` false), it waits; past the bound it answers
`failed: <machine> did not report it applied within 30m0s`. Nothing in it looked at the gate's verdict.
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the walk tries again on
the next step). At 16:49:12 the other walk's send made the laptop's last report no longer current. From
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the rollout tries again on
the next step). At 16:49:12 the other rollout's send made the laptop's last report no longer current. From
then on `nextRollout` waited for a report it would never get about this build. At the bound it failed, and
`advanceOnce` called `failFirstSend`.
@@ -47,7 +47,7 @@ laptop. That is the line seen at 17:14:54.
- The person-wait of issue 318 (ADR 0254), `putBackBroken` and the carried-module paths: the plan had one
module, nothing was carried, and nothing was found broken.
- Sleep: the laptop did not sleep until 18:01, and the failure was at 17:14:54.
- The release itself (`mesh-delivery.release`): it let the walk go at 16:44, and the walk ran as any other.
- The release itself (`mesh-delivery.release`): it let the rollout go at 16:44, and the rollout ran as any other.
**Located.** mesh-controller, `cmd/mesh-controller/release_plan.go`: `nextRollout` judged a passed gate's
first machine again, and `failFirstSend` put back a module whose gate had passed. The path has been there
@@ -57,4 +57,4 @@ step as the pass and another send reaches the first machine before the 30-minute
**What it needs to deliver dunst.** The build is recorded as failed at its gate, and the controller does
not send a build marked that way again on its own. The registered build is back at the one before. Once the
fix is delivered, a new build of dunst (`rebuild dunst`, or a newer merge of it) is judged at the gate
again and walked to both machines.
again and rolled out to both machines.
@@ -20,10 +20,10 @@ From the merge on, every send to the control-node was held. `mesh-controller.pla
> planned step: sending control-node would replace the bus (nats 88135ad0 → 32307bd1), which is a planned
> step ... — tried again for 27m45s
The same refusal held the controller's own walk and the release of builds kept back for a gate.
The same refusal held the controller's own rollout and the release of builds kept back for a gate.
For those 28 minutes `mesh-controller.conditions` listed **no condition** naming the hold or saying it
waited for a person. The operator was not told. The hold was found only because a walk looked stuck, and
waited for a person. The operator was not told. The hold was found only because a rollout looked stuck, and
the operator asked: "where would you be waiting then? Where is my approve button?"
The operator then gave the word, the bus upgrade started at 18:52 and the bus answered healthy at 18:54.
@@ -33,7 +33,7 @@ The held sends then went through.
The design makes the bus wait for a person, and that part worked: nothing replaced the bus unasked. But a
wait that only a person can end is said nowhere a person looks. The refusal is kept as the note on each
held walk, which is read only by somebody already asking `plans`. Rule 4 of
held rollout, which is read only by somebody already asking `plans`. Rule 4 of
ADR 0227
(no failure is silent) is met in the letter and missed in effect: the mesh held every send to its
most important machine and said nothing until asked.
@@ -47,10 +47,10 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
## Impact
- Every send to the control-node stops: the controller's own walk, the node-engine's, every module's walk
- Every send to the control-node stops: the controller's own rollout, the node-engine's, every module's rollout
that includes the control-node, and gated builds waiting for release. Other machines in the same tier
wait with it.
- The operator learns of it only by noticing a stuck walk, which can take hours when nobody is watching.
- The operator learns of it only by noticing a stuck rollout, which can take hours when nobody is watching.
- The only condition that could have fired, a tier past its bound (watchdog S3, "stalled"), fires no
sooner than 30 minutes in, and its words ("stuck halfway ... longer than usual. Nothing is lost") name
neither the bus nor the verb a person runs. It never fired in this case, because the step was taken
@@ -62,9 +62,9 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
| Rule | Check |
|---|---|
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a walk wants to send raises a condition on the next watchdog tick, before any tier bound |
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held walks or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a walk whose only refusal is the bus's planned step |
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a rollout wants to send raises a condition on the next watchdog tick, before any tier bound |
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held rollouts or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a rollout whose only refusal is the bus's planned step |
| it clears when the step starts | the test ends with a bus step started: the condition clears, `bus-maintenance` opens |
| the merge says it before it happens | the merge gate's test: a change that moves the bus's build reports that a bus upgrade by a person is needed and which machine waits until then |
@@ -81,8 +81,8 @@ why none is possible.
with the operator as its resolver). Its headline says, in plain words, what waits ("sends to the
control-node: the controller, the node-engine, …"), behind what (the bus build, from and to), since
when, and the verb that ends it (`mesh-controller.bus upgrade --why … --reversible|--irreversible`). It
is raised on the first refusal, not after a bound. A walk held only by it is excluded from S3, as a
walk held by a paused build seat is.
is raised on the first refusal, not after a bound. A rollout held only by it is excluded from S3, as a
rollout held by a paused build seat is.
2. **An ask, once asks exist.** When the decision flow of ADR 0259 (an ask is a message on the bus, any
channel answers it; on an open pull request, not yet merged) is built, the condition becomes an ask
to the operator: "start the bus upgrade now?", whose yes runs the step with the operator's word
@@ -13,13 +13,13 @@ The send is refused whole. `busHeld` (`bus_step.go`) gives the words "sending X
(nats from → to), which is a planned step: `bus upgrade --why …` …". So the remedy is in the words, but
only in the error.
**Where the error goes.** In `release_plan.go`, the walk's step loop catches the error, keeps it as the
**Where the error goes.** In `release_plan.go`, the rollout's step loop catches the error, keeps it as the
plan's note ("tier N: … — tried again"), saves the plan when the note changed, prints it to the
controller's log once, and tries again on the next tick. Nothing raises a condition from it. That is why
`plans` showed it and `conditions` did not.
**Which watchdog could have said it.** Only S3 (`watchPlans`, `signals.go`), "plan tier progress", raises
`plan.<id>.stalled` when a walk has been in one tier past its bound. The bound is
`plan.<id>.stalled` when a rollout has been in one tier past its bound. The bound is
`max(30 min, 3 × the p90 of that repository's measured tiers)` (`tierAtLeast = planWaitBound = 30 min`,
`watchdogs.go` `boundsOf`). So:
@@ -29,14 +29,14 @@ controller's log once, and tries again on the next tick. Nothing raises a condit
mesh keeps it where it is."). Its summary carries the plan's note, so the bus's reason would have
appeared there, but the headline a person reads names neither the bus nor the verb, and its resolver is
not the operator;
- a walk held by a paused build seat is excluded from S3 (`pausedWaiting`), and a walk waiting for its
delivery's word has its own watchdog (S16). A walk held for the bus step has neither.
- a rollout held by a paused build seat is excluded from S3 (`pausedWaiting`), and a rollout waiting for its
delivery's word has its own watchdog (S16). A rollout held for the bus step has neither.
**What the bus step does raise.** Probe DB (`probeBusStep`, `bus_step.go`) raises `bus-maintenance` only
while a step runs, and `bus-upgrade-failed` when one ended unhealthy. It reads only the latest bus step.
Before a step is started there is nothing for it to read, so a pending bus build raises nothing.
**Before the merge.** `delivery.go` lists the bus among the modules whose walk starts at the merge
**Before the merge.** `delivery.go` lists the bus among the modules whose rollout starts at the merge
(`onTheControllersPath`), and the merge gate (`merge_gate.go`) composes the machines with the change. Neither
reads `pendingBus` or says that the change will hold the bus's machine until a person runs the step. So a
merge that moves the bus's build passes green and blocks every later send to the control-node.
@@ -64,12 +64,12 @@ person, and the person must be told.
ADR 0271
(proposed) takes suggested fixes 1 and 3. mesh-controller #173 builds them:
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a walk's send was refused
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a rollout's send was refused
for the bus's planned step, while the new bus build would still replace the bus on its machine. It is a
warning, for the operator. It names the machines, each waiting walk and the modules of its tier, the bus build
warning, for the operator. It names the machines, each waiting rollout and the modules of its tier, the bus build
from and to, since when, and the `mesh-controller.bus` call with `upgrade`. It clears once the bus's machine has
been sent the build the mesh holds.
- **S3 leaves out a walk held only by it.**
- **S3 leaves out a rollout held only by it.**
- **The delivery plan says it before the merge**: "merging this needs a person's bus upgrade: … nothing else is
sent to <machine> either", and its summary, which `mesh/merge-gate` carries, says the same.
@@ -20,7 +20,7 @@ The plan rebuilt and moved them whatever the merge changed. Two examples:
- Mesh-controller PR #159, merged as `ca09a07f`, changed files under `cmd/mesh-controller`,
`internal/broker`, `internal/link`, `internal/catalogue/verbs.go` and the root `module.json`. Nothing
under `examples/route-proxy` changed. Its walk said, on the home server, "recreates 1 of route-proxy's
under `examples/route-proxy` changed. Its rollout said, on the home server, "recreates 1 of route-proxy's
3 container(s) (with a new image: server)" and "recreates 1 of build-agent's 1 container(s)".
- Mesh-controller PR #163, merged as `9b028b42`, did the same. The route proxy's program at that commit
is byte for byte the program at the commit before it (measured in the diagnosis), and it was still
@@ -43,8 +43,8 @@ The operator stated the rule the delivery plan should follow:
## Why it is an issue
- **Deliveries are slow.** A controller merge walked for about fifteen minutes instead of about five,
and the walks of one repository must not overlap (issue 335), so every merge after it waits too. The
- **Deliveries are slow.** A controller merge rolled out for about fifteen minutes instead of about five,
and the rollouts of one repository must not overlap (issue 335), so every merge after it waits too. The
operator considers this a major bug and a main reason deliveries are slow.
- **A public service restarts for nothing.** The route proxy on the home server was recreated at
merges that did not change its program.
@@ -31,7 +31,7 @@ Read in mesh-controller at `8170fc58`.
4. `mesh-controller` itself is built from the repository's root (its manifest's bundle compiles
`cmd/mesh-controller`; its source path is empty), so every changed file in the repository touches
it as well.
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the walk serial here:
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the rollout serial here:
- a `packages` edge, from a module whose build read a repository to the module whose source that
repository is. So `route-proxy` and `build-agent` both depend on `mesh-controller`, because they
share its repository, not because either is built from the controller's artifacts;
@@ -108,9 +108,9 @@ change. The build source of the route proxy is `examples/route-proxy`, `internal
## Measured: what the extra tiers cost
From `mesh-controller.plans` (read only), the first-node gate of each tier of tonight's controller
walks:
rollouts:
| walk | mesh-controller | build-agent | route-proxy | three gates |
| rollout | mesh-controller | build-agent | route-proxy | three gates |
|---|---|---|---|---|
| `3f68a495` | 3m40s | 3m43s | 3m46s | 11m09s |
| `0cf5a3a5` | 3m39s | 3m26s | 3m34s | 10m39s |
@@ -120,12 +120,12 @@ walks:
| `ca09a07f` | 3m43s | 3m46s | 3m40s | 11m09s |
| `efcdd5dd` | 3m38s | 3m40s | 3m37s | 10m55s |
Two more walks (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still walking at
21:50. The gates alone are 10 to 11 minutes per walk, of which 7 minutes are the two packaging tiers.
Two more rollouts (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still rolling out at
21:50. The gates alone are 10 to 11 minutes per rollout, of which 7 minutes are the two packaging tiers.
Each tier also builds before it sends: the build seat's image took 53 seconds at 21:36. Estimated from
the plan's creation time and the build records, `ca09a07f` walked from 21:07 to about 21:23 and
the plan's creation time and the build records, `ca09a07f` rolled out from 21:07 to about 21:23 and
`efcdd5dd` from 21:32 to about 21:47, about 15 minutes each. The controller's own tier, alone, is about
4 to 5 minutes. Of the seven completed walks, five recreated `route-proxy` on the home server
4 to 5 minutes. Of the seven completed rollouts, five recreated `route-proxy` on the home server
(`3f68a495`, `e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`), and its program had changed in one of
them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`).
@@ -133,7 +133,7 @@ them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07
The deploy plan names a build of a packaging module by the module's own source commit, which is the
catalogue's. `moveFacts.identical` (cmd/mesh-controller/release.go) calls two builds identical when
their commits are equal. In `efcdd5dd`'s walk both modules went `7b307a06 → 7b307a06`, nothing was
their commits are equal. In `efcdd5dd`'s rollout both modules went `7b307a06 → 7b307a06`, nothing was
said to be recreated, and the gate still took about 3.5 minutes for each. Yet the route proxy's
program did change at `efcdd5dd` (table above). **I do not know whether that program reached the home
server.** Either it was sent and the plan said nothing, or it was not sent and the route proxy runs the
@@ -146,7 +146,7 @@ commit would also make issue 280's "repeat" rule register a different program as
- **The changed paths not reaching the planner.** PR #159's delivery record carries its paths and
`module_dirs_said: true`. They are read, and `touchedBy` uses them for the controller's own module.
They are never consulted for a packaging module.
- **The catalogue moving at the same time.** In some walks the catalogue's commit also changed
- **The catalogue moving at the same time.** In some rollouts the catalogue's commit also changed
(`32307bd1 → 7b307a06`), but the README-only what-if shows the controller's repository alone is enough
to plan both modules.
@@ -181,7 +181,7 @@ the commit into the program.
with the build; that is exact for Go and needs a reader per language.
2. **Reproducible builds**: `-buildvcs=false` (or a `.dockerignore` that leaves `.git` out of the
context) in both recipes, so an unchanged source gives an identical program. This alone does not
stop the walk: the image digest still differs (issue 280), and the planner would still plan and gate
stop the rollout: the image digest still differs (issue 280), and the planner would still plan and gate
three tiers. It is cheap and worth doing beside 1, so the program's identity no longer depends on the
commit.
3. **Split the route proxy and the build seat into repositories of their own.** Then the existing
@@ -191,5 +191,5 @@ the commit into the program.
and a change to that library then moves all three again, through a real build edge.
Recommendation: 1 with its check, and 2 beside it. A further question for the fix: whether `built-by`
should order a walk at all. It is a running-order edge for the build seat's worker (issue 206); the
should order a rollout at all. It is a running-order edge for the build seat's worker (issue 206); the
operator's rule 2 counts only modules built *from* a changed module.
@@ -23,7 +23,7 @@ What happened, in order (UTC):
**under the same JSON key `declared`**. The two edits touched different lines, so git merged them
without a conflict. The result does not compile:
`internal/store/store.go:145:2: Declared redeclared`.
4. **00:55.** The walk the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
4. **00:55.** The rollout the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
in tier 0". `mesh/delivery` on the head reads `failure`. Nothing was sent to a node, and every node kept
`d28ca95`.
5. **01:09.** Node-engine #67 repaired the trunk. It renamed #66's field to `DeclaredUnit`, key
@@ -33,7 +33,7 @@ The facts behind the "different lines" claim, read from the node-engine's reposi
contain `d28ca95`. Their common ancestor is `a2c2c35` (#63's merge). `dfb1415` holds one `Declared`
(store.go line 107), `d28ca95` holds the other (line 139), and `690966d` holds both (lines 107 and 145).
No node was harmed, because a walk builds before it sends. But the merge check said "this may merge" of
No node was harmed, because a rollout builds before it sends. But the merge check said "this may merge" of
a merge that did not build, and both required statuses were green when it merged.
## The cost before this: checks that went stale
@@ -33,7 +33,7 @@ merging made, with two, was never built before the merge.
or ready delivery back to proposed. Its comment says "a person asked for it again, or its group
changed".
- `holder.go`, `PullMerged` (line 347): a merge records the merge commit on that pull request's own
delivery and links its walk. It does not touch the repository's other open deliveries into the same
delivery and links its rollout. It does not touch the repository's other open deliveries into the same
trunk.
- The forge marks a head pending only when it announces a new head (issue 305). A merge into the trunk is
not a new head of any other pull request.
@@ -60,14 +60,14 @@ note. Each gap says what is open, why it matters, and what would close it.
9. **On a large data pool the root search may never finish.** The node-engine's search for programs that run as
root now has one bound of fifteen minutes (`mesh-host` #61; the inner two-minute limit is gone). On a machine with
a very large data pool the walk can still run past it, and the verdict then stays "not judged". That is loud,
a very large data pool the traversal can still run past it, and the verdict then stays "not judged". That is loud,
not silent.
*Closes with:* setuid turned off on the data datasets, so the walk skips them.
*Closes with:* setuid turned off on the data datasets, so the traversal skips them.
10. **An agent can merge its own pull request.** No mesh repository's main requires an approval on the forge
(`required_approvals: 0`), and the forge tool merges as the forge's administrator once the required checks are
green. An agent can therefore change a module that runs on every node, open the pull request, wait for the
checks and merge it, and the controller walks the change out. The checks judge whether every machine still
checks and merge it, and the controller rolls the change out. The checks judge whether every machine still
composes, not what the change is for. The operator's yes in a session is a convention, and nothing checks it.
*Closes with:* a merge that needs the operator's own approval: a required approval on the forge from the
operator's personal account only, or an approval through ADR 0259's asks. That holds only when the forge's
@@ -98,7 +98,7 @@ note. Each gap says what is open, why it matters, and what would close it.
14. **Ways to root the node-engine's judge does not see.** ADR 0266 §3 lists them under *not judged, so not seen*:
among them a root-run unit or script the agent account can write, a directory on root's `PATH` it can write,
root's ssh keys readable by it, and a setuid program on a filesystem the search does not walk. None is known
root's ssh keys readable by it, and a setuid program on a filesystem the search does not traverse. None is known
on the control-node, but a healthy verdict says nothing about them. The record left them "for a later record"
with nothing tracking it; added here on 2026-10-10.
*Closes with:* each way judged by the node-engine beside the others, or a record that decides against judging
@@ -16,14 +16,14 @@ stops answering while it runs.
| 10:57:57 | the route proxy's first "lookup … on <private address>:53: … connection refused" | the route proxy's log |
| 10:58:36 | `machine.<home server>.network` raised: one of its two resolvers does not answer | conditions |
| 10:58:45 | `machine.<control node>.network` raised; the home server's cleared, held under it | conditions |
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the walks |
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the rollouts |
| 10:59:37 | `module.step-ca.<control node>.unhealthy` raised | conditions |
| 10:59:42 | `machine.<control node>.network` **cleared**, its newest evidence (10:59:37) still "connection refused" | conditions |
| 11:00:23 | `machine.<control node>.network` reopened, raised time 11:00:23 | conditions |
| 11:06:00 | `seat.mesh-dns-resolver.<control node>.wrong` raised by D2 | conditions |
| 11:10:29 | both builds failed their gate: "<control node> as a whole: raised since it was sent: machine.<control node>.network" — put back | conditions |
| 11:18:12 | `dnsmasq.service` restarted by hand; every question answered again | the operator |
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release walked the node-engine | conditions |
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release rolled out the node-engine | conditions |
### What is measured, and what is guessed
@@ -54,12 +54,12 @@ answered none of its questions for the next run (`Confirm`, issue 277), because
question can be lost. Its first run after 10:57:57 held it; the next, at 11:06:00, raised it. Every one
of those questions had been refused, which no load explains.
**Measured: why the second walk of the node-engine waited on its own first machine.** A release keeps no
**Measured: why the second rollout of the node-engine waited on its own first machine.** A release keeps no
record per module — its modules are in its tier only — and D10 read only the records to know what a plan
is rolling out. So while a release walked the node-engine at 11:21, D10 raised `core-behind` on every
is rolling out. So while a release rolled out the node-engine at 11:21, D10 raised `core-behind` on every
machine, "and no plan is rolling them out", and the gate on the home server, its first machine, waited
on it (issue 281's settle window) until D10's next run five minutes later found the new engine there.
The walk then passed. It did not fail, but it waited five minutes for a condition its own send caused.
The rollout then passed. It did not fail, but it waited five minutes for a condition its own send caused.
**Not explained: why the second resolver did not save the lookups.** Every machine's resolver file lists
both resolvers (`options timeout:1 attempts:2`), and the control node's own file lists its private address
@@ -153,7 +153,7 @@ registry, is the operator's decision.
- An undecided part of a network statement — `unknown` or `starting`, as a node-engine just started says
— holds the open network conditions whose newest evidence names that part, and only those. A statement
undecided as a whole holds every part. The next statement decides.
- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on
- D10 counts what a release's tier names as rolling out, so a rolled out node-engine is not `core-behind` on
its own first machine.
- D2 is unchanged: a resolver that answered nothing, refused or silent, is held for the next run, because
a few hundred milliseconds of refusals is a resolver restarting as well as one that stopped. D2 says it
@@ -61,8 +61,8 @@ A replay in mesh-lab's register is still owed before this core issue resolves (A
## A sequel (2026-10-10)
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a walk:
mesh-delivery matches a walk to the delivery merged as its commit, so two catalogue merges 18 seconds apart
left the earlier one held for a person and the walk carrying it waiting for a word nothing gave, for 58
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a rollout:
mesh-delivery matches a rollout to the delivery merged as its commit, so two catalogue merges 18 seconds apart
left the earlier one held for a person and the rollout carrying it waiting for a word nothing gave, for 58
minutes, with no condition raised. See
[issue 362](../362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md).
@@ -2,7 +2,7 @@
opened: 2026-10-09
---
# 351. A held delivery whose module a backlog walk carried to every machine is never covered
# 351. A held delivery whose module a backlog rollout carried to every machine is never covered
## Symptom
@@ -19,12 +19,12 @@ What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `ha
| time | what happened |
|---|---|
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word |
| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded |
| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours |
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its rollout waited for its word |
| 10-08 11:06 | three later catalogue merges took its rollout over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the rollout ended `failed`, then superseded |
| 10-08 14:27 | #127 merged as bf542128; its rollout built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog rollout sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog rollout, as it had for 24 hours |
No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever
move it again (ADR 0267):
@@ -32,22 +32,22 @@ left as it is, the delivery stays held for ever.
## Why it is an issue
ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running
ADR 0270 covers a held merge when a later delivered merge's rollout "sent the module to every machine running
it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge
commit, which the forge says contains a6385479. The decision's question — does this change go on? — was
answered by the operator's own release of the build. Yet the record cannot end:
- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what
- **A backlog rollout's end is read by nothing.** While it judges, the controller's record of it names what
its gate carried, and mesh-delivery credits the delivery that published each carried build one step per
machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing
from that: the steps stay `judging` for ever.
- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running
it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the
- **`covered` reads only a rollout's own aggregate steps.** A rollout's own record says "every machine running
it" or "the rest running it"; a step per machine, which is all a backlog rollout gives, never counts as the
module sent everywhere.
ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module
whose build reached its machines on the operator's word, after the mesh held it back, rather than by the
walk of the merge that built it. A held delivery in that case has no end but a person's wrong act
rollout of the merge that built it. A held delivery in that case has no end but a person's wrong act
([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and
the condition it raises every hour is one nobody can clear correctly.
@@ -56,7 +56,7 @@ the condition it raises every hour is one nobody can clear correctly.
- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and
`gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128.
- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps.
- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
- The rollout that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
rightly no coverer.
- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module
- A person's `release` of #125 (the verb): it would roll out a6385479, an older commit than the one the module
was built from, and is what ADR 0270 ruled out.
@@ -2,56 +2,56 @@
**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`,
at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127,
`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`,
`mesh-controller.delivery-walks` for #127's rollout and for the backlog rollout, `plans`, `hand-acts`, `module list`,
the forge's `gitea_contains`, and the holder's journal on the control node.
**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a
**Where a backlog rollout's steps come from.** `walkMoved` (holder.go) hands a rollout whose record carries a
`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives
the delivery whose merge commit each carried build was made from one step per machine: `judging` while the
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog rollout
holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`.
`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43
and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the
module was not in that walk's tiers, and the backlog walk is a walk of its own.)
and 14:46 UTC were never moved on. (The controller's own record of #127's rollout never named the module: the
module was not in that rollout's tiers, and the backlog rollout is a rollout of its own.)
**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a
later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only
on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running
on a `sent` step whose machine is the rollout's own aggregate, "every machine running it" or "the rest running
it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is
found, and the forge is never asked — which is why the journal has no line.
**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a
backlog walk that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
**What "done on a machine" means for a backlog rollout.** Read from the controller's records, not its code: a
backlog rollout that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done,
with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of
the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis
is wrong there.
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb —
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed rollout, and the `release` verb —
see the report.
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog rollout's `done` machines for
nothing, and `carries`, which counts no per-machine step.
## The fix
In mesh-catalog #163, one pull request, in `modules/mesh-delivery`:
- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that
walk (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
step it failed on, and a step another walk carried, are left as they are.
- **A finished backlog rollout passes the steps it carried.** `creditReleased`: every step credited to that
rollout (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
step it failed on, and a step another rollout carried, are left as they are.
- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when
the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
the rollout's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still
judging, one failed or one never sent leave the delivery held.
- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its
open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was
- **A rollout that carried a step still judging is asked for by its id.** The controller lists back only its
open rollouts and the last fifty ended at a start; the backlog rollout of this case, ended the day before, was
no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for
such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
such a rollout by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
its record as said.
Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own
walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
rollout of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
The record's consequence that named the one-machine gap gains a progressive insight naming this one.
**How it is checked**, in `covered_test.go`:
@@ -59,15 +59,15 @@ The record's consequence that named the one-machine gap gains a progressive insi
- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered;
still judging on one, failed on one, never sent to one, or a plan naming no machines stays held.
- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one
and leaves the other judging; a failed step stays failed; a step another walk carried is not touched.
and leaves the other judging; a failed step stays failed; a step another rollout carried is not touched.
- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held
while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the
while the backlog rollout judges, covered once it is done and read by its id, naming the later merge and the
module.
All three fail on the catalogue's main before the change and pass after it.
**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are
asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once
asked for by the backlog rollout's id and pass; #127 then carries the module for #125; the forge is asked once
whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads
delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the
acceptance of this issue, read with `mesh-delivery.show` of #125.
@@ -21,7 +21,7 @@ On 2026-10-09, local times on the control node (its controller's and node-engine
migrations than the store had applied, and served what it could.
- 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3
times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent,
which the put-back had removed from the machine. The plan walked on, believing the controller delivered.
which the put-back had removed from the machine. The plan rolled out on, believing the controller delivered.
## Why it is an issue
@@ -35,8 +35,8 @@ Four rules did not hold.
gate of the build it had replaced.
- **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than
itself"). A controller older than the store's schema starts behind its own records. Nothing refused it.
- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to
the merge plan's send: a release keeps no module records, and the check for a walk in progress read
- **One rollout of a module on a machine at a time** (issue 281, ADR 0236). A release's rollout was invisible to
the merge plan's send: a release keeps no module records, and the check for a rollout in progress read
only those.
The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
@@ -25,10 +25,10 @@ read as a module's) are the nearest; neither is this.
to the one the first machine ran before (`PreviousBuild`) and sends it. Nothing asked whether that
build can read the store as the failed build left it. The controller says so itself when it starts
("this control plane is behind the mesh-controller row"), and serves anyway (ADR 0185).
- **The merge plan's send crossed the release's gate.** `walkedBy` finds a walk in progress by a plan's
module records (`p.Modules[module].First`). A release (the walk of the builds waiting for a gate) keeps its walk in its gate's carried moves,
- **The merge plan's send crossed the release's gate.** `walkedBy` finds a rollout in progress by a plan's
module records (`p.Modules[module].First`). A release (the rollout of the builds waiting for a gate) keeps its rollout in its gate's carried moves,
not in module records, so the release's open gate on the control node held nothing back. And a plan's
own modules were never checked against another walk at all: the check ran over the moves a machine
own modules were never checked against another rollout at all: the check ran over the moves a machine
had waiting, and skipped the plan's own.
### What is guessed
@@ -62,13 +62,13 @@ In mesh-controller, pull request 185.
never recorded what it reaches, is not put back; the current build is kept, and the condition
(`rollback-failed`, urgent) says why and that a person decides. The lease holder's `Build` is that
version too.
- **A walk of a module on a machine holds every other send of another build of it there.** `walkedBy`
reads a release's open gate as its walk, and a plan's own modules are held by it as any other move is:
- **A rollout of a module on a machine holds every other send of another build of it there.** `walkedBy`
reads a release's open gate as its rollout, and a plan's own modules are held by it as any other move is:
a merge plan whose first machine is under a release's gate for the same module waits, said as
"waiting for … which release-… is walking". The same build to the same machine is not held: it is
already there.
Not changed: whether a release's walk and a merge plan's walk of the core should be one walk (the walk
Not changed: whether a release's rollout and a merge plan's rollout of the core should be one rollout (the rollout
carrying over) is left to a design; serialising them is what this fixes.
## How it is checked
@@ -79,7 +79,7 @@ carrying over) is left to a design; serialising them is what this fixes.
| A send that moves the module supersedes the judging, puts nothing back | `TestReplay352`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule` |
| A report is held against the gate's own send, by digest or sequence | `TestAReportIsHeldAgainstTheGatesOwnSend`, inventory `TestASendAndAReportAreKnownByTheirDeclaration` |
| Only the build last sent is judged | `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent` |
| A walk holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` |
| A rollout holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` |
| No put-back to a controller behind the store, or one unproved | `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, inventory `TestASchemaReachIsKeptByBuild`, catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`, `TestTheHolderNamesTheBuildTheDeclarationToldIt` |
| Every condition of the change is tested | 31 mutations, each failing a test |
@@ -9,7 +9,7 @@ opened: 2026-10-09
On 2026-10-09, local times on the control node, from its controller's journal (one process throughout):
- 17:32 the router (the `operator-channel` holder) was unassigned, so that a merge gate of the catalogue had
nothing to send there; 17:52:54 a walk of another module pushed the control node and composed the bus's user
nothing to send there; 17:52:54 a rollout of another module pushed the control node and composed the bus's user
list **without** the controller's grant to ask: the grant is composed from a holder of the seat that is
*assigned*, and none was.
- 17:54:05 the controller started asking (the router was assigned again by then, so its check of the record
@@ -73,9 +73,9 @@ one. The tool runner has the same shape at a smaller scale: the bus account and
controller's user list and in every node's assignment.
Hence: **one release in which both names are accepted, on every side, before any release in which the old
one is dropped**, each release walked one node first — the laptop, where a halted node is the operator at a
one is dropped**, each release rolled out one node first — the laptop, where a halted node is the operator at a
keyboard — and then the rest. I think no stage should move before the delivery fixes of issues 351 and 352
are live, because a core walk is exactly what those broke; that is a guess from their symptoms, not a
are live, because a core rollout is exactly what those broke; that is a guess from their symptoms, not a
measurement.
### The staged plan
@@ -90,7 +90,7 @@ measurement.
tool runner's as another (runtime module, bus account, kind). The node-engine's launcher and upgrade
package read the old and the new library and state directories, and the node-engine moves its state
once, the first time it runs under the new name. Both unit names run the same launcher. The conditions
and the `node` verb say node-engine. The merge gate composes every node green; walked one node first.
and the `node` verb say node-engine. The merge gate composes every node green; rolled out one node first.
3. **The rename proper.** The new module names registered, assigned beside the old on the first node, the
old unassigned there, the acceptance sentence checked on that node, then the rest. The repository
renamed on the forge, with its build source and its Go module path; the packaging renamed. The tool
@@ -9,7 +9,7 @@ opened: 2026-10-09
ADR 0270
(2026-10-09) ends a held delivery by itself when a later merge that contains it was delivered to every
machine running its modules — the delivery is *covered*. Its consequences say: "A module that runs on one
machine only is never covered. A walk records 'the rest running it' only after its first machines passed
machine only is never covered. A rollout records 'the rest running it' only after its first machines passed
and the rest were sent. A module on one machine has no rest, so its steps never say it reached every
machine. mesh-delivery itself, on the control node alone, is such a module: a held merge of it stays held
for a person, as before this record. Reading 'every machine running it' from the first machines when they
@@ -29,8 +29,8 @@ done (issue 354).
## Open
- Whether a walk whose first machines are all the machines should record that it reached every machine
- Whether a rollout whose first machines are all the machines should record that it reached every machine
(the record's own suggestion), or whether *covered* should read the machines running a module from the
placement rather than from the walk's steps.
placement rather than from the rollout's steps.
- The owner is mesh-delivery, in the controller's repository, as the record says; `located-in:` is filled
when the diagnosis confirms where the walk's record is written.
when the diagnosis confirms where the rollout's record is written.
@@ -13,7 +13,7 @@ The node-engine sets a directory's mode and owner by path when the directory is
an account's home: `applyDirectory` calls `chmodDirPath` and `own`, which call `os.Chmod` and `os.Chown` on the
path. Both follow a link anywhere in the path. The directory applier refuses a link at the directory's own name,
and issue 339's placement guard refuses the machine's own trees, but nothing refuses a link in a parent at the
moment of use. Below a home the engine already walks every component without following a link (ADR 0266).
moment of use. Below a home the engine already traverses every component without following a link (ADR 0266).
Issue 356 closed this for a directory handed to the mesh by the operator: it is changed only through one
descriptor opened from `/` with no link followed, on every apply. The same treatment for every other mesh
@@ -19,8 +19,8 @@ the controller's `root-free` verb for that machine answered:
> again later
The control-node holds its root and home filesystems on a software RAID of spinning disks, with millions of
files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) walks every local
filesystem mounted without `nosuid`; that walk, cold, takes far longer than fifteen minutes there. Each attempt
files between them. The node-engine's search for setuid programs (ADR 0266 §3 item 10) traverses every local
filesystem mounted without `nosuid`; that traversal, cold, takes far longer than fifteen minutes there. Each attempt
was killed at its bound, judged nothing, and was tried again after a back-off, which ends the same way.
## Why it is a design fault, not one slow disk
@@ -29,7 +29,7 @@ was killed at its bound, judged nothing, and was tried again after a back-off, w
controller could tell "still running" from "stuck" (`rootsearch.Bound`, read by both). It turned a slow
machine into one that is never judged, failing closed for good rather than for a while.
- **Nothing was kept across a restart.** The search's result lived in the engine's memory only, so every
engine restart (every update of the engine) started from nothing and judged nothing until a whole walk
engine restart (every update of the engine) started from nothing and judged nothing until a whole traversal
finished again.
- **The controller's quiet was the same value.** The self-check stayed quiet while a search was "within its
bound" — fifteen minutes — so a search that runs for hours would raise the urgent condition on every
@@ -51,7 +51,7 @@ was killed at its bound, judged nothing, and was tried again after a back-off, w
**Why a day-old search is sound.** Only root can make a setuid- or setgid-root file: the agent account can
neither set the bit on a file root owns nor give a file of its own to root. So a complete search stays true until
root acts, and root's acts through the mesh (applies) start a new search. A file that appears behind a running
walk's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age
traversal's cursor was made by root too. What is left is root acting by hand — the operator — which the stated age
bounds.
The trail and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
@@ -15,25 +15,25 @@
health — which this issue does not change).
- **Nothing else** used the bound: `SetuidCache` had one caller (`moreWays`), and the engine made one cache.
- **Ruled out:** a stuck `find` (each attempt ended at exactly its bound) and a filesystem that should not be
searched (the walk covered only the local root, home and boot filesystems, as ADR 0266 asks).
searched (the traversal covered only the local root, home and boot filesystems, as ADR 0266 asks).
## The fix (`mesh-host` pull request 75, `mesh-controller` pull request 190)
**The walk is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as
**The traversal is the engine's own and resumable.** `filepath.WalkDir` per filesystem, staying on its device as
`find -xdev` did, with the same prunes and the same match (a regular file, setuid and owned by root, or setgid
and of root's group), never following a link. It visits each directory's entries in lexical order, so its
progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is walked before
`/a-c` although `-` sorts before `/`). Every minute the progress (when the walk started, its filesystems, those
progress is one cursor per filesystem — the last path visited — compared name by name (`/a/b` is traversed before
`/a-c` although `-` sorts before `/`). Every minute the progress (when the traversal started, its filesystems, those
done, the cursor, the matches so far) is written beside the engine's state as a new file renamed into place,
readable by root alone. After a restart the walk carries on from the cursor: what is before it is skipped
without being read, the directories leading back to it are read again. A walk older than the stated age, or one
readable by root alone. After a restart the traversal carries on from the cursor: what is before it is skipped
without being read, the directories leading back to it are read again. A traversal older than the stated age, or one
an apply asked to redo, is thrown away and begun again.
**Low priority, one at a time, no bound.** The walk runs on a goroutine locked to an operating-system thread of
**Low priority, one at a time, no bound.** The traversal runs on a goroutine locked to an operating-system thread of
its own, set to nice 19 and the idle I/O class; on Linux both are a thread's, so the engine's other work keeps
its priority, and the thread ends with the goroutine. The package manager is run the same way. A priority that
cannot be lowered fails the search rather than walk at full priority. There is no deadline: the search ends when
it has walked everything, or with an error (a directory it could not read), which is incomplete.
cannot be lowered fails the search rather than traverse at full priority. There is no deadline: the search ends when
it has traversed everything, or with an error (a directory it could not read), which is incomplete.
**A verdict only from a complete, fresh search.** The last complete search is kept beside the engine's state
with its start and finish, and judges while it is younger than `rootsearch.FreshFor`, counted from its start.
@@ -44,7 +44,7 @@ link, owned by the engine's account and writable by it alone, in a directory tha
**When a search runs.** Once a day (`rootsearch.Every`, from the last complete one's start); after an apply
whose outcomes moved a user, a package, an action, or a file of the user and group databases or of sudo's,
doas's or polkit's rules (`apply.MovesRoot`), at most once an hour from the last search's end, the last complete
search judging meanwhile; and at once when a filesystem appears that the last search did not walk, which that
search judging meanwhile; and at once when a filesystem appears that the last search did not traverse, which that
search does not judge.
**The values, one place** (`rootsearch`, read by both repositories):
@@ -57,29 +57,29 @@ search does not judge.
`FreshFor` is `Every` plus `Quiet` so that a daily search which finishes within the controller's quiet never
leaves an hour unjudged; the controller's test holds that relation. The operator's direction named 26 hours as an
example; 36 is chosen because a full walk at idle priority on the control-node is expected to take hours, and
example; 36 is chosen because a full traversal at idle priority on the control-node is expected to take hours, and
with 26 hours a daily search longer than two hours would leave the account unjudged — and phone linking refused —
every day until it finished.
## 2026-10-10 — the independent review, and what it changed
An independent review of both pull requests asked four questions: can a stale or incomplete search yield "free";
can the agent account steer the kept files or the walk; does the walk hurt the machine; do the controller and the
can the agent account steer the kept files or the traversal; does the traversal hurt the machine; do the controller and the
engine agree. What it found, and what was done:
- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed walk
- **A cursor of a name that is not UTF-8 came back changed.** JSON strings replace such bytes, so a resumed traversal
skipped every name that sorted between the real cursor and its replacement, and could finish "free" without
having read them. Any account can make such a name. The walk's paths are now kept as their bytes.
- **An ask to search again was lost to a restart**, and the restarted engine carried on the walk that started
before root acted. The ask now forgets the walk under way and stops it keeping progress.
- **A fresh search judged a filesystem it never walked.** A complete search now judges only while it walked every
filesystem a search would walk now; a filesystem mounted since makes the verdict "not judged yet" until walked.
- **A mount point inside a walked filesystem was read before it was known to be another filesystem.** A FUSE
mount root cannot read failed the walk, and a network mount whose server is away could hang it. Every mount
having read them. Any account can make such a name. The traversal's paths are now kept as their bytes.
- **An ask to search again was lost to a restart**, and the restarted engine carried on the traversal that started
before root acted. The ask now forgets the traversal under way and stops it keeping progress.
- **A fresh search judged a filesystem it never traversed.** A complete search now judges only while it traversed every
filesystem a search would traverse now; a filesystem mounted since makes the verdict "not judged yet" until traversed.
- **A mount point inside a traversed filesystem was read before it was known to be another filesystem.** A FUSE
mount root cannot read failed the traversal, and a network mount whose server is away could hang it. Every mount
point the kernel lists is now left unread. Mount points are read with all the kernel's escapes.
- **A file found and gone before the package manager was asked failed every resumed search.** The walk looks at
- **A file found and gone before the package manager was asked failed every resumed search.** The traversal looks at
what it found again when it ends, and drops what went or is no longer setuid-root.
- **Outcomes that moved nothing started a search**, and a search asked for could follow a walk of hours at once.
- **Outcomes that moved nothing started a search**, and a search asked for could follow a traversal of hours at once.
Only created, updated, corrected and removed outcomes count now, and an ask waits an hour from the last
search's end.
- **Not changed, because the mesh cannot do it:** the review asked that a file or archive declared setuid start a
@@ -87,53 +87,53 @@ engine agree. What it found, and what was done:
## Open, after the fix
- **How long the first walk on the control-node takes is not measured.** Until the first complete search, the
- **How long the first traversal on the control-node takes is not measured.** Until the first complete search, the
verdict there is `not judged yet`, and linking stays refused. The engine says each search's end and how long
it ran in its journal; that line is the measurement. If a walk takes longer than `Quiet`, the self-check
it ran in its journal; that line is the measurement. If a traversal takes longer than `Quiet`, the self-check
raises the urgent condition, which is right: it is then not known to finish.
- **The idle I/O class is honoured only by a disk scheduler that has classes** (BFQ). Under another scheduler
the walk still runs at nice 19 but competes for the disk as any reader does.
- **An ask to search again is not kept across a restart.** The walk under way is forgotten when it is asked,
so a restarted engine does not carry an older walk on; the ask itself is lost, and the daily search covers it.
the traversal still runs at nice 19 but competes for the disk as any reader does.
- **An ask to search again is not kept across a restart.** The traversal under way is forgotten when it is asked,
so a restarted engine does not carry an older traversal on; the ask itself is lost, and the daily search covers it.
The last complete search keeps judging meanwhile, on the reasoning in the report.
- **The agent account can make the search fail, never pass.** A tree deeper than the longest path the kernel
takes, or a directory unreadable to root through FUSE, ends every walk incomplete; a directory of millions of
takes, or a directory unreadable to root through FUSE, ends every traversal incomplete; a directory of millions of
entries makes the engine hold them all at once. Each is "not judged" — the urgent condition, and linking
refused — never a pass. Reading by directory handle, as `find` did, would close the first.
- **A filesystem mounted only on use** (an automounted boot partition) makes the verdict "not judged yet" each
time it appears unwalked, until a walk covers it.
- **Part of the walk's cost escapes its priority**: Go's garbage collector runs on threads of normal priority,
time it appears untraversed, until a traversal covers it.
- **Part of the traversal's cost escapes its priority**: Go's garbage collector runs on threads of normal priority,
and a software RAID's own threads do not carry the idle class to the disks.
- **ADR 0266 §3 item 10** still says "at most hourly" and "at most 15 minutes". On resolution it gains a
progressive insight naming this issue and the values above.
- **A later option, not built now:** a fanotify filesystem mark with `FAN_ATTRIB` (Linux 5.1 and later) on
every searched filesystem would report each change of mode or owner as it happens, turning the daily walk into
every searched filesystem would report each change of mode or owner as it happens, turning the daily traversal into
a true incremental search.
- **Resolution** of this core issue needs a replay in mesh-lab's register, or `replay-none:` with the reason.
## 2026-10-10 — the search shows its progress, and the operator can ask for a fresh one
- **The progress was kept and seen by nobody.** At 01:40 the operator asked how far the first walk on the
control-node had got. The walk kept its progress every minute beside the engine's state, but `root-free` said
- **The progress was kept and seen by nobody.** At 01:40 the operator asked how far the first traversal on the
control-node had got. The traversal kept its progress every minute beside the engine's state, but `root-free` said
only "not judged yet (search running), started 01:27", and no tool read further. The node-engine serves no
tools of its own, so the progress now travels where the verdict already does: the agent account's verdict
carries it as fields, and its reason gains one line that `root-free` repeats. The line holds filesystems done
of the total, the mount point being walked, the entries visited, the setuid- or setgid-root files found so far,
of the total, the mount point being traversed, the entries visited, the setuid- or setgid-root files found so far,
when the progress was last kept, and how long each finished filesystem took. Counts, times and mount points
only: the cursor and every file found are often under an account's home, and stay in the engine's state and
journal. The line moves at each checkpoint (once a minute) and is never part of what makes a verdict
"changed", so the journal does not repeat it every minute; the statement is said every minute anyway while
the account is not healthy. An entry is counted once across a resume, as it is kept with the cursor it
matches. The time a filesystem took counts only the time the engine walked it, not the time it was stopped.
matches. The time a filesystem took counts only the time the engine traversed it, not the time it was stopped.
- **A change by hand was not searched for until the next day.** The first complete search on the control-node
ended at about 01:48 and found two setuid-root programs no package owns, copies in the operator's home. When the
operator removes them by hand, no apply says so, and nothing searches again until the daily search. The
operator now asks for one at the controller's terminal: `nox node setuid-search <node>` on the control-node.
The controller signs the ask with the mesh's key, as it signs a hand-over, under a signing context of its own.
The engine throws away the last complete search and any walk under way or kept, and the next look starts a
The engine throws away the last complete search and any traversal under way or kept, and the next look starts a
full search. Until it completes, the verdict is "not judged yet", with its progress.
- **Why at the terminal alone, not a verb.** A search can never make a machine root-free wrongly: it judges only
once complete. But every ask throws a walk away. Asked again and again, it would keep the machine unjudged
once complete. But every ask throws a traversal away. Asked again and again, it would keep the machine unjudged
for good and its disks busy, and whoever may call a verb includes agents. The operator needs the line only
after a change made by hand.
- **A find counts only where the agent account can execute it.** The operator asked how the agent account could
@@ -11,17 +11,17 @@ Shortly after midnight on 2026-10-10 (local time) two pull requests of the catal
their checks.
`mesh-delivery.deliveries` then showed every delivery of the dunst pull request `superseded`, and the
claude-code delivery `held`, waiting for "a person: merged, and the controller opened no walk for it within
claude-code delivery `held`, waiting for "a person: merged, and the controller opened no rollout for it within
10m0s — nothing it holds follows that branch, or the merge was not heard: a person decides". The
controller's `plans` said of the walk for the catalogue's head, which carried both merges: "published; its
walk waits for mesh-delivery's word — `plans go … --why …` starts it by hand".
controller's `plans` said of the rollout for the catalogue's head, which carried both merges: "published; its
rollout waits for mesh-delivery's word — `plans go … --why …` starts it by hand".
Both reasons were wrong. A walk had been opened, and it contained the held merge. Nothing gave it the word.
Both reasons were wrong. A rollout had been opened, and it contained the held merge. Nothing gave it the word.
The operator's change sat undelivered for 58 minutes, until the operator's hand-act `mesh-controller.plans
go` started the walk. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours
before `mesh-delivery.stalled` lists it, and the walk's own wait raised only a warning, at 00:51, which said
go` started the rollout. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours
before `mesh-delivery.stalled` lists it, and the rollout's own wait raised only a warning, at 00:51, which said
there was nothing to do and pointed at the ended delivery (the diagnosis's second entry; this sentence said
before that the walk's wait raised nothing).
before that the rollout's wait raised nothing).
## Why it is an issue
@@ -33,41 +33,41 @@ cause.
## Where it lives
The diagnosis ([01-diagnosis.md](01-diagnosis.md)) traces it to two parts that each behave as written and
disagree on what a walk's commit means:
disagree on what a rollout's commit means:
- The controller plans a merge it hears after a later merge of the same branch at the later merge's commit,
and supersedes the earlier walk (the fix for
and supersedes the earlier rollout (the fix for
[issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md)).
So the claude-code merge got a walk at the dunst merge's commit.
- mesh-delivery finds the delivery a walk walks by its walk id or by the commit it was merged as. The new
walk matched the dunst delivery, which its first walk's supersession had already ended, so no live
delivery held the walk and nothing gave it its turn. The claude-code delivery, merged as a different
commit, never got a walk and was held after ten minutes.
So the claude-code merge got a rollout at the dunst merge's commit.
- mesh-delivery finds the delivery a rollout rollouts by its rollout id or by the commit it was merged as. The new
rollout matched the dunst delivery, which its first rollout's supersession had already ended, so no live
delivery held the rollout and nothing gave it its turn. The claude-code delivery, merged as a different
commit, never got a rollout and was held after ten minutes.
## Open
- How mesh-delivery learns which merges a walk carries: the controller's walk record naming every merge it
- How mesh-delivery learns which merges a rollout carries: the controller's rollout record naming every merge it
answers (its commit and the commits planned into it), or mesh-delivery reading containment itself, as it
already does to decide *covered*.
- A superseded walk whose successor matches only final deliveries: whether the successor should be adopted
as a delivery of its own, so that something always holds a waiting walk.
- A walk waiting for mesh-delivery's word past a bound, and a delivery held because no walk was opened,
- A superseded rollout whose successor matches only final deliveries: whether the successor should be adopted
as a delivery of its own, so that something always holds a waiting rollout.
- A rollout waiting for mesh-delivery's word past a bound, and a delivery held because no rollout was opened,
raise a condition to the operator instead of waiting silently (failure must be loud).
## Decided
The operator decided the model that removes this shape, not only its bookkeeping:
ADR 0276. Merges are collected in a rolling merge window and each batch is walked once, with at most one
commit per repository, one walk at a time. The bookkeeping that tied a walk to an ended delivery is repaired as
ADR 0276. Merges are collected in a rolling merge window and each batch is rolled out once, with at most one
commit per repository, one rollout at a time. The bookkeeping that tied a rollout to an ended delivery is repaired as
well, and every wait of this shape has a bound and a condition. It is designed in to-be 47, *The merge window
and the batch*.
## Resolved
2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a walk is tied to every live delivery whose
merge it answers, never to one that has ended, and a waiting walk no live delivery holds is listed by `stalled`
at once. The controller's side is mesh-controller#197: the merge window and the batch, one walk per batch with
one commit per repository, the walk naming every merge it answers, and the bounds of decision 9. The replay
2026-10-10. mesh-delivery's side is mesh-catalog#171 (merged): a rollout is tied to every live delivery whose
merge it answers, never to one that has ended, and a waiting rollout no live delivery holds is listed by `stalled`
at once. The controller's side is mesh-controller#197: the merge window and the batch, one rollout per batch with
one commit per repository, the rollout naming every merge it answers, and the bounds of decision 9. The replay
R362 is registered in mesh-lab's replays register (mesh-lab#75), proved to fail on the commit before the
mesh-delivery fix and pass on it. The build raised one rule the record had not settled, decided during the
build and written into ADR 0276: a merge on the controller's own path never shares a batch with a catalogue
@@ -9,38 +9,38 @@ request as 48bda475 at 00:21:09. 48bda475 is the branch's head and contains 553b
**What the controller did** (its journal, 00:21:14 to 00:21:15):
1. It heard the dunst merge first and opened a walk at 48bda475 for dunst, waiting for mesh-delivery's word.
1. It heard the dunst merge first and opened a rollout at 48bda475 for dunst, waiting for mesh-delivery's word.
2. It then heard the claude-code merge, and printed "553b7191 was merged before 48bda475 (…, building): what
it moved is built from 48bda475, which contains it". This is the rule that answered
[issue 349](../349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md):
`SourceMoved` replaces the merge's commit and merge time with the newest planned merge's, when that merge
is later on the branch (`laterOnTheBranch`).
3. The second walk, also at 48bda475 and with the same merge time, superseded the first by when each was made
3. The second rollout, also at 48bda475 and with the same merge time, superseded the first by when each was made
(`supersededBy`, `earlierOnTheBranch`), and carried claude-code and dunst. It waited for mesh-delivery's
word.
So every walk the controller opened was at 48bda475. None was at 553b7191.
So every rollout the controller opened was at 48bda475. None was at 553b7191.
**What mesh-delivery did** (`mesh-delivery.show` for both deliveries):
1. The dunst delivery (merged as 48bda475) took the first walk, matched by its commit, and became
1. The dunst delivery (merged as 48bda475) took the first rollout, matched by its commit, and became
`published` at 00:21:14.
2. The first walk's record came back superseded, and the table's `superseded` transition ended the dunst
2. The first rollout's record came back superseded, and the table's `superseded` transition ended the dunst
delivery at 00:21:15 ("a newer delivery to the same trunk took over its walk").
3. The second walk was matched to a delivery by `deliveryWalking`, which looks first for a delivery holding
that walk id and then for any delivery whose `merged_as` is the walk's commit, final or not. That is the
dunst delivery again, already final. The walk was linked there, so it was never kept as unmatched and
3. The second rollout was matched to a delivery by `deliveryWalking`, which looks first for a delivery holding
that rollout id and then for any delivery whose `merged_as` is the rollout's commit, final or not. That is the
dunst delivery again, already final. The rollout was linked there, so it was never kept as unmatched and
never adopted as a delivery of its own. A turn (`turn`) is given only to a `published` or `delivering`
delivery, so nothing ever gave the second walk its word.
4. The claude-code delivery, merged as 553b7191, matched no walk. Ten minutes after its merge
delivery, so nothing ever gave the second rollout its word.
4. The claude-code delivery, merged as 553b7191, matched no rollout. Ten minutes after its merge
(`unwalkedPast`) it was held for a person at 00:30:58, with a reason that names two causes, neither of
which happened.
After the operator's `plans go`, the second walk's machine steps (claude-code and dunst, each on a node) were recorded on the superseded dunst delivery, which confirms that mesh-delivery reads that walk as
After the operator's `plans go`, the second rollout's machine steps (claude-code and dunst, each on a node) were recorded on the superseded dunst delivery, which confirms that mesh-delivery reads that rollout as
the dunst delivery's.
**Why nothing was raised.** A delivery held for a person is bounded at 24 hours before
`mesh-delivery.stalled` lists it (`Bounds` in its table). The walk waiting for mesh-delivery's word is
`mesh-delivery.stalled` lists it (`Bounds` in its table). The rollout waiting for mesh-delivery's word is
counted against the controller's plan wait bound of 30 minutes for `status`, but no condition reached the
operator in the 58 minutes it waited. I did not trace why the controller's side stayed quiet; that belongs
to the fix.
@@ -48,28 +48,28 @@ to the fix.
## Ruled out
- **The merge was not heard.** The controller's journal shows it was heard and planned within 23 seconds.
- **Nothing follows that branch.** claude-code follows the catalogue's trunk; the second walk built it.
- **Nothing follows that branch.** claude-code follows the catalogue's trunk; the second rollout built it.
- **A failed check.** Both heads passed the merge gate and the repository's check.
## Located
Two parts, each correct by its own rule, disagree on what a walk's commit means. The controller (since issue
349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a walk's commit is the
merge commit of exactly one delivery. `located-in` names both. The fix is to make the walk say which merges
it answers, or to make mesh-delivery match by containment, and to let no waiting walk be held by nothing.
Two parts, each correct by its own rule, disagree on what a rollout's commit means. The controller (since issue
349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a rollout's commit is the
merge commit of exactly one delivery. `located-in` names both. The fix is to make the rollout say which merges
it answers, or to make mesh-delivery match by containment, and to let no waiting rollout be held by nothing.
## 2026-10-10 — what the controller's side said
The first entry says no condition reached the operator in the 58 minutes. The controller's condition history
says more precisely what happened. S16 (a walk waiting for its delivery's word) raised
`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the walk opened, as its bound says.
says more precisely what happened. S16 (a rollout waiting for its delivery's word) raised
`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the rollout opened, as its bound says.
It was a warning, its explanation was "Nothing for you to do … It becomes a question for you if it still waits
after 4 hours", and its summary sent the reader to "`mesh-delivery.show` for the delivery that landed as
48bda475", which is the dunst delivery the supersession had already ended. It cleared at 01:20:32, when the
operator's `plans go` started the walk.
operator's `plans go` started the rollout.
So the controller's side was not silent; it was mild and pointed at the wrong delivery. S16 cannot tell a
group member waiting its turn from a walk nobody holds; only mesh-delivery knows that. That is why ADR 0276
decision 9 has mesh-delivery list a waiting walk no live delivery holds through `stalled` at once, and has
S16 name the merges a walk answers.
group member waiting its turn from a rollout nobody holds; only mesh-delivery knows that. That is why ADR 0276
decision 9 has mesh-delivery list a waiting rollout no live delivery holds through `stalled` at once, and has
S16 name the merges a rollout answers.
+1 -1
View File
@@ -14,7 +14,7 @@ and why. Today that is almost entirely **Novox Mesh**, the foundation everything
| [`04-ISSUES`](04-ISSUES/) | The front door for "something is wrong" at the level of design or governance. |
**The numbering is the flow.** Research produces a decision; the decision authorises a design.
That is why decisions are `02` and design is `03` — a reader following the numbers walks the
That is why decisions are `02` and design is `03` — a reader following the numbers rollouts the
process in the order it happens. The research efforts and decision records themselves live in the knowledge
base, the seat `knowledge-base`, read and written through the mesh MCP server from any node (ADR 0290).
-6
View File
@@ -1,6 +0,0 @@
# Release 57: config and provisioning
This branch carries the work of design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.