Compare commits

...
Author SHA1 Message Date
jschoubben 0944437d4e Playbooks 07 and 08: the integration tests run once before every push, through mesh-controller.try or merge-check.sh (the operator, 2026-10-18) 2026-10-18 13:53:03 +02:00
jschoubben 2e6a989031 Playbook 07: a watcher also wakes the lead when nothing changed for two minutes (issue 1121) 2026-10-18 12:59:29 +02:00
jschoubben ec373556d0 Playbooks 07 and 08: an acceptance criterion is numbered and cited by its test as <issue>/AC<n>, so its check is a grep (review round 1)
mesh/merge-gate head ec373556d0bb merged into trunk 672011ab92ad — pass: the change touches no module of the mesh's graph
mesh/repo-check head ec373556d0bb merged into trunk 672011ab92ad — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-18 12:50:30 +02:00
jschoubben f242b8763b Design 47 builds with ADR 0426 (no timer where an event can end the wait) 2026-10-18 12:49:18 +02:00
jschoubben bb6af05f5c Playbooks 07 and 08: tests are written from acceptance criteria the larger model writes onto the issue (issue 1118); design 47 gets milestone 5b, the planning out of the controller (issue 1117) 2026-10-18 12:42:06 +02:00
mesh-tools 672011ab92 Merge pull request 'Design 55: designs and the glossary become records, and the knowledge base owns a design's state (ADR 0423, ADR 0424)' (#419) from design/055-designs-and-glossary-as-records into main 2026-10-18 02:28:14 +00:00
jochen 05ff547358 Design 55: hq owns part of the third slice's code, its checks and links (review round 1)
mesh/delivery delivered
mesh/merge-gate head 05ff547358fe merged into trunk 76a671b8c5da — pass: the change touches no module of the mesh's graph
mesh/repo-check head 05ff547358fe merged into trunk 76a671b8c5da — pass: its merge-check.sh passed
2026-10-18 04:25:11 +02:00
jochen 1def984c07 Design 55: designs and the glossary become records, and the knowledge base owns a design's state (ADR 0423, ADR 0424) 2026-10-18 04:23:10 +02:00
mesh-tools 76a671b8c5 Merge pull request 'Retire the word epic, and define milestone as a group of a design's issues delivered together (issue 1105)' (#418) from fix/1105-retire-epic into main 2026-10-18 02:10:49 +00:00
jschoubben 19bd7fd960 Retire the word epic, and define milestone as a group of a design's issues delivered together (issue 1105)
mesh/merge-gate head 19bd7fd96066 merged into trunk f82a90dfa021 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 19bd7fd96066 merged into trunk f82a90dfa021 — pass: its merge-check.sh passed
mesh/delivery delivered
The operator decided on 2026-10-18 that a domain word is used only when clearly defined. epic had three uses and no code; design 47 now names two ways, hotfix and feature.
2026-10-18 04:08:15 +02:00
mesh-tools f82a90dfa0 Merge pull request 'Playbook 07: a new integration test enters through the path the live trigger takes (issue 1086)' (#417) from fix/1086-test-enters-live-path into main 2026-10-18 00:23:21 +00:00
jschoubben 17c27ddf83 Playbook 07: a new integration test enters through the path the live trigger takes (issue 1086)
mesh/merge-gate head 17c27ddf834f merged into trunk 5c6b645c6ba9 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 17c27ddf834f merged into trunk 5c6b645c6ba9 — pass: its merge-check.sh passed
mesh/delivery delivered
Milestone 4's test saved its plan by hand, skipped the cut from a batch, and passed while the backup step never ran.
2026-10-18 02:19:20 +02:00
mesh-tools 5c6b645c6b Merge pull request 'Design 47: milestone 3 fix names issue 1079 (1084 was its duplicate)' (#416) from fix/design-47-milestone-row into main 2026-10-17 23:51:00 +00:00
jschoubben b6cf49d8ce Design 47: milestone 3 fix names issue 1079, of which 1084 was closed as a duplicate, and the forge module it touched
mesh/merge-gate head b6cf49d8ce12 merged into trunk 1484ff76826c — pass: the change touches no module of the mesh's graph
mesh/repo-check head b6cf49d8ce12 merged into trunk 1484ff76826c — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-18 01:49:37 +02:00
mesh-tools 1484ff7682 Merge pull request 'Design 47: milestone 4 delivered; a fix of milestone 3 comes next. Playbook 07: new integration tests are read against the code before the first push (issues 980, 1082)' (#415) from feat/milestone-4-delivered into main 2026-10-17 22:32:35 +00:00
jschoubben c10efa4d8c Playbook 07: the new test failed on its first two runs, not three
mesh/merge-gate head c10efa4d8c8f merged into trunk be5284f7b86c — pass: the change touches no module of the mesh's graph
mesh/repo-check head c10efa4d8c8f merged into trunk be5284f7b86c — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-18 00:30:41 +02:00
jschoubben 24fd567c31 Design 47: milestone 4 delivered and split; a fix of milestone 3 comes next. Playbook 07: new integration tests are read against the code before the first push (issues 980, 1082) 2026-10-18 00:29:35 +02:00
mesh-tools be5284f7b8 Merge pull request 'Say where nine half-built designs stand (issue 1072)' (#414) from fix/1072-design-states into main 2026-10-17 21:19:27 +00:00
jochen 21c4d212ec Correct three facts in the designs' states (issue 1072, review round 1)
mesh/merge-gate head 21c4d212ec8d merged into trunk 9d59c2ce1453 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 21c4d212ec8d merged into trunk 9d59c2ce1453 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 22:55:02 +02:00
jochen 1bd6f4611e Say where nine half-built designs stand (issue 1072)
The records cleanup found them saying proposed or designed although their
code is on main; each now says in-progress and what is built and left.
2026-10-17 22:53:10 +02:00
mesh-tools 9d59c2ce14 Merge pull request 'Playbooks 07 and 08 without the ready mark; a fixing agent runs no integration test; large issues split; the merge script ships first (issues 1022, 895)' (#413) from feat/one-group-one-plan into main 2026-10-17 20:10:56 +00:00
jschoubben 64e357da7d Playbook 07: a change the new merge script must judge ships the script first (issue 895)
mesh/merge-gate head 64e357da7d7e merged into trunk a6db6a70cffa beside mesh-catalog#352@07717974 +1 more — pass: the change touches no module of the mesh'…
mesh/repo-check head 64e357da7d7e merged into trunk a6db6a70cffa beside mesh-catalog#352@07717974 +1 more — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/one-group-one-plan merged: every member is merged
2026-10-17 21:32:35 +02:00
jschoubben 9a9a78f882 Playbook 07: report every ten minutes (the operator, 2026-10-17) 2026-10-17 21:28:46 +02:00
jschoubben e816b61cb7 Playbook 07: the prepare step splits an issue sized over an hour (the operator, 2026-10-17) 2026-10-17 21:25:39 +02:00
jschoubben 610be93803 Playbook 07: a fixing agent never runs an integration test locally (the operator, 2026-10-17) 2026-10-17 21:23:05 +02:00
jschoubben d5ddd1e715 Playbooks 07 and 08: no ready mark; the review record rides in the pull request and the Approve is asked once the checks pass (issue 1022, ADR 0421) 2026-10-17 20:55:47 +02:00
mesh-tools a6db6a70cf Merge pull request 'Playbook 07: a milestone has steps, runs the whole unit suite, writes its findings into its issues, watches on the smaller model; design 47 names milestone 2b and takes 895 (issue 1053)' (#412) from fix/1053-steps-not-phases into main 2026-10-17 18:29:24 +00:00
mesh-tools e44ec3f612 Merge pull request 'Playbook 07: a new node-engine key needs the controller first, on a branch of its own (issue 1051)' (#411) from fix/1051-pb07-controller-first into main 2026-10-17 18:28:25 +00:00
mesh-tools dbc7406490 Playbook 07: a new node-engine key needs the controller's pin moved first, and a release that must merge first has a branch name of its own (issue 1051)
mesh/merge-gate head dbc740649039 merged into trunk 90c67930969b — pass: the change touches no module of the mesh's graph
mesh/repo-check head dbc740649039 merged into trunk 90c67930969b — pass: its merge-check.sh passed
mesh/delivery delivered
Feature owned-paths (issue 1024) planned no controller release for a new key on directory resources, though the controller judges every machine with the node-engine's vendored validator; and the controller's release, on the feature's branch name, joined the catalogue's delivery group and could not merge before it.
2026-10-17 18:19:35 +00:00
jschoubben 2001d6e063 Playbook 07: a milestone has steps, runs the whole unit suite, writes its findings into its issues and watches on the smaller model; design 47 names milestone 2b and takes 895 into milestone 3 (issue 1053)
mesh/merge-gate head 2001d6e06334 merged into trunk e44ec3f61235 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 2001d6e06334 merged into trunk e44ec3f61235 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 20:19:16 +02:00
mesh-tools 90c6793096 Merge pull request 'Design 29: a folder a module owns says how it treats files it did not write (issue 1024)' (#409) from feat/owned-paths into main 2026-10-17 18:14:10 +00:00
mesh-tools 499407c669 Take in main up to 2cf1c64e24 (issue 1024)
mesh/merge-gate head 499407c66936 merged into trunk 2cf1c64e24f5 beside mesh-catalog#349@7104df6f — pass: the change touches no module of the mesh's graph
mesh/repo-check head 499407c66936 merged into trunk 2cf1c64e24f5 beside mesh-catalog#349@7104df6f — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/owned-paths merged: every member is merged
hq main moved after this change's check; it touches neither design 29 nor anything this change edits. The branch's tree is the reviewed design 29 change merged with main.
2026-10-17 18:09:09 +00:00
mesh-tools 2cf1c64e24 Merge pull request 'Playbook 07: the reviewer is kept for the whole release, and a watcher waits for its checks' (#410) from fix/pb07-reviewer-kept into main 2026-10-17 16:34:55 +00:00
jschoubben e1987b3654 Playbook 07: the reviewer is kept for the whole release, and a watcher waits for its checks (issue 1042)
mesh/merge-gate head e1987b365427 merged into trunk c87ab9af0f9a — pass: the change touches no module of the mesh's graph
mesh/repo-check head e1987b365427 merged into trunk c87ab9af0f9a — pass: its merge-check.sh passed
mesh/delivery delivered
The operator asked why review takes so long: a fresh reviewer per round or per mark reads
everything again, and a reviewer waiting on the forge's checks spends its run waiting.
2026-10-17 18:26:06 +02:00
mesh-tools c87ab9af0f Merge pull request 'Playbook 07: a design names its milestones' issues, never their delivery state' (#408) from fix/pb07-no-state-in-designs into main 2026-10-17 16:23:52 +00:00
mesh-tools 4e0dd1b0d3 Say in design 29 how a folder a module owns treats files it did not write (issue 1024, ADR 0422)
mesh/merge-gate out of date: judged against trunk 7a1d3d89f0b9, the trunk is now c87ab9af0f9a; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 7a1d3d89f0b9, the trunk is now c87ab9af0f9a; judged again on a recheck or when it is merged
mesh/delivery-group group feat/owned-paths checking: 0 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
Design 29 is ADR 0422's home: it now says what `unknown` and `knows` mean on a directory under a home, that the found class no longer holds in ~/.ssh (ADR 0415, 0417), and lists ADR 0412, 0415, 0416, 0417 and 0422 among its decisions.
2026-10-17 16:23:05 +00:00
jschoubben b5d3f23947 Design 57: its milestones name their issues, not their delivery state (issue 1039, folded into 1038)
mesh/merge-gate head b5d3f2394798 merged into trunk 7a1d3d89f0b9 — pass: the change touches no module of the mesh's graph
mesh/repo-check head b5d3f2394798 merged into trunk 7a1d3d89f0b9 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 18:20:58 +02:00
jschoubben c753ff068a Playbook 07: a design names its milestones' issues, never their delivery state (issue 1038)
mesh/merge-gate head c753ff068a55 merged into trunk 7a1d3d89f0b9 — pass: the change touches no module of the mesh's graph
mesh/repo-check head c753ff068a55 merged into trunk 7a1d3d89f0b9 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The tracker and the delivery records hold whether a milestone is delivered, when and by what;
a copy in the design's table is a central status kept by hand (AGENTS.md). Design 47 loses its
copied cells.
2026-10-17 18:18:45 +02:00
mesh-tools 7a1d3d89f0 Merge pull request 'Design 47: phase II release 1 delivered (issue 1025)' (#407) from fix/design-47-phase-2-release-1-delivered into main 2026-10-17 16:11:22 +00:00
mesh-tools cb79a28a80 Merge pull request 'Playbook 07: every wait on the wait list has a watcher agent on the bus' (#406) from fix/pb07-read-the-whole-log into main 2026-10-17 16:09:28 +00:00
jschoubben 0b9fda46cc Design 47: phase II release 1 delivered, 2026-10-17 (playbook 07, write-back)
mesh/merge-gate head 0b9fda46cc72 merged into trunk cb79a28a8062 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 0b9fda46cc72 merged into trunk cb79a28a8062 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 18:04:17 +02:00
jschoubben 6b58a6af82 Playbook 07: the glossary's word, the builder (review of #405)
mesh/merge-gate head 6b58a6af8246 merged into trunk c0b1c47be6b4 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 6b58a6af8246 merged into trunk c0b1c47be6b4 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 18:03:52 +02:00
jschoubben 9a39785116 Playbook 07: every wait on the wait list has a watcher agent on the bus (the operator, 2026-10-17)
mesh/merge-gate waiting in the build queue since 18:01
2026-10-17 18:00:35 +02:00
mesh-tools c0b1c47be6 Merge pull request 'Playbook 07: read a failed check's whole log; a change to merge-check.sh is judged by main's version' (#405) from fix/pb07-read-the-whole-log into main 2026-10-17 15:58:25 +00:00
jschoubben 06440c1522 Playbook 07: the build seat's holder, not a retired word (review of #405's check)
mesh/merge-gate head 06440c152298 merged into trunk cf73f9679771 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 06440c152298 merged into trunk cf73f9679771 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 17:56:21 +02:00
jschoubben 0e7eb9c2da Playbook 07: read a failed check's whole log; a change to merge-check.sh is judged by main's version
mesh/merge-gate head 0e7eb9c2dac8 merged into trunk cf73f9679771 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 0e7eb9c2dac8 merged into trunk cf73f9679771 — fail: its merge-check.sh failed: words: 1 problem(s)
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 17:29:43 +02:00
mesh-tools cf73f96797 Merge pull request 'Design 47: one plan per group, no order lines, the Approve as the only gate (ADR 0418, 0419, 0421)' (#404) from fix/design-47-records-0418-0421 into main 2026-10-17 15:25:14 +00:00
jschoubben b0cfd08f7c Design 47: ADR 0420 accepted, ADR 0249 superseded, milestone 3's issues named (review of #404)
mesh/merge-gate head b0cfd08f7c97 merged into trunk aaa0647ec31c — pass: the change touches no module of the mesh's graph
mesh/repo-check head b0cfd08f7c97 merged into trunk aaa0647ec31c — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 17:20:16 +02:00
jschoubben 0a6613f0d4 Design 47: one plan per group, no order lines, the Approve as the only gate (ADR 0418, 0419, 0421)
mesh/merge-gate head 0a6613f0d4b8 merged into trunk aaa0647ec31c — pass: the change touches no module of the mesh's graph
mesh/repo-check head 0a6613f0d4b8 merged into trunk aaa0647ec31c — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 17:18:03 +02:00
mesh-tools aaa0647ec3 Merge pull request 'Design 47: milestones 1 and 2 delivered' (#403) from fix/design-47-milestone-2-delivered into main 2026-10-17 13:10:39 +00:00
mesh-tools de32fb82ea Merge pull request 'Playbook 07: the full report shows the whole design's milestones' (#402) from fix/pb07-report-shows-the-design into main 2026-10-17 13:08:59 +00:00
jschoubben 85a729c607 Design 47: milestones 1 and 2 delivered, 2026-10-17 (playbook 07, write-back)
mesh/merge-gate head 85a729c607a1 merged into trunk de32fb82ea73 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 85a729c607a1 merged into trunk de32fb82ea73 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 15:06:38 +02:00
jschoubben fb284fa6ee Playbook 07: the full report shows the whole design's milestones, the running one's issues under it (the operator, 2026-10-17)
mesh/merge-gate head fb284fa6ee2a merged into trunk 2b04a5f0a1aa — pass: the change touches no module of the mesh's graph
mesh/repo-check head fb284fa6ee2a merged into trunk 2b04a5f0a1aa — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 14:13:28 +02:00
mesh-tools 2b04a5f0a1 Merge pull request 'Playbooks 07 and 04: a milestone is one release, never cut for size or time' (#401) from fix/pb07-a-milestone-is-one-release into main 2026-10-17 11:50:34 +00:00
jschoubben addbd74c39 Playbook 04 and design 47: the milestone is one release, not a feature step (review of #401)
mesh/merge-gate head addbd74c3924 merged into trunk 0f63a73def93 — pass: the change touches no module of the mesh's graph
mesh/repo-check head addbd74c3924 merged into trunk 0f63a73def93 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 13:47:37 +02:00
jschoubben 6bf18062c4 Playbooks 07 and 04: a milestone is one release, never cut for size or time; milestones are cut by release order (the operator, 2026-10-17) 2026-10-17 13:47:21 +02:00
mesh-tools 0f63a73def Merge pull request 'Describe the mesh MCP server as it runs and mark its design implemented (issue 1008)' (#400) from fix/1008-mcp-server-shipped into main 2026-10-17 11:43:03 +00:00
mesh-tools d0ee14712e Merge pull request 'Playbooks 07 and 08: a repository's own clone stays on main (issue 1010)' (#399) from fix/1010-clone-stays-on-main into main 2026-10-17 11:40:24 +00:00
jochen f6199fafce As-is 13: fix five review findings on the mesh MCP server (issue 1008, review round 1)
mesh/merge-gate head f6199fafcee4 merged into trunk d0ee14712e18 — pass: the change touches no module of the mesh's graph
mesh/repo-check head f6199fafcee4 merged into trunk d0ee14712e18 — pass: its merge-check.sh passed
mesh/delivery delivered
node-tools still has a manifest (invokes *, waits, the mcp-endpoint provision) —
describe it in prose instead of saying it has none. The TypeScript sources
remain on main; only the built artifact is Go. The mesh-controller seat
answers 63 verbs as its own runtime, not 60-93 per node (that is node-tools'
own node-scoped seat verbs). Prefer reading mesh_overview live over a count
that drifts. An as-is document amends nothing — reworded accordingly.
2026-10-17 13:39:14 +02:00
mesh-tools d901156614 Merge pull request 'Playbook 07: the lead writes a design's state back (issue 1009)' (#398) from fix/design-status-written-back into main 2026-10-17 11:38:12 +00:00
jochen e934a0286f Playbook 08: check that the clone stays on main (issue 1010, review round 1)
mesh/merge-gate head e934a0286f7f merged into trunk d9011566146e — pass: the change touches no module of the mesh's graph
mesh/repo-check head e934a0286f7f merged into trunk d9011566146e — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 13:37:09 +02:00
jochen 8dbfd8d8ff Playbooks 07 and 08: a repository's own clone stays on main (issue 1010)
A clone left on an old work branch hands the next session stale playbooks
and designs as if current.
2026-10-17 13:36:37 +02:00
jochen 2cea6c9b0c Date design 57's delivered milestones (issue 1009, review round 2)
mesh/merge-gate head 2cea6c9b0cf4 merged into trunk 84b464d109c0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 2cea6c9b0cf4 merged into trunk 84b464d109c0 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 13:35:56 +02:00
jochen 79dcc0a6ec Give design 57 its milestones and check every write-back moment (issue 1009, review round 1) 2026-10-17 13:35:20 +02:00
jochen 65f69d96a9 Describe the mesh MCP server as it runs and mark its design implemented (issue 1008)
mesh-console is gone: the tool runner (node-tools) is a core component on
every node, host-side and never a container (ADR 0175), announcing seven
tools by address (ADR 0195, 0367), not the five-tool container the old
as-is doc described. The to-be document's unsettled items get owners:
the mesh's own verbs already shipped (ADR 0154); identity and
per-assignment grants are design 58 milestone 6 (issue 883).
2026-10-17 13:35:12 +02:00
jochen b83a1e748e Playbook 07: the lead writes a design's state back (issue 1009)
Design 57 was merged as a release with three tasks left and still said
designed, so another session took it for unbuilt. Say it in-progress, with
what is left and when it is built.
2026-10-17 13:33:37 +02:00
mesh-tools 84b464d109 Merge pull request 'Playbook 07: one design is built at a time (issue 1007)' (#397) from fix/pb07-one-design-at-a-time into main 2026-10-17 11:26:12 +00:00
jochen 88d7db36c8 Playbook 07: the check counts milestone issues only (issue 1007, review round 1)
mesh/merge-gate head 88d7db36c866 merged into trunk c61f01d63eb8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 88d7db36c866 merged into trunk c61f01d63eb8 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 13:23:21 +02:00
jochen 974b3e7d42 Playbook 07: one design is built at a time (issue 1007)
The operator decided on 2026-10-17 that a second design's milestones wait
until the design being built is delivered; nothing in the playbooks said so.
2026-10-17 13:22:41 +02:00
mesh-tools c61f01d63e Merge pull request 'Playbook 07: one milestone is one branch per repository' (#395) from fix/pb07-one-branch-per-milestone into main 2026-10-17 10:20:34 +00:00
mesh-tools 4325467b4d Merge pull request 'Design 58: milestones 11 to 14 from research 057, as-is 17, and role names in place of machine and login names' (#396) from design/058-milestones-11-to-14 into main 2026-10-17 10:19:33 +00:00
mesh-tools 0a010b8b79 Design 58: milestone 13 does not wait for milestone 7; until then nox's dangerous acts are refused (review round 1 of #396)
mesh/merge-gate head 0a010b8b79a1 merged into trunk 85f0aecf5ad6 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 0a010b8b79a1 merged into trunk 85f0aecf5ad6 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 10:17:00 +00:00
mesh-tools cc0503ef66 Design 58 and 36: role names for the machines and no operator logins, as a public repository needs (fixes what #394 wrote)
mesh/merge-gate head cc0503ef663a merged into trunk 85f0aecf5ad6 — pass: the change touches no module of the mesh's graph
mesh/repo-check head cc0503ef663a merged into trunk 85f0aecf5ad6 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 10:13:02 +00:00
mesh-tools 56d8d546c7 Design 58: milestones 11 to 14 from research 057 with their issues, and as-is 17 on today's accounts (playbook 04) 2026-10-17 10:10:42 +00:00
jschoubben edff1f25a1 Playbook 07: one milestone is one branch per repository, its steps reuse the name (the operator, 2026-10-17)
mesh/merge-gate head edff1f25a10b merged into trunk 4325467b4d59 — pass: the change touches no module of the mesh's graph
mesh/repo-check head edff1f25a10b merged into trunk 4325467b4d59 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 12:04:37 +02:00
mesh-tools 85f0aecf5a Merge pull request 'Design 58 and 36: identities carry logins, run for an identity, a bus password per identity (research 057)' (#394) from design/057-identities into main 2026-10-17 09:54:42 +00:00
mesh-tools 0dc8cab178 Design 58 and 36: cite ADR 0266's ways to root, and make acceptance tests 7 and 9 and the nox glossary entry agree with ADR 0411, 0412 and 0414 (review round 1)
mesh/merge-gate head 0dc8cab1781b merged into trunk 375e77690b8a — pass: the change touches no module of the mesh's graph
mesh/repo-check head 0dc8cab1781b merged into trunk 375e77690b8a — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 09:40:48 +00:00
mesh-tools 98b731d5d1 Design 58 and 36: identities carry logins, run for an identity, a bus password per identity (ADR 0411 to 0414, research 057)
mesh/merge-gate head 98b731d5d1ff merged into trunk 375e77690b8a — pass: the change touches no module of the mesh's graph
mesh/repo-check head 98b731d5d1ff merged into trunk 375e77690b8a — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 09:35:37 +00:00
mesh-tools 375e77690b Merge pull request 'Feature agents-write-git, step 2: design 29 cites ADR 0399 and ADR 0400 (issue 955)' (#390) from feat/agents-write-git-ssh into main 2026-10-16 22:31:07 +00:00
jschoubben 24701df23e Merge branch 'main' into feat/agents-write-git-ssh
mesh/merge-gate head 24701df23e8a merged into trunk dd5a06529556 beside mesh-catalog#340@ae9cda85 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 24701df23e8a merged into trunk dd5a06529556 beside mesh-catalog#340@ae9cda85 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/agents-write-git-ssh merged: every member is merged
2026-10-16 22:27:02 +00:00
mesh-tools dd5a065295 Merge pull request 'Design 47's milestones and issues, and the as-is delivery as it runs today' (#393) from handoff/47-milestones into main 2026-10-16 22:09:55 +00:00
jschoubben 748641dd22 Review of #393: rollouts run one at a time today (issue 525); Approve on Telegram (issue 953 for the terminal); 976's place in the order; milestone 1 takes the rest of 525 and acceptance test 9
mesh/merge-gate head 748641dd22ae merged into trunk 73b9cc3257a1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 748641dd22ae merged into trunk 73b9cc3257a1 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 00:06:59 +02:00
jschoubben 75d3ac0f82 Playbooks 04 and 07 after research 055: a milestone's shape from what-if; the Approve on every merge into main; when the two-step exception ends (issue 922)
mesh/merge-gate head 75d3ac0f8258 merged into trunk 73b9cc3257a1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 75d3ac0f8258 merged into trunk 73b9cc3257a1 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 00:04:48 +02:00
jschoubben b6a191d0a6 Design 47's milestones and issues (playbook 04), and the as-is delivery rewritten from research 055's findings
mesh/merge-gate head b6a191d0a6dc merged into trunk 73b9cc3257a1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head b6a191d0a6dc merged into trunk 73b9cc3257a1 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The operator confirmed six milestones on 2026-10-17; issues 973-981 and 971
carry their repository parts, linked depends-on in order. The as-is described
the predecessor's pipeline; it now says what runs today, which design 47
replaces.
2026-10-17 00:04:00 +02:00
mesh-tools 73b9cc3257 Merge pull request 'Playbook 08: a fix that moves a file to a new filler compares the values first (issue 957)' (#391) from fix/957-playbook-08-first-fill into main 2026-10-16 21:56:23 +00:00
mesh-tools f795356ae1 Merge pull request 'Design 47 rewritten from research 055's decisions: hotfix, feature and epic from the trunk, delivery on the bus' (#392) from graduate/055-nats-native-delivery into main 2026-10-16 21:50:05 +00:00
jschoubben aa4bfe289e Design 47 and the glossary: the reviewer's findings (three work queues, a feature in one repository, what-if, ADR 0406 clause numbers, backups by protection, no suite-time claim, the Go module's judge account)
mesh/merge-gate head aa4bfe289ee4 merged into trunk f9c240a957b8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head aa4bfe289ee4 merged into trunk f9c240a957b8 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 23:44:59 +02:00
jschoubben 6f0db74ee4 Design 47: the SDK commits the catalogue accepts (ADR 0402 §4)
mesh/merge-gate head 6f0db74ee47f merged into trunk f9c240a957b8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 6f0db74ee47f merged into trunk f9c240a957b8 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 23:42:56 +02:00
jschoubben e7854a31a6 Design 47: a member naming the SDK member's head is proven by its own merge check (ADR 0402's review)
mesh/merge-gate head e7854a31a6a9 merged into trunk f9c240a957b8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head e7854a31a6a9 merged into trunk f9c240a957b8 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 23:41:37 +02:00
jschoubben 8bfb11da2c Design 47 and the glossary: the batch's record in the chain is the rollout (ADR 0408's review) 2026-10-16 23:40:11 +02:00
jschoubben 96f2ae72c1 Rewrite design 47 from research 055's decisions: hotfix, feature and epic from the trunk, delivery on the bus
ADR 0401-0408 graduate research 055: no release branch, two more ordering
rules, the composed check as each member's gate, a backup before the send,
installed toolchains, events and work queues on the bus, the merge window
as keys, and one chain kept on the bus alone. Design 47 is rewritten from
them (one design per thing); the glossary takes hotfix, feature, toolchain,
milestone and epic and retires release branch; design 18 loses the
toolchain image; playbooks 07 and 08 drop the release-branch wait.
2026-10-16 23:39:12 +02:00
nox 78353ad272 Playbook 08: a fix that moves a file to a new filler compares the values first (issue 957)
mesh/merge-gate head 78353ad272a3 merged into trunk f795356ae1d6 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 78353ad272a3 merged into trunk f795356ae1d6 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 22:37:59 +02:00
nox fa5fd9fc69 Cite ADR 0399 and 0400 in design 29: the account's forge key lives in ~/.ssh (issue 955)
mesh/merge-gate out of date: judged against trunk f9c240a957b8, the trunk is now f795356ae1d6; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f9c240a957b8, the trunk is now f795356ae1d6; judged again on a recheck or when it is merged
mesh/delivery-group group feat/agents-write-git-ssh checking: 0 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 22:37:15 +02:00
mesh-tools f9c240a957 Merge pull request 'Playbook 07: report the whole issue tree every five minutes, every two while a delivery runs' (#389) from fix/pb07-overview-every-five into main 2026-10-16 19:34:36 +00:00
mesh-tools fff89b7c62 Playbook 07: report the whole issue tree every five minutes, every two while a delivery runs
mesh/merge-gate head fff89b7c6218 merged into trunk 80e36c75957f — pass: the change touches no module of the mesh's graph
mesh/repo-check head fff89b7c6218 merged into trunk 80e36c75957f — pass: its merge-check.sh passed
mesh/delivery delivered
The operator asked for a status update more often, with an overview of the issues worked on.
2026-10-16 19:18:42 +00:00
mesh-tools 80e36c7595 Merge pull request 'Playbook 09 and the hq-cleanup skill: a cleanup that deletes only what is proven done' (#388) from process/09-cleanup into main 2026-10-16 19:08:33 +00:00
jschoubben 3c49b389d1 Playbook 09: a deletion is approved as an ask on the operator's warrant once design 58 milestone 7 is built; until then the operator runs the approved list
mesh/merge-gate head 3c49b389d1e7 merged into trunk b29008ad36f4 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 3c49b389d1e7 merged into trunk b29008ad36f4 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 20:51:00 +02:00
jschoubben 222fbe95f4 Playbook 09 and the hq-cleanup skill: a cleanup reads first, deletes only what the operator approves, and reports what it kept (the operator, 2026-10-16)
mesh/merge-gate head 222fbe95f4bb merged into trunk b29008ad36f4 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 222fbe95f4bb merged into trunk b29008ad36f4 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 20:01:13 +02:00
mesh-tools b29008ad36 Merge pull request 'Playbook 02: one design per thing' (#387) from process/02-one-design-per-thing into main 2026-10-16 17:41:22 +00:00
jschoubben 101d7f30f7 Playbook 02: one design per thing; a graduation rewrites the design that covers it instead of writing a second (the operator, 2026-10-16)
mesh/merge-gate head 101d7f30f7e2 merged into trunk 225eee14d0bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 101d7f30f7e2 merged into trunk 225eee14d0bf — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 19:40:48 +02:00
mesh-tools 225eee14d0 Merge pull request 'Design 47: a move of the trunk asks every active check again (ADR 0395)' (#381) from design/47-recheck-on-trunk-move into main 2026-10-16 17:35:52 +00:00
jschoubben 12e75cdfa2 Design 47: the transition paragraph, the recheck verb and the section's bullets say ADR 0395 whole (review)
mesh/merge-gate head 12e75cdfa215 merged into trunk f1081c4f9768 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 12e75cdfa215 merged into trunk f1081c4f9768 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 19:27:52 +02:00
jschoubben 377fa3f88a Design 47: say ADR 0395 as accepted — a conflict is asked again at every move, an ended check's builds are not reused, a group's composed check reruns through its members, and the cost bound 2026-10-16 19:26:47 +02:00
jschoubben 11e32ce5be Merge remote-tracking branch 'origin/main' into design/47-recheck-on-trunk-move 2026-10-16 19:26:35 +02:00
mesh-tools f1081c4f97 Merge pull request 'Playbook 08: a second fault the delivery shows joins the issue only in the same files' (#386) from fix/pb08-second-fault into main 2026-10-16 17:09:53 +00:00
mesh-tools 152cc0c0e7 Playbook 08: a second fault the delivery shows joins the issue only in the same files
mesh/merge-gate head 152cc0c0e77c merged into trunk a8b951f3b488 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 152cc0c0e77c merged into trunk a8b951f3b488 — pass: its merge-check.sh passed
mesh/delivery delivered
Taken from issue 937, where mesh-catalog #335 was taken into the issue
and issue 943 was opened for a fault in other code.
2026-10-16 17:06:25 +00:00
mesh-tools a8b951f3b4 Merge pull request 'Playbook 08: needing a decision keeps a fix small' (#385) from process/08-a-decision-keeps-a-fix-small into main 2026-10-16 15:58:01 +00:00
jschoubben 0eadc27fc7 Playbook 08: needing a decision keeps a fix small; a feature is several modules with research or a design before it (the operator, 2026-10-16)
mesh/merge-gate head 0eadc27fc722 merged into trunk 4e437eb1488f — pass: the change touches no module of the mesh's graph
mesh/repo-check head 0eadc27fc722 merged into trunk 4e437eb1488f — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 17:46:23 +02:00
mesh-tools 4e437eb148 Merge pull request 'Design 58: agent work as the mesh's own (graduation of research 053)' (#372) from design/58-agent-work-as-the-meshs-own into main 2026-10-16 14:09:25 +00:00
jschoubben 0e7a115183 Merge pull request 'Playbooks 07 and 08: mark ready with mesh-delivery.mark-ready (issue 913)' (#383) from feat/913-mark-ready into main
Reviewed-on: #383
2026-10-16 13:47:15 +00:00
jschoubben 18428baf4b Merge remote-tracking branch 'origin/main' into design/58-agent-work-as-the-meshs-own
mesh/merge-gate head 18428baf4b23 merged into trunk 0e7a115183a9 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 18428baf4b23 merged into trunk 0e7a115183a9 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 14:53:11 +02:00
jschoubben 7b5f3582be Playbooks 07 and 08: mark ready with mesh-delivery.mark-ready, label then comment in the order the forge counts (issue 913)
mesh/merge-gate head 7b5f3582be90 merged into trunk 6dcfa2226e63 beside mesh-controller#357@114c99e0 +1 more — pass: the change touches no module of the me…
mesh/repo-check head 7b5f3582be90 merged into trunk 6dcfa2226e63 beside mesh-controller#357@114c99e0 +1 more — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/913-mark-ready merged: every member is merged
2026-10-16 14:31:49 +02:00
jschoubben 6dcfa2226e Merge pull request 'Playbooks 01 and 02: place research in the domains; its words reach the glossary at graduation' (#382) from playbooks/01-place-it-in-the-domains into main
Reviewed-on: #382
2026-10-16 12:24:26 +00:00
jschoubben 1ddc2f5cbf Merge pull request 'Design 41: every file on a node is managed through node-files (WP9, ADR 0394)' (#380) from design/41-node-files into main
Reviewed-on: #380
2026-10-16 12:24:18 +00:00
jschoubben 06998cc8d9 Merge pull request 'Playbooks 07 and 08: the five rules from the delivery-time audit' (#375) from playbooks/07-08-time-sinks into main
Reviewed-on: #375
2026-10-16 12:24:07 +00:00
jschoubben 353bfdcef3 Merge remote-tracking branch 'origin/main' into design/58-agent-work-as-the-meshs-own-tmp
mesh/merge-gate out of date: judged against trunk b9af4e373dc0, the trunk is now 6dcfa2226e63; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk b9af4e373dc0, the trunk is now 6dcfa2226e63; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 14:00:12 +02:00
jschoubben 91f82cd38e Design 47: a group's own merge in order asks nothing again (ADR 0395 §1, review)
mesh/merge-gate out of date: judged against trunk b9af4e373dc0, the trunk is now 6dcfa2226e63; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk b9af4e373dc0, the trunk is now 6dcfa2226e63; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 13:32:39 +02:00
jschoubben 300be92814 Design 41 WP9: readlink and the account and group lookup are node-files verbs too (ADR 0394 §1; review)
mesh/merge-gate head 300be928144e merged into trunk b9af4e373dc0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 300be928144e merged into trunk b9af4e373dc0 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 13:31:45 +02:00
jschoubben 427fe4c9c5 Playbooks 01 and 02: place research in the domains before its direction, and carry its words into the glossary at graduation (ADR 0244)
mesh/merge-gate head 427fe4c9c58b merged into trunk b9af4e373dc0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 427fe4c9c58b merged into trunk b9af4e373dc0 — pass: its merge-check.sh passed
mesh/delivery delivered
The operator, 2026-10-16: research defines the modules a feature lands in and the domain extensions and
reductions that come with it.
2026-10-16 12:38:59 +02:00
jschoubben a65e1076a2 Design 47: a pull request that no longer merges cleanly is told, not checked (ADR 0395 §2)
mesh/merge-gate head a65e1076a2e5 merged into trunk b9af4e373dc0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head a65e1076a2e5 merged into trunk b9af4e373dc0 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 11:42:03 +02:00
jschoubben 09ec150bd6 Design 47: a move of the trunk asks every active check again, one at a time per pull request (ADR 0395, issue 888)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate error: the check could not run: killed by hand
mesh/repo-check error: the check could not run: killed by hand
2026-10-16 11:40:14 +02:00
jschoubben 1efa37957d Playbooks 07 and 08: is the coast clear? Hold a push while a release branch in flight shares its files; the glossary's release branch
mesh/merge-gate head 1efa37957db2 merged into trunk b9af4e373dc0 siblings unknown — pass: the change touches no module of the mesh's graph
mesh/repo-check head 1efa37957db2 merged into trunk b9af4e373dc0 siblings unknown — pass: its merge-check.sh passed
mesh/delivery delivered
The operator, 2026-10-16: the file module waited for feature 57's release branch, which changes four of the
same controller files.
2026-10-16 11:28:32 +02:00
jschoubben 26073f30b0 Design 41: node-reads goes in the delivery after node-files is held on every node (ADR 0394 §9)
mesh/merge-gate head 26073f30b0de merged into trunk b9af4e373dc0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 26073f30b0de merged into trunk b9af4e373dc0 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 11:14:01 +02:00
jschoubben 1f5a12e07e Design 41: WP8's node-reads is replaced by WP9's node-files (ADR 0394 supersedes ADR 0310)
mesh/merge-gate head 1f5a12e07e42 merged into trunk b9af4e373dc0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 1f5a12e07e42 merged into trunk b9af4e373dc0 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 11:11:53 +02:00
jschoubben 7ac5194bcb Design 41 WP9: mesh-files runs as the operator account with sudo where needed, and replaces node-reads (ADR 0394)
mesh/merge-gate head 7ac5194bcbdc merged into trunk b9af4e373dc0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 7ac5194bcbdc merged into trunk b9af4e373dc0 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 11:06:32 +02:00
jschoubben 330d341618 Design 41 WP9: node-files is held by a module of its own, mesh-files, beside mesh-reads
mesh/merge-gate waiting in the build queue since 11:01
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 11:01:43 +02:00
jschoubben cf56649845 Design 41: every file on a node is managed through node-files (WP9, ADR 0394, issue 900)
mesh/merge-gate waiting in the build queue since 11:01
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 11:00:46 +02:00
mesh-tools b9af4e373d Merge pull request 'Feature 57 (config and provisioning): release hq's record' (#378) from release/57-config-and-provisioning into main 2026-10-16 08:41:58 +00:00
mesh-tools e7417cf124 Feature 57 (config and provisioning): release hq's record (glossary words, release notes) on top of main
mesh/merge-gate head e7417cf1248c merged into trunk c22b58143b74 beside mesh-host#116@bff1ac14 — pass: the change touches no module of the mesh's graph
mesh/repo-check head e7417cf1248c merged into trunk c22b58143b74 beside mesh-host#116@bff1ac14 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group release/57-config-and-provisioning merged: every member is merged
2026-10-16 08:24:14 +00:00
jschoubben 47f7321c05 Playbooks 07 and 08: integration tests only in the merge check, no try before it, no lab and no end-to-end tests yet (the operator, 2026-10-16)
mesh/merge-gate out of date: judged against trunk c22b58143b74, the trunk is now b9af4e373dc0; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk c22b58143b74, the trunk is now b9af4e373dc0; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 10:15:31 +02:00
jschoubben f95043e379 Playbooks 07 and 08: a module's acceptance is an integration test the forge runs, and try runs on the builder (ADR 0391)
mesh/merge-gate head f95043e3797e merged into trunk c22b58143b74 — pass: the change touches no module of the mesh's graph
mesh/repo-check head f95043e3797e merged into trunk c22b58143b74 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 10:14:42 +02:00
jschoubben cb98a08384 Playbooks 07 and 08: prove a module whole on a throwaway bus and store, not in the lab while its runs are too slow (the operator, 2026-10-16) 2026-10-16 10:14:25 +02:00
jschoubben 4e822c56a0 Playbooks 07 and 08: checks for every new rule, the ready mark's bound in both, the lead marks an unreviewed edit (review round 1) 2026-10-16 10:14:25 +02:00
jschoubben c5fa1de264 Playbooks 07 and 08: review asks only what makes a change wrong, try before opening, mark ready on approval, prove a module whole first, hand back stopped issues
From the delivery-time audit of 2026-10-16: seven review rounds on issue 704, a feature branch's first
check failing on 29 problems, mesh-catalog #300 waiting 12 h unmarked, the mail modules failing live,
98 issues held in fixing.
2026-10-16 10:14:25 +02:00
jschoubben ea359ea4e6 Try a small fix's branch before its pull request, since local runs miss the check's toolchain (playbook 08) 2026-10-16 10:14:25 +02:00
mesh-tools c22b58143b Merge pull request 'Test levels and time budgets in how-we-build and playbooks 07 and 08 (issue 891, ADR 0391)' (#374) from fix/891-test-levels-and-budgets into main 2026-10-16 08:13:26 +00:00
mesh-tools d9dd8a63d8 Say ADR 0391's test levels and time budgets in how-we-build and playbooks 07 and 08 (issue 891)
mesh/merge-gate head d9dd8a63d83b merged into trunk 08d6476e5287 — pass: the change touches no module of the mesh's graph
mesh/repo-check head d9dd8a63d83b merged into trunk 08d6476e5287 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 08:09:33 +00:00
jschoubben 43c3f2c098 Design 58: the operator confirmed the milestones; issues 878 to 887, building after design 57's delivery (playbook 04)
mesh/merge-gate out of date: judged against trunk 08d6476e5287, the trunk is now c22b58143b74; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 08d6476e5287, the trunk is now c22b58143b74; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 09:55:11 +02:00
jschoubben 45e2d7e972 Design 58: milestones for the operator to confirm, owners named, and the as-is of logins on the bus (playbook 04)
mesh/merge-gate head 45e2d7e9722d merged into trunk 08d6476e5287 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 45e2d7e9722d merged into trunk 08d6476e5287 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 03:03:21 +02:00
jschoubben 9844020589 Glossary: Uses lines, the thread's two records, the grant from ADR 0386 §4, the provisioner's harness, issue 872 for the seat-sense role (review round 1)
mesh/merge-gate head 9844020589df merged into trunk 08d6476e5287 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 9844020589df merged into trunk 08d6476e5287 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 02:49:22 +02:00
jschoubben 535fd8bb54 Glossary: the words of research 053's graduation, and harness and role as homonyms
mesh/merge-gate head 535fd8bb5498 merged into trunk 08d6476e5287 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 535fd8bb5498 merged into trunk 08d6476e5287 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Playbook 02 adds a graduated effort's words; design 58 now says agent harness, never the bare word.
2026-10-16 02:47:48 +02:00
jschoubben cb4c18cedc Design 58: the identity store is read by grant, and sessions never call through the tool runner's login (review round 1)
mesh/merge-gate head cb4c18cedcf0 merged into trunk 08d6476e5287 — pass: the change touches no module of the mesh's graph
mesh/repo-check head cb4c18cedcf0 merged into trunk 08d6476e5287 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 02:45:58 +02:00
jschoubben 8f8f1b4337 Design 58: agent work as the mesh's own, graduated from research 053 with ADR 0383 to ADR 0390
Threads on the bus, a headless thread handler in a sandbox per session, a workspace toolkit with no
general command runner, one identity seat, effects on every tool, signed bus logins and the switch
to them; names the build's acceptance tests for the milestones.
2026-10-16 02:44:15 +02:00
mesh-tools 08d6476e52 Merge pull request 'Playbooks 01, 02 and 04: settle decisions before versions, test at the cheapest level, acceptance tests through to milestones' (#371) from playbooks/01-settle-before-versions into main 2026-10-16 00:15:00 +00:00
jschoubben 80b647a2ed Name the acceptance tests in the milestone table and say what the review returns (review round 2)
mesh/merge-gate head 80b647a2ed66 merged into trunk f8c0676b3a1d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 80b647a2ed66 merged into trunk f8c0676b3a1d — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 02:12:17 +02:00
jschoubben a4c5ef69bb Give the build's acceptance tests a home in the design and milestones, and answer review round 1
The research playbook named acceptance tests the milestones carry, which
playbooks 02 and 04 did not know; its pre-check also disagreed with the
knowledge base's deferral rule.
2026-10-16 02:11:27 +02:00
jschoubben ada703e790 Write a graduation's records side by side, from the confirmed direction (playbook 02)
Proposing records one after another serialises their reviews and the
operator's asks; retyping the effort's findings into each record makes them
long and gives the review more to send back.
2026-10-16 02:09:57 +02:00
jschoubben 2a61c57258 Settle research decisions before proposing versions, and test on the cheapest bed (playbook 01)
Research 053's identity part took six versions and about two hours: options
that broke the mesh's own rules were offered and argued back, a version was
proposed after every answer, and reviews returned findings a pre-check
would have caught. A lab run was planned where an in-process test answered.
2026-10-16 02:09:22 +02:00
mesh-tools f8c0676b3a Merge pull request 'Cite ADR 0382 in the delivery design (issue 867)' (#370) from fix/867-group-siblings into main 2026-10-15 23:52:50 +00:00
mesh-toolsandClaude e4a55051cc Cite ADR 0382 in the delivery design: a group's member is checked beside its siblings (issue 867)
mesh/merge-gate head e4a55051cc3b merged into trunk 9bfd57174fab — pass: the change touches no module of the mesh's graph
mesh/repo-check head e4a55051cc3b merged into trunk 9bfd57174fab — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group fix/867-group-siblings merged: every member is merged
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-15 23:27:14 +00:00
mesh-tools 9bfd57174f Merge pull request 'Playbook 08: a small fix may cover two or three related issues in one repository' (#368) from playbooks/08-related-issues into main 2026-10-15 20:38:39 +00:00
jschoubben a227f98bdd Let a small fix cover two or three related issues in one repository (playbook 08)
mesh/merge-gate head a227f98bddd4 merged into trunk 409fa9bbc979 — pass: the change touches no module of the mesh's graph
mesh/repo-check head a227f98bddd4 merged into trunk 409fa9bbc979 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 22:37:12 +02:00
mesh-tools 409fa9bbc9 Merge pull request 'Playbook 08: a small fix gets three review rounds, as a feature does' (#367) from playbooks/08-three-rounds into main 2026-10-15 19:29:07 +00:00
jschoubben 747fc546c1 Allow a small fix three review rounds, as a feature has (playbook 08)
mesh/merge-gate head 747fc546c1df merged into trunk 046bbd3b308d — pass: the change touches no module of the mesh's graph
mesh/repo-check head 747fc546c1df merged into trunk 046bbd3b308d — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 21:28:29 +02:00
mesh-tools 046bbd3b30 Merge pull request 'Playbooks 01, 02 and 04: one cycle from idea to delivered milestones' (#366) from playbooks/01-02-04-the-cycle into main 2026-10-15 19:23:08 +00:00
jschoubben 99243ec47e Confirm milestones before opening their issues, name issue 845 for the check, keep 01's withdrawal and history lines
mesh/merge-gate head 99243ec47efa merged into trunk 5ff97ec3aa02 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 99243ec47efa merged into trunk 5ff97ec3aa02 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 21:19:01 +02:00
jschoubben 0d66edb24c Make playbooks 01, 02 and 04 one cycle from idea to delivered milestones
The operator found 04 odd on 2026-10-15: it went from a settled design straight to building,
with no work breakdown, and contradicted playbook 07 on pull requests. 01 now starts from the
operator's idea with setup questions, findings and a direction the operator confirms; 02
graduates that direction into one record per reversible choice and hands the design on; 04
breaks it into milestones usable on their own, one issue per repository part, each milestone
one feature step of 07. 01 notes the knowledge base's insight-only rule for an active effort
until issue 844 is delivered.
2026-10-15 21:18:15 +02:00
mesh-tools 5ff97ec3aa Merge pull request 'Say rollout, not walk (issue 646)' (#365) from fix/646-rollout-not-walk into main 2026-10-15 17:59:22 +00:00
jschoubben 75ac87c72b Merge remote-tracking branch 'origin/main' into fix/646-rollout-not-walk
mesh/merge-gate head 75ac87c72b6a merged into trunk 058d76ee8a4e — pass: the change touches no module of the mesh's graph
mesh/repo-check head 75ac87c72b6a merged into trunk 058d76ee8a4e — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 19:57:54 +02:00
jschoubben 437523554e Say rollout, not walk, for the sending of a batch across nodes (issue 646) 2026-10-15 19:57:20 +02:00
mesh-tools 058d76ee8a Merge pull request 'Playbook 07 as five bounded phases with reused context; playbook 08 for small fixes' (#364) from playbooks/07-08-fast-flow into main 2026-10-15 17:54:22 +00:00
jschoubben db6430ccbf Show the report's issues as a tree, and keep the agent's brief, the red-check rule and the forge-commit check, in playbook 07
mesh/merge-gate head db6430ccbf39 merged into trunk 99a9da095010 — pass: the change touches no module of the mesh's graph
mesh/repo-check head db6430ccbf39 merged into trunk 99a9da095010 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-15 19:52:48 +02:00
jschoubben 458f248c05 Give the lead's ten-minute report a full and a short layout, in playbook 07 2026-10-15 19:52:31 +02:00
jschoubben 464833d0ef Rewrite playbook 07 as five bounded phases with reused context, and add playbook 08 for small fixes
mesh/merge-gate head 464833d0ef3f merged into trunk 99a9da095010 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 464833d0ef3f merged into trunk 99a9da095010 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A feature waited a day for its last issue, each small issue paid for its context again, and
waits nobody saw cost the most time. The operator set the flow on 2026-10-15: prepared context
per repository, one to two hours of local development on unit tests, one review of at most three
rounds back to the same agents, one push for the integration checks, then the delivery plan;
steps for a larger feature; a separate playbook for small fixes; models per step; a wait list
with bounds; a prepared clear at each phase end.
2026-10-15 19:26:02 +02:00
mesh-tools 99a9da0950 Merge pull request 'Add three lessons from feature mail-hardening to playbook 07' (#363) from playbook-07-lessons into main 2026-10-15 15:14:16 +00:00
jochen de8a7176ab Add three lessons from feature mail-hardening to playbook 07
mesh/merge-gate head de8a7176abb8 merged into trunk a7d4c3953599 — pass: the change touches no module of the mesh's graph
mesh/repo-check head de8a7176abb8 merged into trunk a7d4c3953599 — pass: its merge-check.sh passed
mesh/delivery delivered
A temporary-directory quota failure, a delivery group's manifest-check
ordering, and an agent that stops or finishes without committing each cost
time during that feature because the playbook did not yet say what to do.
Writing them down so the next feature avoids the same detours.
2026-10-15 17:08:43 +02:00
mesh-tools a7d4c39535 Merge pull request 'Playbook 07: an issue is one repository and five to ten minutes; a long run says where it stands' (#362) from pb07-issue-size into main 2026-10-15 12:40:48 +00:00
mesh-toolsandClaude Opus 5.5 048f14540a Size an issue for one repository and five to ten minutes, and have a long run say where it stands, in playbook 07
mesh/merge-gate head 048f14540a50 merged into trunk c2ee44255e5b — pass: the change touches no module of the mesh's graph
mesh/repo-check head 048f14540a50 merged into trunk c2ee44255e5b — pass: its merge-check.sh passed
mesh/delivery delivered
On 2026-10-15 issues spanning one repository finished in two to four minutes, while those spanning two or
three ran for thirty to sixty, mostly merging the feature head and checking again as other agents landed.
The operator asked for agents that finish in five to ten minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:37:33 +00:00
mesh-tools c2ee44255e Merge pull request 'Playbook 07: the ready note and the brief a fixing agent follows' (#361) from pb07-agent-brief into main 2026-10-15 12:33:46 +00:00
mesh-toolsandClaude Opus 5.5 ddf23a7861 Put the ready note and the brief a fixing agent follows into playbook 07
mesh/merge-gate head ddf23a786101 merged into trunk f3bbfd4bf628 — pass: the change touches no module of the mesh's graph
mesh/repo-check head ddf23a786101 merged into trunk f3bbfd4bf628 — pass: its merge-check.sh passed
mesh/delivery delivered
They lived in a file on one machine that only one session's agents read. They are how every feature's
agents work, so they belong with the method; a feature keeps only its local facts (paths, branch, locks).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:33:00 +00:00
mesh-tools f3bbfd4bf6 Merge pull request 'Playbook 07: up to five Sonnet agents at once, at most two on the larger model' (#360) from pb07-five-agents into main 2026-10-15 12:02:56 +00:00
mesh-toolsandClaude Opus 5.5 893d90fb41 Allow up to five Sonnet agents at once and keep the larger model to two, in playbook 07
mesh/merge-gate head 893d90fb417e merged into trunk 1e577168bf0c — pass: the change touches no module of the mesh's graph
mesh/repo-check head 893d90fb417e merged into trunk 1e577168bf0c — pass: its merge-check.sh passed
mesh/delivery delivered
The operator, 2026-10-15: the smaller model is as good as free against the weekly allowance, so the
limit that matters is how many agents the lead can settle and follow (three to five); subagents on the
larger model stay at two at most and as rare as possible.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 12:00:09 +00:00
mesh-tools 1e577168bf Merge pull request 'Playbook 07: three to five review rounds, the reviewer told its round, and the allowance in every report' (#359) from pb07-review-rounds into main 2026-10-15 11:22:56 +00:00
mesh-toolsandClaude Opus 5.5 f349c940f0 Cap review rounds at three to five and report the allowance used, in playbook 07
mesh/merge-gate head f349c940f0fe merged into trunk 1699375476c1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head f349c940f0fe merged into trunk 1699375476c1 — pass: its merge-check.sh passed
mesh/delivery delivered
A reviewer that does not know its round keeps finding the next smaller thing: one decision record went
through nine rounds on 2026-10-15, each spending the operator's shared weekly allowance. The operator
asked for three to five rounds, the round told to the reviewer, and the allowance in every report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-15 11:21:37 +00:00
mesh-tools 1699375476 Merge pull request 'Playbook 07: the lead settles an issue's questions before a smaller-model agent builds it, two at a time' (#358) from pb07-builders into main 2026-10-14 23:26:01 +00:00
mesh-toolsandClaude Opus 5.5 912fd428fb Have the lead settle an issue's questions before a smaller-model agent builds it, two at a time
mesh/merge-gate head 912fd428fb89 merged into trunk a75faad88240 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 912fd428fb89 merged into trunk a75faad88240 — pass: its merge-check.sh passed
mesh/delivery delivered
Five agents at once, each stopping on an undecided design question and restarting, spent a
week's allowance in a night. Also write down the lessons of feature 57: never land on a red
check, a new manifest word lands with the controller that reads it, and review the tree that
is pushed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 23:24:14 +00:00
mesh-admin a75faad882 Merge pull request 'To-be 42: the kernel module holds the kernel's run-time settings, huge pages first (ADR 0369, issue 746)' (#357) from design/42-kernel-module-holds-runtime-settings into main 2026-10-14 21:52:48 +00:00
jschoubben 9f4757a8e4 Let the kernel module hold the kernel's run-time settings, huge pages first (ADR 0369, issue 746)
mesh/merge-gate head 9f4757a8e49d merged into trunk 93f8bbfc8eaf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 9f4757a8e49d merged into trunk 93f8bbfc8eaf — pass: its merge-check.sh passed
mesh/delivery delivered
Huge pages at always flooded the control-node's spinning disks with journal
write-back and cost the controller its lease; the setting needs an owner,
and the kernel module is the one to-be 42 already names for the kernel.
2026-10-14 23:48:35 +02:00
mesh-admin 93f8bbfc8e Merge pull request 'To-be 48: cite ADR 0368, the resolver restarts itself for that module alone' (#356) from design/48-cites-adr-0368 into main 2026-10-14 21:29:04 +00:00
jschoubben 58edc4b4fb Cite ADR 0368 in design 48: the resolver restarts itself, for that module alone
mesh/merge-gate head 58edc4b4fb90 merged into trunk 010a34cc33e8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 58edc4b4fb90 merged into trunk 010a34cc33e8 — pass: its merge-check.sh passed
mesh/delivery delivered
Redoes the design-48 edit of the unmerged pull request 284, which cited the
record under its file number 0280; the record now lives in the knowledge base
as ADR 0368.
2026-10-14 23:18:38 +02:00
mesh-admin 010a34cc33 Merge pull request 'To-be 08: the five connectivity contexts walked through the network layers' (#275) from design/connectivity-through-the-layers into main 2026-10-14 21:14:49 +00:00
mesh-admin 35d5b2814f Merge pull request 'Playbook 07: review the local commits before anything is pushed' (#355) from playbook-07-review-before-push into main 2026-10-14 21:12:20 +00:00
jschoubben 3c5edc3879 Cite decisions by number now that they live in the knowledge base, and date the design today
mesh/merge-gate head 3c5edc387955 merged into trunk 35d5b2814f09 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 3c5edc387955 merged into trunk 35d5b2814f09 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 23:12:18 +02:00
jochen ab5025d508 Connectivity: walk the five contexts through the network layers, so a reader can see which layer each one acts at and read a failure from the bottom up 2026-10-14 23:11:53 +02:00
mesh-adminandClaude Opus 5.5 6c8532ce70 Review the local commits before anything is pushed
mesh/merge-gate head 6c8532ce7090 merged into trunk c032cae10030 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 6c8532ce7090 merged into trunk c032cae10030 — pass: its merge-check.sh passed
mesh/delivery delivered
A push starts the forge's checks, which build the modules and run every
test for up to a quarter of an hour, so a review finding fixed after the
push costs another full run (the operator, 2026-10-14).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 20:03:11 +00:00
mesh-admin c032cae100 Merge pull request 'Playbook 07: how several agents work one feature on one branch' (#354) from playbook-07-parallel-issues into main 2026-10-14 19:55:30 +00:00
mesh-adminandClaude Opus 5.5 071fc1c29d Name the lead, and have it take over a finished issue
mesh/merge-gate head 071fc1c29dce merged into trunk 21b79a04a283 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 071fc1c29dce merged into trunk 21b79a04a283 — pass: its merge-check.sh passed
mesh/delivery delivered
A finished issue's agent is gone, so left in fixing it goes stale; the lead
takes it over until the feature is delivered. One repository is one
delivery, not a group; readiness comes from depends-on links (review
comment 12466).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:53:20 +00:00
mesh-adminandClaude Opus 5.5 c2d5323dea Say how several agents work one feature on one branch
mesh/merge-gate head c2d5323dea5a merged into trunk 21b79a04a283 — pass: the change touches no module of the mesh's graph
mesh/repo-check head c2d5323dea5a merged into trunk 21b79a04a283 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Feature 57 was first worked as a pull request per issue into protected
branches and lost most of a day waiting on merge checks. Record the issue
as the unit, a review and checks once at the end, landing by fast-forward
under a lock, and the lessons met on the way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 19:50:14 +00:00
mesh-admin 21b79a04a2 Merge pull request 'Fit design 57 to walks with no first-machine gate (ADR 0360, ADR 0362)' (#353) from design/57-adr-0360 into main 2026-10-14 18:15:22 +00:00
mesh-adminandClaude Opus 5.5 bafc7b7d05 A silent machine is caught up by the backlog walk, not fixed forward
mesh/merge-gate head bafc7b7d05dd merged into trunk dcb5d5ce3883 — pass: the change touches no module of the mesh's graph
mesh/repo-check head bafc7b7d05dd merged into trunk dcb5d5ce3883 — pass: its merge-check.sh passed
mesh/delivery delivered
A machine that says nothing is often asleep; its builds wait for the
backlog walk (ADR 0360 clause 10), so no new pull request is needed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:12:52 +00:00
mesh-adminandClaude Opus 5.5 5e493934a5 Say the rehearsal is the only live check, and that a fix goes into main
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 5e493934a583 merged into trunk dcb5d5ce3883 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 5e493934a583 merged into trunk dcb5d5ce3883 — pass: its merge-check.sh passed
The pull requests' checks still run before the walk, and by then the
release's branches have reached main. Name the silent machine too, and
list ADR 0360, which the passages rest on (review comment 12348).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:11:52 +00:00
mesh-adminandClaude Opus 5.5 c957fcc6d7 Fit design 57 to walks with no first-machine gate
mesh/merge-gate head c957fcc6d751 merged into trunk dcb5d5ce3883 — pass: the change touches no module of the mesh's graph
mesh/repo-check head c957fcc6d751 merged into trunk dcb5d5ce3883 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
ADR 0360 sends a merge's builds to every machine at once, so release 57
has no first node to hold it and P6 can no longer wait at a gate. ADR 0362
moves the wait for the provisioner to the walk's end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 18:09:02 +00:00
mesh-admin dcb5d5ce38 Merge pull request 'To-be 45: the bus module says every user's slow consumer, and S9 raises it (issue 270, ADR 0357)' (#352) from design/270-the-bus-module-says-every-users-slow-consumer into main 2026-10-14 07:00:14 +00:00
jochen bb8075b37b Give ADR 0357 a home in design 45: the bus module says every user's slow consumer, and S9 raises it (issue 270)
mesh/merge-gate head bb8075b37bc1 merged into trunk 40a39c860278 — pass: the change touches no module of the mesh's graph
mesh/repo-check head bb8075b37bc1 merged into trunk 40a39c860278 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-14 08:42:54 +02:00
mesh-admin 40a39c8602 Merge pull request 'To-be 57: building config and provisioning, as 37 packages each an issue (ADR 0348, ADR 0349)' (#350) from design/57-building-config-and-provisioning into main 2026-10-13 23:57:18 +00:00
mesh-adminandClaude Opus 5.5 e5b98e3b0a Design 57: answer the second review of hq#350
mesh/merge-gate head e5b98e3b0a42 merged into trunk fd416cef91bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head e5b98e3b0a42 merged into trunk fd416cef91bf — pass: its merge-check.sh passed
mesh/delivery delivered
The word setting waits for R2, P3 waits for F3, the group's order is declared with after: lines, and a stop's fix is a new merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:53:59 +00:00
mesh-adminandClaude Opus 5.5 2f3e6e6641 Design 57: answer the review of hq#350
mesh/merge-gate head 2f3e6e6641d0 merged into trunk fd416cef91bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 2f3e6e6641d0 merged into trunk fd416cef91bf — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
Role names instead of node names, the failure rule as ADR 0276 and 0299 have it, readiness and adoption of existing credentials said exactly, and the counts and refusals corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:47:32 +00:00
mesh-adminandClaude Opus 5.5 1405dd39e8 Design 57: ship the packages as one release, rehearsed in the lab
mesh/merge-gate head 1405dd39e8bb merged into trunk fd416cef91bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 1405dd39e8bb merged into trunk fd416cef91bf — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The operator chose one delivery over one per package; the packages meet on one branch per repository and reach main as one delivery group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:40:15 +00:00
mesh-adminandClaude Opus 5.5 11294372b7 Design 57: the work of ADR 0348 and ADR 0349 as packages, each an issue, in dependency order
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate head 11294372b769 merged into trunk fd416cef91bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 11294372b769 merged into trunk fd416cef91bf — pass: its merge-check.sh passed
So config and provisioning are built in small, provable steps with at most five worked at once.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-13 23:12:23 +00:00
mesh-admin fd416cef91 Merge pull request 'To-be 47: a busy container on the first node holds only its walk, and the wait counts (ADR 0335, issue 594)' (#349) from design/335-a-busy-container into main 2026-10-13 22:21:01 +00:00
jschoubben 74e45e9b48 Merge remote-tracking branch 'origin/main' into design/335-a-busy-container
mesh/delivery delivered
mesh/merge-gate head 74e45e9b4834 merged into trunk c2a7016b620e — pass: the change touches no module of the mesh's graph
mesh/repo-check head 74e45e9b4834 merged into trunk c2a7016b620e — pass: its merge-check.sh passed
# Conflicts:
#	03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md
2026-10-14 00:12:11 +02:00
mesh-admin c2a7016b62 Merge pull request 'To-be 47: a change is built once, at its check (ADR 0350, issue 649)' (#348) from design/350-one-build-per-change into main 2026-10-13 21:30:32 +00:00
jschoubben 18ce19b85d To-be 47: a change is built once, at its check (ADR 0350, issue 649)
mesh/delivery delivered
mesh/merge-gate head 18ce19b85d71 merged into trunk 3a0b33501dfe — pass: the change touches no module of the mesh's graph
mesh/repo-check head 18ce19b85d71 merged into trunk 3a0b33501dfe — pass: its merge-check.sh passed
The design the decision lands in did not name it, so the knowledge base could
not accept the record's insight with nothing citing it.
2026-10-13 23:18:20 +02:00
jschoubben a0eccc713a To-be 47: a busy container on the first node holds only its walk, and the wait counts (ADR 0335)
mesh/delivery-group group feat/335-a-busy-container-is-replaced-when-idle rejected: its order contradicts itself: novox/mesh-catalog@190608193764, novox/mesh-c…
mesh/delivery stopped: the pull request closed unmerged
mesh/merge-gate head a0eccc713a81 merged into trunk c2a7016b620e — fail: head a0eccc713a81 does not merge cleanly into trunk c2a7016b620e: in conflict 03-D…
mesh/repo-check head a0eccc713a81 merged into trunk c2a7016b620e — fail: head a0eccc713a81 does not merge cleanly into trunk c2a7016b620e: in conflict 03-D…
2026-10-13 00:54:03 +02:00
112 changed files with 3398 additions and 1158 deletions
+44
View File
@@ -0,0 +1,44 @@
---
name: hq-cleanup
description: Use when the operator asks to clean up the forge and the working copies — stale or merged branches, old pull requests, finished worktrees, leftover folders, stashes. Triggers on "cleanup", "clean up our repos", "stale branches", "old worktrees", "tidy the forge". Reads first, deletes only what the operator approves, and reports what it cleaned, kept and left.
---
# hq-cleanup
Runs playbook [09 — Cleanup](../../../00-META/process/09-cleanup.md); the playbook is authoritative. This skill
adds the mechanics.
## 1. Take stock (one read-only agent, smaller model)
Brief it to change nothing and to write one report to the session's scratchpad:
- **Forge.** `gitea.gitea_list_repos`; per repository its default branch and **every branch from the git refs**
(`gitea.gitea_api` GET `/repos/{owner}/{repo}/git/refs/heads`; the list tool misses branches, issue 949);
`gitea.gitea_branch_protection_get`; open pull requests (`gitea.gitea_open_pulls`, titles and update times
from `gitea.gitea_api` `/repos/{owner}/{repo}/pulls?state=open`). Per branch: contained in the default branch
(`gitea.gitea_contains`, with the default branch's sha), patch-equal (`git cherry` in a local clone), tip date.
- **Local.** Under the projects folder: each clone's `git worktree list --porcelain`, `git branch -vv`,
`git branch --merged origin/<default>`, `git stash list`, `git status --porcelain` per worktree; folders and
loose files beside the clones with `du -sh` and newest file. A worktree under another session's scratchpad is
Kept.
- A **Missing tools** section: each read it did without a mesh tool.
## 2. Classify and ask
Classes as playbook 09 step 2: Safe, Likely safe, Kept, Unknown. Tell the operator the counts and the report's
place, and ask one question: delete the Safe list? Likely-safe and Unknown items are asked separately.
## 3. Delete the approved list only
- Forge branch: `gitea.gitea_delete_branch`.
- Worktree: `git worktree remove <path>` (never `--force`), then `git worktree prune`.
- Local branch: `git branch -d <name>` (never `-D`).
Once design 58 milestone 7 is built (issue 884), each deletion is a call held as an ask: the operator approves
the exact list on Telegram and the mesh carries it out on the warrant. Until then a session's permission check
refuses these; hand the operator the approved list as a file, with the commands, and do not work around the
refusal.
## 4. Report
Four lists: **cleaned** (with its evidence), **kept** (why), **unknown** (what it looks like), **the operator
kept** (their word). Open each missing tool with `issue-tracker.open`, or note it on an existing issue.
+214 -70
View File
@@ -69,7 +69,8 @@ is listed under [Homonyms](#homonyms), with the qualified form each domain uses.
**Recorded in** the catalogue, one manifest per module. **Upstream of** every other domain; its words
are a published vocabulary, fixed by the manifest's schema. **Decided by** ADR 0040, 0126, 0174, 0188,
0207, 0236, 0262.
**Uses:** module, seat, provision (Provisioning), data class (Data), health (Health and repair).
**Uses:** module, seat, config value (Config), provision (Provisioning), data class (Data), health (Health
and repair).
- **manifest** — the file a module is defined by (`module.json`): what it is, what it provides and
requires, the seats it claims, the resources it declares, its tools, its data and its health. Say
@@ -83,34 +84,77 @@ are a published vocabulary, fixed by the manifest's schema. **Decided by** ADR 0
it guards: the `mesh-controller`, `postgres` and `nats` modules claim the `mesh-controller`,
`mesh-store` and `mesh-broker` seats (ADR 0079,
ADR 0116).
- **setting** — a value a manifest declares and an assignment gives, one of the two ways a node varies
a module (ADR 0174).
What a "flavor" once varied is a setting or a separate module.
*Not:* ~~flavor~~
- **preference** — a setting whose manifest gives it a default, because its value is true of the
software and only tuned per node: a font size, a width
(ADR 0262).
A setting with no default is the operator's own value: a domain, an identity.
*Not:* ~~tunable~~ (hq)
- **kept region** — a marked block in a managed file the mesh writes *into*, where the operator's own
lines survive every send and are given back when the module goes (ADR 0174). The other way a node
varies a module.
varies a module, besides a config value; a **managed** file's marked block is one (Provisioning).
- **bundle** — the artifact a module's own code is built into — its tools, a seat's implementation, a
daemon — in any language the mesh has a toolchain for; never an image. A **tools bundle** speaks MCP to
the tool runner. One module may declare several
(ADR 0188).
- **tool** — one capability a module serves through the tool runner, addressed `<module>.<tool>`. A
seat's **verb** is the same thing carried by a seat rather than a module.
- **mesh-sdk** — the library a module's own code is written against, including the harness that serves a
tools bundle to the tool runner. The harness is part of it, not a library of its own.
- **mesh-sdk** — the library a module's own code is written against, including the SDK harness that serves a
tools bundle to the tool runner. The SDK harness is part of it, not a library of its own.
*Not:* ~~tools-sdk~~
- **invokes** — the manifest word for the tools a module calls, `<module>.<tool>` each or `*` for every
one. A grant on the publish side and nothing else; a module that declares none calls nothing.
- **run for** — an assignment, or a start of a module, that serves one agent identity with that identity's machine
account, home and bus password. Only a module whose manifest declares it may be run for an identity; running for
the operator is the default run, as the operator account, with no setting (ADR 0412).
- **upgrade policy** — how a module's new builds reach its nodes: `roll` (one node first, judged, then
the rest), `together`, or `record` (register the build, send it nowhere)
(ADR 0236).
The controller's verb `upgrade` shows it.
## Config — the values someone may change
**Purpose.** To hold, check and resolve every value a module reads that someone may change, and to say
where each resolved value came from. **Recorded by** the `mesh-config` module, in its buckets on the bus,
with its store as the history behind them. **Upstream of** Provisioning, which places the values;
downstream of Module and Placement. **Decided by** ADR 0262, 0277, 0348.
**Uses:** module, node, manifest (Module), assignment (Placement), provision (Provisioning), proposal
(Operator and conversation).
- **config value** — anything a module reads that someone may change: a font size, a domain, an address.
A module declares every one it reads under `config` in its manifest, with a type, a description,
validation, whether it is trusted, optionally the predefined key it falls back to, and a default or
`required`; it lands in a template file as `${config:<key>}` or is read by the module's code at runtime.
It resolves from four sources, the most specific winning: the default in the manifest; **mesh config**,
the value of the predefined key it falls back to for the whole mesh; **node config**, that key's value
on one node; and **module-assignment config**, the module's own value on one node. There is no module
value for the whole mesh. *Config* is the concept, the verb (`config`) and the manifest field
(ADR 0348). Mesh config is a source level, the value set for the whole mesh; `mesh-config` is the module
that keeps every level. The controller holds, mirrors and resolves no config value (ADR 0348 §9).
Older records and documents say "setting" for a config value and "settings" for the concept; new writing
says config value and config. The word goes on this entry's *Not:* line once the documents in force that
still use it in running prose are reworded, which is issue 683; until then it is checked by review. The
manifest field and the verb "settings" are kept as aliases until every module has moved (ADR 0348 §1,
issue 630), and are named on the line below with the rest once issue 683 is done.
*Identifier until renamed:* `takes-settings`, `${setting:}` — manifest forms kept as aliases until every
module has moved to their config names (ADR 0348 §1, issue 630).
What a "flavor" once varied is a config value or a separate module.
*Not:* ~~flavor~~
- **predefined key** — a key of the catalogue's `predefined.json`, the fixed list of keys for which mesh
config and node config hold values, each with its type, description, validation and whether it is
trusted, such as `font-size`. A config value may name the predefined key it falls back to, of its own
type; one that names none resolves from its default and the module's own value only. The list is the
catalogue build's: a change to it is validated with every manifest naming a key, and removing a key still
named or still holding a value is refused (ADR 0348).
- **trusted** — said of a config value that decides what runs or who can reach what. At every level it is
set only at the controller's terminal or through `config` with propose, on the operator's Approve; a
module value that falls back to a trusted predefined key is itself trusted (ADR 0277, ADR 0348 §8). Not
the trusted sender of an operator message (see [Homonyms](#homonyms)).
- **preference** — a config value whose manifest gives it a default, because its value is true of the
software and only tuned per node: a font size, a width
(ADR 0262).
A config value with no default is `required`: the operator's own value, such as a domain or an
identity, and while nothing sets it the module's files are not written (ADR 0348).
*Not:* ~~tunable~~ (hq)
- **`mesh-config`** — the module that owns every config value, and not the source level mesh config: it
holds the seat `mesh-config` once for the mesh, validates, takes a trusted value only from the
controller's terminal or the operator's Approve, keeps the history with who set it and why, and resolves
the four sources (ADR 0348).
## Core — the mesh's own machinery
**Purpose.** To keep the controller, the bus, the store and every node's engine running and agreed, so
@@ -178,10 +222,11 @@ every other domain has something to run on. **Recorded by** the controller and t
(ADR 0229).
- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its
records (nodes, assignments, walks, builds, settings), after each change commits in its store and only
records (nodes, settings, declared config, predefined keys), after each change commits in its store and only
when the value differs, with one event per write. The store stays the record; the self-check compares
the two and raises `mirror-drifted` on any difference
(ADR 0289). Shortened to *mirror* where the
(ADR 0289, ADR 0356). Builds, rollouts and assignments are no mirrors: they are
the chain, kept on the bus as their only source (ADR 0408). Shortened to *mirror* where the
text is about the controller's records; the builder's *mirrored* `upstream` image (Change and delivery) is
a different thing, a copy of an outside image in the artifact store.
@@ -189,10 +234,11 @@ every other domain has something to run on. **Recorded by** the controller and t
**Purpose.** To decide, send and apply what each node runs, and to say why.
**Recorded by** the controller's `inventory` database. **Upstream of** Provisioning, Connectivity, Data
and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 0252, 0261.
**Uses:** node, machine, module, seat, setting (Module), manifest (Module).
and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 0252, 0261, 0411.
**Uses:** node, machine, module, seat, manifest (Module), config value (Config).
- **assignment** — a module put on a node, with the settings that node gives it.
- **assignment** — a module put on a node. Its config values on that node (module-assignment config,
Config) are `mesh-config`'s, never part of what the controller records (ADR 0348 §9).
- **pending assignment** — an assignment of a module whose build is still running, kept, and made by the
controller when that build registers the module, or ended with why when it does not
(ADR 0261).
@@ -221,15 +267,15 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
other things (see [Homonyms](#homonyms)).
- **apply** — the node-engine making its node match its declaration; **reconcile** is doing so again
until nothing differs.
- **operator account** — the login name of the person who works on a node, stated on the node record;
empty for a node nobody logs into. Everything the mesh places under a person's home is resolved
against this account's home and owned by it
(ADR 0181).
- **operator account** — the machine account of the operator's person identity on a node, read from the
identity and no longer stated on the node record; empty for a node nobody logs into. Everything the mesh
places under a person's home is resolved against this account's home and owned by it
(ADR 0181, ADR 0411).
Not a name a manifest carries.
- **agent account** — the login every coding agent on a node runs as, when the node names one of its own;
stated on the node record at the controller's terminal, never by a verb or a setting. Empty means agents
run as the operator account. It cannot become root without a person, and the node-engine judges that on
every look (ADR 0266).
- **agent account** — the machine account of the agent identity the identity seat marks as a node's agent account,
on the operator's Approve, never by an assignment or a config value. Where none is marked, agents run as the
operator account. It cannot
become root without a person, and the node-engine judges that on every look (ADR 0266, ADR 0411).
- **account group** — a group of the machine's group database that a module puts an account in, by
declaring the account with that group and nothing else of it. Several modules may each add one to the
same account; the mesh takes back only an account group it added
@@ -238,15 +284,21 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
## Provisioning — one module serving another
**Purpose.** To resolve which module serves a provision for which consumer, and to wire the two.
**Recorded by** the controller. **Upstream of** Data; downstream of Module, Placement and Identity.
**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263.
**Uses:** module, seat, credential (Identity and access).
**Purpose.** To resolve which module serves a provision for which consumer, to wire the two, and to
place every value a module needs into its files on its node. **Recorded by** the controller and
`mesh-provisioner`. **Upstream of** Data; downstream of Module, Placement, Config and Identity.
**Decided by** ADR 0027, 0084, 0138, 0225, 0230, 0263, 0349.
**Uses:** module, seat, node-engine (Core), declaration (Placement), config value (Config), credential
and vault (Identity and access).
- **provision** — a service one module `provides` and others `require`; the mesh resolves a provider
and wires the two with an endpoint and a credential. A provision is a service you offer, a seat is a
role you occupy, and the two meet where a seat delivers a provision: occupying the seat is what makes
a module *the* provider of it.
Every value a module needs is a provision from the domain that owns it: placement from the controller,
config values from `mesh-config`, shared secrets from the vault, a consumer's values from the module
providing it, an item of a collection from its contributor. A module says only what it needs and where
it lands (ADR 0349).
- **provider** and **consumer** — the module that provides a provision, and the module that requires
it. Which of several providers serves a consumer is resolved
(ADR 0084).
@@ -264,27 +316,70 @@ and Health and repair. **Decided by** ADR 0100, 0126, 0176, 0181, 0207, 0221, 02
node seat `node-mounts`, as a mount unit and an automount unit, mounted on access (ADR 0263).
- **occasional source** — a share on a device that is not a node and comes and goes, such as a USB disk;
its absence is a state, *not present*, never a fault (ADR 0263).
- **mesh-provisioner** — the module that places provisions on the nodes: it holds the seat
`mesh-provisioner` once for the mesh, keeping which domain answers which kind of placeholder, what each
module on each node needs, what is applied where and the items of every collection, and the seat
`node-provisioner` on every node, which its provisioner client serves (ADR 0349).
- **provisioner client** — mesh-provisioner's part on one node. For each module there it reads the
placeholders of its template files, records the needs, watches each owner's values for the node, fills a
template only once every value is there, has the node-engine write the file through a narrow local verb,
makes the program take the change as the file declares (`restart-on`, `reload-on`, `read-on-use`), and
reports `prov.applied`. It runs as a service with a bus account of its own, never in the tool runner
(ADR 0349).
- **collection** — a provision with many providers. The **collector**, a module that takes additions from
others (a shell's snippets, a bar's blocks), requires it; each **contributor** ships its **item** as a
template in its own repository and provides it. The provisioner client fills the collector's template
from the items of the contributors placed on its node, and an item goes when its contributor leaves the
node (ADR 0349).
- **owned** — a placed file that is the mesh's whole: the provisioner writes it back whenever it differs.
It is one of three answers a manifest gives for each file it places (`owner-at-runtime`): **owned**,
**seed** or **managed**. Whichever it is, the file has one writer: the node-engine writes it only on the
provisioner client's word (ADR 0349).
- **seed** — a placed file the provisioner writes once; from then on it is the program's.
- **managed** — a placed file of which only its named keys or a marked block (a kept region, Module) are
the mesh's; the program keeps the rest.
## Identity and access — who may do what
**Purpose.** To issue, hold, rotate and check the credentials and grants by which modules, nodes, agents
and the operator act. **Recorded by** the controller's `identity` and `licences` databases and the vault.
**Upstream of** Provisioning, Change and delivery, and Operator and conversation.
**Decided by** ADR 0085, 0113, 0183, 0225, 0234.
**Uses:** module, node, operator, consumer (Provisioning).
**Decided by** ADR 0085, 0113, 0183, 0225, 0234, 0386, 0387, 0388, 0411, 0413, 0414.
**Uses:** module, node, seat, tool (Module), operator, consumer and provision (Provisioning), agent and ask (Operator and conversation).
- **credential** — anything that proves an identity: a password, a key, a token. A **pair credential**
is the two ends of one credential, the consumer's and the provider's.
- **secret** — a credential or other value the vault keeps sealed and the node-engine unseals into a
file at apply (ADR 0085).
- **secret family** — a module's own secrets declared as one name ending in `-*`; each member is the prefix
and a name (`smb-password-<source>`, one per source the module's settings name), given by its full name
and a name (`smb-password-<source>`, one per source the module's config values name), given by its full name
and never made by the mesh
(ADR 0283).
- **vault** — the module that owns every secret (`mesh-vault`,
[24 — The secrets vault](../03-DESIGN/01-to-be/24-the-secrets-vault.md)).
- **grant** — what an identity may call or reach, bounded by the provisions it requires
(ADR 0225).
- **identity** — a person, an agent or a module as the mesh knows it: a name, a kind, a role, and whether it
is switched on; a person or an agent also has a login name, the machines it is present on and, for a person,
the login kept on a machine where it differs. Every identity and role is held by the identity seat, in the
bus's key-value store, and changes only through its verbs (ADR 0386, ADR 0411). Each person and agent
identity has a bus account of its own, carrying its role (ADR 0413).
- **machine account** — an identity's login on one machine, made by the mesh from the identity: the
controller composes it and the node-engine applies it, adopting a login that exists and never renaming or
deleting one (ADR 0411).
- **role** — what an identity may call, as a set of grants (ADR 0386). Not a seat: a bare *role* is a
homonym (see [Homonyms](#homonyms)).
- **grant** — one line of a role: an effect everywhere, an effect on one seat, or one named tool, and may be
limited to named machines; a dangerous tool is granted only by name (ADR 0386). A module's role
is made from what its manifest declares it calls (ADR 0386 §4).
- **effect** — a tool's mark, set by its module in its manifest: **reads**, **acts** or **dangerous**; a
tool served without one counts as dangerous (ADR 0387).
- **break-glass agent** — the one agent that holds every tool and never asks: sealed until the operator, or
a condition the operator named, opens it, open until the operator closes it, and every act it takes told
at once (ADR 0387).
- **signed login** — a bus login signed by a key the bus trusts, carrying its expiry, which the bus checks by
its signature instead of reading a list (ADR 0388). The identity seat signs with one **signing key** per
role.
- **root key** — the key at the top of the signed logins, which vouches for the signing keys: sealed in the
vault and used only to replace them, on the operator's Approve (ADR 0388).
- **bus account** and **membership** — a module's or a node's account on the bus, and the subjects it
is issued. The bus server calls its accounts *users*; the mesh says bus account.
- **rotate** — replacing a credential without a consumer holding one the provider does not know.
@@ -306,8 +401,8 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
**Purpose.** To take one commit from its pull request to every node that should run it, and to put it
back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the
planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323.
planner, the rollout). **Downstream of** Module, Core, Health and repair and Data.
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323, 0401–0408.
**Uses:** module, node, declaration and send (Placement), health (Health and repair), person.
- **merge check** — the two statuses a pull request carries before it may merge: the **merge gate**
@@ -322,7 +417,10 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
**builder**. Any node may hold the seat, so there is no build node by nature
(ADR 0237).
*Not:* ~~build machine~~
- **build queue** — the controller's queue of builds to run; one entry in it is a **build request**.
- **build queue** — the builds waiting to run, kept by the bus as three work queues a builder takes in order:
delivery builds, then checks of pull requests whose Approve is asked or given (ADR 0421 §10), then every other check; each check queue keeps one
request per pull request, the newest (ADR 0406); one entry in it is a
**build request**.
The queue's verbs still say *ask* for an entry; the conversation's ask is another thing
(see [Homonyms](#homonyms)).
- **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by **version**.
@@ -348,7 +446,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
- **delivery** — one commit in one repository on its way to the nodes, from its pull request's head
being announced to one of its final states: one pull request, one status, one note
(ADR 0239).
From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk
From its merge on it is delivered by the rollout of the batch that answers it, and ends with that rollout
(ADR 0276).
Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`,
`held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`;
@@ -357,13 +455,21 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
mechanism, which the as-is designs describe; it is never a word for a delivery.
*Not:* ~~deployment~~ (hq)
- **covered** — the end of a delivery that a later merge containing it delivered: every module it moves was
sent to every machine running it by the walk of a later merge into the same trunk, delivered on its own
sent to every machine running it by the rollout of a later merge into the same trunk, delivered on its own
passing check, whose merge commit the forge says contains this one. Not *superseded*, which is a delivery
whose walk a newer delivery took over before it ended; nor *delivered*, which its own walk or an empty plan
whose rollout a newer delivery took over before it ended; nor *delivered*, which its own rollout or an empty plan
ends in (ADR 0270).
- **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's
repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One
level: a group holds deliveries, never groups. Its state is derived from its members, never set.
- **hotfix** — one pull request in one repository, delivered at its merge (ADR 0401).
- **feature** — one branch name across its repositories: in two or more, its pull requests are one delivery group,
merged together and delivered member by member in the group's order; in one, a single pull request delivered as a
lone delivery (ADR 0401). Work is never held back on a branch other than a trunk: the trunk is what runs.
*Not:* ~~release branch~~ (hq)
- **toolchain** — a language's compiler and tools, installed by that language's build module on every node holding
the build seat, pinned to one version that is the same on every builder; its version is part of a build's reuse
key (ADR 0405). No longer an image.
- **delivery plan** — what a delivery does to the mesh: its **build plan** (the modules it moves and
their dependents, in tiers), its **deploy plan** (per node, what it receives and what waits for a
person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a
@@ -371,19 +477,21 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
*Not:* ~~change plan~~, ~~release plan~~
- **merge window** — the rolling wait in which the controller collects merges before it plans them: each
merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest
10 minutes after its first merge (ADR 0276).
- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more
than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*,
10 minutes after its first merge (ADR 0276). Kept as two keys with a time to live on the bus; the first to
expire closes the batch (ADR 0407).
- **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more
than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*,
which is a delivery group.
- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat
of the same name. It records and decides; the controller sends, judges and rolls back when it is asked.
- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236,
ADR 0276). One walk is open at a time; it names every merge it answers. A primitive the delivering stage
asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules
built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own:
a module *rolls out* by its `roll` policy.
*Not:* ~~rollout~~ (hq)
- **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
repository, tier by tier, to every node that runs each module at once (ADR 0276, ADR 0360). Rollouts that share a
repository, a module or a graph edge run one after another; others run side by side (ADR 0407). It names every
merge it answers, and its record is one key of the bus's rollouts bucket, a link of the chain (ADR 0408). A primitive the delivering stage
asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules
built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll`
policy.
*Not:* ~~walk~~ (hq)
- **first-node gate** — the judgement on a delivery's first node: healthy three times, no witness put it
back, no condition raised since the send (ADR 0236 §2). Not *the gate* bare (see [Homonyms](#homonyms)).
- **core module** and **leaf module** — a core module is one on the controller's own path (the controller,
@@ -394,7 +502,7 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
the core modules add mesh-delivery and the resolver.
- **delivery time** — from a delivery's merge to every node of its deploy plan that is heard from running its
modules' builds and reporting them healthy. Its **delivery budget** is what it is held to: three minutes for
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the walk's time, which ends at the send. A
a leaf module (ADR 0304), ten for a core module (ADR 0282). Not the rollout's time, which ends at the send. A
measure inside the fix budget, not the operator's target (ADR 0313).
- **fix time** — from an issue's first move to `fixing` to the moment the last of its fixes is delivered: the
operator's measure of a fix, the agent's work, the review and the checks included, the delivery time its last
@@ -422,7 +530,7 @@ becomes a message). **Decided by** ADR 0227, 0231, 0240, 0252, 0283.
running session, started before, does not have that group yet. Said by the node-engine as the module's
resource of kind `account`; it clears at the first look after a new login (ADR 0252).
- **waiting** — a resource's health when its module's tool check says it waits for the operator: a named own
secret or setting of the module, not given yet, without which a part cannot work. Not a fault: the
secret or config value of the module, not given yet, without which a part cannot work. Not a fault: the
controller checks it and says it as the condition `needs-operator`, naming the act, and the first-node gate
reads it as *waits for a person*
(ADR 0283).
@@ -503,8 +611,8 @@ it should not. **Recorded by** the controller. **Upstream of** Provisioning and
trust. **Recorded by** the router (asks, warrants and the operator's identities).
**Downstream of** Health and repair, Change and delivery, and Identity and access. An authorising
answer acts in whichever domain asked, through that domain's own verb; the conversation owns the
asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
**Uses:** operator, person, proof (Identity and access), release (Change and delivery).
asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272, 0383, 0384, 0385, 0390.
**Uses:** operator, person, node, module, seat, tool (Module), proof and identity (Identity and access), release (Change and delivery).
- **mesh MCP server** — the endpoint on a node's loopback through which every agent and person on that
node reaches the mesh: the MCP server named `mesh`, with its five tools `mesh_overview`,
@@ -518,8 +626,8 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
line as the **controller's terminal** or as an ordinary call
(ADR 0272).
- **nox** — `mesh-cli`'s alias, in interactive zsh only. A script, `sudo` and a document meant to work everywhere
say `mesh-cli`. The operator also calls the mesh itself *nox mesh*; on this page and in the records *nox* is the
alias alone. It is the one command the operator types: every line the mesh addresses to the operator — a
say `mesh-cli`. The operator also calls the mesh itself *nox mesh*. *nox* is also the agent (ADR 0414 §7): see
[Homonyms](#homonyms), and say which is meant. It is the one command the operator types: every line the mesh addresses to the operator — a
condition's words, a verb's answer, a README, a design — names a `nox` line, never a daemon's (the controller's
or the node-engine's binary followed by a command), which `words.py` and the catalogue's check refuse
(ADR 0272,
@@ -546,7 +654,7 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
published for that asker alone. The only thing an asker acts on (ADR 0259).
- **proposal** — a change an agent, or anyone the bus admits, puts forward through a verb and that takes
effect only on the operator's warrant, bound to the exact thing shown: a shared instruction section (ADR
0274), a trusted setting's layer (ADR 0277).
0274), a trusted config value (ADR 0277, ADR 0348).
A proposal changes nothing by itself; declined, expired or refused, it is discarded.
- **approval mark** — a tool's or verb's declaration in its manifest (`needs-approval`) that each call to it runs
only on the operator's warrant for that one call, with its exact arguments; the tool runner that serves it, or
@@ -559,11 +667,24 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
instructions.
- **reference** — an opaque token in a message's words standing for a detail the content rule keeps out
of them (a path, an address); opened with `detail` only on a `private` channel or through the mesh MCP server.
- **agent** — a coding agent: the operator's, or one the mesh runs. Not the node-engine, and not the
- **agent** — a coding agent: the operator's, or one the mesh runs; to the mesh, an identity of kind agent
(ADR 0386). Not the node-engine, and not the
build seat's holder, whose seat name `node-build-agent` is an identifier to rename.
- **thread** — one line of agent work, owned by the mesh: its turns, tool calls and results, its workspace
and usage, kept whole for good on the bus by the thread seat; it outlives any session, node or front end
(ADR 0383, ADR 0384).
- **session** — one run of an agent harness on a thread.
- **agent harness** — the program that runs an agent's sessions, its context manager and loop: Claude Code
today (ADR 0384). Not the SDK harness (see [Homonyms](#homonyms)).
- **thread handler** — the node seat whose holder runs threads' sessions headless, on the always-on nodes
(ADR 0384).
- **workspace toolkit** — the tools a session works with on its thread's workspace, the same for every agent
harness: the node seat `workspace` (ADR 0385).
- **sandbox** — the small virtual machine one session runs in, with its thread's workspace mounted
(ADR 0390).
- **instruction section** — one named section of the managed instruction file, which every agent session on the
nodes it is for reads first, as the mesh's word: set at the controller's terminal in the agent module's
settings, or **proposed** by an agent and made active on each node only by the operator's warrant, which that node
config, or **proposed** by an agent and made active on each node only by the operator's warrant, which that node
proves itself (ADR 0274). A hand-over between sessions is one.
- **personal rule** — one short text in the operator's own words about how the operator wants their agent
to work with them, registered for the operator (the **operator scope** of the agent module) and rendered
@@ -578,10 +699,10 @@ asking, never the act. **Decided by** ADR 0152, 0175, 0234, 0259, 0272.
## Media — what a person watches, listens to and reads
**Purpose.** To find, fetch and file copies of the works a person watches, listens to and reads, each in
the library its kind names. **Recorded by** the `media-importer` module's settings and its memory of what
the library its kind names. **Recorded by** the `media-importer` module's config values and its memory of what
it was told, and the servarr apps' own libraries. **Upstream of** nothing. **Decided by** ADR 0295, 0296,
0302.
**Uses:** module, seat, setting (Module).
**Uses:** module, seat, config value (Config).
- **content** — a work a person watches, listens to or reads: a film, an album and its tracks, an
episode, a book, a session of a sport event. It is known by an identity from a catalogue outside the
@@ -592,7 +713,7 @@ it was told, and the servarr apps' own libraries. **Upstream of** nothing. **Dec
- **media release** — one copy of a work an engine can fetch, such as an NZB or a video. Downloaded, it
sits in the downloads folder, not yet filed. ADR 0302 says *release*.
- **library** — one folder of one kind, filed by one importer: the media-importer's, named in its
settings, or a servarr app's. Several libraries may hold one kind, as TV Shows and Anime both hold
config, or a servarr app's. Several libraries may hold one kind, as TV Shows and Anime both hold
series.
- **media file** — a file on disk together with its metadata: its kind, the fields that identify the
work, and what part of the work the file is. It is what the media-importer takes.
@@ -611,7 +732,7 @@ it was told, and the servarr apps' own libraries. **Upstream of** nothing. **Dec
Not a domain of the mesh but of the way it is built, listed because its words meet the mesh's.
**Recorded in** this repository, the issue tracker and the knowledge base. **Decided by** ADR 0019, 0080, 0244, 0275,
0284, 0290, 0291.
0284, 0290, 0291, 0423, 0424.
- **research effort**, **graduation**, **hand-off**, **playbook** — an investigation, held whole in the
knowledge base and named "research NNN";
@@ -624,17 +745,29 @@ Not a domain of the mesh but of the way it is built, listed because its words me
`changed`, `superseded`): a later record may change one clause and leave the rest
(ADR 0290). Not a *rule*, which is a line of
`00-META/how-we-build.md`.
- **knowledge base** — the role that holds the decision records and research efforts whole (number, status, topic,
typed links, clauses and reasoning, with every version), takes them as proposals, has them reviewed and accepted,
- **knowledge base** — what holds the decision records and research efforts whole (number, status, topic,
typed links, clauses and reasoning, with every version), and, once they are moved, the designs and the glossary
entries too (ADR 0423); takes them as proposals, has them reviewed and accepted,
and answers what holds: the seat `knowledge-base`, held by the module `mesh-knowledge`
([55 — The knowledge base](../03-DESIGN/01-to-be/55-the-knowledge-base.md)). Not **the record**, the literal reader
of this repository.
- **design** — a document in `03-DESIGN/`: **as-is** (what runs) or **to-be** (what is being built).
- **design** — a record of the knowledge base, named "design NN", with a **to-be** section (what is being built)
and an **as-is** section (what runs), its owning repositories and its links to its decisions and other designs
(ADR 0423). Until the move, a document in `03-DESIGN/`, in `01-to-be/` or `00-as-is/`.
- **design state** — of a design: proposed, designed, in progress or implemented, worked out by the knowledge base
from the issues with a design link to it and whether their deliveries are done; never written by anyone
(ADR 0424).
- **glossary entry** — one entry of the glossary as a record of the knowledge base: its head word or words, its
domain, its meaning, its *Not:* line and its identifier line (ADR 0423). Until the move, an entry of this page.
- **milestone** — a group of a design's issues delivered together, as one delivery group, or one pull request when
it touches one repository; named by what the operator or the mesh can do once it is delivered (playbook 04,
ADR 0401).
*Not:* ~~epic~~ (hq)
- **issue** — a question about the mesh that someone must answer, numbered and tracked by the issue tracker, which
holds it whole: its report, diagnosis and resolution included (ADR 0291);
the folders in `04-ISSUES/` are imported there and removed.
An **incident** is a past event that taught a rule.
- **issue tracker** — the role that holds the mesh's issues and hands out their numbers: the seat `issue-tracker`,
- **issue tracker** — what holds the mesh's issues and hands out their numbers: the seat `issue-tracker`,
held by the module `mesh-issues` ([53 — The issue tracker](../03-DESIGN/01-to-be/53-the-issue-tracker.md)).
Not hits's *item*; bare *ticket* is a homonym (see [Homonyms](#homonyms)).
*Not:* ~~ticketing system~~ (hq), ~~ticket system~~ (hq)
@@ -672,7 +805,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|---|---|---|---|
| plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement |
| | what a delivery would do to the mesh | **delivery plan** | Change and delivery |
| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery |
| | the sending of one batch of commits across nodes | **rollout** (verb `plans`) | Change and delivery |
| | a step a person starts, like the bus's | **planned step** | Core |
| membership | the subjects a bus account is issued | **membership** | Provisioning |
| | an account in a group of the machine | **account group** | Placement |
@@ -686,8 +819,13 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
| | the pull request status | **merge gate** | Change and delivery |
| | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement |
| tier | a level of the mesh | **layer** | Core |
| | a step of a walk | **tier** | Change and delivery |
| | a step of a rollout | **tier** | Change and delivery |
| | how much proof an ask needs | **assurance level** | Identity and access |
| harness | the part of `mesh-sdk` that serves a tools bundle | **SDK harness** | Module |
| | the program that runs an agent's sessions | **agent harness** | Operator and conversation |
| | the TypeScript `provisioner` helper for database providers | **`provisioner`**, renamed by design 57 | Provisioning |
| role | what a seat names, at a scope | **seat**; the bare word in older designs is tracked by issue 872 | Module |
| | what an identity may call | **role** | Identity and access |
| ask | a request for the operator's input | **ask** | Operator and conversation |
| | an entry in the build queue | **build request** | Change and delivery |
| store | the one database server | **store** | Core |
@@ -696,6 +834,8 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
| record | a numbered decision | **decision record** | The record |
| | the literal reader of this repository, searched first for a symptom | **the record** (the `records` module) | Operator and conversation |
| | the upgrade policy that sends nowhere | `record` | Module |
| nox | the mesh's one agent identity, its logins and accounts (ADR 0414) | **`nox`, the agent** | Identity and access |
| | the operator's terminal command, `mesh-cli` | **the `nox` command** | Operator and conversation |
| knowledge base | the module holding the decision records and research efforts whole | **knowledge base** | The record |
| | the predecessor's operational memory, in older playbooks (issue 430) | none: it no longer exists | — |
| check | a pull request's two statuses | **merge check**, **merge gate**, **repository check** | Change and delivery |
@@ -712,6 +852,10 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
| | a question someone must answer | **issue** | The record |
| deploy | the per-node part of a delivery plan | **deploy plan** | Change and delivery |
| | a wrapped program's own deploy | its own word | — |
| managed | a placed file of which only named keys or a marked block are the mesh's | **managed** (`owner-at-runtime`) | Provisioning |
| | a file the mesh writes, such as the agent's instruction file or `managed_settings` | the file's own name, or *a file the mesh places* | — |
| trusted | a config value that decides what runs or who can reach what | **trusted** config value | Config |
| | a sender on the operator's own list | **trusted** sender, its message an **operator message** | Operator and conversation |
## How this page is kept
+26 -1
View File
@@ -1,9 +1,10 @@
---
status: canonical
updated: 2026-10-11
updated: 2026-10-16
derives: the constitution page
decisions:
- "ADR 0020"
- "ADR 0391"
---
# How we build
@@ -172,6 +173,30 @@ quietly stop being true, and nobody will learn that from a document.
Not test-driven development as a blanket rule — a test written first against undiscovered
behaviour asserts a guess. The obligation is that every decision has a defender.
### Tests run at the lowest level that proves them, within a time budget
Three levels, by how much real system a test uses (ADR 0391):
| Level | Uses | Budget | Runs |
|---|---|---|---|
| **Unit** | one piece of logic alone: no database, network, bus, running service or other process | well under a second per test | locally, the tests of what changed, while working; and in the forge's checks |
| **Integration** | the real thing beside it: a database, the bus, a catalogue tree | the repository's whole suite in seconds, never over sixty | only in the forge's checks, once per push |
| **End to end** | the whole mesh, or a lab mesh, from the outside | — | only when the operator asks |
- **Test each behaviour at the lowest level that can prove it.** A defect found higher up gets a test at the
lowest level that reproduces it, before its fix.
- **A test that needs longer than a second, or a database, the network or a service, is not a unit test.**
- **A suite over sixty seconds is a defect with its own issue**: it is made faster or split, not accepted.
- **Never run a merge check, a throwaway database or an integration suite locally.** The forge's checks do,
on the builder.
- **A report names its tests and how long they took**, so a test over its budget is seen when it is added.
### Tests within a time budget — how it was earned
On 2026-10-16 a feature's fixing agents ran the catalogue's whole merge check (about ten minutes), a
throwaway database and a locally built gate during local development. Each round cost tens of minutes and
caught neither real failure; the forge's checks found both.
### A report is read from the system, never from what asked for it
The same rule as the two above, pointed at reporting rather than at verification. **Anything
+6 -4
View File
@@ -50,13 +50,15 @@ the expensive half.
| # | Playbook | Trigger |
|---|---|---|
| [01](01-research.md) | Research | An idea worth investigating before committing to design |
| [02](02-graduation.md) | Graduation & design change | Research concludes, or a design must change |
| [01](01-research.md) | Research | An idea from the operator, or one worth investigating before committing to design: setup questions, findings, a direction the operator confirms |
| [02](02-graduation.md) | Graduation & design change | Research concludes (its direction confirmed), or a design must change: decision records, the to-be design |
| [03](03-issues.md) | Issues | Something is wrong — often with the owner unknown |
| [04](04-build-handoff.md) | Build handoff | A design is ready to be built |
| [04](04-build-handoff.md) | Work breakdown and handoff | A design is ready to be built: milestones usable on their own, issues per repository, each milestone one feature step |
| [05](05-constitution-sync.md) | Constitution sync | `how-we-build.md` changed a rule the mesh enforces |
| [06](06-writing-a-module.md) | Writing a module | Something that runs today must run on the mesh |
| [07](07-feature-branches.md) | Feature branches across repos | Work that changes code, in one repo or several at once |
| [07](07-feature-branches.md) | Features | Work that changes code across issues or repositories: prepared context, local development, one review, one check, one delivery |
| [08](08-small-fixes.md) | Small fixes | One issue in one repository, about an hour of work: one review, one push, one delivery |
| [09](09-cleanup.md) | Cleanup | Merged branches, stale pull requests, finished worktrees and folders: read first, delete only what the operator approves, report the rest |
## The cycle is checked
+113 -39
View File
@@ -1,66 +1,140 @@
# Playbook 01 — Research
**Trigger.** An idea, technology or approach worth investigating before it is committed to
design. Also: an as-is document that raises a question nobody can answer.
**Trigger.** An idea the operator expresses, or an idea, technology or approach worth investigating before
it is committed to design. Also: an as-is document that raises a question nobody can answer. A symptom (a
bug, something broken) is not research: it follows playbook [03](03-issues.md).
**Who runs it.** Anyone.
**Who runs it.** Anyone; in an agent session, the session that heard the idea leads it.
**Where it lives.** A research effort lives whole in the knowledge base, the seat `knowledge-base`, reached
through the mesh MCP server from any node and any folder (ADR 0290). This repository holds none: an effort
is written with the knowledge base's verbs, never as a folder here.
through the mesh MCP server from any node and any folder (ADR 0290). This repository holds none.
## The cycle it starts
```
idea ─► research (01) ─► decisions and design (02) ─► work breakdown in milestones (04)
─► each milestone built and delivered as one feature step (07) ─► what is reported afterwards (03, 08)
```
An idea arrives in one of two forms, and the first step differs:
- **A new idea**, where only the mesh's own context matters: read what the knowledge base holds on it.
- **An idea for an existing application**: read that application's to-be and as-is designs, its decision
records and its open issues first; the effort `touches` them.
## Steps
1. **Read what is known first.** `knowledge-base.ask` with the question in plain words returns the decision
records and research efforts that bear on it; `knowledge-base.get` with `kind: research` and a number
reads one effort, and `reasoning: true` (or `section` with a document's name) reads its documents.
2. **Take a number.** `knowledge-base.reserve` with `kind: research` and `for`: the effort's title. The
number comes from the issue tracker's counter and is spent whether or not the effort is ever proposed;
one never proposed is withdrawn after a week. Never pick a number by hand.
3. **Propose it.** `knowledge-base.propose` with `kind: research`, the `number` reserve gave, a one-line
`title`, `status: active`, `date` (the day it began), and `sections`: its documents in order, each a
`name` and its markdown `text`. The first is `00-overview.md`: **what** is being investigated, **why**,
and **what it touches**. Name what it touches as `links` of kind `touches` (`"0276"`, `"research 045"`,
`"issue 365"`, or a design's path), not only in prose, and what it names for another reason as `cites`.
4. **Do the research** in further documents of the same effort: notes, option analyses, evidence, draft
designs. Each change is `knowledge-base.propose` again with the effort whole and `why`, one line saying
why this version. Every version is kept; `knowledge-base.history` lists them and gives the difference
between two.
1. **Read what is known.** `knowledge-base.ask` with the idea in plain words returns the decision records and
research efforts that bear on it; `knowledge-base.get` reads one (`reasoning: true`, or `section`, for its
documents). Read the issue tracker for open issues in the same area (`issue-tracker.list`).
2. **Ask the operator to set it up.** A few questions, one at a time, each answerable with a choice and
carrying a recommendation: what comes first, what is out of scope, which options must be tried, what
done looks like. Three to five questions is usual; the answers are the effort's first findings.
- **Check every option against the mesh's own rules before offering it.** An option that breaks one
(the core values in [`mission.md`](../mission.md), among them *the mesh owns the truth* and
*sovereignty: external dependencies are deliberate and few*; the rules in
[`how-we-build.md`](../how-we-build.md); and everything between modules over the bus, research 036) is
dropped, or offered only with the rule it breaks named. A recommendation the operator has to argue back
costs a version.
- **Explain each question with a concrete picture** in the operator's words: what they would see, do or
lose under each option. No internal names (a record number, a verb, a word the glossary lacks) without
what it is beside it.
3. **Take a number.** `knowledge-base.reserve` with `kind: research` and `for`: the effort's title. The number
is spent whether or not the effort is ever proposed; one never proposed is withdrawn after a week. Never
pick one by hand.
4. **Propose it.** `knowledge-base.propose` with `kind: research`, the `number`, a one-line `title`,
`status: active`, `date` (the day it began), and `sections`, each a `name` and its markdown `text`:
- `00-overview.md`: the idea in the operator's words, what exists, the operator's answers (dated), and the
words the effort uses that the glossary lacks, each defined in one line;
- `01-questions.md`: the questions to answer, the first ones first;
- `02-method.md`: what will be tried and measured, and on what.
`propose` refuses an effort that breaks a rule that judges one record (an unknown status, a link to
nothing, a retired word of the glossary, a daemon's line), naming every reason at once and writing nothing.
The knowledge base then reviews the proposal itself, through a judgement that sees only the record, the
records it links and the rules; on `changes` the effort stays proposed with the findings, and its author
proposes it again. An effort that needs the operator's word is proposed with `needs-operator: true` and
waits for the operator's Approve on a channel that proves who answered.
Name what it touches as `links` of kind `touches` (`"0276"`, `"research 045"`, `"issue 365"`, or a
design's path), and what it names for another reason as `cites`.
5. **Do the research.** Try, measure and read, smallest question first. Each finding goes into
`04-findings.md`: what was tried, what it showed (with numbers), what it means. Each change of the effort
is `knowledge-base.propose` again, whole, with `why`; every version is kept, and `knowledge-base.history`
lists them and gives the difference between two. Until issue 844 is delivered, the knowledge base
takes a new version of an accepted, active effort only as a progressive insight: give `insight`, saying
what the effort held before the new finding, and mark the finding in the text with its date.
- **Propose a version per settled set, not per answer.** While the operator answers a run of questions,
or while experiments run, keep the answers and findings in the session's save point and propose once
when the set is settled. Each version is reviewed and resent whole, so a version per answer multiplies
the reviews.
- **Pre-check before proposing**, against what the review sends back most: every record, issue or design
the text names is in `links`; work put off names the issue that tracks it (an acceptance test is not
put-off work: it is what the build proves, step 7); each figure is given as measured (a range, not a
rounded claim); no word on a glossary *Not:* line.
- **Try at the cheapest level that answers the question.** A program's own behaviour (a library, a
server, a protocol) is tried in process, on the machine's loopback, in minutes. Lab runs (about 12 to
16 minutes fresh) are for the mesh's own code; before that code exists, a lab run tests nothing an
in-process test does not, and its lab scenario is written as the build's acceptance test instead.
6. **Place it in the domains.** Before the direction, write `03-domains.md`: the domain or domains the idea
belongs to (the glossary's sections, ADR 0244); the modules and seats it lands in, each new, changed or
retired; what it does to each domain's words, each word added (with its domain and a one-line meaning),
changed or retired, and none on a *Not:* line or already meaning something else (the glossary's homonyms);
and whether a domain grows or shrinks, a concept moving between domains included. The direction names its
decisions in those words. Asked when the idea is set up, too: an option that lands a concept in the wrong
domain is offered with that cost named, like one that breaks a rule.
7. **Settle the direction with the operator.** When the questions are answered, write `05-direction.md`:
the proposed decisions in a few lines each, the options rejected and why, and the build's acceptance
tests: what the build must prove before it is done, carried by its milestones (playbook
[04](04-build-handoff.md)). Nothing else is left open: each question closes by a test, by a decision the
effort records with its reason (a small design limit is the lead's), or by the operator. Ask the
operator to confirm or change the direction, one point at a time, then propose it once. Then the effort
graduates through playbook [02](02-graduation.md).
## Bounds and models
| Step | Bound | Model |
|---|---|---|
| reading and the setup questions (1, 2) | 30 minutes | the session's own, larger |
| proposing (3, 4) | 15 minutes | larger |
| each experiment or reading task (5) | 30 minutes | smaller, one agent at a time unless the operator allows more |
| placing it in the domains (6) | 15 minutes | larger |
| the direction (7) | 30 minutes | larger |
A step past its bound is said to the operator at once, with what is being done about it. The lead reports
as playbook 07 does: every ten minutes and at once when something is found, stops or needs the operator,
with the allowance used.
## What makes research worth reading
State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"*
outranks *"the dependency rule is not followed"*. An effort that measured nothing has not
finished.
State evidence, not assertion. *"Zero of 124 modules declare `brain` as a dependency"* outranks *"the
dependency rule is not followed"*. An effort that measured nothing has not finished.
Research describes real observations but never identifies the mesh it observed. The shape of a
finding survives anonymisation intact — *a node publicly named but behind a household NAT*
carries the whole lesson without naming anything.
Research describes real observations but never identifies the mesh it observed. The shape of a finding
survives anonymisation intact — *a node publicly named but behind a household NAT* carries the whole
lesson without naming anything.
`propose` refuses an effort that breaks a rule that judges one record (an unknown status, a link to
nothing, a retired word, a daemon's line), naming every reason at once and writing nothing. The knowledge
base then reviews the proposal itself; on `changes` the effort stays proposed with the findings and its
author proposes it again, at most three times before the findings go to the operator with a
recommendation. An effort that needs the operator's word is proposed with `needs-operator: true` and waits
for the operator's Approve on a channel that proves who answered.
## Do not
- Do not put status in prose. It is the effort's `status`, and the prose must not restate it.
- Do not give a `became` link while the effort is open — playbook 02 gives it at closure.
- Do not write into `03-DESIGN` from an open effort.
- Do not write an effort, or a copy of one, into this repository: the knowledge base is the only place one
is read, and `records.py` refuses a new path under `01-RESEARCH/`.
- Do not write an effort, or a copy of one, into this repository: `records.py` refuses a new path under
`01-RESEARCH/`.
## Closing
An effort never just stops. It closes through playbook [02](02-graduation.md) as `graduated`
or `abandoned`, always with a `became` link to what it turned into. Nothing is deleted —
what was rejected, and why, is the more expensive half to rediscover.
An effort never just stops. It closes through playbook [02](02-graduation.md) as `graduated` or
`abandoned`, always with a `became` link to what it turned into. Nothing is deleted — what was rejected,
and why, is the more expensive half to rediscover.
## How it is checked
The rules on an effort are the knowledge base's: `propose` refuses one that breaks a rule that judges one
record (the module's table test holds one case per rule), and the rules that judge the whole record set run
after every acceptance and raise a condition when one fails. That this repository holds no effort is
`records.py`'s `no-records` check, on every pull request.
after every acceptance and raise a condition when one fails. The habits of steps 2, 5, 6 and 7 (options checked
against the rules, one version per settled set, the cheapest level, nothing left open) are the lead's, read by
the reviewer; of the pre-check, `propose` refuses a link to nothing and a retired word, and the review returns
a record named but not linked, an untracked deferral and a figure not as measured. That this repository holds
no effort is `records.py`'s `no-records` check, on every pull request. An active effort with no new version
for seven days is the smell of research left behind: its lead closes or continues it.
+35 -5
View File
@@ -13,12 +13,30 @@ clauses in force for a topic or a record).
## Graduating research
The effort's `05-direction.md`, confirmed by the operator (playbook [01](01-research.md), step 7), is what
graduates. The lead writes the records and the design on the larger model; it is one sitting, bounded at
about an hour, and said to the operator if it runs past.
1. **Check it against GENESIS.** An effort graduates only if its conclusion is traceable to
[`mission.md`](../mission.md), [`context.md`](../context.md) and
[`effect.md`](../effect.md). If it conflicts, either the effort is wrong or GENESIS is —
say which, in writing, before proceeding.
2. **Record the decision** (below, *Writing a decision record*).
3. **Write the design.** Create the document under `03-DESIGN/01-to-be/` with frontmatter:
2. **Record the decisions** (below, *Writing a decision record*): one record per choice that could be
reversed on its own, not one record for the whole effort.
- **Write them together, not one after another.** Reserve every number first, then propose each record
as soon as it is written, without waiting for the review of the one before; the reviews run side by
side, and the operator's asks for the records that need them arrive together. A record that
`rests-on` or `extends` a sibling is proposed once that sibling is accepted.
- **Take the clauses from the confirmed direction** and cite the effort (`cites`, `"research NNN"`) for
the evidence instead of retyping its findings: a record's Context holds the few facts its decision
rests on, with the effort named for the rest.
- **Pre-check each record before proposing it**, as for research (playbook [01](01-research.md), step
5): every record, issue or design it names is in `links`; each deferral names its issue; each figure
is given as measured; no word on a glossary *Not:* line.
3. **Write the design.** **One design per thing** (the operator, 2026-10-16): when a to-be design already
covers what the decisions change, they rewrite that design, as *Amending an existing design* below says,
and no second design is written beside it that refers to it. Only for something no design covers, create
the document under `03-DESIGN/01-to-be/` with frontmatter:
```yaml
---
@@ -31,7 +49,16 @@ clauses in force for a topic or a record).
---
```
4. **Close the effort.** `knowledge-base.propose` the effort again, whole, with `status: graduated` and a
The design lists the direction's acceptance tests: what the build proves before it is done.
The words of the effort's `03-domains.md` go into the glossary in the same change, each in its domain: a
word added gets its entry, a word whose meaning changed gets its entry rewritten, and a retired word moves to
its replacement's *Not:* line. A word that already means something else in another domain is named as a
homonym with the qualified form each domain uses.
4. **Hand it to the work breakdown.** The design goes on through playbook [04](04-build-handoff.md):
milestones, issues, and each milestone built as one feature step (playbook [07](07-feature-branches.md)).
5. **Close the effort.** `knowledge-base.propose` the effort again, whole, with `status: graduated` and a
`became` link to the decision record (`"NNNN"`) and to the design document (its path). An effort that
stops without a decision is proposed with `status: abandoned` and a `became` link to what replaced it.
@@ -66,7 +93,8 @@ clauses in force for a topic or a record).
superseded record cited alone, an unknown topic, a retired word of the glossary, a daemon's line, a
deferral no issue tracks. The knowledge base then reviews the proposal itself, through a judgement that
sees only the record, the records it links and the rules, and answers `approve` or `changes`. On
`changes` the record stays proposed with the findings: revise it by proposing it again, whole, with `why`.
`changes` the record stays proposed with the findings: revise it by proposing it again, whole, with `why`;
at most three rounds, then the findings go to the operator with a recommendation.
On `approve` it is accepted, or, when it says `needs-operator`, it waits for the operator's Approve on a
channel that proves who answered. Nothing proposed is in force. `knowledge-base.get` shows a record's
reviews and the operator's ask when it waits for one.
@@ -133,4 +161,6 @@ record (its module's table test holds one case per rule), and the four rules tha
changes, an accepted record extending a superseded one) run after every acceptance and raise a condition
when one fails. In this repository, `records.py` checks that a design's `decisions:` entries and every
"ADR NNNN" in the text are well formed, and that no record is written here; `index.py` checks the reading
order's six topics, which the knowledge base reads.
order's six topics, which the knowledge base reads. The three habits of graduating step 2 (side by side, clauses
from the direction, the pre-check) are the lead's, read by the reviewer; `propose` refuses a link to nothing,
a deferral no issue tracks and a retired word, and the review returns a record named but not linked.
+63 -25
View File
@@ -1,35 +1,73 @@
# Playbook 04 — Build handoff
# Playbook 04 — Work breakdown and handoff
**Trigger.** A to-be design is settled and work is about to start in a code repository.
**Trigger.** A to-be design is settled (playbook [02](02-graduation.md)) and its building is about to start.
**Who runs it.** Whoever starts the build.
**Who runs it.** The lead who will run the feature, on the larger model. It is one sitting, bounded at about
an hour, and said to the operator if it runs past.
## What it produces
A design becomes **milestones**, and each milestone becomes **issues**:
- **A milestone is usable on its own.** Once its delivery plan is done, the operator or the mesh can do
something new, said in one sentence ("a Claude Code session is mirrored into a thread on the bus").
A milestone that changes nothing anyone can use is not a milestone; fold it into the next.
- **A milestone is one release** of playbook [07](07-feature-branches.md): prepared, developed whole, reviewed,
checked and delivered as one, never cut for its size or for time (the operator, 2026-10-17).
- **Milestones are cut by release order.** A milestone ends where one repository's change must be released before
another can use it (a new manifest word before the catalogue reads it, an SDK change before its users, until
design 47's milestones 3 and 6 lift those orders); the lead checks this when cutting, so no milestone needs a
second release.
- **An issue is one repository's part of a milestone,** sized so a fixing agent finishes it inside the
milestone's development time. Issues that must land in order are linked `depends-on`.
## Steps
1. **Confirm the design is settled.** Its frontmatter reads `status: designed`, and every
claim in it traces to a decision record in the knowledge base (`knowledge-base.get` for each entry of
its `decisions:`). An open question in the text is a
reason to run playbook [01](01-research.md), not to start building around it.
2. **Name the owner.** Set `code:` in the design's frontmatter to the repositories from
[`repos.md`](../repos.md). If the repository does not exist yet, add it to `repos.md` in
the same change.
3. **Check the as-is.** Read the matching `03-DESIGN/00-as-is/` document. What is being
replaced is stated there; if it is not, write it before changing it. Building against an
undocumented as-is is how a shipped behaviour gets lost.
4. **Flip the status.** `status: in-progress`, `updated:` today.
5. **Build in the code repository.** HQ is not a code repository and never carries
implementation.
6. **On completion**, run the "when something ships" section of playbook
1. **Confirm the design is settled.** Its frontmatter reads `status: designed`, and every claim in it traces
to a decision record in the knowledge base (`knowledge-base.get` for each entry of its `decisions:`). An
open question in the text is a reason to go back to playbook [01](01-research.md), not to build around it.
2. **Name the owners.** Set `code:` in the design's frontmatter to the repositories from
[`repos.md`](../repos.md). A repository that does not exist yet is added to `repos.md` in the same change.
3. **Check the as-is.** Read the matching `03-DESIGN/00-as-is/` document. What is being replaced is stated
there; if it is not, write it before changing it. Building against an undocumented as-is is how a shipped
behaviour gets lost.
4. **Break it into milestones.** Write them into the design, in a section *Milestones*: for each, its number,
its one-sentence outcome, the acceptance tests of the design it proves, the repositories it touches, and
what it depends on. Order them so each is usable as soon as it is delivered, the smallest useful one
first. A milestone is named by what it lets someone do; how it ships is shaped by the plan its changes would
have, asked through `mesh-delivery.what-if`: the modules it moves and their fan-out (ADR 0401 §6).
5. **Ask the operator to confirm the milestones**, in a short table: number, outcome, the acceptance tests it
proves, repositories, the parts each repository builds. Change what they change.
6. **Open the issues.** For each confirmed milestone, `issue-tracker.open` one issue per repository part, its
title the symptom or capability in plain words, its report naming the design, the milestone and the
decisions it builds with; link the order with `depends-on`. Write the issue numbers into the milestone's
entry in the design.
7. **Flip the status.** `status: in-progress`, `updated:` today; the design change goes through playbook
[08](08-small-fixes.md) (one hq pull request).
8. **Hand each milestone to playbook 07, one at a time.** The milestone is one release of playbook 07; its issues
are the release's issues; the next milestone starts when the previous one's delivery plan is done. A design waits
while another design's milestones are being built: one design is built at a time (playbook
[07](07-feature-branches.md)), so this step starts only when the design before it is delivered.
9. **On completion of the last milestone**, run the "when something ships" section of playbook
[02](02-graduation.md).
## Rules
- Every merge is a human checkpoint, without exception.
- What the handoff leaves for later is an issue with an owner, named in the design or the decision that
leaves it — never a sentence saying "later". The design's `code:` names who owns what is built; the issue
names who owns what is not. the knowledge base's rule `deferral` checks the decision record when it is proposed
- What the handoff leaves for later is an issue with an owner, named in the design or the decision that leaves
it — never a sentence saying "later". The design's `code:` names who owns what is built; the issue names who
owns what is not. The knowledge base's rule `deferral` checks the decision record when it is proposed
([issue 354](../../04-ISSUES/354-a-decided-rename-was-never-scheduled-and-the-operator-believed-it-live/00-report.md)).
- Never open a pull request unprompted. A permissions list saying it is allowed is not a
request.
- Work in an isolated worktree, never a shared checkout. A failed `cd` in a shared checkout
commits to the wrong branch, and the error scrolls past.
- HQ is not a code repository and never carries implementation.
- How the building is done — worktrees, agents, review, checks, the Approve and the delivery — is playbook
07's, not this one's.
## How it is checked
Read by the lead and the reviewer today; a check in hq's merge-check is tracked by issue 845.
- A design in `status: in-progress` has a *Milestones* section, each milestone with its outcome and its
issues; an in-progress design without one is the smell of building without a breakdown.
- Every issue a milestone names exists in the tracker (`issue-tracker.get`), and an issue in `fixing` names
its milestone in its report.
- A milestone whose issues are all `resolved` has had its delivery plan done; its outcome sentence and
acceptance tests are proven on the mesh before the next milestone starts.
+387 -55
View File
@@ -1,69 +1,401 @@
# Playbook 07 — Feature branches across repos
# Playbook 07 — Features
**Trigger.** Work that changes code — in one code repo or in several at once (`mesh-sdk`,
`mesh-controller`, `mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a decision rides along).
**Trigger.** A feature: work that changes code in one repository or several (`mesh-sdk`, `mesh-controller`,
`mesh-catalog`, `mesh-host`, `mesh-lab`, and `hq` when a design rides along), too large to be one small fix.
A small fix (one repository, one issue, about an hour of work) follows
[playbook 08](08-small-fixes.md) instead.
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly —
it is the guard against the failure it was written for.
**Who runs it.** Anyone who writes code, engineers and agents alike. Agents follow it exactly.
## The failure it prevents
## The failures it prevents
A feature was worked as a branch-and-MR per *unit of thought* — one per decision, one per
stacked increment — and each MR was treated as finished when it was *opened*, not when it was
*merged*. Across repos the same feature took a different branch name in each. The MRs piled up
unmerged: one session left **sixteen** stacked intermediate MRs that had to be consolidated and
closed by hand. An MR is a review checkpoint, not a scratchpad.
- **Pull requests as scratchpads.** A feature worked as one pull request per unit of thought left sixteen
stacked pull requests unmerged, consolidated by hand afterwards. A pull request is the end of a feature,
not a step in it.
- **Waiting on checks mid-feature.** A feature worked as one pull request per issue into protected
branches spent most of a day waiting for merge checks while the code was ready.
- **Context bought again and again.** A fresh agent per small issue spends most of its run reading the
playbook, the issue and the code before it writes a line. On 2026-10-14 an issue sized for five to ten
minutes took forty-one.
- **Finished work that never ships.** A feature that kept taking new issues held thirteen landed issues
undelivered for a day: an issue done at 21:09 was still not on the machines at 15:00 the next day.
- **Work that waits and nobody notices.** Most lost time was not work at all, but a check, a delivery plan
or an agent waiting on something that had stopped, unseen.
- **Review that keeps hardening.** One fix went through seven review rounds in five and a half hours, each
asking for more hardening than its issue asked (issue 704, 2026-10-14). A round is for what makes the
change wrong; everything else is an issue of its own.
- **The first check after the whole branch is written.** A feature's catalogue branch, pushed once after
hours of local work, met its first check on 2026-10-16 and failed on 29 problems, TypeScript errors among
them, that a unit suite run continuously while working finds at once.
- **A green pull request nobody marks.** An approved, green pull request that was not marked ready waited
12 h 17 min for someone to merge it (mesh-catalog #300, 2026-10-14). There is no ready mark since ADR 0421:
the operator's Approve is asked as soon as the checks pass.
- **A module proven only after it reached the machines.** The mail modules were delivered with unit tests
only, then failed live three times (issues 859, 865, 868) and took five more hours of fixes.
## The rule
## The shape
One feature is **one branch name**, **one worktree per repo**, **one MR per repo**, opened
**once, at the end**.
One feature is **one branch name**, **one worktree per repository**, **one pull request per repository**,
opened **once**, after the review. It runs in five steps, and its time is bounded. A milestone has no phases
(the operator, 2026-10-17): these are its steps, and a report names the step it is in:
1. **Name the feature once.** `feat/<slug>`. The *same* branch name in every repo the feature
touches — never a different name per repo, never a fresh branch per increment within the
feature.
2. **Isolate each repo.** One git worktree per touched repo under `.work/<slug>/<repo>`, branched
off `main`:
```
git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main
```
Parallel features never collide, and no shared checkout is edited.
| Step | What happens | Bound |
|---|---|---|
| 0. Prepare | the lead settles the open questions and writes one context file per repository | 15 minutes |
| 1. Develop | one fixing agent per repository builds every issue locally, the whole unit suite before every commit | the milestone's issues, 30 minutes each |
| 2. Review | one reviewer reads all of it; reports go both ways; at most 3 rounds | 15 minutes |
| 3. Check | push; the forge's merge checks run the integration tests; no further review | one check run |
| 4. Deliver | merge; the delivery plan builds, publishes and sends the whole feature | its delivery plan |
**Add the untouched siblings the lab reads.** The lab beds find the other repositories by
sibling path from the lab checkout (`../mesh-tools/module.json`, `../mesh-sdk`, …), the way
the main layout has them. A `.work/<slug>/` directory holding only the touched repos fails a
bed at once with `no manifest for mesh-tools at …/.work/<slug>/mesh-tools/module.json`, after
genesis has already passed. Give the directory those repos as **detached worktrees on `main`**
— never symlinks:
```
git worktree add --detach .work/<slug>/mesh-tools main
```
3. **Commit as you go — locally.** Increments land on the one branch. Nothing is pushed and no
MR is opened mid-feature.
4. **Finish, then publish.** When the whole feature is done — every repo, tests green — push
every branch and open **one MR per touched repo**, together.
5. **Merge promptly, once approved.** Every merge into `main` is notified and approved
(ADR 0023); once it is, merge —
do not leave it sitting. The branch is deleted on merge.
6. **Leave nothing behind.** After the MRs merge, no `feat/<slug>` branch and no `.work/<slug>`
worktree survive.
**One milestone is one release** (the operator, 2026-10-17). A milestone is developed whole, reviewed once,
pushed once and delivered once. It is never cut into steps for its size or for time: every cut adds a review, a
merge check, an Approve and a delivery plan of waiting, so the fewest cuts is the fastest. Milestones are cut by
release order when the design is broken up (playbook [04](04-build-handoff.md)); a release-order dependency found
inside a milestone is a fault of that cut, said to the operator, never a silent extra step.
## What this is not
**One design is built at a time** (the operator, 2026-10-17). While the milestones of one design are being
built, no milestone of another design starts: its issues stay `open`, and nobody moves them to `diagnosing` or
`fixing`. The next design starts when the last milestone of the one being built has its delivery plan done. A
lead who finds another design's milestone already in `fixing` asks its owner before starting its own. A small
fix ([playbook 08](08-small-fixes.md)) and an issue that is in no design's milestones are not held by this rule.
- **Not a licence to batch unbounded work.** A feature is a *bounded* unit; if it sprawls for
days, end-of-feature bloat merely replaces per-increment bloat. Split it into features, each
its own branch and MR.
- **Not a second trunk.** Every repo branches off `main`. There is no longer an `initialization`
trunk.
**Scope is fixed at the prepare step.** An issue found during a feature goes to the next milestone or the next feature,
never into the running one, unless it blocks the running one.
## Step 0 — Prepare (the lead, larger model)
1. **Name the feature once:** `feat/<slug>`, the same branch name in every repository it touches (but see the
exception for a new manifest word in phase 3: a release that must merge first has a name of its own). The
branch is never protected. **One milestone is one branch per repository** (the operator, 2026-10-17), with one
folder `.work/<slug>/`; never a second name for part of it (`-2`, `-step2`).
2. **Make the worktrees:** one per touched repository under `.work/<slug>/<repo>`, branched off `main`
(`git worktree add .work/<slug>/<repo> -b feat/<slug> origin/main`). The lab beds find the other
repositories by sibling path, so add the untouched siblings the lab reads as detached worktrees on
`main` (`git worktree add --detach .work/<slug>/mesh-tools main`), never symlinks. **A repository's own
clone stays on `main`** (the operator, 2026-10-17): no work is done in it and no branch is checked out
there, so a session that opens in it reads what holds now; fetch and fast-forward it before reading.
3. **Settle every open question** in the feature's issues before any agent starts: on the issue, through
the knowledge base, or by asking the operator. A fixing agent that meets an undecided design question
stops, and its run is spent again.
4. **Write one context file per repository**, `.work/<slug>/<repo>.context.md`. It is what a fixing agent
reads instead of rediscovering the code, and it is reused by every agent and every round:
- the issues it builds, in order, each with: the files and functions it changes, the decisions it builds
with (quoted, with where each was decided), and its acceptance criteria (item 5), each with the test that
proves it;
- the repository's map for this feature: the packages and files involved, with one line each on what they
hold, and the patterns to copy (file:line);
- what not to touch, and what lands beside it in other repositories;
- the local facts: the worktree, the branch, the check commands, a temporary directory for tests outside
the home (`TMPDIR`), and the unit tests to run (never a database server: tests that need one are
integration tests, which the forge's checks run).
When the reading is large, a read-only agent on the smaller model answers the lead's exact questions with
file:line, and the lead writes the file from its answers.
5. **Write the findings into the issues** (the operator, 2026-10-17: the better the content of the issues, the
better work the agents do, and the less review is needed). Before any agent starts, each issue carries, with
`issue-tracker.note`: every file and test that touches the change, found by search and not from memory
(file:line); how the change is checked and delivered end to end (which test proves its done-when, which
modules its delivery plan moves, to which machines); and for a group, its merge order. The context file
points to these notes; an issue reread by another session or a later agent then holds what was found.
When the reading is large, it starts with a read-only survey agent on the smaller model.
**Each issue carries its acceptance criteria, written by the lead** (the operator, 2026-10-18: the larger
model writes them, and they guide every test). Each criterion is one sentence, given, when, then, about what
the module or the operator observes, never about how the code does it, and names the level of the test that
proves it: unit, integration (a bus, a store or a process started) or end to end (none exist yet, so such a
criterion is proven live after the delivery and said so). Together they cover the issue's done-when and the
cases its life passes through: on 2026-10-18 mesh-delivery's group life (a group of three, a member's own
check, a close in a group, a second pull request at the same commit, a stop and a resend) had no criterion,
and each case was found by a failure on the live mesh. Each criterion is numbered in its note and cited as
`<issue>/AC<n>` (`1118/AC2`). They are an `issue-tracker.note` headed `Acceptance criteria`, a revision a new note that replaces the older one, until the tracker holds them as a
field (issue 1118). A criterion the reviewer adds in step 2 is written onto the issue the same way.
**The survey sizes each issue, and an issue sized over an hour is split before any agent starts** (the operator,
2026-10-17), into issues of about 30 minutes each, one change each, in the order they land: the bound of a
fixing agent's issue is then a real bound. On 2026-10-17 one issue held four changes sized at four to six
hours; it ran past its 30-minute bound for over an hour, and the reports could only say "still on part D".
6. **Move the issues to `fixing`**, `by` naming the feature's agent for that repository.
## Step 1 — Develop (one fixing agent per repository, smaller model)
- **One agent per repository, kept for the whole feature.** It reads its context file once and builds the
issues one after another, committing each locally with a message naming the issue ("Keep the assignments
on the bus (issue 652)"). It is not replaced per issue: the lead continues the same agent (by its id) with
the next instruction, so its context is paid for once.
- **Unit tests only, each well under a second** (how-we-build §5, ADR 0391). **The whole unit suite runs before
every commit** (the operator, 2026-10-17), not only the tests of what changed: it takes under a second, and a
change breaks tests in packages it never touched. Each issue's first test fails before its change and passes
after; `gofmt`, `go vet` and the whole unit suite stay green. **The tests are written from the issue's
acceptance criteria**: one test or more per criterion at the level it names, each test's comment citing
the criterion it proves by its label (`// 1118/AC2`), and no criterion left without its test; an integration criterion is a test
that compiles while you work and runs in the one merge check before the push.
- **Nothing is pushed** and no pull request is opened in this step. Never land on a red check: run the
checks, read the result, then land in a separate step.
- **The fixing agent's brief** (the lead gives it with the context file): name your issue in every step and
message ("issue 805: landing in mesh-catalog"); reach the mesh only through the mesh MCP server, never ssh;
never push, open, merge or close a pull request; never write to a store's database by hand, a schema change
is a numbered migration and the consolidated schema; Go only unless the context file says otherwise; never
move another issue; **run the integration tests once before every push, not while developing** (the
operator, 2026-10-18): compile them with `go vet -tags integration ./...` while you work, and before the push
run the repository's whole merge check with `mesh-controller.try` on the pushed branch (the store and bus a
pull request's check gets) or `merge-check.sh` against a throwaway store, and push only on its pass. On
2026-10-18 three pushes in a row failed their first merge check on faults an integration run finds in a
minute (mesh-catalog #361, mesh-controller #380's stale keys, #381's 400 tests without the layout marker),
each costing a check, a fix and another check. Report back in under 150 words: commits, tests added and that they failed first, the
check results, anything you stopped on.
- **An agent that finds the code differs from its context file stops and reports** (file:line); the lead
settles it, updates the context file, and continues the agent.
- **Two repositories that depend on each other** keep their order in the context files: the reader before the
writer, the controller's new manifest word before the catalogue manifest that uses it.
- **Several agents in one repository** (only for a large repository with independent areas): each works on a
local branch of its own from the feature branch's head and lands by fast-forward under a lock
(`flock <feature worktree>.lock git -C <feature worktree> merge --ff-only <its branch>`), after its unit
tests pass; if another landed first, it merges again and repeats. In a shared clone, fetch the feature
branch by name before branching from it. Never resolve a conflict with a blind `git merge-file --union`.
## Step 2 — Review (one reviewer, larger model; at most 3 rounds, about 15 minutes)
1. **Bring every feature branch up to the current trunk** (fetch by name, then merge), so the reviewed tree is
the tree that is pushed.
2. **One reviewer reads every repository's local commits** together, against the context files and the
decisions. **It is one agent, kept for the whole release** like the fixing agents: the lead continues it by
its id for every later round, for the extra cycle after a failed check (step 3) and for its review record, so it
reads the decisions, the issues, the context files and the commits once (issue 1042). A fresh reviewer per
round or per mark pays for all of that again. Its report is per repository: each finding as file:line, the failure it causes, and the fix.
It says which round it is in ("round 1 of at most 3").
3. **The report goes to the same fixing agents** that wrote the code, continued by their ids, so they fix
with the context they already hold. Each answers the reviewer: what it changed, per finding, or why not.
4. **Every new integration test is read against the code it drives before the first push.** It has never run: fixing agents only compile it, and its first run is the merge check, so each fault it hides costs a whole check. The reviewer traces each step of each new integration test through the code (fixtures against the manifest rules, what the store keeps and reads back, who holds a seat), and says which steps it could not trace. On 2026-10-17 a new test failed on its first two runs, on a fixture and then on a fault of the change; reading it against the code after the first failure found another fault, which would have failed a run too. The reviewer also checks that each new integration test enters through the same path the live trigger takes, not through a state it writes by hand: on 2026-10-17 milestone 4's test saved its delivery plan directly, so it skipped the cut from a batch that every live plan takes, and it passed while the backup step never ran (issue 1086).
5. **The reviewer reads the answers and the new commits**, and approves or reports again (step 7 says what a round
may ask). After round 3 the lead stops and brings the operator what is left, with a recommendation.
6. **The reviewer reads every acceptance criterion against its test**: each criterion has a test at its level
that would fail without the change, and no test proves less than its criterion says. A criterion missing
for a case the issue's done-when needs is written onto the issue (step 0, item 5) and is a finding; a test
missing for a criterion is a finding.
7. **A finding outside the issues' done-when and their acceptance criteria is never a change asked, in any round.** More hardening, a
further case, a nicer form: the reviewer opens it with `issue-tracker.open` and names the number in its
report, and the round does not wait for it. What a round asks is only what makes the change wrong against
its issues, their acceptance criteria and its decisions.
## Step 3 — Check (no review)
1. **Push every feature branch and open one pull request per repository, together.**
The integration tests ran once before the push (step 1); the merge check runs them again as the gate. Pull requests in two or
more repositories that share the branch name are one delivery group (ADR 0239 §2).
Where a session pushes through the forge tool `gitea_commit` (one commit on the forge's current base), the
reviewer first confirms that commit's tree equals the reviewed one. **Each pull request's description carries
the review record** (ADR 0421 §5), one line the lead writes from the reviewer's approval:
`Reviewed-by: <who@node> at <full sha>` (the issue 1022 note gives its form). mesh-delivery reads it at every new
head and puts it in the Approve's ask; a missing line, or one naming another commit, is said first in the ask.
It is never a gate.
2. **The forge's merge checks run the integration tests once** (`mesh/merge-gate`, `mesh/repo-check`). No
second review: step 2 reviewed this tree.
3. **A check that fails** goes back to the same reviewer, continued by its id, with the failure, for **one more cycle** of step 2 with
the same fixing agents; then push once again. **The failure is read from the check's whole log**, never
reasoned from its summary: `mesh-delivery.show` names the check's build id, and `mesh-controller.builds` with
`log` set to that id prints everything the builder said. A check's summary picks only some failing lines;
on 2026-10-17 it showed a store test's `FAIL` without its cause, and a fix was first guessed from the code.
**A change to `merge-check.sh` is judged by main's version of it**, so a test the change adds must pass under
either version: one that runs the new script's behaviour runs a file of its own, never `merge-check.sh` itself.
**So a change that needs the new script to judge it ships the script first, as a small fix of its own**, and the
rest waits for its delivery: on 2026-10-17 milestone 3 of design 47 moved its integration tests behind a build
tag in the same pull request as the script that runs that tag, and main's script would have skipped every
tagged test, its acceptance tests among them. The prepare step looks for this: a change to `merge-check.sh` or to
what it selects (build tags, the store-tests list) is cut ahead of the milestone.
4. **The milestone's acceptance is proven by its check.** A milestone that adds or changes a module
carries an integration test that starts the module whole, its process against the real bus and store, and
shows the milestone's acceptance sentence; the forge's checks run it (ADR 0391 §1, §6). Unit tests alone are not
that proof; the mesh has no end-to-end tests yet, and the lab is not used (the operator, 2026-10-16).
5. **The operator's Approve is asked as soon as the checks pass** (ADR 0421). Nobody marks, labels, comments or
rechecks: mesh-delivery asks the Approve on Telegram, the channel that proves who answered today (ADR 0398 §3;
an Approve at the terminal is issue 953), once both checks passed, for a delivery group once for all its
members, naming the review record. On the Approve it merges the approved heads, a group's in one pass
(ADR 0418). A new head ends the Approve and is asked again once its checks pass. A watcher follows the checks,
the Approve and the merge, and wakes the lead. Beyond that
Approve, the operator's word is waited for only when the milestone changes what the operator sees, removes
data, or needs a step only the operator's terminal can take; the lead says which before the milestone starts.
**One exception: a catalogue change that uses a new manifest word.** The catalogue's merge check reads
manifests with the controller the mesh runs, so the controller's pull request is merged and delivered first;
then `mesh-delivery.recheck` the catalogue's and merge it on the new verdict. Such work is two releases, the controller's first, and playbook 04 cuts its milestones that way.
**A new key the node-engine checks is such a word too** (issue 1051): the controller judges every machine with the
node-engine's own validator, vendored at the commit `go.mod`'s `replace` pins, so a key added to the node-engine is
refused by the gate until the controller's pin moves to the delivered node-engine (`go mod vendor`); the feature is
then three releases, the node-engine's, the controller's, the catalogue's. **Every release that must merge before
the catalogue's is on a branch name of its own** (`feat/<slug>-node-engine`, `feat/<slug>-controller`): on the
feature's name it joins the catalogue's delivery group, and a group merges only whole, so neither merges.
The exception ends when milestone 3 of design 47 is delivered (issue 977, ADR 0402, ADR 0403): the group's composed
check, judged by the group's controller, is then each member's gate, and the feature is one group.
## Step 4 — Deliver
The merge starts the delivery plan: build, publish, send. A merge into a branch other than `main` is not a
delivery: no machine receives it (issue 889 while it still starts one). The lead follows it to done
(`mesh-controller.plans` with `wait` and `until: done`), then resolves the feature's issues with `fixed-by`
naming its pull requests, and removes what is left: no `feat/<slug>` branch and no `.work/<slug>` worktree
survive.
## Models
| Who | Model | Why |
|---|---|---|
| The lead (step 0 decisions, the context files, following and reporting) | larger | judgement |
| Read-only questions about the code for the context files | smaller | reading, answered with file:line |
| Fixing agents (step 1, and the fixing in steps 2 and 3) | smaller | building to a written context |
| The reviewer (step 2, the extra cycle in step 3 and the review record; one agent, continued by its id) | larger | judgement, once per round |
| Decision records | larger | judgement |
Every agent the mesh runs draws on one weekly allowance shared by every session, and the larger model
spends it many times faster. Up to five agents on the smaller model at once, at most two subagents on the
larger one (the lead not counted).
## Following and reporting
Most lost time is waiting nobody saw. The lead therefore keeps a **wait list**: everything the feature waits
on, with when it started and its bound.
| Waiting on | Bound | Past the bound |
|---|---|---|
| a fixing agent's issue | 30 minutes | ask it where it stands; split or take over |
| the review | 15 minutes a round | ask the reviewer; stop at round 3 |
| a merge check | 15 minutes | read the build queue (`mesh-controller.queue`) and the check's log |
| the Approve's ask | 5 minutes after both checks pass | read `mesh-delivery.show`: an ask not raised is a fault, said to the operator |
| a merge after the Approve | 5 minutes | read `mesh-delivery.show` for the delivery |
| a delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
**Every wait on the list has a watcher** (the operator, 2026-10-17). A session runs only when a message reaches
it, so a check, a merge, a delivery plan, a record's review or the operator's Approve on a channel that lands
while the lead's turn has ended wakes nobody: on 2026-10-17 an approved group sat half merged for 14 minutes
unseen (issue 1032). For each wait the lead starts a watcher, an agent on the smaller model that changes nothing,
never the smallest (the operator, 2026-10-17).
It loops `mesh_wait` on the event families of what it watches (`mesh-delivery.transition` and
`mesh-delivery.group` for a delivery, for example) with a timeout under 60 seconds, since the call itself ends near
a minute. After every answer or timeout it reads the state from the verb that owns it (`mesh-delivery.show`,
`mesh-controller.plans`, `knowledge-base.get`), because an event only wakes it (ADR 0367). It finishes, and so wakes
the lead, when the state changes, when the wait passes its bound, **or when nothing changed for two minutes**
(the operator, 2026-10-18): a wait that stopped moving is a stall, never a wait. On 2026-10-18 a delivery plan stopped
at 12:46 because the bus refused the new controller's writes, its watcher waited for "done or failed", and the stall
was found ten minutes later only because the operator asked what was happening (issue 1121). The lead never sleeps or polls by hand to wait.
**Every ten minutes (the operator, 2026-10-17), every two minutes while a delivery plan runs, and at once when something lands, stops or
needs the operator,** the lead reports, in one of the two layouts below (the operator, 2026-10-16: an overview
of the issues, often). Times are local and absolute, from the clock (`date`), never guessed; every number has its
unit and its bound or baseline; an issue is named by its number and a few words, never by its number alone.
**The full layout** (in the session) is the whole design as a tree (the operator, 2026-10-17: always the full
overview): every milestone of the design with its state, and under the running milestone its repositories and
their issues, each issue under the one it waits for, marked ✓ landed or delivered, ▶ fixing, ⏸ waiting, ✗ stopped:
```
Feature <slug> · milestone <n> of <m> · step <name>, <minutes used> of <bound> min
Issues: <landed> of <total> landed · <fixing> fixing · <waiting> waiting
Design <nn> · <its title>
├── ✓ milestone 1 <its outcome, a few words> delivered 12:58
├── ▶ milestone 2 <its outcome, a few words>
│ ├── mesh-controller (agent: fixing, 2 of 3 landed)
│ │ ├── ✓ 652 assignments on the bus landed 14:02
│ │ ├── ▶ 736 placement words leave the layers fixing since 13:51, 23 of 30 min
│ │ │ └── ⏸ 751 config keys named (mesh-catalog) waits for 736
│ │ └── ✓ 654 a file names its template landed 13:20
│ └── mesh-catalog (agent: waiting)
│ └── ⏸ 751 (above, under 736)
└── ⏸ milestone 3 <its outcome, a few words> <its issues>
Opened along the way: <each issue opened during the feature, its number and a few words, and where it goes> — or "none"
Past its bound: <the wait, how long, what is being done> — or "none"
Needs you: <one line per question, with my recommendation> — or "nothing"
Allowance: week <n> % (<+d> since the last report), session <n> %
Next: <the next actions up to the delivery, in order>
```
**The short layout** (on a channel such as Telegram, or when nothing changed but time):
```
<slug> milestone <n>/<m> · <step> <used>/<bound> min · <landed>/<total> landed · past bound: <none | what> · needs you: <nothing | what> · week <n> %
```
A report says what changed since the last one first. In the session every report is the full layout, the whole
tree of the milestone's issues included, also when only time passed: the operator reads the state from it without
asking. The short layout is for a channel.
## The context of the lead's own session
At the end of each step, and whenever the session's context grows large or spread over unrelated work, the
lead prepares a `/clear` as the mesh's instruction "prepare before clear" says: a save point with the
feature's step, its branches and heads, the context files' paths, the agents' ids, the wait list and the
next action; then the prompt the operator pastes. The context files make the feature resumable after a clear.
## The design's state is written back
Other sessions know what is built only from the design's frontmatter and the issue tracker; what the lead
knows and does not write back, nobody else knows. A design merged in one delivery with three of its tasks
left, its frontmatter still saying `designed`, read to another session as not built, and a second design
was started beside it (issue 1009, 2026-10-17). So the lead writes the design's state back at each of these
moments, the same hour, an hq change going through [playbook 08](08-small-fixes.md):
- **The first milestone starts:** the design's `status: in-progress` ([playbook 04](04-build-handoff.md) step 7),
if it is not already.
- **A milestone's delivery plan is done:** its issues are resolved (step 4), with `fixed-by` naming its pull
requests. The design's milestone entry is not changed: it names the milestone's issues, and whether they are
delivered, when and by what is read from the issue tracker when it is asked, never copied into the design
(issue 1038; AGENTS.md: cross-cutting views are generated when they are read).
- **The building pauses with work left** (another design takes its turn, a hand-over, the session's end): the
design stays `in-progress`, and each issue left gets a note with `issue-tracker.note` saying what it waits
for and what comes after it ("built after design N is delivered, before design M"). The first issue of
a design that waits its turn gets the same note.
- **The last milestone is delivered:** [playbook 02](02-graduation.md), "When something ships": the as-is is
written and the design says `implemented`.
## Issues that stop
An issue stays in `fixing` only while an agent works on it. A milestone that stops, for a wait past its bound, a
hand-over, or the session's end, moves each of its issues not delivered back to `open` with
`issue-tracker.move` and a note saying how far it got (its branch, its last commit, what is left). A list of
issues in `fixing` that nobody works on reads to the operator as stuck work.
## How it is checked
The end state is visible, and its absence is the smell:
- A feature's issues show their commits on the feature branch (`git log --grep 'issue <n>'`), and no pull
request into a feature branch exists. A pull request per issue, or an agent waiting on a merge check in
step 1, is the failure this playbook prevents.
- A context file exists per touched repository under `.work/<slug>/` while the feature runs.
- No issue of a running milestone is sized over an hour in its prepare note; an issue whose fixing agent passes
twice its bound was not split.
- Every issue of a running milestone has a note from the prepare step naming its files and tests (file:line), how
it is checked and delivered, and for a group its merge order (`issue-tracker.get`); an issue moved to `fixing`
without one is a prepare step skipped.
- Every issue of a running milestone has a note headed `Acceptance criteria` before it moves to `fixing`
(`issue-tracker.get`), each criterion naming its test level, and every criterion's label is cited by a test in the
pushed tree (`git grep -n '<issue>/AC<n>'`); an issue without them is a prepare step skipped, and a
criterion no test names is a review that missed it (until issue 1118 makes the tracker refuse it).
- Every fixing agent's report gives the whole unit suite's result and time; a report naming only some packages'
tests is the whole-suite rule not kept.
- The lead's save point names each watcher with its model; a watcher on the smallest model is the failure of
"Following and reporting".
- A report or a save point that names a "phase" of a milestone is the old wording: a milestone has steps.
- Every repository's own clone says `main` (`git -C <clone> branch --show-current`); another branch there is
work done outside a worktree (issue 1010).
- After a feature's delivery, `git branch -r | grep feat/<slug>` and `git worktree list` return nothing for it.
- A design whose milestone issues are partly resolved says `in-progress`, and one whose issues are all
resolved and delivered says `implemented`: read its Milestones section against `issue-tracker.get` for each
issue. A design that says `designed` with a resolved milestone issue is a state not written back.
- A design's Milestones section names each milestone's issues and nothing of their state: a "delivered", a date
or a pull request in an entry is a copy of the tracker's state (issue 1038), read with
`grep -n 'delivered' <design>` over its Milestones section.
- Every open issue of a design that says `in-progress` and has no agent on it shows, in `issue-tracker.get`, a
note saying what it waits for; one without such a note is a pause not written back.
- The milestone issues in `fixing` (`issue-tracker.list`) belong to one design at most: milestone issues
of two designs in `fixing` at once is the failure of "one design is built at a time".
- More than one open pull request in a repository that share no feature name, or a milestone whose issues
stay in `fixing` for more than a working day, is the smell: stop and bring it to the operator.
- A review report that asks a change outside the issues' done-when and acceptance criteria, or a fourth round, is the failure of the
review rules; `issue-tracker.list` with `flag: abandoned` lists issues left in `fixing` with nobody on them.
- A pull request green for more than 5 minutes with no Approve asked is a fault of mesh-delivery
(`mesh-delivery.show`), said to the operator.
- Every review record of a release names one reviewer (`mesh-delivery.show`, the review record), and the lead's
save point names one reviewer id per release; a second reviewer within one release is a reviewer not kept
(issue 1042). A pull request whose ask says it has no review record at its head is a record not written.
- A milestone that adds or changes a module whose pull request has no integration test that starts the module
whole is the failure of the check step's fourth item.
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it cannot find is a
missing sibling worktree, not a mesh fault.
- After a feature merges, `git branch -r | grep feat/<slug>` and `git worktree list` return
nothing for it. A surviving branch or worktree means step 6 was skipped.
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it
cannot find is a missing sibling worktree, not a mesh fault.
- More than one open MR in a repo that share no feature name, or a stack of MRs none of which is
merged, is the failure this playbook exists to prevent — stop and consolidate before opening
more.
When this playbook meets a problem, the lesson is added here, so the next feature avoids it.
+110
View File
@@ -0,0 +1,110 @@
# Playbook 08 — Small fixes
**Trigger.** One issue in one repository, small enough to build in about an hour: a bug with a known cause,
a missing check, a wording, a playbook or design edit. Anything larger, or anything that needs a change in a
second repository first, is a feature and follows [playbook 07](07-feature-branches.md). A feature is the
larger kind: it touches several modules and has a research effort or a design before it. **Needing a decision
does not make a fix a feature**: the decision is settled first (step 1) and the fix stays small (the operator,
2026-10-16).
**Related issues in one fix.** A small fix may cover two or three related issues when they are in the same
repository, mostly in the same module, and together still fit in about an hour: they touch the same files, or
one would be wrong or unsafe without the other (switching a clipboard's history on before its history tool stops
handing out what was copied). Each issue is named in the commit and the pull request and moved to `fixing`,
and each is resolved on its own once the fix is delivered. More issues, a second repository, or more than an
hour is a feature.
**Who runs it.** Anyone who writes code, engineers and agents alike.
## The aim
A small fix spends its time on the fix, and runs the long part once: **one review, one push, one check run,
one delivery**. Everything before the push is local and fast.
## Steps
1. **Read the issue and the code it names** (the session's own model, or one read-only agent on the smaller
model answering exact questions with file:line). If a design question is open, it is settled before any
code is written: on the issue, through the knowledge base, or by the operator. A fix that turns out to
need a second repository, or more than about an hour, stops here and becomes a feature; one that needs a
decision stays a small fix. **The session's own model (the larger) then writes the issue's acceptance
criteria** as playbook [07](07-feature-branches.md) step 0 item 5 says: each one sentence, given, when, then,
naming the level of the test that proves it, as an `issue-tracker.note` headed `Acceptance criteria`. A
playbook or design edit has none: its check is `sh merge-check.sh`.
2. **Move the issue to `fixing`** with `by` as who@node.
3. **Build on a local branch** `fix/<issue>-<slug>`, in a worktree of its own branched from `main`, never in
the repository's own clone, which stays on `main` (playbook [07](07-feature-branches.md), step 0, item 2). Write the
test that fails today first; make the change; keep `gofmt`, `go vet` and the repository's whole unit suite green
before every commit (each test well under a second; no database in the unit tests: how-we-build §5, ADR 0391); before the push the repository's merge check runs once, integration tests included, through `mesh-controller.try` or `merge-check.sh` against a throwaway store, as playbook 07 step 1 says.
The tests are written from the acceptance criteria, at least one per criterion at its level, each citing the
criterion it proves by its label (`<issue>/AC<n>`, playbook 07 step 0 item 5). Commit, naming the issue. For a playbook or design edit, run the repository's own checks (in hq,
`sh merge-check.sh`).
4. **One review of the local commit, before anything is pushed** (a reviewer that did not write it, larger
model; at most 3 rounds, as playbook 07; from round 2 only what is wrong; it reads each acceptance criterion
against its test, as playbook 07 step 2 item 6; a finding outside the issue's done-when and its acceptance
criteria is opened as an issue of its own, never asked as a change). The reviewer's findings go back to the same agent that wrote the fix, continued by
its id, so the fix keeps its context. A playbook or design edit of a few lines may skip the reviewer: the
operator's Approve of its merge is its review, and the review record says so.
5. **Push once and open the pull request.** The trunk is what runs and nothing holds merged work back
(ADR 0401), so a fix waits for no other change's delivery. The integration tests ran once
before the push (step 3); the forge's merge check runs them again as the gate. A failed check
goes back to the same agent with the failure, and the reviewer reads only the change it makes.
6. **The pull request's description carries the review record** (`Reviewed-by: <who@node> at <full sha>`,
playbook [07](07-feature-branches.md) step 3), and mesh-delivery asks the operator's Approve as soon as its
checks pass and merges the approved head (ADR 0421); nobody marks, labels, comments or rechecks. A fix to a module that changes how it runs carries an
integration test starting the module whole against the real bus and store, run by the forge's checks
(ADR 0391 §1, §6); unit tests alone are not that proof, and the lab is not used. The operator's word is waited for only when the fix changes what the operator sees, removes data,
or needs the operator's terminal.
7. **Follow its delivery plan to done** (`mesh-controller.plans` with `wait` and `until: done`), prove the fix on
the mesh (the symptom gone, measured), and resolve the issue with `located-in` and `fixed-by` the same hour.
A fix that stops before its delivery moves its issue back to `open` with a note saying how far it got
(branch, last commit, what is left), as playbook [07](07-feature-branches.md) says.
**A second fault found by the delivery.** When a fix's delivery shows a second fault that keeps the issue's
done-when from holding, and that fault is in the same files, it is taken into the same issue as a second pull
request with a review round of its own. A fault in other code is opened as an issue of its own, and the first
issue waits for it (issue 937: mesh-catalog #335 was taken in; issue 943 was opened).
**A fix that moves a file to a new filler compares the values first.** When a fix makes a module's file filled a
new way for the first time (the provisioner client in place of the controller, a provider's entry in place of a
composed value), the fix lists every value the old way put into that file on each machine it reaches and shows,
in its review, that the new way gives the same values, or says which differ and why. A difference nobody looked
at is an outage: on 2026-10-16 the forge and the identity provider on the control node were down for 17 minutes
because postgres's consumers were served the port inside its container instead of the port it is published on
(issue 957). The comparison is read from the machines (the file as it is today) before the merge, not from the
code.
## Bounds
| Waiting on | Bound | Past the bound |
|---|---|---|
| building the fix | 1 hour | it is a feature: stop and plan it under playbook 07 |
| the review | 15 minutes | ask the reviewer where it stands |
| the merge check | 15 minutes | read `mesh-controller.queue` and the check's log |
| the Approve's ask | 5 minutes after both checks pass | read `mesh-delivery.show`: an ask not raised is a fault, said to the operator |
| the merge after the Approve | 5 minutes | read `mesh-delivery.show` for the delivery |
| the delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
A wait past its bound is said to the operator at once, with what is being done about it.
## Models
The fixing agent runs on the smaller model; the reviewer on the larger. A session fixing alone does the
reading itself and uses one subagent only for the review.
## How it is checked
- A small fix has one pull request with one or few commits naming its issue, opened after its review: a pull
request with many heads is a fix reviewed after its push.
- Its issue moves to `resolved` within the hour its delivery plan ends.
- A pull request with no review record at its head (the Approve's ask says so first), a fix to a module's
running pushed with no integration test of the module whole, or a review report
asking a change outside the issue's done-when and acceptance criteria, is the failure of steps 4 to 6.
- A fixed code issue shows a note headed `Acceptance criteria` (`issue-tracker.get`) older than its first
commit, and each criterion's label is cited by a test in the pushed tree
(`git grep -n '<issue>/AC<n>'`); one without is step 1 skipped.
- An issue in `fixing` with nobody on it shows in `issue-tracker.list` with `flag: abandoned`; a stopped fix
moves it back to `open` with its note.
- The repository's own clone says `main` (`git -C <clone> branch --show-current`); another branch there is a
fix built outside a worktree (issue 1010).
+49
View File
@@ -0,0 +1,49 @@
# Playbook 09 — Cleanup
**Trigger.** The operator asks for a cleanup, or the forge and the working copies have gathered what nobody
uses: merged branches, stale pull requests, worktrees and folders of finished work.
**Who runs it.** A session on the operator's word, with the operator deciding every deletion. Agents read and
propose; a deletion is the operator's act or is done on the operator's word for the list it was shown.
## The aim
Remove what is provably done, and nothing else. A cleanup that deletes one piece of unfinished work costs more
than every stale branch it removed. So every item is shown with its evidence, and anything not provably safe is
reported and left.
## Steps
1. **Take stock, reading only.** One agent on the smaller model reads, changing nothing:
- the forge: every repository, every branch **as git refs** (the branch list tool misses some, issue 949),
every open pull request with its head, title and last update;
- each working copy on the machine: its worktrees, local branches, stashes, and the folders beside the
clones, with size and newest file.
It writes one report with each item, its class and its evidence, and a list of the tools it lacked.
2. **Classify each item**, by evidence, never by name or age alone:
- **Safe** — a forge branch whose tip is contained in its repository's default branch, with no open pull
request, not protected and not a long-lived branch the mesh treats as a trunk; a worktree that is clean,
whose branch is contained in the remote default branch, untouched for two hours and not in another
session's working folder; a local branch contained in the remote default branch.
- **Likely safe, not proven** — a branch whose every commit is patch-equal to one on the default branch
(the forge lands some commits under new hashes): listed, deleted only on the operator's word for it.
- **Kept** — unmerged commits, a dirty worktree, a stash, an open pull request, a protected or trunk-like
branch, a repository of another owner, another session's worktree.
- **Unknown** — a folder or file outside git: described, never opened where a secret may be, left.
Stale open pull requests (no update for three days) are reported with their author's issue, never closed.
3. **Ask the operator** with the counts per class and the report's place. One question: delete the safe
items as listed? The likely-safe and unknown items are separate questions, item by item when few.
4. **Delete only the approved list**, each with the command that refuses an unsafe case (a worktree removal
that refuses a dirty one, a branch deletion that refuses an unmerged one). A refusal is reported, never
forced. The operator approves a deletion on a channel that proves who answered, as an ask bound to that
exact list and carried out on the warrant (ADR 0387, design 58 milestone 7, issue 884). Until that is built,
an agent session's permission check refuses these deletions, and the approved list is handed to the
operator to run.
5. **Report**: what was cleaned, what was kept and why, what was unknown, what the operator kept, and the
missing tools, each opened as an issue (`issue-tracker.open`) or added to an existing one.
## How it is checked
- The report names, for every deleted item, the evidence it was safe (contained, clean, no pull request).
- Nothing classed Kept or Unknown is deleted without the operator's word for that item.
- A cleanup's missing tools are issues, not notes in a session.
+1 -1
View File
@@ -34,7 +34,7 @@ one-for-one — `mesh-catalog`, `mesh-sdk` and `mesh-tools` exist where the tabl
| `mesh-foundation` | 1 | the four pinned services, as declarations |
| `mesh-controller` | 2 | **exists.** The controller and its contexts — one of seven built (ADR 0006) |
| `mesh-surfaces` | 3 | tools, web, cli |
| `mesh-sdk` | — | the stable spine modules build against — the tool-serving harness, the messaging/event framework, the contracts and core primitives. Holds nothing per-module and nothing volatile (ADR 0039). |
| `mesh-sdk` | — | the stable spine modules build against — the tool-serving SDK harness, the messaging/event framework, the contracts and core primitives. Holds nothing per-module and nothing volatile (ADR 0039). |
| `mesh-lab` | — | **exists.** The lab — scenario lifecycle, networking, placement. Ships to nobody; runs on a workstation. |
Tier 4's shape is open, and deliberately so: see ADR 0019 and
+71 -81
View File
@@ -1,100 +1,90 @@
---
layer: as-is
status: implemented
code: [hal]
updated: 2026-08-23
code: [mesh-controller, mesh-catalog modules/mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/build-agent]
updated: 2026-10-17
decisions:
- "ADR 0010"
- "ADR 0010"
- "ADR 0010"
- "ADR 0239"
- "ADR 0267"
- "ADR 0276"
- "ADR 0282"
- "ADR 0289"
- "ADR 0304"
- "ADR 0350"
- "ADR 0358"
- "ADR 0360"
- "ADR 0364"
- "ADR 0382"
- "ADR 0395"
- "ADR 0398"
---
# Delivery — from a push to a running node
# Delivery — from a merge to the nodes, as it runs
One trigger, three silos, and a fan-out point that is the most consequential boundary in the
mesh.
What runs on the mesh's main branches on 2026-10-16, read from the code and the live record by research 055
(findings 1 to 12). It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) replaces, milestone
by milestone. Until 2026-10-16 this document described the predecessor's pipeline, which no longer runs.
## The trigger
## Who does what
A push to the forge. The forge calls a webhook; the receiving module verifies its signature
and emits an event. The coordinator resolves which modules the pushed commits affect, orders
them into dependency levels, and creates a pipeline per level.
- **mesh-delivery** owns deliveries and delivery groups (ADR 0239): one delivery per commit, a group per branch
name shared across repositories. It records each transition in its state on the bus (key-value buckets
`deliveries` and `groups`), marks pull requests, asks the controller to deliver, and merges a pull request marked
ready at its head once both required checks pass against the trunk's head (ADR 0364) and the operator approved
the merge into main on a channel that proves who answered (ADR 0398 §3).
- **The controller** plans, orders, sends and judges. It computes a delivery plan for a repository and the files a
merge changes, moves only modules whose build source changed with their build dependents (ADR 0267), and
reuses a merge check's build of the same reuse key (ADR 0350).
- **The builders** (holders of the build seat) take build requests and checks from a work-queue stream on the bus
and build inside a toolchain image another module provides, with the network off; on 2026-10-13 no node had Go
or npm installed (issue 626).
- **The forge module** publishes every forge event on the bus through the forge's system webhook since 2026-10-14
(issue 729), and still polls the forge every 30 s, or every 300 s once it hears events. It keeps files of what it
announced so it does not announce twice.
There is no second path. A manual trigger exists for a module the push detection missed, and
using it routinely is a sign the detection is wrong rather than a workflow.
## From a merge to a rollout
Detection is the pipeline's most fragile input. It has failed for reasons that have nothing to
do with the change: a webhook truncating its commit list on a large merge, a forge address
whose port broke the module-path matching. The characteristic outcome is the bad one — **a
merge that created no pipeline, and nothing said so**.
- **The merge window.** The controller collects merges into an open batch; each merge restarts the window, which
closes after 90 seconds without a merge or 10 minutes after the batch's first merge (ADR 0276). A lone merge is
cut at once (ADR 0282 §2, as ADR 0304 §1 changed it). The batch and its merges are rows in the controller's
store, cut by a timer every 5 s, with a catch-up timer every 5 minutes for merges no batch holds.
- **The rollout.** A closed batch becomes one rollout, at most one commit per repository, tiered along the module
graph. Rollouts run one at a time: a batch cut while a rollout is open is queued behind it, whatever it shares
with it. Running unrelated rollouts side by side (ADR 0282 §3, ADR 0304 §2) is decided and not built (issue 525). A rollout's builds go to every machine at once, with no first-machine gate, and the rollout ends on
the machines' reports or after fifteen minutes of silence (ADR 0360). A controller timer advances rollouts every
30 s.
- **A merge during a rollout.** Its place in the next batch is a store row the open rollout's holder writes. On
2026-10-16, when another controller worked the plans, a merge was not recorded and waited until a person released
it (issue 921).
- **"Release branches".** A pull request into a branch other than a trunk is judged by the controller built from that
branch (ADR 0358); feature 57 shipped about 37 packages that way over about 2.5 days and held the controller for
55.7 h.
## Three silos
## Checks
Cardinality is the whole point, and the three differ
(ADR 0010):
- Every pull request's head gets `mesh/merge-gate` (the composed machines) and `mesh/repo-check` (the repository's
`merge-check.sh`), each judged against one trunk commit (ADR 0323). A trunk move asks the check of every active,
cleanly merging pull request again (ADR 0395).
- A delivery group's members are each checked beside their siblings' heads, and the group also gets a composed
check, `mesh/delivery-group`; each member keeps its own gate beside it (ADR 0382).
- The queue's order (delivery builds first, then ready checks, then the rest) is kept by pausing the worker and
deleting and republishing a request (issue 558); an old head's check is ended after asking the forge for the
current head (issue 726).
- mesh-delivery ticks every 10 s, reconciles with the controller every 30 s and polls the forge for ready marks
every minute, although the forge module already publishes labeled, commented and synchronized events.
| Silo | Runs | Where | Does |
|---|---|---|---|
| **build** | once per module feature | the build node | compile and bundle into a self-contained output |
| **publish** | once per module feature | the build node | package that output and upload it; a package-registry feature publishes here |
| **deploy** | once per module feature **per node** | every assigned node | install, configure, start, verify |
## Where the record of what runs lives
Commands and events are addressed per feature, not per module.
Builds and build requests, rollouts with their tiers, batches and merges, and assignments are rows in the
controller's store; artifacts are blobs in the artifact store by digest (ADR 0251); packages are versions in the
package registry. The controller writes three record mirrors to the bus: assignments, declared config and
predefined keys; of the five ADR 0289 §7 decided, the nodes, rollouts, builds and settings mirrors are not in the
code. An assignment names its module and node but not the build, artifact or rollout that put it there: which commit
runs where, and which delivery brought it, takes three verbs and a join by hand.
Build hands over a **staged tree**, not a package. Packaging belongs to publish, so a failed
upload retries by re-packaging rather than by re-sending something stale, and build never needs
to know how each module composes its artifact. The handover is a tree in a known location,
because the build's own working directory is reference-counted and may be gone by the time a
later stage runs.
## The artifact
The artifact is **build output** — compiled and bundled with its dependency graph inlined —
never a filtered copy of source (ADR 0010).
A deploy is extract-and-run and touches no network.
The consequence is the whole cost of the decision: **anything not in the build output does not
ship.** Every file kind had to be brought into that rule separately, and each was discovered by
something silently not happening after deploy — migrations reading a source layout, provisioning
scripts reading a source layout, selection files never packaged at all.
## The fan-out, and the build node
After publish, work fans out to every assigned node. The build node is the only node that has
already passed through two silos when this happens, and that asymmetry has its own defect
class: anything advancing a node's stage must account for **both** pre-fan-out stages. Code
that knew only about the first parked the build node forever while every other node deployed
cleanly — and the recovery sweeper, which knew the same subset, reported nothing to recover.
A recovery mechanism that knows less than the thing it guards is worse than none, because it
reports success over a stall it cannot see.
## Levels
A level completes before the next begins, so a module builds against its dependencies as they
were just published. The shared library is at level zero, which means anything that breaks it
breaks the first module of every cascade.
## What green proves
**A green pipeline proves transport, not effect.** The stages report that a message was
dispatched and accepted, which is not the same as the thing being running, correct, or present.
This is the mesh's most consistent failure shape and it is not incidental to the design — it is
what the stage reporting currently measures. Documented instances include a service reported
started when the container command merely returned, an image pull failure that did not fail the
deploy, a package install that 404ed from every mirror while the job went green (see
[`04-ISSUES/001`](../../04-ISSUES/001-failed-package-install-reports-success/00-report.md)),
and a node left on old code after a failed artifact download with a version marker that had
already advanced.
A verify stage exists to close this gap. It has been built and, for a period, was never
scheduled, because the coordinator's stage list did not include it.
No backup is taken before a delivery moves a module that holds data; backups run nightly and on demand (ADR 0214).
## What is not covered
The end-to-end harness for this pipeline has not built since the workspace was removed, and
nothing reports that nothing runs it
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). The
pipeline's coverage is currently assumed rather than checked — which is the same class of claim
this repository exists to make people stop making.
The integration tests run only in each repository's merge check; there are no end-to-end tests of delivery yet,
and the lab is not used (the operator, 2026-10-16).
+122 -47
View File
@@ -1,69 +1,144 @@
---
layer: as-is
status: implemented
code: [mesh-catalog modules/mesh-console, mesh-tools src/mesh.ts, mesh-tools src/http.ts, mesh-tools src/runtime.ts, mesh-controller internal/broker, mesh-controller cmd/mesh-controller/check.go, mesh-controller cmd/mesh-controller/seatverbs.go]
updated: 2026-10-10
code: [mesh-tools, mesh-host, mesh-controller]
updated: 2026-10-17
decisions:
- "ADR 0152"
- "ADR 0095"
- "ADR 0037"
- "ADR 0175"
- "ADR 0195"
- "ADR 0197"
- "ADR 0367"
- "ADR 0154"
---
# The mesh MCP server, as it runs
**The mesh's tools reach a person through a module the mesh assigned to their machine.** Since
2026-09-30 a workstation that is a node can be assigned `mesh-console`; the mesh mints a bus account
`<node>.mesh-console`, seals its credential to the machine, and the container binds
`127.0.0.1:<port>` with the port the mesh assigned for the manifest's declared one. An agent on the
machine is pointed at `http://127.0.0.1:<port>/mcp` and sees the mesh's tools; a person uses the same
endpoint. Nothing on the machine holds a credential a person had to carry.
**The `mesh-console` container no longer exists.** Read 2026-10-17 from `mesh-tools` (package
`node-tools`), `mesh-host` and `mesh-controller` on their forges' main, and from the live mesh
through `mesh_overview`, `mesh_machine`, `mesh_runtimes` and `mesh_describe`. Checked: the
catalogue (`mesh-catalog`) carries no module named `mesh-console` any more — the name survives only
in the glossary's retired-words list and in a test fixture — and none of the four nodes answers a
module of that name.
## What it is now
**The tool runner is one of the mesh's core components**, named `node-tools` beside
`mesh-controller`, `mesh-host` (the node-engine) and `nats` (the bus) — the mesh's own repositories'
check treats the four the same way, as the core whose health the mesh itself defines rather than a
module's own. `mesh-host` and `mesh-controller` supervise, compose and account for it as a core
process rather than as an ordinary assignment (`ComponentNodeTools`/`BusNodeTools` in both, the
witness and lease code that restarts and rolls it back, the account named `<node>.node-tools` on
the bus).
**It still carries a manifest, `node-tools/module.json` in `mesh-tools`,** and that manifest is
where its grants live: `invokes` names `*`, so its account may call every tool on the mesh;
`waits` names the five event families `mesh_wait` may subscribe to (the knowledge base's reviews,
mesh-delivery's transitions, the forge's status, the controller's checked event, and the operator
channel's decision for the agent module) — the bus refuses a subscription outside this list
(ADR 0367 §2); and it
`provides` one provision, the MCP endpoint, at node scope, and `listens` on the port it serves from,
opened only to the machine itself. Its code lives under `node-tools/` in the `mesh-tools`
repository (`internal/console`, `internal/announce`, `internal/bus`).
**One runtime per node, run by the node-engine as a sibling process, never a container**
(ADR 0175). It holds the node's one bus credential and may call every tool on the mesh; its account
publishes nothing else. It binds `127.0.0.1:<port>` and refuses to bind anything else, said before
binding rather than after (`node-tools/internal/console/http.go`, citing ADR 0152): whoever is on
the machine owns the mesh there, and nobody else may reach this port. An agent or a person on the
node is pointed at `http://127.0.0.1:<port>/mcp` once; the `serverInfo.name` the handshake answers
is `mesh`.
## What it answers
`initialize`, `tools/list`, `tools/call`, over HTTP, one JSON body per request, no session and no event
stream. `tools/list` is what the running modules answered: every tool runner built on or after that day
serves a `tools` verb for its module, and the mesh MCP server asks the catalogue for the roster and each module
for its tools. A module that did not answer is named in the list's `_meta.notAnswering`. On the day it
shipped that was 36 of 51 modules — those that serve no tools at all, and those whose rebuilt runtime the
mesh records rather than rolls out — and 62 tools from the rest.
`initialize`, `notifications/initialized`, `ping`, `tools/list`, `tools/call`, over HTTP, one JSON
body per request, no session and no event stream (`node-tools/internal/console/mcp.go`).
`tools/call` reaches any tool by `<module>.<tool>`, listed or not. The mesh MCP server's grant is `*`, so what it
may call is every tool on the mesh; its account may publish nothing else and subscribes nothing.
**It announces seven tools, not five** (ADR 0195, extended by ADR 0367): `mesh_overview`,
`mesh_machine`, `mesh_search`, `mesh_describe`, `mesh_call`, `mesh_runtimes` and `mesh_wait`. The
handshake's `instructions` text names all seven and the four address forms. `mesh_overview` is
where the mesh-wide count of modules and tools is read live — a number copied into this document
would drift; asked on 2026-10-17 it answered 103 modules and 645 tools across the whole mesh, with
two module instances (`route-proxy` on two of the nodes) named as not answering rather than silently
absent.
No call is refused for want of a person's approval: a tool runner serves every call it is issued, whoever made
it, and cannot tell an agent from the operator (ADR 0259 §6, issue 340). An act waits for the operator's warrant
only where its module asks first, in its own code: the agent module's instruction proposals, and the
controller's settings proposals and the actions it offers on a condition.
**Addresses, exactly as the handshake states them:** `<seat>.<verb>` for a seat held once for the
mesh (`mesh-controller.status`); `<node>/<seat>.<verb>` for a seat every machine holds
(`<node>/node-packet-filter.rules`); `<node>/<module>.<tool>` for a module assigned to one machine;
and `<module>.<tool>` also for a module whose instances are interchangeable. A tool not listed can
still be called by address — listing is discovery, calling is the grant — and a module that is not
interchangeable is refused with the machines it runs on rather than guessed at.
**What exists is what announced itself** (ADR 0197). Every node's tool runner answers the NATS services
protocol's `$SRV.PING`, `$SRV.INFO` and `$SRV.STATS`, one service per process, naming in each
endpoint's metadata which module, seat, scope and machine serves it. `mesh_overview` and
`mesh_machine` gather one request's answers: at least 750ms so a runtime that only just answered
`$SRV.PING` is still waited for, up to 5s for one that said it is there but has not yet sent its
full announcement — a fixed shorter window lost the largest runtime's listing once a live
measurement found how slow its round trip over the bus could be. An announcement unchanged since
last asked (by its digest) is not re-fetched, which is why `mesh_runtimes` can show a runtime
answering in under 100ms for hundreds of kilobytes of discovery.
## mesh_wait
**The seventh tool, added by ADR 0367**, lets a session wait on the bus for an event instead of
polling: a call names one or more granted event families and an optional filter, and either answers
the first match or says it timed out. The tool runner's own bus account hears only the families its
manifest's `waits` list names; an event it hears is never taken as the state — a session reads the
state from the verb that owns it once woken. It is meant to be called in a loop from a background
command, not inline, because one call blocks until it answers or times out.
## The mesh's own verbs
*Since 2026-09-30 evening (ADR 0154).*
The mesh MCP server asks the `mesh-controller` seat's `tools` verb beside the modules and lists every role's
tools as `<seat>.<verb>` — `mesh-controller.status`, `mesh-controller.push` and the other ten. A call
to `<prefix>.<name>` reaches the seat when the prefix is a seat declaring that verb, and the module
otherwise; `seat:<seat>.<verb>` says so outright. When the controller does not answer, the list
names `mesh-controller (seat)` as not answering and carries the modules' tools regardless. The
`mesh-controller` *module* is always named as not answering: it serves no module tools, only its seat's.
**These already answer directly, not as a stated future.** Beside every assigned module's tools,
the tool runner asks the `mesh-controller` seat for its own verbs the same way (ADR 0154): measured
live, the mesh-controller runtime is one instance, mesh-wide, and it answers 63 verbs for that one
seat (`status`, `push`, `assign`, `build`, `artifacts` and the rest), and the `mesh-controller`
*module* answers none of its own — it serves no module tools, only its seat's, exactly as design 33
described while it was still only designed. The 56-to-93 seat-verb counts each node's tool runner
answers (below) are a different thing: the tool runner serves the verbs of whichever node-scoped
seats are held on that node (`node-backup`, `node-build-agent`, `node-service-manager` among them),
on behalf of the modules holding them, not the mesh-controller seat's — that one is served by the
mesh-controller runtime itself, a separate process, and the tool runner only asks it and relays what
it answers.
## Around it
## Where it runs
- **`invokes`** in a manifest is the grant. It is composed into the bus's user list exactly as a
person's account is; the mesh MCP server is the only module that declares it.
- **`module check <file|dir>…`** on the controller's binary judges a manifest with no mesh: the strict
parse, every per-manifest problem, and the rules between the manifests given. It prints what it cannot
judge without a store rather than refusing. The mesh MCP server's own manifest was the first thing checked
with it, and the whole catalogue passes.
- **The person's client remains.** `operator issue` and `mesh tools|call|mcp` with a credential file
still work, for a machine that is not a node and for a mesh not yet able to assign anything.
`mesh tools --console <url>` goes through a running mesh MCP server with no credential; it is covered by the
runtime repository's tests and was not exercised on the live mesh.
**Every node, not a per-operator assignment.** Measured live: all four of the mesh's machines
answer a `node-tools` runtime, each with its own module and tool count (between 294 and 312 tools,
56 to 93 seat verbs, in this reading). This is a fact that shipped past the design it came from: the
to-be document still describes it as assigned to "whichever machines an operator sits at"; what
actually runs is the opposite — every node the mesh holds carries one, because it is core
infrastructure the mesh itself depends on (discovery, the controller's own verbs, every module's
tools), not a convenience assigned to a workstation. There is accordingly no assign or unassign verb
for it: removing it from a node is not an operation the controller or the catalogue exposes.
## What shipped bent
## What this repository's prior text got wrong
- A module registered by hand from the catalogue with `--source <url> --path modules/<m>` records a URL,
not a place on the git seat: `--self` takes the forge path form (`<owner>/<repository>`), which the
operator did not pass. The rebuild-on-merge matched the URL anyway.
- Modules whose upgrade policy is *record* — the forge among them — answered `tools` only once
something pushed their rebuilt runtime; until then they are listed as not answering while still
callable. That is the policy doing what it says, not a fault of the mesh MCP server.
The document this replaces described `mesh-console` as a catalogue module served by a container and
assigned only to workstations. That module and its container are gone; the manifest is not — it
moved to `node-tools/module.json`, still declares `invokes: ["*"]`, and gained `waits` and the
`mcp-endpoint` provision, described above. What also moved is which language builds it: the
manifest's `build.artifacts` names a Go binary built from `cmd/node-tools`, and that is what a node
runs. The repository still carries the earlier TypeScript sources (`node-tools/src/mesh.ts`,
`http.ts`, `runtime.ts`, `announce.ts`, `broker-nats.ts`, `client.ts`, `launch.ts`, `main.ts`,
`mcp.ts`) on main, but the manifest does not build from them. The quirks the prior text recorded
from the `mesh-console` era (a module registered by hand recording its source as a URL; a
*record*-policy module staying listed as not-answering until its rebuilt runtime is pushed) are not
re-checked here: they were about the `mesh-console` container's registration path, which no longer
exists.
## How it is checked
- `mesh_overview` and `mesh_machine <node>`: the live tool and seat-verb counts, and which modules
are named not-answering.
- `mesh_runtimes`: which runtimes answered discovery, how long they took, how large their
announcement was, and whether it was shortened to fit the bus — read this when an address is said
to be missing or on another machine.
- `mesh_describe <address>` for one tool's schema; a call through `mesh_call` reaches the module or
seat over the bus and the answer is its own, unshaped.
- The code: `mesh-tools` (`node-tools/internal/console`, `internal/announce`, `internal/bus`),
`mesh-host` and `mesh-controller`'s core-component handling of `node-tools` (witness, lease,
bus-user naming). Their unit tests hold the loopback-only refusal, the seven-tool handshake, the
discovery gather window and `mesh_wait`'s event-then-read-state rule.
- `knowledge-base.get` with `kind: decision` and number 152, 175, 195, 197, 367 or 154.
@@ -0,0 +1,41 @@
---
layer: as-is
status: implemented
code: [mesh-controller, mesh-tools]
updated: 2026-10-16
decisions:
- "ADR 0106"
---
# Logins on the bus, as they run
**The bus reads one static list of logins from its settings file, in one account, written by the controller from
its own records. Every module has a login per assignment; a person's login may call only the tools it was given;
the sessions on the operator's machines call through their machine's tool-runner login, which may call every tool.
Nothing marks what a tool does.** Read from the controller's and the tool runner's code on main on 2026-10-15
(research 053, question 8). Design 58 replaces it.
## The list
- **One list, one account.** The controller writes every login from its records into the bus's settings file;
the bus re-reads it without dropping connections (ADR 0106 §3).
- **A module's login per assignment**, named by machine and module, may publish and subscribe only what its
manifest declares it emits, consumes, serves and calls.
- **A person's login** may publish only to the tools it was given and listen only on its own inbox.
- **The caller is in the address.** A call's address carries the caller's name, so the runtime takes the caller
from the address, never from the message. For a seat every machine holds, the address also names the machine.
## What it cannot do
- **No expiry.** A login lives until its record is deleted and the list rewritten on the bus's node.
- **No per-agent login.** Agent sessions reach the bus through each machine's tool-runner login, which may call
every tool on the mesh; what one session did cannot be told from another's.
- **No effect.** No tool or seat verb declares whether it reads, acts or is dangerous; some module documents say
it in prose, and nothing reads that.
- **Asks and warrants exist** (ADR 0259): a module can have one act authorised by the operator, bound to that
act; nothing yet asks for a dangerous call by itself.
## How it is checked
By reading: the controller's code that renders the bus's settings and the tool runner's caller check, on main.
When design 58's switch is done, this document is replaced by the as-is of signed logins.
@@ -0,0 +1,116 @@
---
layer: as-is
status: implemented
code: [mesh-controller, mesh-catalog modules/claude-code, mesh-catalog modules/gitea, mesh-catalog modules/ssh-client]
updated: 2026-10-17
decisions:
- "ADR 0266"
- "ADR 0399"
- "ADR 0409"
- "ADR 0410"
---
# Accounts on a machine, as they run
**A node record names at most two logins: the operator account, always, and an agent account, on one node of
four. Where a node names no agent account, agents run as the operator account. Nothing ties an identity to a
login, and a module that needs one declares it itself.** Read on 2026-10-17 from the controller's and the
catalogue's code on main (both local clones were a few commits behind the forge, so each claim below was
also checked against a live read) and from the live node records. The change planned in ADR 0411 to 0414
replaces this document when it is built.
## The node record
- **The operator account is stated at the controller's terminal** with `node account <node> <account> [home]`.
It is the person's login on that machine. The controller refuses to set it to the node's agent account.
- **The agent account is stated at the controller's terminal only** with `node agent-account <node> <account>
[home]`, and cleared with `--clear`. It is no verb and no setting, so an agent cannot name itself one
(ADR 0266 §1). The controller refuses root, a malformed login, a well-known service account and the
operator account.
- **Five machine facts come from the record.** `account` and `account-home` are the operator's, absent when
the node has no operator account. `agent-account` is the agent account, or the operator account where none is
named; `agent-home` is its home (`/home/<account>` when no home is stated); `agent-root` is `never` only where
the node names an agent account of its own, otherwise empty. Modules read them as `${machine:...}` words.
- **One node of four names an agent account today**, the control node. The workstation, the laptop and the
home server name none: `mesh-controller.node` says there "agents run as the operator account; no agent
account is named", and the plan for the workstation declares the `claude-code` account resource with the
operator's name and no `root` field. The tool runner also receives `MESH_AGENT_ACCOUNT` and `MESH_AGENT_HOME`.
- **Not verified:** whether the mesh-config snapshot carries either account. The controller's code reads both
from its own node table (columns for account, account home, agent account and agent home); no second place
that states them was found.
## What claude-code does with the agent account
- **It declares a `user` resource** named `${machine:agent-account}` with home `${machine:agent-home}` and
`root: ${machine:agent-root}`, and a `~/.claude` directory under that home owned by it, mode 0700. On the
control node the plan holds the resource with `root: never`; elsewhere it holds the operator's account.
Where the account is new, the mesh creates it with a home, its own group and no password (ADR 0266 §2).
- **`claude-agent`** is a file the module writes at `/usr/local/bin/claude-agent`. Run as the agent account
it starts `claude`; run as any other account it runs `sudo -iu <agent account> claude` through that
account's own sudo, carrying the operator's guard override only if it is set in that shell. The file
`/etc/claude-code/agent-account` holds the account's name.
- **The guard of ADR 0266 §28** is the module's hook on every tool call. On a node with an agent account
of its own it refuses each call of a session that runs as another account, with no override, and tells the
session to start with `claude-agent`. Its record goes to the machine's journal. The same guard also refuses
ssh to a mesh machine and writes to `/etc/hosts` and the resolver file (ADR 0245).
- **Not verified live:** this rests on the guard's code and README, not on a refusal seen on a running machine.
## The judgement that the agent account cannot become root
- **The node-engine judges `root: never` on every look**, by reading only: uid 0, a group that grants root,
a sudo, doas or polkit rule, a container runtime's socket or API port, a secret the mesh placed for another
account, a setuid program no package owns, and kernel link protection off (ADR 0266 §12 to §16). A way found,
or a question unanswered, makes the account resource unhealthy. It does not prevent any way to root.
- **The controller's self-check holds** that a node naming an agent account has it judged, in a verdict heard
within 15 minutes, as unable to become root (ADR 0266 §21 to §22). Otherwise it raises the urgent condition
`agent-can-become-root`. `mesh-controller.node` shows the verdict: for the control node it reads "cannot
become root without a person", judged the same day.
- **On the three nodes without an agent account** the check does not apply. The controller instead holds an
urgent condition `root-not-free` on each machine that runs reads or the mail notifier. Two of the three are
silenced until 2026-10-23 on the operator's decision; the home server's is open and not silenced.
## The accounts agents push as on the forge
- **Today the forge has a user per machine**, made by the gitea module's `forge-account` provision and
holding that machine's key, plus a write token the module still mints for each (ADR 0398, ADR 0400). ADR 0410
decides one forge user, `nox-agent`, in the agents' write team, each machine's key titled by machine. No tool
reads the forge's user list, so whether `nox-agent` exists yet is **not verified**.
- **The forge key is meant for the agent account** (ADR 0409 §2), but a module cannot run as that account:
the controller refuses it (issue 989, open). So the `ssh-client` module's key keeper still runs as the
operator account.
- **An ordering deadlock** between `ssh-client`, which wants `forge-account`, and `gitea`, which provides it,
stopped every node's declaration from composing on 2026-10-17 at 00:35 until gitea was registered three
minutes later (issue 986). The four urgent conditions it raised were not observed again but stay raised,
because the controller's declaration check times out and never clears them (issue 992).
Git over ssh was proven from the laptop only; the proof on the other three, and retiring the forge token the
module still places, wait on issues 989 and 991 (the latter in `fixing`).
- **Main stays closed**: a push to the main branch is refused, and a merge waits for a person (ADR 0398).
## Logins on the bus
Not here: see [`16-logins-on-the-bus.md`](16-logins-on-the-bus.md).
## What it cannot do
- **No identity carries a login.** An agent, a person and a module are not tied to an account on a machine;
the node record names a login per node and nothing else.
- **A module declares a login itself.** Each module that needs an account writes a `user` resource of its own,
and nothing checks that two modules agree or that a login exists on every machine that needs it.
- **Agent and operator share a login on three of four machines.** There the agent runs as the operator account,
which may become root; the guard's account rule does not apply, and the exposure is a held condition, not a fix.
- **No per-agent bus password.** Every session on a machine calls through that machine's tool-runner login
(document 16), so one agent's calls cannot be told from another's.
- **Detects, does not prevent.** The judgement above lists the ways to root it looks for; a way it does not list
is not seen (ADR 0266 §17).
## How it is checked
- `mesh-controller.node` with a node name: "agents run as" line, the verdict and when it was judged, the open
conditions (`agent-can-become-root`, `root-not-free`, `uncomposable`).
- `mesh-controller.plan` with a node name: the `claude-code.agent-account` resource and its `root` field; with
`files` set, the files given, including `claude-agent` and `agent-account`.
- `mesh-controller.conditions` for the held conditions; `claude_code_guard` for the guard's record, `claude-code`
tools for its status.
- `knowledge-base.get` with `kind` decision and number 266, 409 or 410; `issue-tracker.get` with 989 or 991.
- The code: the controller's node-record and agent-account files, and the `claude-code` module's manifest and
guard in the catalogue. Their unit tests hold the node record's refusals and the guard's account rule.
+2
View File
@@ -24,6 +24,8 @@ Where the two disagree, the implementation wins and the disagreement is stated.
| [`13-the-console.md`](13-the-console.md) | The mesh's tools on the machine a person sits at, served by a module the mesh assigned there |
| [`14-a-modules-state.md`](14-a-modules-state.md) | A module's current state on the bus: what the controller creates, the runtime serves, and the first live use showed |
| [`15-the-agent-and-its-licences.md`](15-the-agent-and-its-licences.md) | The operator's agent on every machine and the licence manager that keeps its licences |
| [`16-logins-on-the-bus.md`](16-logins-on-the-bus.md) | The bus's one list of logins, a login per module assignment, persons limited to their tools, sessions calling through the tool runner's login, and no effect on any tool |
| [`17-accounts-on-a-machine.md`](17-accounts-on-a-machine.md) | The operator and agent accounts on the node record, claude-code's own login and guard, the judgement that the agent account cannot become root, and the forge accounts agents push as |
## What these documents are not
+1 -1
View File
@@ -340,7 +340,7 @@ once, and maintain for a year. The modules are the input; a person reads what on
writes what it declares tomorrow.
**It also changes what "safe" means for the system being retired.** A fix to it has to be safe on
its own, because there is no careful walk to sequence it into: the thing is being switched off
its own, because there is no careful rollout to sequence it into: the thing is being switched off
by hand, not managed into retirement. A change needing three steps in the right order is a change
that will be half-applied.
+110 -1
View File
@@ -22,8 +22,10 @@ code:
- mesh-host internal/apply (the service that reflects a rule set; a whole file handed to its new owner)
- mesh-controller cmd/mesh-controller/nodes.go (a token issued for a machine's tunnel key, ADR 0169)
- mesh-host cmd/mesh-host/join_tunnel.go (a machine joins through the tunnel, ADR 0169)
updated: 2026-10-13
updated: 2026-10-14
decisions:
- "ADR 0241"
- "ADR 0117"
- "ADR 0340"
- "ADR 0247"
- "ADR 0226"
@@ -1119,6 +1121,113 @@ against today's behaviour, where both are always composed; an unrouted endpoint
named; an assignment naming an endpoint the module does not declare refused where it is said; and a
silent assignment rendering byte-identically to today.
## The five through the network layers
*2026-10-09. A reading aid: it decides nothing and states no rule. Each claim names the section or the
record that decided it.*
The OSI model is the standard seven-layer picture of a network: every layer wraps what the layer above
hands it, and the receiving side unwraps in reverse. The five contexts of this document act at
different layers, and seeing which explains why they fail independently. A resolver that stopped
answering leaves the private network working, and a tunnel that cannot handshake leaves every name
resolving to an address nothing can reach.
| Layer | What it carries | Where the mesh acts there |
|---|---|---|
| 7 Application | the message itself | the bus (ADR 0106); resolution (§2); exposure, where the proxy chooses a route by the name a request asks for (§3) |
| 6 Presentation | how what is said is encoded and encrypted | certificates (§5): TLS, which is usually drawn between 4 and 7 and placed here by the book |
| 5 Session | a conversation kept open | a node's link to the bus, opened once and reopened after the machine sleeps |
| 4 Transport | ports, TCP and UDP | filtering (§4), which opens a port to a source; the private network itself travels as UDP |
| 3 Network | addresses and the path between them | the private network (§1): every node's address, the peer graph, the hub relaying |
| 2 Data link | frames on one network segment | the machine's own uplink (ADR 0117), on the underlay |
| 1 Physical | cable, fibre, radio | nothing of the mesh's |
### Two networks, one inside the other
**The private network is a layer 3 network carried at layer 4 of the underlay.** A packet for another
node's private address is encrypted whole and becomes the payload of a UDP datagram addressed to a
peer's underlay address. Every hop on the underlay reads only the outer envelope, up to its layer 3,
and never sees inside.
```
underlay │ L1 radio │ L2 frame │ L3 to the hub's public address │ L4 UDP │ ▓▓ encrypted ▓▓ │
│
decrypted at the hub │
┌────────────────────────────────────────────────┘
▼
private network │ L3 to a private address │ L4 TCP │ L5–L7 a message on the bus │
```
### The shape the layers are drawn on
The shape is §1's: a hub, direct peering between nodes at the same site, and a roaming node that goes
through the hub only. In a small mesh one node is both the anchor and the hub, and holds the bus.
```
┌──────────────────────────────────────────┐
│ hub and anchor — reachable from outside │
│ holds the bus │
└─────▲──────────────────▲─────────────▲───┘
│ │ │
│ each tunnel is dialled out by the node below it
│ │ │
┌─────┴─────┐ ┌──────┴──────┐ ┌────┴────────┐
│ laptop │ │ home-server │ │ home-server │
│ no site: │ └──────▲──────┘ └──────▲──────┘
│ hub only │ └──── direct ───┘
└───────────┘ same site, peered
```
**Every arrow points at the hub.** A node with no public address still has a tunnel, because it dials
out and the hub answers; only the hub's tunnel must be dialable, at a stable address
(ADR 0169).
### One tool call, from the laptop to a home-server
The laptop wraps the call one layer at a time:
| Layer | On the laptop |
|---|---|
| 7 | a message on the bus naming the tool and its arguments |
| 6 | the message encoded for the wire |
| 5 | the laptop's link to the bus, already open |
| 4 | TCP to the bus at its private address |
| 3 | a packet from the laptop's private address to the bus's |
| 3 inside 4 | the tunnel: that packet encrypted, in a UDP datagram to the hub's public address |
| 2 | a frame to the gateway of whatever network the laptop is on |
| 1 | radio |
**At the hub the outer envelope comes off** — frame, packet, datagram — and the tunnel decrypts what was
inside. The inner packet is addressed to the bus, which is on this node, so the hub keeps unwrapping up
to layer 7. The bus finds the home-server's subscription and sends the message down the home-server's
own link, which that node opened earlier through its own tunnel. Neither spoke ever accepted a
connection; both dialled the hub.
**How far the hub unwraps depends on what the packet is for.** A packet addressed to a home-server's
private address, not to the bus, is opened only to layer 3: the hub relays it, encrypted again for the
home-server's tunnel. That is the whole of the glossary's *a hub relays for nodes that cannot reach
each other directly*. Two home-servers at one site skip the hub: their tunnel crosses the home network
at layer 2, and nothing of theirs reaches the hub at all.
**Why a roaming node has one path is a layer 3 fact.** A more specific route to an endpoint that is
gone takes the packets and drops them; it does not fall back (§1). A laptop that moves between networks
therefore has exactly one route, the hub's.
### Reading a failure by layer
The machine's own network checks
(ADR 0241)
each look at one layer:
| Check | Layer | Failing alone means |
|---|---|---|
| **route** — a default route | 3, underlay | the machine has no way out; nothing above can work |
| **tunnel** — a handshake with the hub within five minutes | 3 carried at 4 | the underlay works and the private network does not: a key, the hub's address, or a filter on the way |
| **bus** — the link is open | 5 to 7 | the private network works and the bus, or the node's account on it, does not |
| **resolv-conf** and **names** | 7 | addresses work and names do not, as when a VPN client rewrites the machine's resolver file |
Read them from the bottom. A layer that fails explains every failure above it, and never one below it.
## What this removes
The list is worth having in one place, because it is most of the argument:
+3 -3
View File
@@ -54,7 +54,7 @@ rather than two kinds of thing
node** — it is a machine with a program on it that has not been told which mesh it belongs to.
There is no state for *the first node*. That is the point of
ADR 0004: the first node walks the
ADR 0004: the first node rollouts the
same path, in an unusual order.
---
@@ -239,7 +239,7 @@ carries, to the machine it is already on.
**After step 3 the first node is not special in any way**, which is the property `adopt.sh` and
the bootstrap script never had. Its specialness lasted two commands.
**And enrolment is exercised on node one.** The path every other node depends on is walked
**And enrolment is tried on node one.** The path every other node depends on is exercised
immediately, against a controller on the same machine, rather than being written and first
used months later on node two.
@@ -524,7 +524,7 @@ remains locally authoritative for *operating*; the copy exists only for this.
The node-engine is delivered like anything else
(ADR 0010), and this is worth
walking through because tier 0 looks like it should be special and is not.
going through step by step because tier 0 looks like it should be special and is not.
```
push to mesh-host
+5 -5
View File
@@ -277,15 +277,15 @@ A transition the table does not hold is refused. Two or more deliveries that sha
together as one future state of the mesh.
**This answers the first open question above.** A fit artifact does not declare itself. Off the trunk it is
only checked. On the trunk it is published, and its walk across the machines starts when its delivery says
only checked. On the trunk it is published, and its rollout across the machines starts when its delivery says
so: by itself for the core, which cannot wait for anything to be repaired, and by `mesh-delivery` for
everything else while that module is held. A person can always deliver by hand.
The controller keeps the comparison, the planner, the gate, sending and rollback. `mesh-delivery` keeps the
record and the order, and asks.
**Merges are walked in batches** (2026-10-10, by ADR 0276). The controller collects
**Merges are rolled out in batches** (2026-10-10, by ADR 0276). The controller collects
merges in a rolling merge window, 90 seconds by default and at most 10 minutes, and plans each batch once, as
one walk with at most one commit per repository: the latest of its branch, which contains the earlier ones.
One walk is open at a time, and a walk that started is never superseded. `plans` shows a batch while it
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's walk.
one rollout with at most one commit per repository: the latest of its branch, which contains the earlier ones.
One rollout is open at a time, and a rollout that started is never superseded. `plans` shows a batch while it
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's rollout.
+7 -4
View File
@@ -1,12 +1,13 @@
---
layer: to-be
status: proposed
status: in-progress
code:
- mesh-controller cmd/mesh-builder
- mesh-controller internal/builder
- mesh-catalog modules/build-agent
updated: 2026-10-08
- mesh-controller modules/build-agent
updated: 2026-10-17
decisions:
- "ADR 0405"
- "ADR 0262"
- "ADR 0233"
- "ADR 0219"
@@ -30,6 +31,8 @@ decisions:
# Building a module
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: mesh-controller's builder (internal/builder, cmd/mesh-builder) implements the recipe kinds described here, and the sections marked built and proven live match the code. Left: not broken down into milestones; issue 1072 found it saying proposed.
[`12-a-module-repository`](12-a-module-repository.md) says what a module may build — an image, an
archive, an upstream mirror — and where the result goes. This says **how a build is modelled**, and
why the current model does not fit what a module is.
@@ -138,7 +141,7 @@ need to run. Third-party software still arrives as an image, because that is how
## What this costs, stated before it is chosen
**Every language is permanent.** It needs an SDK — broker client, sealed-credential reading, the
event envelope, tool serving — a toolchain image, and a bundler the builder understands. And the
event envelope, tool serving — a build module that installs its toolchain on every builder (ADR 0405), and a bundler the builder understands. And the
spine changes rarely but cascades when it does
(ADR 0039): with one language a contract
change is one edit; with four it is four that must land together, and a mesh whose SDKs disagree
+4 -2
View File
@@ -1,13 +1,13 @@
---
layer: to-be
status: proposed
status: in-progress
code:
- mesh-sdk src
- mesh-tools src/broker-nats.ts (and broker-amqp.ts until the rollout)
- mesh-controller internal/link
- mesh-controller cmd/mesh-controller (status)
- mesh-catalog modules/postgres, modules/keycloak (the Go provisioner loop)
updated: 2026-10-06
updated: 2026-10-17
decisions:
- "ADR 0224"
- "ADR 0230"
@@ -24,6 +24,8 @@ decisions:
# The module protocol
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: the wire this design describes is in mesh-sdk and mesh-controller's internal/link, and the design's own checks against the code (2026-09-16) agree. Left: not broken down into milestones; issue 1072 found it saying proposed.
**What a module's code and the mesh say to each other.** An SDK is an implementation of this in one
language and nothing more (ADR 0074).
+1 -1
View File
@@ -215,7 +215,7 @@ and recreate a container naming a step after the step ran.
*Written 2026-10-02, from ADR 0189
and [issue 108](../../04-ISSUES/108-the-registry-has-no-garbage-collection-once-it-has-two-doors/00-report.md).*
Some work cannot be done underneath a running service: an artifact store's collector walks the
Some work cannot be done underneath a running service: an artifact store's collector rollouts the
storage and requires every writer stopped. A `run-once` step runs *beside* containers and a
scheduled one is the same container again, so until this a module had no way to say it — and the
mesh inherited a store that has never collected anything, because the predecessor said it with a
@@ -1,11 +1,11 @@
---
layer: to-be
status: proposed
status: in-progress
code:
- mesh-host internal/bootstrap
- mesh-host cmd/mesh-bootstrap
- mesh-lab test/integration/one-node-mesh.test.ts
updated: 2026-10-06
updated: 2026-10-17
decisions:
- "ADR 0226"
- "ADR 0067"
@@ -16,6 +16,8 @@ decisions:
# The installation, in full
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: the node-engine's bootstrap (`mesh-host` internal/bootstrap) carries phases 2 and 3 of the installation, so the steps this text calls manual are partly automated. Left: not broken down into milestones; the text is not yet corrected against the code.
Every step from a machine with nothing to a mesh that maintains itself. Written out explicitly
because it is the procedure everything else depends on, and because the parts that do not exist
yet are easier to see beside the parts that do.
+3 -1
View File
@@ -6,7 +6,7 @@ code:
- mesh-controller
- mesh-catalog
- mesh-lab
updated: 2026-09-15
updated: 2026-10-17
decisions:
- "ADR 0067"
- "ADR 0074"
@@ -17,6 +17,8 @@ decisions:
# The work ahead
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Phase 2's blocker (issue 053) is resolved and delivered, and the forge's module declares the package registry in full, Phase 2's done-when. Whether every phase holds is checked under issue 1073, which marks it implemented or names what is left.
Everything decided this cycle and not yet built, in the order its dependencies allow. Each phase
ends at something provable on a running mesh, because a phase that ends at a claim is a phase that
went missing without anything complaining.
+5 -3
View File
@@ -1,8 +1,8 @@
---
layer: to-be
status: designed
code: []
updated: 2026-09-25
status: in-progress
code: [mesh-controller]
updated: 2026-10-17
decisions:
- "ADR 0084"
- "ADR 0027"
@@ -11,6 +11,8 @@ decisions:
# 23 — Choosing a provider
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: mesh-controller's internal/catalogue/resolve.go implements the model described here, and designs 26, 27 and 29 rely on it. Not yet checked against this text clause by clause, nor given an as-is: issue 1074.
A provision is named for what the consumer's code is coupled to
(ADR 0027):
`postgres-database`, not `database`. That decides *what kind* of provider satisfies a requirement.
+1 -1
View File
@@ -163,7 +163,7 @@ again restarts nothing. The provider asks when it starts, when the software's va
consumer that binds in between holds a value the mesh made, which the software refuses, for at most the hour until the next ask. A change is recorded in the
hand-act log as a decision, not a repair, and `rotate` of such a provision is refused. *How it is checked:*
the tests ADR 0305 §13 names, in both repositories; the live row is `hand-acts` showing `secret issued` for
the media server's provision once both walks are done.
the media server's provision once both rollouts are done.
## Beyond generate and hold
+1 -1
View File
@@ -75,7 +75,7 @@ and uses, and the subjects, streams, consumers and permissions all follow from t
(ADR 0039).
**Added 2026-10-10** (ADR 0289): the controller's own records join
the bus as **record mirrors** — nodes, assignments, walks, builds and settings, each a bucket written from
the bus as **record mirrors** — nodes, assignments, rollouts, builds and settings, each a bucket written from
the controller's store only when the record changed, with one event per write, and audited against the
store on every self-check run. Designed in [to-be 54](54-the-controllers-records-on-the-bus.md). The
rule it states for every bucket, that a write is made only when what the bucket describes changed, holds
+1 -1
View File
@@ -709,7 +709,7 @@ healthy while reacting to nothing.
Found on the way: **no build had ever recorded what it stood on.** A recipe reads its base from
a build argument, so the digest was never in the file the builder derived edges from, and every
order that says *bases first* — `build --on`, `build --behind`, the merge follow-up of
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — walked a
[issue 131](../../04-ISSUES/131-nothing-tells-the-mesh-a-source-moved/00-report.md) — traversed a
graph with no edges. The builder now reports the bases it was handed, the controller records
them by artifact path, and the graph is read from the newest build of each module — a recorded
manifest carries no `build.on`, so the edge is derived from the build or it does not exist.
@@ -5,7 +5,7 @@ code:
- mesh-controller internal/inventory
- mesh-controller internal/catalogue
- mesh-controller cmd/mesh-controller
updated: 2026-10-02
updated: 2026-10-17
decisions:
- "ADR 0181"
- "ADR 0182"
@@ -14,6 +14,13 @@ decisions:
- "ADR 0051"
- "ADR 0113"
- "ADR 0117"
- "ADR 0399"
- "ADR 0400"
- "ADR 0412"
- "ADR 0415"
- "ADR 0416"
- "ADR 0417"
- "ADR 0422"
---
# 29 — A node has operator accounts, and the mesh owns what lives under a home
@@ -56,6 +63,15 @@ These are a **family**, not one module: an `ssh-client` module, a shell module,
module, each a *universal-tier* consumer of the account fact — assigned wherever a person logs in,
which is every node, unlike the graphical stack that a capability gates.
**A folder a module owns says how it treats a file it did not write** (ADR 0416, ADR 0422). Declaring a
directory under a home owns it (ADR 0182 §1), and the owner alone writes there and reacts to every file
that enters it, never losing what the file held. The manifest says how, on the directory itself:
`unknown` is `move-out` (the node-engine moves every path the manifest does not declare and `knows` does
not name into `~/.local/state/mesh-moved/<module>/`, with a list), `module` (the module's own process
reacts, as ssh-client does in `~/.ssh`) or `program` (everything in it is the program's, as in a
service's data directory). `move-out` is only for a folder under a home, the home of the account the
assignment runs for (ADR 0412 §5). A folder many modules write in is not declared at all.
## 3. The whole of `~/.ssh` is the mesh's — with one boundary drawn inside it
The predecessor owned a single file (`~/.ssh/config`) and left the rest alone; it drifted, because
@@ -82,6 +98,9 @@ ADR 0117 draws for the uplink and the sshd
module draws for the firewall: **the mesh must never be able to arrange the one failure that severs
its own way back in.** The carve-out is not a convenience; it is that rule, in `~/.ssh`.
Since ADR 0415 and ADR 0417 the found class no longer holds in `~/.ssh`: ssh-client moves out what it
does not know into a kept folder, and never moves a key or `authorized_keys`, which keeps this boundary.
## 4. Keys are the mesh's to generate — through a CA, and existing keys are adopted, not replaced
Key *generation* is the mesh's, not each node's improvising its own. The clean form is an **SSH
@@ -220,6 +239,17 @@ each need a decision before the modules that replace the generators can be writt
The modules §2 calls *a family* — the shell, the terminal, the desktop, everything under a home that is not `~/.ssh` — are designed in [37 — The operator's machine](37-the-operators-machine.md), under ADR 0173 to ADR 0177. This document keeps `~/.ssh`, the CA and the roster files. Two things it listed as not built are decided there: user-scoped services (ADR 0177) and the one-off steps a hook used to run (declared state, or a seat's verb).
## The forge over SSH — 2026-10-16
`~/.ssh` gains the account's own key for the forge. ADR 0399 decides that each machine's account has an SSH
key made on the machine, that the forge registers its public half on the machine's agent user, and that
`ssh-client` writes the forge's SSH key into `known_hosts`, ssh's configuration entry for the forge and git's
rule that rewrites the forge's `https://` address to its SSH address. ADR 0400 decides how the public half
travels: `ssh-client` writes it to its own state, keyed by node and account, and the forge's provider reads it
there, since a manifest's contribution cannot carry a value made at run time. The forge's SSH address is its
machine's name on the mesh, where its SSH port is open; the key's file is `~/.ssh/mesh_ed25519`, beside the
operator's own keys, which stay theirs (§3). Built under issue 955.
## References
- The gap was found generating `~/.ssh/config` from the *HAL* registry (`hal/terminal`'s
@@ -236,3 +266,4 @@ The modules §2 calls *a family* — the shell, the terminal, the desktop, every
- ADR 0117,
ADR 0118 —
the never-sever-the-channel rule and the found-vs-owned semantics, applied here to `~/.ssh`.
- ADR 0399, ADR 0400 — git over SSH with a key made on each machine, and how its public half reaches the forge.
@@ -1,8 +1,8 @@
---
layer: to-be
status: proposed
code: []
updated: 2026-10-06
status: in-progress
code: [mesh-controller]
updated: 2026-10-17
decisions:
- "ADR 0238"
- "ADR 0221"
@@ -15,6 +15,8 @@ decisions:
# 30 — The mesh updates itself on a push
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: mesh-controller's delivery plans, merge gate and merge window (plan.go, release_plan.go, merge_gate.go, merge_window_bus.go, changeplan.go, with their migrations), which design 47 builds on. Not yet checked against this text nor given an as-is: issue 1075.
**Today the mesh does not update itself; a person drives the pipeline by hand, and one class of
change freezes it.** A code change lands in `mesh-controller` or `mesh-catalog`, and getting it onto
the machines is a sequence somebody types. The predecessor's pipelines rebuilt and redeployed on a
@@ -141,7 +143,7 @@ Revision, ADR 0218. Three rules on how a plan delivers what it built.
A membership that could not be issued fails the send.
- **One machine first.** Unless a module's upgrade policy says *together*, a plan sends it to one machine,
the first by name, and to the rest only once that machine reports the new declaration applied and
current. A first machine that fails stops the module's walk there, with the reason in the plan.
current. A first machine that fails stops the module's rollout there, with the reason in the plan.
- **A newer merge takes over.** A merge's plan supersedes every older open plan for the same repository
and branch, and takes in the modules they had not yet built. A plan that waits on nothing can be closed
by hand, by its id.
@@ -182,7 +184,7 @@ policy or a plan is holding back
builds are not known is left too. The push names the machine, the module, the builds and the reason,
and says `push <node>` sends it.
- The machine a push names, a push of every machine, and `push --behind` send held builds as before.
Walking a change through the mesh under `record` is one `push <node>` per machine.
Rolling out a change through the mesh under `record` is one `push <node>` per machine.
## Why now, and why not yet
@@ -253,7 +253,7 @@ the controller creates from the catalogue, mesh-wide, existing from registration
watch before the owner runs anywhere ([design 25](25-the-bus-on-nats.md) §3). Every instance of the
owner writes; a reader reads and watches. A key may name a machine by the module's own convention;
the mesh keeps one bucket per name, not one per machine, because "every server, for every machine"
is then one list rather than a walk.
is then one list rather than a rollout.
What a module sees is what it named. Its assignment's membership lists its buckets by those names,
with whether it may write, and the runtime answers `get`, `put`, `delete`, `keys` and `watch` for
@@ -338,7 +338,7 @@ module, version, digest. An event again: a fact about the builder.
**The build cascade is that event fanning out through a graph the mesh already has.** A module
whose image is built *on* another's artifact declares that in `build.on`. So `built` reaches the
controller, which walks the declared graph and submits rebuild jobs for everything downstream. A
controller, which traverses the declared graph and submits rebuild jobs for everything downstream. A
dependency cascade is not special machinery — it is one event, one derived graph, and the same job
queue.
@@ -351,7 +351,7 @@ entrypoint that brings its state to the shape that version needs — the same vo
tools and its provisioner, and nothing about how a machine runs it. The mesh runs that entrypoint as it
runs the module's own code, to completion, in the module's own context, and a version whose preparation
did not succeed does not run: the step gates that module and nothing else on the machine
(ADR 0136), and the walk stops
(ADR 0136), and the rollout stops
at the first machine that did not take it
(ADR 0135, superseding
ADR 0133).
+15 -8
View File
@@ -1,11 +1,13 @@
---
layer: to-be
status: designed
code: [mesh-catalog, mesh-tools, mesh-controller]
updated: 2026-10-03
status: implemented
code: [mesh-tools, mesh-host, mesh-controller]
updated: 2026-10-17
decisions:
- "ADR 0197"
- "ADR 0195"
- "ADR 0175"
- "ADR 0367"
- "ADR 0160"
- "ADR 0159"
- "ADR 0152"
@@ -163,11 +165,16 @@ matched it by URL.
## What this does not settle
- Narrowing a mesh MCP server's grant per assignment. ADR 0046 makes it a setting; nothing reads one yet.
- The mesh's own verbs on the bus. Design 33's third family; this document only says where they appear
once they exist.
- A person's identity behind the mesh MCP server. The mesh sees the mesh MCP server's account; design 15 keeps the
question open.
- **The mesh's own verbs on the bus.** Design 33's third family: these now answer directly, through
the `mesh-controller` seat, under ADR 0154 — settled, not open. [As-is 13](../00-as-is/13-the-console.md)
records it live.
- **Narrowing the tool runner's grant per assignment.** ADR 0046 makes it a setting; nothing reads
one yet. Owned by [design 58](58-agent-work-as-the-meshs-own.md) milestone 6 (issue 883): an
identity asked for each assignment is one of that milestone's parts.
- **A person's identity behind the mesh MCP server.** The mesh still sees only the tool runner's own
account, never the caller's; design 15 raised the question and leaves it open. Owned by the same
milestone (design 58 milestone 6, issue 883): its outcome is that every person, agent and module
becomes an identity with a role, and a call carries which one made it.
## References
@@ -2,8 +2,10 @@
layer: to-be
status: in-progress
code: [mesh-catalog modules/claude-code]
updated: 2026-10-13
updated: 2026-10-17
decisions:
- "ADR 0411"
- "ADR 0412"
- "ADR 0343"
- "ADR 0274"
- "ADR 0273"
@@ -306,21 +308,27 @@ operator's word — and, for the operator's own home, only at the terminal or on
## 9. The account the agent runs as
*Added 2026-10-08 by ADR 0266.*
Everywhere above, "the operator's home" and "the operator account" mean **the account the agent runs as**.
That is the operator account, unless the node names an **agent account** of its own. The control node does.
*Added 2026-10-08 by ADR 0266; changed 2026-10-17 by ADR 0411 and ADR 0412.*
Everywhere above, "the operator's home" and "the operator account" mean **the account the agent runs as**:
the machine account of the identity the module runs for (ADR 0412). claude-code declares that it can run for
an agent identity, in its assignment or when a session is started: started for `nox` it runs with the agent's
home; started for the operator, it is the default run, with the operator's configuration. On a
machine where no agent is marked as the agent account, its assignment is the default run, as every module's is.
**The fact.** The operator states the agent account at the controller's terminal. No verb and no setting
writes it. The controller resolves the machine facts `agent-account`, `agent-home` and `agent-root`, and gives
the tool runner the account and its home. Where the node names none, the facts name the operator account and
`agent-root` is empty, so nothing changes there.
**The fact.** The machine accounts come from the identity seat (ADR 0411); the operator no longer states the
agent account at the controller's terminal. A machine's agent account is the machine account of the agent the
identity seat marks as the agent account there, on the operator's Approve: `nox` on the control node. There this module's
assignment is run for that agent, and the controller refuses one that is not. The controller resolves the
machine facts `agent-account`, `agent-home` and `agent-root` from it, and gives the tool runner the account and
its home. Where no agent is marked (every other machine today), the facts name the operator account and `agent-root`
is empty, so nothing changes there, though a login `nox` exists for the modules run for it.
**What the module declares.**
- A `user` named by `agent-account`, with `root` set from `agent-root`. Where the account is the agent's own,
the mesh creates it (home, own group, no password) and the node-engine judges on every look that it cannot
become root without a person, by the ways ADR 0266 §3 lists. It detects those ways and no others; it
prevents nothing.
- No login. The mesh makes the agent's machine account (home, own group, no password) from the identity, and the
node-engine judges on every look that it cannot become root without a person, by the ways ADR 0266 §12 to §17 list.
It detects those ways and no others; it prevents nothing. The `user` resource named by `agent-account`,
which this module declared until ADR 0411, goes.
- `~/.claude` under that account's home, owned by it.
- `claude-agent`, a command that starts the agent as that account through the operator's own sudo, or
directly when already run as it.
@@ -2,8 +2,9 @@
layer: to-be
status: in-progress
code: [mesh-host, mesh-controller, mesh-catalog]
updated: 2026-10-12
updated: 2026-10-16
decisions:
- "ADR 0394"
- "ADR 0310"
- "ADR 0268"
- "ADR 0252"
@@ -90,6 +91,7 @@ WP5 assign and prove (operator-gated) needs WP
WP6 a module puts the account in a group (mesh-host, mesh-controller, mesh-catalog) issue 247
WP7 the control-node withholds execute (mesh-controller, mesh-catalog) ADR 0268
WP8 a node's files and accounts are read (mesh-sdk, mesh-catalog, mesh-controller) ADR 0310, issue 418
WP9 every file on a node is managed (mesh-catalog, mesh-controller) ADR 0394, issue 900
```
WP1, WP2 and WP4 are independent, and are built in parallel on one feature branch per repository
@@ -341,6 +343,10 @@ out, the control-node's value is set to `withhold` and zsh is pushed to it. Then
*`mesh-sdk`, `mesh-catalog`, `mesh-controller`. ADR 0310, extending ADR 0268; tracked by issue 418.*
**Replaced by WP9.** ADR 0394 supersedes ADR 0310: `node-files` takes over every read below, with the same
bounds, content rule and refusal of secret places; once `node-files` is held on every node, the next delivery
removes `node-reads` and `mesh-reads`.
ADR 0268 §5 has narrow read tools take the common reads over from `execute`, so that an approval is asked for an
act and not for a read, and ADR 0286 §44 sets the approval mark on `execute` only once those reads have a tool on
every node. Two had none: reading or searching a file, and looking up an account or a group.
@@ -366,3 +372,38 @@ under the same issue.
`mesh-sdk` test of the content rule and one in each of its users; `mesh_search` for `cat`, `grep` and
`getent passwd` finding the verbs. Live, the kernel's RAID status file is read through the seat on the home
server, and the agent's credentials file on the laptop is refused.
## WP9 — Every file on a node is managed through `node-files`
*`mesh-catalog`, `mesh-controller`. ADR 0394; tracked by issue 900.*
`node-reads` only reads, and as an account that cannot open a person's home, so nothing in the mesh could copy,
move, write or delete a file, and a cleanup needed a shell on the machine. Most files belong to no module: homes,
project folders, downloads, logs, scratch.
- **A node seat, `node-files`, on every node**, held by a module of its own, `mesh-files`, beside `mesh-reads`
in the catalogue; the controller holds the seat's definition beside the other node seats. It has a verb for each file action: list,
stat, readlink, read, search, write, copy, move, delete, make a folder, change mode, change owner, account and group
lookup, and the trash's list and restore. Each says what it replaces (ADR 0245): `ls`, `stat`, `readlink`, `cat`,
`grep`, `cp`, `mv`, `rm`, `mkdir`, `chmod`, `chown`, `getent passwd`, `getent group`, `id`, and `ssh <node> cp` for a
copy between nodes.
- **It may touch every file.** It declares no account of its own: the tool runner starts it as the node's
operator account, which has sudo without a password, and it uses `sudo -n` only where that account's own rights
do not reach. It keeps each file's owner and modes as well as it can: a copy or a move keeps them, a new file
takes its folder's owner. Which files to touch is the caller's responsibility; it judges no permission.
- **Nothing is lost at once.** A file deleted or overwritten goes to the node's trash with who asked and why, is
restorable for 7 days, and is emptied after.
- **A path a module owns is refused**, naming the module and its tool, unless the call says `override` with a
reason; an override is recorded and said to the operator.
- **It replaces `node-reads`.** Its reads keep WP8's bounds, content rule and kernel reads. Every node is
assigned `mesh-reads`, so `node-reads` and `mesh-reads` are removed by the next delivery, once `mesh-files`
holds `node-files` on every node.
- **No Approve is asked**; every act is recorded with its caller. Read and search never return the content
of a known credential file or a secret place (the mesh never prints a token); copying or moving one is
allowed.
**Proof.** The module's tests: each verb against a temporary tree keeps owner and modes, a new file takes its
folder's owner, a delete is restorable from the trash and gone after 7 days, an owned path is refused and an
override recorded, a credential file's content is never returned. Live: the operator's `~/.claude` on a workstation is listed, a file in `/etc` is read through sudo, and a file copied from one node to another
arrives with its owner and modes.
@@ -2,8 +2,9 @@
layer: to-be
status: in-progress
code: [mesh-catalog, mesh-controller, mesh-host]
updated: 2026-10-12
updated: 2026-10-14
decisions:
- "ADR 0369"
- "ADR 0303"
- "ADR 0215"
- "ADR 0173"
@@ -75,7 +76,7 @@ In order:
| 8 | `docker` | the runtime's packages, base configuration, group | four configurations, one owner on one machine |
| 9 | `ssh-client` | everything under `~/.ssh` (research 027/03) | a predecessor's entries winning over the mesh's; stale keys |
| 10 | scripts | the operator's own scripts, shared and per role (research 027/03) | under no version control, copied by hand |
| 11 | `kernel` | kernel, microcode, boot entries | two machines without microcode |
| 11 | `kernel` | kernel, microcode, boot entries, and the kernel's run-time settings per machine (transparent huge pages first) | two machines without microcode; huge pages left at the distribution's `always` on a machine whose slow disks they flood (issue 746) |
`systemd`, `pacman` and `docker` hold the three seats that apply resources
(ADR 0207):
@@ -83,7 +84,19 @@ every module that declares a service, a package or a container depends on them b
They go on every machine before the rest, and once they have, an unmet dependency is refused rather
than reported.
`kernel` is last because a mistake in it costs a boot. `docker` holds the runtime seat
`kernel` is last because a mistake in it costs a boot. Its run-time settings are the exception and come
first (ADR 0369, issue 746): they change nothing at boot that the kernel does not already accept at run time, and
each is a per-machine setting whose default is what the kernel boots with, so assigning the module changes
no machine until that machine is given another value. The first is transparent huge pages. On the
control-node, with `always`, the journal's memory-mapped files were held in 2 MiB pages, so each small journal
write cost a 2 MiB write-back; the journal daemon and a log-reading intrusion-prevention service wrote
gigabytes in minutes onto a mirrored pair of spinning disks that also holds the bus's store, and the
controller lost its lease. That machine is set to `madvise`. The module writes the mode at boot and, on a
change, the new value directly; it reads it back and fails its unit if it did not take, gives back the mode
it found when unassigned, and on a running machine rotates the journal once so the open journal files are
mapped anew in small pages. How it is checked: the module's tests run its script against
a stand-in directory, its unit's health is the unit's state, and its tool `kernel_huge_pages` shows the
mode in force and the file pages each unit holds in huge pages. `docker` holds the runtime seat
(ADR 0166), with its settings declared as
ADR 0164 decides and the capability detected as
ADR 0165 decides; what of the three is not built yet is tracked by issue 398.
@@ -53,9 +53,9 @@ bucket, runs a bad migration — and not for the day a disk dies (ADR 0214).
*Added 2026-10-06, ADR 0233.*
The mesh composes a second file for the holder: every data item every module on the machine declares,
with its class, its path, the path that covers it and its protection. Once an hour and after every
night the holder measures each item that is not a cache, by the item's `measure`: a bounded walk at
night the holder measures each item that is not a cache, by the item's `measure`: a bounded rollout at
most daily for a small item (stopped after ten minutes or two million files and said as partial), a ZFS
dataset's own counters hourly, or the top-level entries only — never an hourly walk of anything large —
dataset's own counters hourly, or the top-level entries only — never an hourly rollout of anything large —
and reads the redundant storage it is on (a ZFS pool's health, an md array's
members, a btrfs filesystem's error counters). `backed-up` says each item with that measurement and its
newest good backup. The holder judges nothing: the controller's self-check (to-be 45, D13) keeps the
@@ -2,8 +2,9 @@
layer: to-be
status: in-progress
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
updated: 2026-10-12
updated: 2026-10-14
decisions:
- "ADR 0357"
- "ADR 0325"
- "ADR 0323"
- "ADR 0318"
@@ -100,7 +101,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
| the operator-channel's open messages | the seat's holder | its own key-value state | — |
| a machine's declared data: what it measured, what is retired, what was deleted (ADR 0233) | controller, from what each `node-backup` holder answers | the controller's store, `data_item` and `data_reading` | the holder measures and says; it records nothing the controller keeps |
| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads |
| the record mirrors: nodes, assignments, walks, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
| the record mirrors: nodes, assignments, rollouts, builds, settings (ADR 0289, *added 2026-10-10*) | controller (lease holder), after each change commits in its store, only when the value differs | key-value `mesh-controller_nodes`, `_assignments`, `_walks`, `_builds`, `_settings`; one event per write | read through `reads` (ADR 0201); D17 compares them with the store; the controller never reads them to decide ([to-be 54](54-the-controllers-records-on-the-bus.md)) |
| a message a consumer gave up on (ADR 0264) | controller, from the server's maximum-deliveries notice | the bus, the stream `DEAD_LETTERS` | read, delivered again or dropped through the controller's `dead-letters` verb, with why; an ask is delivered again only when the controller made it itself, every other ask is only dropped (ADR 0293) |
A bus subject two components may publish on is refused when the controller composes grants, unless
@@ -175,7 +176,7 @@ process of its own reads them back from the store.
*Added 2026-10-08, ADR 0253:*
**a condition says itself to the operator in plain words, beside its summary, and says whether the operator
is needed.** The headline, needs, explanation, resolved line and actions are made where the condition is
made: by the producer when it knows more than the subject (a walk's modules, a failed unit's name, a
made: by the producer when it knows more than the subject (a rollout's modules, a failed unit's name, a
hand-act's cause), otherwise by the wording registered for the kind, otherwise from the scope, which is
reported. The explanation opens with its verdict. All of it is held to the plain rule when the controller
takes an observation: no hash, numbered id, dotted name, duration in code, clock time, flag, markup or line
@@ -259,25 +260,39 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — |
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted; every user's connection the server closed as a slow consumer or for breaking the bus's rules (ADR 0357) | bus server's advisory subjects; the controller's connection; the bus module's watch of its server (`bus.slow-consumer`, `bus.connection-closed`) | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `connection-closed` (one per bus user), `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
| S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — |
| S14 | facts snapshot exported | controller | when it moved, and daily | 2 days, or none kept by a controller up that long (Phase 5) | `facts-stale` | warning | — |
| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — |
| S16 | a walk waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose walk waits | 30 min; urgent after 4 h | `waiting` (names the walk and `plans go`) | warning | — |
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a walk out | warning | — |
| S18 | a batch of merges is cut into its walk (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no walk is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
| S19 | a batch waiting behind an open walk is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a walk is open | that walk's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the walk) | warning | — |
| S16 | a rollout waiting for its delivery's word is let go | mesh-delivery, through `deliver` (ADR 0239) | each merge whose rollout waits | 30 min; urgent after 4 h | `waiting` (names the rollout and `plans go`) | warning | — |
| S17 | a send held for the bus's planned step is told to a person (ADR 0271) | controller's plan | each send refused because it would replace the bus outside its step | none: the first refusal; cleared once the bus's machine has been sent the build the mesh holds | `bus-step-waiting`, for the operator (names the machines, what waits, the bus build from and to, since when and the `mesh-controller.bus` call); S3 leaves such a rollout out | warning | — |
| S18 | a batch of merges is cut into its rollout (ADR 0276) | controller's merge window | each batch | a minute past `merge-window-at-most` while no rollout is open: the controller failed to cut it | `batch-not-cut` (names the batch and what it holds) | warning | — |
| S19 | a batch waiting behind an open rollout is cut when it ends (ADR 0276) | controller's merge window | each batch closed while a rollout is open | that rollout's bound, a tier's bound for each of its tiers, from when the batch closed | `batch-behind-walk` (names the rollout) | warning | — |
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
controller translates each into a condition naming the thing in the mesh's words, as the refused reply
of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer
is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a
system account the controller hears a slow consumer and a refused subject for its own connection only
([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md),
open question 2).
is said only for one the mesh names and expects, and clears when it exists again.
**Every user's slow consumer, from the bus module (ADR 0357).** The server closes every client it judges
a slow consumer and keeps its user and reason in its record of closed connections; it publishes neither
in the mesh's account. So the `nats` module's tool bundle reads that record on its own machine every 30
seconds and publishes `bus.slow-consumer` for each connection closed as a slow consumer, and
`bus.connection-closed` for each the server closed for something its client did against the bus's
rules or limits — naming the bus user, the connection and the reason, never a credential. A closure by
the client, by the network or by the server's own shutdown is not said: D15 says a user dropped in a
loop. A closure — its user, connection id and opening time — is said at least once: again when publishing
it failed, and again for the last hour when the bundle starts. The controller's watchdogs read the last
hour of both events back from the events stream on every tick, count each closure once, and raise one
condition per bus user and kind, naming the module and its machine; it clears an hour after the newest
closure. A closure is a condition at most about a minute after the server closed it.
**A refused subject is still heard for the controller's own connection only.** The server tells the
client and writes its log, and says it nowhere else — not in its monitoring endpoint and not to a system
account (issue 702, open).
## 4. The self-check: `doctor` (rule 6)
@@ -525,7 +540,7 @@ except an act recorded by a verb that is a person's decision by design, which is
counts, whatever cause it gives. The controller keeps one table of the verbs that write the log, and
each says whether it records a repair or a decision, and why: approving or rejecting a retirement and
deleting what was retired (ADR 0230); `bus upgrade`, because the bus is never rolled by the mesh, and
`upgrade release-backlog`, because after a failed walk the next opens only on a person's word
`upgrade release-backlog`, because after a failed rollout the next opens only on a person's word
(ADR 0236); and `secret rotate` when its cause is a leak (`leaked-in-logs`) — a person judges what was
disclosed, one leak rotates several values, and a leak that recurs is a defect of the module that
prints them, an issue against it, not a healer that rotates. A rotation for any other cause counts. A
@@ -604,7 +619,7 @@ the other machines follow. "Reported applied" is not enough.
**As built** (ADR 0236):
- **Every walk is gated, not only the core's.** A module is judged by its own health: it reported
- **Every rollout is gated, not only the core's.** A module is judged by its own health: it reported
applied, no witness put it back, no condition was raised since its send about the machine or about the
module there, and its tools are served on that machine where it has tools. Three healthy judgings at
least forty seconds apart and two minutes after the send, within ten minutes of it. A policy of
@@ -627,7 +642,7 @@ the other machines follow. "Reported applied" is not enough.
- **No build reaches a machine without a gate**: a gated send carries and judges everything waiting on
its machine; every other send — a plan's rest, a cascade, a healer's, a whole-mesh push — is refused
or leaves the machine while a build no gate has seen waits there; a rebuild with the same artifacts and
manifest is no move. What waits is moved by a **walk**, one machine at a time, the control
manifest is no move. What waits is moved by a **rollout**, one machine at a time, the control
node last, each judged; one that fails holds the next until `upgrade release-backlog --why`.
- **A wait for a person passes, and each module of a send keeps its own pass**
(ADR 0254): a module whose only fault is a new login the operator owes
@@ -787,10 +802,10 @@ has a week of entries; the durations are recorded for every machine.
| Repository | Delivers |
|---|---|
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9), and every user's slow consumer and closed connection read back from the bus module's events (ADR 0357); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
| `mesh-host` | the heartbeat carries its interval; D1's validator published as a package the controller imports |
| mesh-tools | the tool runner's heartbeat (S11) |
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) |
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5); the `nats` module's watch of its server's closed connections, publishing `bus.slow-consumer` and `bus.connection-closed` (S9, ADR 0357) |
| mesh-lab | R9: each signal suppressed in turn |
**Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a
@@ -923,7 +938,7 @@ and "the commit is the build at hand"), on branches, not yet merged:
| hq | its own `merge-check.sh`, running `records.py`, `index.py` and `cycle.py` |
**Not yet:** the delivery plan kept (id, inputs, result) and the release comparing its plan with it; the
state machine replacing the walks' states, with its table, its test, its conditions and healer H2;
state machine replacing the rollouts' states, with its table, its test, its conditions and healer H2;
the status's link to the kept plan; the notes under `refs/notes/mesh-plan`; check builds in a scratch
namespace (the check builds no module today, so there is none to keep apart yet); the statuses made
required — prepared as calls of the forge module's tool, applied by the operator.
@@ -327,36 +327,36 @@ Presence facts are events on the bus. The router keeps them as **current state o
machine and per intake kind, overwritten — never as a history, and never in a message's words. A
consumer that wants a timeline of the operator's day is a decision of its own.
### While a delivery walks
### While a delivery rollouts
*2026-10-12, ADR 0308; the build is tracked by issue 532.*
A delivery restarts what it moves, and those restarts raise conditions that clear once it is done; a router
that restarted used to say again what it had been told to keep quiet (issue 531). So the router holds what a
walk touches and tells the walk's result once.
rollout touches and tells the rollout's result once.
- **A walk names what it touches.** The controller's walk events say, when a walk starts and again when it
- **A rollout names what it touches.** The controller's rollout events say, when a rollout starts and again when it
ends with its outcome (delivered, failed or stopped), the modules it moves and the machines it sends them to.
What it touches is those modules on those machines, and every module a moved tool runner or node-engine
carries there.
- **The router holds what an open walk touches.** A message whose subject is a touched module or machine is
- **The router holds what an open rollout touches.** A message whose subject is a touched module or machine is
held instead of sent, kept with when it was held, and `history` says it was held. A message several open
walks touch is held until the last of them ends or its bound passes.
rollouts touch is held until the last of them ends or its bound passes.
- **Some messages are never held:** an ask (§6), a secret given or replaced, a message about anything no open
walk touches, and a walk's own failure.
- **A walk's end is told once:** one message with its outcome, the modules and machines it reached and how
rollout touches, and a rollout's own failure.
- **A rollout's end is told once:** one message with its outcome, the modules and machines it reached and how
long it took, then each held message whose condition is still open, one line each. One whose condition
cleared during the walk is only counted. A walk delivered with nothing held and nothing open is one line, or
cleared during the rollout is only counted. A rollout delivered with nothing held and nothing open is one line, or
nothing on a channel set to say only what needs the operator.
- **A hold has a bound:** twice the largest delivery budget in force among the walk's deliveries
- **A hold has a bound:** twice the largest delivery budget in force among the rollout's deliveries
([to-be 47](47-delivery-from-commit-to-delivered.md)): 20 minutes for a core module, 10 for leaf modules at
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a walk none of whose
deliveries has a budget. Past it, every held message still open is sent with a line saying the walk still
runs and since when, and the cleared ones are counted. A walk that hangs never silences the mesh.
a five-minute budget and 6 at a three-minute one; 15 minutes, not doubled, for a rollout none of whose
deliveries has a budget. Past it, every held message still open is sent with a line saying the rollout still
runs and since when, and the cleared ones are counted. A rollout that hangs never silences the mesh.
- **A restart says nothing it has said:** a router that restarts says again only what is open and was never
answered, never a silenced message or one withheld for its content.
*How it is checked:* the controller's tests of the walk events, and the router's tests of holding, never
*How it is checked:* the controller's tests of the rollout events, and the router's tests of holding, never
holding, the end message, the bound and the restart, as ADR 0308 lists them; live, the next delivery of the
tool runner gives the operator one message at its end, and the channel's `history` shows what was held and
released.
@@ -467,13 +467,13 @@ the factors are the router's state.
### An approved settings change takes effect (ADR 0344)
*2026-10-13, issue 641.* Approving a settings proposal is the act that makes it take effect. The
serving controller sets the layer and, in the same act, opens a **settings walk**: every machine
serving controller sets the layer and, in the same act, opens a **settings rollout**: every machine
that runs the module and that the layer reaches, the control node last, each sent and its module
judged at the first-node gate before the next. A settings walk sends the layer and no build: every
judged at the first-node gate before the next. A settings rollout sends the layer and no build: every
module on the machine stays at the build it was last sent, so a build held for a person, or waiting
for a first-node gate, stays where it is. A send that would move a running module's data stops the
walk and names the `push --move` that sends it. The walk says its start and end like every walk, so
its end is told as every walk's is. A failure stops the walk and leaves the layer as approved; the
rollout and names the `push --move` that sends it. The rollout says its start and end like every rollout, so
its end is told as every rollout's is. A failure stops the rollout and leaves the layer as approved; the
previous layer is in the settings history. The ask says all this before the Approve, and no longer
names a push. A layer set by hand still waits for a push.
@@ -481,9 +481,9 @@ A delivered file is not yet an applied one: a module whose program reads its set
starts applies them itself (a service's `restart-on`, or a tool of its own). Betterbird is such a
module; issue 643 makes it restart Betterbird at a quiet moment.
How it is checked: the controller's tests open the walk on the warrant, send one machine at a time
How it is checked: the controller's tests open the rollout on the warrant, send one machine at a time
with every build kept and the module judged, stop before a data move and on a failure without
holding other walks back, and read the ask's words.
holding other rollouts back, and read the ask's words.
### Proofs and tiers
@@ -2,62 +2,122 @@
layer: to-be
status: in-progress
code: [mesh-controller, mesh-catalog modules/mesh-delivery]
updated: 2026-10-13
updated: 2026-10-18
decisions:
- "ADR 0426"
- "ADR 0421"
- "ADR 0420"
- "ADR 0419"
- "ADR 0418"
- "ADR 0408"
- "ADR 0407"
- "ADR 0406"
- "ADR 0405"
- "ADR 0404"
- "ADR 0403"
- "ADR 0402"
- "ADR 0401"
- "ADR 0398"
- "ADR 0395"
- "ADR 0382"
- "ADR 0377"
- "ADR 0364"
- "ADR 0362"
- "ADR 0360"
- "ADR 0350"
- "ADR 0323"
- "ADR 0313"
- "ADR 0306"
- "ADR 0304"
- "ADR 0299"
- "ADR 0298"
- "ADR 0282"
- "ADR 0276"
- "ADR 0270"
- "ADR 0267"
- "ADR 0250"
- "ADR 0249"
- "ADR 0239"
- "ADR 0238"
- "ADR 0236"
- "ADR 0233"
- "ADR 0214"
- "ADR 0201"
---
# 47 — Delivery: from a commit to delivered
**A delivery is one commit in one repository, from the moment its pull request's head is announced to the
moment every machine of its deploy plan runs it, or to the moment it fails, is superseded or is stopped. A
delivery group is two or more deliveries that share a head branch name, delivered as one unit in order. Both
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
(ADR 0239.)
**A change reaches the machines in one of two ways, both owned by mesh-delivery: a hotfix (one pull request,
delivered at its merge) or a feature (one branch name across its repositories: in two or more, one delivery
group merged in one pass on the operator's one Approve and delivered by one delivery plan; in one, a lone
delivery). A design's milestones are features delivered one after another. The trunk is what runs; there is no
"release branch"** (ADR 0401, ADR 0418 §1, ADR 0421 §2).
**From its merge on, a delivery is delivered by the walk of its batch.** The controller collects merges in a
rolling merge window and plans each batch once, as one walk with at most one commit per repository, one walk
at a time. Every delivery the walk answers ends with it
(ADR 0276).
**Every step starts on an event, and the state lives on the bus.** The forge's events, the controller's
events and the nodes' reports move a delivery; the bus's own rules keep the queues, the merge window and the
order of a plan's tiers, which only the module graph gives (ADR 0419); and one chain, from module to commit to build to rollout to assignment, is kept in the bus's
key-value store as its only source (ADR 0406, ADR 0407, ADR 0408). A running delivery never touches merging:
every merge is recorded at once for the next delivery plan of what it touches.
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
of source and artifacts decides, and nothing here changes that. This design says who answers for one
commit's journey and in what order the steps are allowed.
commit's journey, in what order the steps are allowed, and where each fact of it is kept. It was rewritten on
2026-10-16 from the decisions of research 055, which replaced the "release branches", the toolchain image, the
controller's delivery tables and the timers that polled them.
## The parts
```
forge's holder ──pull.updated / .merged / .closed──► mesh-delivery ◄──checked / plan-moved── controller
(statuses, the view, (deliveries, groups, (planner, gate,
the notes) the table, its state) registration,
▲ │ │ sending, walk)
└── its tools: note, view, status ◄───────────────┘ └── its verbs: delivery-plan, -order,
-check, deliver, delivery-stop,
delivery-walks
forge (system webhook) ──forge.pull-request.* / merged / status──► the bus ◄──built / checked / plan-moved── controller
│ ▲ (planner, order,
│ │ registration,
▼ │ sending)
mesh-delivery │
(deliveries, groups, a key per │
pull request, the state table) ▼
node-engine on every node
builders (build seat) ◄── work queues: delivery builds, approve-asked checks, other checks ───────────────── (reports; backups first)
```
- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants,
registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk
rule, a gated send, the rollback and the walk of one trunk commit's tiers across machines. It still answers
every pull request's head with its check itself, so a check never depends on mesh-delivery.
- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the
group, the state table and their state, and nothing else.
- **The forge's holder** sets a commit's statuses, keeps the delivery's view as one comment on the pull
request, and appends to the commit's note under `refs/notes/mesh-plan`.
- **The build seat** builds and checks, asked by the controller as before.
- **mesh-delivery** (the seat `mesh-delivery`) owns hotfixes, features and milestones as deliveries and delivery
groups, decides each step on events, keeps one key per pull request, and asks the controller for each act
and the nodes' backup seats for the backups a delivery needs. It never sends to a machine, writes a
declaration or registers a build (ADR 0239 §25).
- **The controller** keeps planning, ordering, registration, sending and judging. It writes the chain's
links it owns — builds, rollouts with their tiers, assignments — and the merge window's
keys, all on the bus, by compare-and-set, while it holds the lease.
- **The builders**, the holders of the build seat, build and check with toolchains installed on their node, as
the build seat's own account, and take their jobs from the bus's work queues, delivery builds first
(ADR 0405, ADR 0406).
- **The forge module** (`gitea`) publishes every forge event on the bus through the forge's system webhook
(issue 729): pull requests updated, synchronized, merged and closed, and commit statuses; no label or comment
moves a delivery, and mesh-delivery subscribes to none (ADR 0421 §11). It sets statuses, keeps the delivery's view as one comment on the pull request, and appends
to the commit's note under `refs/notes/mesh-plan`.
- **Each node's backup seat** (`node-backup`) backs up the data a delivery plan moves there before the plan
sends anything (ADR 0404).
## Two ways a change reaches the machines
*ADR 0401.*
| Way | What it is | How it is delivered |
|---|---|---|
| **hotfix** | one pull request in one repository | at its merge, as a lone delivery; a lone merge is cut at once (ADR 0282 §2, as ADR 0304 §1 changed it) |
| **feature** | one branch name across its repositories; in two or more, its pull requests are one delivery group; in one, a single pull request | a group is merged in one pass once its composed check passed at every member's head and the operator gave the group's one Approve (ADR 0418 §4, ADR 0421 §2); its merges are one batch and one delivery plan, each module built once and sent once (ADR 0418 §1); in one repository, as a lone delivery like a hotfix |
- **The trunk is what runs.** Every delivery is cut from a repository's trunk. A merge into any other branch
starts no delivery, and the controller says so for that merge (issue 889). Holding merged work back, on a
branch beside the trunk or per module, was dropped: from 2026-10-13 23:52 to 2026-10-16 09:32 feature 57's
branches held the controller for 55.7 h, and 28 of 68 other merges touched a held module (research 055,
finding 3).
- **A design's milestones are features, one after another.** Each milestone is delivered as a feature when its
own group is ready; the next starts when the previous one's delivery plan is done (playbook 04).
- **A milestone is functional, its delivery shaped technically.** A milestone is named by what the operator or
the mesh can do once it is delivered; how it ships is shaped by the delivery plan its changes would have, asked of
the controller's planner through mesh-delivery's `what-if` (ADR 0239 §3): the modules it moves and their fan-out.
- **What a merge waits for.** After its checks, a merge waits for one act of a person: the operator's Approve,
asked as soon as both required checks passed at the head (ADR 0421 §1, §3). For a group the Approve is asked
once, when every member's checks and the group's composed check passed, and covers every member's head
(ADR 0421 §2); it is the review ADR 0382 §6 asks (ADR 0421 §9). No ready mark, label, comment, forge review or
recheck stands before it. The operator's Approve is asked for every merge into main, a hotfix's too
(ADR 0398 §3; the operator, 2026-10-16 23:40).
## A delivery
@@ -66,12 +126,21 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
- the pull request (number, base, head branch, description), or none for a commit that reached the trunk
without one;
- its **delivery plan**: build plan, deploy plan and verdict, as the controller's planner computed them for
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
its diffset (ADR 0238 §2, as ADR 0323 §1 changed it);
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
**machine steps** while delivering;
- once merged, **the commit it landed on the trunk as**, and the walk that delivers it: the walk of the batch
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
window and the batch*).
- its **Approve**: the ask, the operator's answer with its warrant, and the review record the pull request was
opened with (ADR 0421 §5, §8);
- once merged, **the commit it landed on the trunk as**, and the delivery plan that delivers it: the plan of the
batch that answers its merge, which may carry a later commit of its repository that contains it. Every member
of a delivery group is delivered by the same plan, and its delivery ends with that plan's result
(ADR 0418 §1).
**One key per pull request.** Beside the delivery, mesh-delivery keeps one key per open pull request on the
bus: its head, the trunk commit its current check is asked against, and its verdict, written by
compare-and-set (ADR 0406 §4). It writes a new head when the forge says one, and a new trunk commit only when it
asks the pull request's check again (ADR 0395 §1). A running check watches that key and stops when the head or
the trunk commit in it moves.
## The state table
@@ -84,372 +153,422 @@ allows.
|---|---|---|---|
| proposed | a head was announced and is being checked | 1 hour from when its check is asked; none while its verdict is out of date and no check is asked | nothing: the check's own watchdog (S6) speaks |
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
| ready | it passed; it waits to be merged | none | — |
| ready | it passed; its Approve is asked (a group member's once the group is ready), and it waits for it and the merge | none | — |
| rejected | it failed its check | none | — |
| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a walk that does not answer its merge took over its walk; `go` when its walk started |
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
| delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says |
| published | merged on its trunk, recorded in a batch; it waits for its plan to start | 30 minutes | `superseded` when a plan that does not answer its merge took over its plan; `go` when its plan started |
| held | it waits for a person: a failed backup (ADR 0404 §4), a policy of record; it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
| delivering | its plan runs: backups, then builds asked and sent tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the plan's key says |
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
**A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head
at the moment the build seat clones it, the **trunk commit**, and the verdict names both. When another merge
moves the trunk, a `ready` delivery goes back to `proposed` on the observed event `trunk moved`, with both
commits in the transition's reason; a `rejected` one stays `rejected`, its statuses saying the verdict is out
of date. Nothing is asked for it: a recheck, a new head or the merge tool asks the check again. The row from
`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for
an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own walk.
when the builder clones it, and the verdict names both. When another merge moves the trunk, a `ready`
delivery goes back to `proposed` on the event `trunk moved`, and mesh-delivery asks the check of every active,
cleanly merging open pull request into that trunk at once (ADR 0395, *A move of the trunk asks every check
again* below). An Approve already asked stays; the merge waits for the fresh check, and the Approve is not asked
again for a trunk move (ADR 0421 §1, §3). The row from `proposed` to `checked` takes only a verdict that names the trunk's head.
What this leaves as it is, said so nobody reads more into it:
**What needs no person ends by itself** (ADR 0270): a merge that moves nothing is `delivered` at its merge; a
merge a later delivered merge contains is `covered`, by the forge's word that the later merge commit contains
it; a merge whose plan said its modules unchanged is `delivered` when that plan ends (issue 611); a stop whose
plan ended without building its modules is read as covered once a later merge covers it (issues 474, 480).
- **mesh-delivery's trunk head is the last merge it heard** into that trunk while the delivery was open. A push
to the trunk that is not a merge it hears does not move it; the forge's holder, which reads the branch, and
the merge tool, which compares commits when asked, still catch it.
- **A verdict that names no trunk commit**, from a build seat older than ADR 0323, counts in mesh-delivery as it
did: such a delivery stays `ready`. The forge reads the same verdict out of date once its trunk moves, and the
merge tool refuses it and asks the check. The two agree once the head is judged again.
- **A verdict no build judges** (a change that touches no module, in a repository with no `merge-check.sh` or
not the mesh's) names the base branch's head the forge announced with the head as its trunk commit.
- **In a busy repository the merge tool can refuse, ask, and refuse again**: another merge may move the trunk
while the asked check runs, and the new verdict is then out of date too. Each refusal asks once more; the
merge goes through once a check finishes before the next merge.
- **A check that was asked and never answers** is listed by `stalled` an hour after the ask, and is asked again
with `mesh-delivery.recheck`, which the merge tool's refusal names.
**History is retired, not held** (ADR 0250). A delivery made from the forge's word at its merge, heard fifteen
minutes or more after it, with no plan of it open, goes to `retired` at adoption. **Nothing of a published
delivery is registered before its plan starts.**
The transitions are those of ADR 0239 decision 2, ADR 0270 and ADR 0323. A delivery that is
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
delivery merged with no walk answering it within ten minutes is held too, with the true reason, and listed by
`stalled` at once. Nothing waits silently.
**The machine steps.** While a delivery is `delivering`, each machine its plan reaches has a row with the
module and the build: `backed-up`, `sent`, `applied` or `failed`, and `silent` when no report came within
ApplySilentAfter (fifteen minutes). They are read from the plan's key, written by the controller from the
nodes' reports, never inferred from time.
**What needs no person ends by itself** (ADR 0270):
## Order without a person
- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and
sends nowhere) is `delivered` as soon as it is merged, whatever its check said;
- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine
running it by the walk of a later merge into the same trunk, delivered on its own passing check, and the
forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge
order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery`
says `covered: a later merge that contains it was delivered` as a success. A module that runs on one
machine only is covered too: its walk records the empty rest as sent once its first machine passes
(ADR 0270, as corrected 2026-10-12);
- *a merge whose walk said its modules unchanged* moves nothing too, and is `delivered` when that walk ends,
whichever commit the walk carried (issue 611). A delivery plan made while another walk built a module reads
that module whole and names it; the walk, cut later, reads the build source that build recorded (ADR 0267) and
may find none of the changed files in it. The walk's record then says so in `delivery.unchanged`: each such
module with why, "its build source holds none of the changed files", and not in its modules, which keep
meaning what the walk builds. A delivery whose every module the walk built or said unchanged is delivered,
saying which matched what runs. One with a module the walk neither built nor said unchanged is read as below.
Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's
`TestAMergeWhoseModulesMatchWhatRunsIsDelivered`;
- *a stop whose walk ended without building its modules* (issue 474) stays `stopped`, a final state, and
is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after
its stop` with the covering merge, once, as a success. No state or transition is added (issue 480);
- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes.
*ADR 0419, ADR 0418 §2.*
**History is retired, not held** (ADR 0250).
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no walk
of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes
to `retired` at adoption, before any check, hold or walk, and its pull request is not told. A held delivery that
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its walk
starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a
build registered before its turn would be carried by somebody else's send.
A delivery group has no order among its members, and nobody writes one (ADR 0419 §1). Inside a plan only the
module graph orders: the planner orders the build and deploy tiers by edges between modules, never between
repositories or members (ADR 0418 §2, ADR 0419 §3). Modules with no edge between them share a tier.
**The machine steps.** While a delivery is `delivering`, each machine its walk reaches has a row with the
module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy
readings so far), `passed`, `failed`, `rolled-back`. They are read from the walk's own record in the
controller's `plan-moved` event, never inferred from time.
| Edge | A module goes before | Read from |
|---|---|---|
| *built by* and *stands on* | the modules built by it or standing on it | the module graph |
| *SDK before its users* | every module built on the SDK, `mesh-sdk` | the module graph |
| *verb before caller* | a module whose change adds to its invoked verbs a verb this module's change adds to a seat's served verbs | the manifests |
| *version skew* | (the controller, when the plan moves it) every module whose manifest the plan changes | the change |
- **Engine before controller is the one default.** When a plan moves both the node-engine and the controller and
no edge orders them, the node-engine's tier comes first; an edge either way wins over it (ADR 0419 §4).
- **Each tier says the edge that placed it.** The plan, `delivery-order`, `show` and the plan's note name, for
each module placed after another, the edge and its rule, and where the default was overruled, the edge that
overruled it (ADR 0419 §5).
- **A cycle is refused before anything merges.** Edges that form a cycle fail the group's composed check, or a
lone pull request's check, naming every module and edge in it. The way out is in the change itself (a verb
promised optional before it is required, ADR 0246), never an order typed by a person (ADR 0419 §6).
- **No `after:` line is read.** mesh-delivery reads no `after:` line, no refusal asks for one, and a description
that holds one is told once on its pull request that the line is not read (ADR 0419 §2).
**The SDK by version.** A Go module names the SDK by its released version and fetches it from the mesh's forge,
checked against its recorded checksum; vendored copies go. Inside a group, a member may name the SDK member's
head: its merge check resolves that commit from the forge and builds against it, and the head stays fetchable
after the group merges because the forge keeps every pull request's head. The catalogue's check accepts an SDK
commit on the SDK's trunk or the head of an SDK pull request the forge holds, open or merged (ADR 0402 §2 to §4).
## Checks
### One check per pull request, kept by the bus
*ADR 0323, ADR 0350, ADR 0395, ADR 0406.*
- **Three work queues, taken in order.** Delivery builds go on a work-queue stream of their own; checks of pull
requests whose Approve is asked or given on a second (ADR 0421 §10); every other check on a third. A builder takes from the first, then the
second, then the third. Each check stream has one subject per pull request and keeps only the newest request:
a newer request replaces one still waiting, by the stream's rule. The fresh check of such a pull request, asked
on the merge path, is published on the second stream; a request left behind on the third for a head and trunk commit already
judged or running is acknowledged and dropped when taken, by reading the pull request's key. Nothing pauses the
worker or deletes and republishes a request (issue 558).
- **A running check stops itself.** It watches its pull request's key and stops when the head or the trunk
commit in it moves, without asking the forge (issue 726). An ended check delivers no verdict, so none of its
builds is reused.
- **A failed builder loses nothing.** A job a builder took and never acknowledged goes to another builder
after its wait.
- **A change is built once** (ADR 0350). A check that passes also builds what its merge would move into the
artifact store's scratch namespace under a reuse key, and the plan of the merge commit copies the build of the
same key, named by the check's verdict, instead of building.
### A move of the trunk asks every check again
*ADR 0395.*
- A trunk move asks the check of every active open pull request into that trunk that merges cleanly, against the
new head; a conflict is told once on the pull request, not checked.
- A pull request with no push for 7 days and no Approve asked or given is left out; its statuses say out of date
and that a push or a recheck asks its check (ADR 0421 §10).
- For a group, a move asks each member's own check, and the composed check runs again through those checks; the
group's own merges, made in one pass on its Approve, ask its members nothing again (ADR 0418 §4).
- Its cost is at most one check per active, cleanly merging pull request per trunk move; issue 944 measures it.
### A delivery group's checks
*ADR 0239 §14, ADR 0323 §6, ADR 0382, ADR 0403, ADR 0418 §4, ADR 0421 §2, §9.*
- **The composed check is each member's gate.** The group's composed check composes every machine with all
members' heads, each merged into its own trunk's head first, judged by the controller built from the group's
controller member when it has one (ADR 0382 §3). Its verdict sets `mesh/delivery-group` and each member's
`mesh/merge-gate`, and names every member it read. No gate runs for a member alone.
- **Each member's own repository check stays required.** Each member's `mesh/repo-check` runs its repository's
suite with the siblings read beside it (ADR 0382 §2).
- **No composed verdict, no gate.** A member's gate is pending while no composed verdict at the members' current
heads exists. A member's new head, close or base move puts the composed verdict out of date, and with it every
member's gate. When mesh-delivery cannot say the groups, the pull request is checked alone and its verdict
says so (ADR 0382 §1).
- **A group merges in one pass, with no order.** Once the composed check passed at every member's head and the
operator's one Approve covers those heads, mesh-delivery merges every member in one pass, no member waiting for
another's merge or delivery (ADR 0418 §4). The Approve is the review ADR 0382 §6 asks: the merge tool reads its
warrant for those heads in place of an approving forge review (ADR 0421 §9). A member the forge refuses is
merged again on its fresh check while the group's batch stays open for it.
## The Approve and the merge
*ADR 0421, ADR 0398 §3, ADR 0406 §1.*
- **Asked as soon as the checks pass.** When a pull request's delivery reaches `ready` (both `mesh/merge-gate` and
`mesh/repo-check` passed at its head against its trunk's head), mesh-delivery asks the operator's Approve of its
merge into main at once, on a channel that proves who answered (ADR 0274; Telegram today). The ask names the
pull request, its issues, the reviewer's verdict, the delivery plan and the trunk commit the checks judged. A
verdict that names a trunk commit the trunk had already left raises no ask: mesh-delivery asks that head's check
against the trunk's head, and its passing verdict raises the ask (ADR 0421 §1).
- **A group is asked once.** One ask names every member and its head, once every member is `ready` and the
composed check passed at those heads; one Approve covers them all (ADR 0421 §2).
- **The reviewer's verdict is attached, not a gate.** The review finishes before the push (playbook 07), and the
session that opens a pull request gives its review record: who reviewed it, the commit the review approved and
the verdict. The ask carries it; with none, or one naming another commit, the ask says so in its first line and
the operator decides (ADR 0421 §5).
- **The merge.** mesh-delivery merges through the forge's merge tool at the heads the Approve names, only while
they are still the heads and both checks passed against the trunk's head. When the trunk moved, the merge tool
asks the fresh check, and the event of its passing verdict asks the merge again; no timer asks it, and no person
rechecks (ADR 0421 §3, §12).
- **A push or a Decline ends the Approve.** A new head ends its ask or its Approve, for a group every member's, and
is said once on the pull request; the new head is asked again once its checks pass. A Decline merges nothing and
is said once (ADR 0421 §6).
- **A failure is said, never merged past.** An approved head whose check failed is said once on the pull request
and listed by `stalled`; so is a merge the merge tool kept refusing for an hour (ADR 0421 §7).
- **Recorded.** The ask, the answer with its warrant and the review record are kept on the delivery, shown by
`show` and said in mesh-delivery's log; the merge's transition names the Approve (ADR 0421 §8).
## The merge window and the batch
*2026-10-10, by ADR 0276.*
*ADR 0276, ADR 0282, ADR 0299, ADR 0304, ADR 0407.*
- **The window.** The controller puts every merge into a branch a module follows into the open **batch**.
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
merges and its times are kept in the controller's store and survive a restart.
- **The cut.** When the window closes and no walk is open, the batch becomes one walk: one delivery plan for
every commit in it, tiered along the module graph across repositories, with at most one commit per
repository, the latest of its branch. The walk names every merge it answers: its own commits, and the
earlier merges of a repository that its later commit contains.
- **One walk at a time.** Merges that arrive while a walk is open join the next batch. A walk that started
is never superseded. A walk still waiting for its word when the next batch is cut is folded into the new
walk, which answers every merge the folded one answered.
- **The controller's own path.** A merge that moves a module whose walk waits for nobody's word (the controller,
the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for
mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its
walk like any other. An own-path walk folds no waiting catalogue walk; that walk's merges are batched again
behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10).
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a walk is
answered by that walk: the assembling batch, the open walk, or, when that walk was delivered, it ends
`covered` at once.
- **The ends.** A walk done ends every delivery it answers: `delivered` for its own commits, `covered` for
an earlier merge of a repository it carried later, naming that merge. A walk that failed fails its own
commits' deliveries and marks none delivered; each earlier merge it answered is then walked on its own
commit, the newest first, one walk each, until one is delivered. A walk stopped stops every delivery whose
merge it walks again nowhere (below).
- **A group inside a walk.** The pairs of a delivery group's order (ADR 0249) are edges of the walk's
tiers beside the module graph. mesh-delivery gives a walk its word only when every delivery it carries is
at its turn.
- **The bookkeeping.** mesh-delivery ties a walk to every live delivery whose merge it answers, follows a
folded walk to the walk that took it over, and never ties a walk to a delivery that has ended. A waiting
walk no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
and a batch waiting behind an open walk names that walk. The controller says the batch on the bus as
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
delivery it holds.
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no walk open, and a batch
waiting behind an open walk past that walk's bound, each raise a condition. A merged delivery that no walk
answers within ten minutes is held with the true reason (the open walks of its repository and that none
names its merge, or that the controller holds none) and listed by `stalled` at once.
- **Every merge is recorded at once.** The forge's merged event records a merge into the open batch of what it
touches, or opens one, by whichever controller holds the plans. A merge is never refused, held or dropped,
and needs no person's release (issue 921). The bus's durable consumer keeps every merge event until it is
recorded; no catch-up timer looks for missed merges.
- **The window is two keys.** An open batch's window is a key whose time to live is `merge-window` (90 s),
refreshed by each merge into the batch, beside a key set once when the batch opens whose time to live is
`merge-window-at-most` (10 minutes). The first to expire closes the batch, heard by a watcher; no cutter
polls. A stop that records merges again into the open batch closes its window too (ADR 0299 §3).
- **A group's merges are one batch.** A batch holding a merge of a delivery group with a member not yet merged
does not close when either key expires; once the group's last member merged, the window runs again from that
merge. A group whose last member has not merged ten minutes (`merge-window-at-most`) after its first raises a
condition naming the member and why, and nothing of the group is delivered while it waits (ADR 0418 §3).
- **A lone merge is cut at once**: no batch it shares a repository, module, graph edge or delivery group member
with is assembling (ADR 0282 §2, ADR 0304 §1, ADR 0306).
- **The plan.** A closed batch becomes one delivery plan: every commit in it, tiered along the module graph
across repositories, at most one commit per repository, the latest of its trunk, each module it moves built once
and sent once to each node that runs it. It answers every merge in the batch, its own commits and the earlier
merges a later commit contains (ADR 0276 §2, as ADR 0299 §7 changed it; ADR 0418 §1). No member of a group has
a turn: only the module graph orders its tiers (ADR 0418 §2).
- **A plan fixes its commits when it starts.** A plan starts at its first send, after mesh-delivery's word, given once for a
group's plan when every member is merged (ADR 0418 §2), and
from then nothing that merges changes it. A plan still waiting for its word is folded into the next batch
(ADR 0276 §4).
- **Plans that share something run one after another.** A batch that shares a repository, a module or a graph
edge with an open plan waits for it, and starts with every merge recorded since when that plan ends. A batch
that shares none runs beside the open ones, so a hotfix is never held by an unrelated feature's delivery.
- **The ends.** A plan done ends every delivery it answers: `delivered` for its own commits, `covered` for an
earlier merge it carried by a later commit. A plan that failed fails its own commits' deliveries; each earlier
merge it answered is then delivered on its own commit, the newest first, until one is delivered (ADR 0276 §3). A
group's plan fails as one: no member is stopped because another failed, what earlier tiers delivered stays, and
the fix is a new change (ADR 0418 §5).
- **Shown.** `plans` lists the open batch first, with its window's time left and its merges, and a batch waiting
behind an open plan names that plan; the controller says the batch as `plan-moved` in the state `assembling`.
## A backup before the send
## A stop on a shared walk
*ADR 0214 §7, ADR 0233, ADR 0404.*
*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.*
When a delivery plan is ready to send, each node it reaches backs up, through its `node-backup` seat, every item
the plan's moved modules keep there whose protection is a backup (a copy or a dump). An item protected by
`redundancy`, one that says `backup: none`, and a cache are never asked, whatever their class. A node with no backup holder backs up
nothing, and the plan says so for that node.
One walk carries every merge of its batch, so ending a walk for one delivery ends it for every other
delivery on it. A stop therefore ends only the change it is for:
- **The send waits for every backup.** The controller sends nothing of the plan to any node until every backup
it asked has answered done.
- **A failed backup holds the delivery.** A backup that fails, or does not answer within ten minutes, holds the
delivery with the node, the module and the backup's own words as the reason, and raises a condition for the
operator. A person's `release` asks the backups again, and the send goes on when they pass.
- **What it costs.** About 85% of data modules back up in under 30 s; the slowest measured, postgres on the home
server, about 165 s (research 055, finding 5). A plan moving no backed-up item costs nothing more.
## The send and the rollout
*ADR 0360, ADR 0362, ADR 0377, ADR 0406 §5.*
- **Every machine at once.** A plan's builds go to every machine that runs them in one step, tier by tier; a tier
is asked once the tier before it is built and sent (ADR 0360 §1, §6). No machine goes first and nothing waits
for a health reading. A module of policy `record` that would recreate a container or move data waits for a
person (ADR 0377).
- **Every move is said.** The controller says every move of a plan as `plan-moved`, a move a command makes among
them, so mesh-delivery needs no periodic reconcile; it reads `delivery-walks` once when it starts.
- **A rollout advances on events.** The `built` and sent events of a tier ask the next; the nodes' reports end
the rollout (ADR 0360 §7, with ADR 0362's provisioning report). Each send puts a key whose time to live is
ApplySilentAfter (fifteen minutes); its expiry names the machine silent. There is no rollout ticker.
- **A failed apply is loud and fixed forward** (ADR 0360 §8); the node-engine's witness still puts back a
controller or tool runner that does not come up (ADR 0360 §9).
## A stop on a shared plan
*ADR 0299.*
One plan carries every merge of its batch, so a stop ends only the change it is for:
- **No stop by member.** A group's plan fails as one (ADR 0418 §5), so mesh-delivery asks no stop of a member
after a failed one. The mechanism stands for a person's stop with the way `again`: the controller ends the plan
and records every other merge it answered again in the open batch of its kind, whose window the stop closes.
- **What is left out with it.** A later merge of the stopped merge's repository, and a merge that moves a
module a left-out merge moves, would deliver the stopped change; each is left out too, its delivery `stopped`,
naming why.
- **A person's stop.** `stop` of a delivery whose plan carries others first answers what it would take down, and
acts on `way`: `again` (the others delivered again without it) or `walk` (the whole plan stopped).
- **What it does not do.** What the stopped plan already sent stays on the machines it reached; putting it back
is issue 509's.
- **A group's stop.** When a member after a failed one is stopped and its walk is open, mesh-delivery asks
`delivery-stop` for that walk with the stopped members' merges to leave out, every member the same pass
stops on that walk in one ask. The controller ends the walk, said as stopped, and walks every other merge
it answered again.
- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A
module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too.
Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped
merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is
walked again.
- **Where the others go.** They join the open batch of their kind, where merges heard while the walk ran
already wait, so an older commit never walks after a newer one; only with none open do they form a new
batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop
before it is cut, are cut without the rest of their window. Its walk withholds the left-out merges' modules
where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes
to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every
merge moved are written in one transaction.
- **The walk's record.** The stopped walk names the batch that walks its merges again, every merge it left
out, and each module it already sent that nothing walks again, with the machines it reached. A delivery
whose merge is walked again goes from `delivering` back to `published` (the row `rewalk`), or stays
`published`, and follows the fresh walk once it is cut. A delivery whose merge was left out, and every
delivery on a walk stopped whole, ends `stopped`, naming the walk and the stop.
- **A stopped merge never ships by a walk.** mesh-delivery asks every open walk or batch that names a stopped
delivery's merge to leave it out, and never gives such a walk its word. When the controller refuses, it
stops the whole walk.
- **A person's stop.** `stop` of a delivery whose open walk carries other deliveries does nothing at first: it
answers with each of them, which of them contain this merge and end either way, and two ways given with
`way`: `again` (the walk ended and the others walked again without this merge) or `walk` (the whole walk
stopped, every delivery on it with it). A delivery alone on its walk is stopped at once. A delivery is
marked stopped only once its walk is; a stop the controller refuses leaves it as it was.
- **What it does not do.** What the stopped walk already sent stays on the machines it reached, and its
asked builds still register, so a later send may carry them further; the stop names the machines, and
putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the
stopped merge's modules, contains the stopped change and delivers it when walked; the stop names each it
can see, and holding a merged change back for good is a revert's job.
## The delivery budget
*Decided 2026-10-10 by ADR 0282.
Where it differs, this section replaces the window for every merge and the one walk at a time above, slice by
slice as each slice below is built; until a slice is built, the text above is what runs.*
*ADR 0282, ADR 0304 §4, ADR 0313, ADR 0360 §12, ADR 0362 §5.*
**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard
from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes;
a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is
shown and held to no budget. Measured on 51 walks on 2026-10-09 and 10, a leaf walk took a median of 6.0 minutes
(7 of 20 within five), a core walk of one tier 5.6 (worst 8.8), and a controller walk dragging two catalogue tiers
14.2 (issue 338).
A delivery's **delivery time** runs from its merge until every machine its plan sent has reported the send
applied, and for a provisioned module also placed. A **leaf** delivery's budget is three minutes, a **core**
delivery's ten. A delivery held for a person, a failed backup among them, is shown and held to no budget. Each
phase is kept — the window, the wait behind a plan that shares something, the backups, the build, the send, the
reports — and a delivery over its budget raises `delivery.<class>.over-budget`, naming its longest phase. The
**fix time** (ADR 0313) runs from an issue's first move to `fixing` to its last fix delivered, and contains the
delivery time.
*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget
above runs from the merge; the operator's own target is wider, the **fix time**, which runs from an issue's
first move to `fixing` to the moment its last fix is delivered. Each issue's class carries a **fix budget** too
(five minutes leaf, ten core, the delivery budget's own figures), and the tracker splits a fix time into four
phases — implementing, review, checks, and merge to running (the delivery time above) — so a fix over its
budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget.
The delivery budget stays as written: a measure inside the fix budget, not replaced by it.
## Builds without a toolchain image
**What changes in the walk.**
*ADR 0267, ADR 0350, ADR 0405.*
- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged.
- **Independent walks run at once**: no shared repository, module or module-graph edge with any open walk. One
node judges one walk at a time; a send to a node under another walk's judgement waits for its verdict; no leaf
build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged.
- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf
module with a long-running resource; one for a leaf module whose every resource is a tools bundle.
- **A tier's first nodes are judged at once.**
- **The bus's grants arrive before the gate** (ADR 0298): when a walk's builds change the bus's user list (a new
tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before
anything is judged; a step that would carry anything else is refused unsent and said on the gate line.
- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy
report; the plan shows an expected landing time from the last seven days' medians.
- **One build module per language** (Go; Node and TypeScript) installs its toolchain as a declared package on
every node holding the build seat, pinned to one version that is the same on every builder. The toolchain
image and its module retire.
- **Every builder runs the same toolchain, and says so.** Each builder names its toolchain versions with every
build outcome, and the controller's self-check raises a condition while two builders differ.
- **Builds and tests run directly**, as the build seat's own account on its node, in no container. A
repository's `# mesh-check-toolchain` line selects which installed toolchain runs its check.
- **A judge has an account of its own.** A judge built from a sibling's merge (ADR 0382 §3) is built and run as a
second account of the Go build module, whose home, configuration and cache the build seat's account cannot write,
so a check's script cannot change it.
- **Every dependency comes by the version its lock file names**, checked against its checksum: the mesh's own
code from the mesh's forge, its own packages from its package registry, any other from its public source.
- **A builder fetches from the forge over SSH with a key of its own**, as the build seat's account, under a forge
account for that builder that reads the mesh's repositories and pushes nothing.
- **A build's reuse key** holds the installed toolchain's version and the build module's recipe version in place
of the toolchain image's digest, so a toolchain update rebuilds only the modules of that language.
*Walks beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
a change shares).* Building the walks at once found what those bullets left open, and ADR 0306 decides it:
## One chain, on the bus alone
- **A walk's builds are its own, never another's backlog.** A send composes a module whose build another open
walk owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
judged only by its own walk, so a failed walk puts back only its own builds. What waits on a node with no open
walk to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
for builds an open walk owns).
- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the
merge window whether or not a batch is assembling, so a group's members walk in one batch, in their order.
- **Related means everything the walks would move.** A module in common is any module a walk plans to build or
send, the dependents of what its merges moved included; a change whose modules are not known yet is related to
every change. Two related walks are neither cut beside each other nor judged beside each other on one node.
- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a
warning naming the walk and the core walks that set it aside.
- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an
unhealthy network or engine) fails every judgement open on it, leaf and core, each walk putting back only its
own builds.
- **A rollback beside another walk's judgement** behaves as it does today; whether it should wait is tracked by
issue 528.
*ADR 0408.*
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of walk A left out
of walk B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
overlapping dependents keeping two walks apart; the warning on the third set-aside; a failed apply failing every
open judgement on its node.
Module, commit, build, published package or artifact, rollout and assignment are one chain, each link
answerable from the next. A **rollout** here is the controller's record of one batch's delivery; a delivery's own
delivery plan (*A delivery* above) stays the delivery's, and names the rollout that delivers it.
**Who builds what.**
| Link | Names | Written by | Bucket |
|---|---|---|---|
| build | its module, repository and commit, its reuse key, what it published: an artifact by digest or a package release by name and version | the controller | `mesh-controller_builds` |
| rollout | the commit it fixed per repository, its tiers, the builds it carried, the machine steps, and every assignment it changed with the build before and after; its states and transitions | the controller | `mesh-controller_rollouts` |
| assignment | the module and node, the build, artifact or package and commit it runs, and the rollout that put it there | the controller | `mesh-controller_assignments` |
| Part | Owner | What it adds |
|---|---|---|
| the walk's phases and readings | mesh-controller | each phase's start and end and each reading in the walk and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` |
| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery.<class>.over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` |
| the lone merge | mesh-controller | the cut at once when nothing waits to join |
| walks at once | mesh-controller | the independence test, one judgement per node, a core walk's ground, a repository's next batch |
| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once |
| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` |
Beside the chain, on the bus too: deliveries and groups (mesh-delivery's `deliveries` and `groups`, ADR 0239 §37),
one key per open pull request (mesh-delivery), and each open batch with its merges and its window's two keys (the
controller, ADR 0407).
**Slices**, each one pull request per repository, each with its acceptance sentence:
| Slice | Delivers | Done when |
|---|---|---|
| 1 — the measure | the walk's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its walk's phases |
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its walk |
| 4 — walks at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
| 5 — the landing time | the expected landing time | `plans` shows it for an open walk, and the week's deliveries land within a minute of it at the median |
Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges
only once it is built.
## A delivery group
- **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull
requests. Two or more make a group. A group that has started delivering takes no new member.
- **Order**: `after: <repository>` lines in a member's description, and the controller's `delivery-order`
over the graph (ADR 0239 decision 4). The plan shows the order and why each pair is ordered: *declared*,
*built by*, *version skew*, *engine before controller*, *by name*.
- **Precedence** (ADR 0249): when rules order one pair both ways, the higher one decides. *Declared* comes
first, then *built by* and *version skew*, then *engine before controller*. The pair shows the rule that
won and the rules it won over. Rules of one rank both ways are a contradiction. The group is `rejected`,
and the reason names both members, both rules and the `after:` line that would settle it.
- **Check**: the members' heads composed together, one gate over every machine with all their definitions,
each head merged into its own trunk's head first and named with that trunk commit (ADR 0323). It is asked
by mesh-delivery through `delivery-check` when the group forms or a member's head moves, and posted on
every member's head as `mesh/delivery-group`. A member rebased or pushed is a new head with its own check:
it takes its old head's place in the group, even once the group is delivering, and the composed check runs
again for the new heads instead of holding the group. A merge into a member's trunk that is not one of the
group's members puts the composed check out of date: the group reads `checking` with the reason, and the
composed check runs again once a member is judged again. Checked by mesh-delivery's
`TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember` and `TestARebasedMemberIsComposedAgainNotHeld`.
- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`.
A member published while its group's composed check is asked and not answered waits there for the answer;
one held because that check had not passed is *lifted* back to `published` once it passes or warns for the
heads that merged and its own walk is still open (ADR 0270).
- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its
walk while the walk's other merges are walked again (ADR 0299). The members before it stay delivered.
Its own builds were put back at its gate.
A group's state is derived from its members and its composed check, and is shown, never stored apart from
them.
- **The bus is the only source.** Each bucket is written by the controller that holds the lease, by
compare-and-set, and the controller reads its own buckets to decide; a rollout resumes from its key after a
restart. No table of the controller's store holds builds, rollouts, batches or assignments, so no record mirror,
audit or healer exists for them; the nodes, settings, declared-config and predefined-keys mirrors stay as
ADR 0289 and ADR 0356 have them. A module reads a bucket by naming it in `reads`; mesh-config reads the
assignments bucket.
- **No key of the chain is removed.** Each bucket keeps up to 64 earlier revisions of each key as its history. A
value is never cut to a summary and stays under the bus's largest message. The bus's storage is declared as data
protected by a backup, so it is backed up nightly and before a delivery moves the bus.
- **One read answers each question.** For any module on any node, one read of its assignment names the commit,
the build, the artifact or package and the rollout that put it there; one read of a rollout lists every
assignment it changed.
- **The tables go by a numbered migration**: builds, rollouts and assignments are written once into the buckets,
waiting build requests onto the work queues and an open batch into its keys; then the tables are dropped from the
controller's store and its consolidated schema.
- **The artifact store and the package registry keep the bytes**; the chain names them (ADR 0251).
## What every transition does
In this order, each by its owner:
1. **kept**: mesh-delivery puts the delivery in its state (`deliveries`, one key per delivery; `groups`, one
key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried, later
each time, from ten seconds doubling to an hour. The forge's answer that what it is for does not exist ends
it at once, and sixty failed tries end it too, each said among the owner's refusals (ADR 0250);
1. **kept**: mesh-delivery puts the delivery in its state by compare-and-set before anything else; what it owes
outside is kept with it and retried from ten seconds doubling to an hour, at most sixty times (ADR 0250);
2. **said**: the event `mesh-delivery.transition` (and `mesh-delivery.group` when a group's derived state
changes), with the delivery's id, from, to, event, why and plan summary, and no secret or address;
3. **noted**: mesh-delivery asks the forge's holder (`gitea_note_append`) to append a line to the commit's
note: the head while the delivery is off the trunk, the commit it landed as once it is on it, and at its
end a line of what was executed there. The forge's holder writes it in its own repository as the forge's
own account, and a line already there is not appended again;
4. **reflected** (not for history, whose pull request the forge may no longer hold): the delivery's view, one comment on the pull request with the state, the plan, the machine
steps and the group, kept current (`gitea_delivery_view`); the status `mesh/delivery` on the head, and
`mesh/delivery-group` on every member's head (`gitea_commit_status`). Every status links to the pull
request, `mesh/merge-gate` among them.
changes), with no secret or address;
3. **noted**: a line on the commit's note, appended by the forge module as the forge's own account;
4. **reflected**: the delivery's view on the pull request, `mesh/delivery` on the head and `mesh/delivery-group`
on every member's head.
A doubled step is refused by compare-and-set and by the bus's message de-duplication, which also replaces the
forge module's files of what it announced (ADR 0406 §6).
## The verbs
**mesh-delivery's seat** (served by its holder; the first five read):
**mesh-delivery's seat**:
| Verb | Answers or does |
|---|---|
| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group |
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the walk |
| `groups` | every group with its members in order and its derived state |
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
| `deliveries` | every delivery not final, the final ones of the last day, and every one `stalled` still lists |
| `show` | one delivery or group whole: plan, the edge that placed each tier, the Approve, transitions, machine steps, the chain's links |
| `groups` | every group with its members and its derived state |
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner |
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting walk no live delivery holds |
| `recheck` | a rejected or ready delivery back to proposed, with why |
| `release` | a held delivery to delivering, with why: a person's word |
| `stop` | any delivery not final to stopped, with why; its walk is ended through the controller. On a walk that carries other deliveries it first answers what ending the walk would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise |
| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing |
| `stalled` | every delivery held past its bound, with the transition H2 may take |
| `recheck` | an exception verb, no step of the flow: a check asked again for a proposed delivery nothing else asks again, an inactive one or a failed verdict gone out of date (ADR 0395 §6, ADR 0421 §12) |
| `release` | a held delivery goes on, with why; for a failed backup it asks the backups again |
| `stop` | a delivery to stopped, with why; on a shared plan it first answers what it would take down (ADR 0299) |
| `close` | H2's only verb: the transition the table names for a stalled delivery |
| `retire-history` | held history to retired (ADR 0250) |
| `times` | each class's budget, the week's median and worst delivery time, and the deliveries over budget |
**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`,
`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which
takes the merges to leave out and walks the walk's other merges again without them, ADR 0299),
`delivery-walks`. A person has `plans go <plan> --why`, which does what `deliver` does under their name and
is recorded as a hand act. The controller says every walk it keeps as `plan-moved`, from a verb's command
too.
**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It
also announces `pull.closed`, and a merge's head and the statuses the forge holds on it.
## The store
Key-value state on the bus, declared by the module (ADR 0201): `deliveries` and `groups`. One writer, the
seat's holder, whose code changes state from a single goroutine. A final delivery is kept for thirty days and
then removed. Its note on the commit keeps its record after that. The bus's own snapshot (ADR 0235) backs it
up.
**The controller's verbs for it**: `delivery-plan`, `delivery-order` (the edges that order a plan's tiers,
ADR 0419 §2), `delivery-check`, `deliver`,
`delivery-stop`, `delivery-walks`, and `plans` (the open batch, the open plans, and a what-if for a repository
and its files). A person has `plans go <plan> --why`.
## The bootstrap
- A walk that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
controller's own, started by the merge and witnessed on the machine. mesh-delivery records it.
- Any other walk waits for `deliver` only while the seat has a holder on record. With none on record, the
controller starts it as it always did.
- A waiting walk is said by the controller whatever mesh-delivery says of itself: S16 raises
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the walk answers and
`plans go <walk> --why` as the way on.
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
- A plan that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
controller's own, started by the merge; mesh-delivery records it (ADR 0239 §31). Such a merge shares one batch
and one plan with the other merges of its delivery group, and that plan is started by the controller when the
group's batch closes, not by its own merge (ADR 0418 §6).
- Any other plan waits for `deliver` while the seat has a holder on record; with none, the controller starts it.
- A replaced controller or mesh-delivery carries on from the bus: the keys, the queues and the durable
consumers hold every step, so nothing is lost or repeated (ADR 0406 §6).
## The switch
## Acceptance tests
1. The controller's change (the seat, its verbs, `plan-moved`, the wait for a holder's word, `plans go`) is
merged and rolls out as the core does. With no holder on record it behaves exactly as before.
2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built
and published.
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Walks that start
from then on wait for its word. Walks already open finish on their own and are adopted as deliveries in
`delivering`.
4. To step back, unassign it. Waiting walks are then started by the controller at its next pass.
What the build proves before this design is implemented (research 055's direction):
## Phases
1. A feature across the controller and the catalogue with a new seat verb is delivered as one group, with no
`after:` line, one composed check, one Approve and one delivery plan: its merges are one batch, each module is
built once and sent once, its commits are carried by that one plan, and the plan names the edge that placed
each tier.
2. An SDK change and a Go module using it are delivered as one group.
3. A data module is backed up before a delivery moves it, and a failed backup holds the delivery.
4. A build on any builder uses the installed toolchain with no container, and a toolchain update rebuilds only
modules of that language when its version changes.
5. Two merges into one repository within a minute leave each open pull request with one queued check, by the
stream's rule.
6. The operator's Approve is asked within seconds of the event of a pull request's checks passing, with no ready
mark, label or comment written by anyone.
7. A controller replaced in the middle of a rollout resumes it with no step lost or repeated.
8. The merge window closes by its key's expiry.
9. A hotfix to a module the running feature's delivery does not touch reaches the machines in minutes, beside it.
10. For any module on any node, one read of its assignment names the commit, build, artifact or package and the
rollout that put it there, and one read of that rollout lists every assignment it changed.
11. Two merges into one repository while a delivery of that repository runs are both delivered by the next plan,
with no person's release and nothing held in mesh-delivery.
12. No table in the controller's store holds builds, rollouts or assignments.
| Phase | Delivers | Done when |
|---|---|---|
| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` |
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a walk never let go is raised by S16 |
| C — the walk itself | the per-tier walk moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
## What retires
- The "release branches" and the controller's judge of them (ADR 0358, superseded by ADR 0401).
- The toolchain image and its module (ADR 0405).
- The window cutter, the merge catch-up timer, the rollout ticker, mesh-delivery's tick, reconcile and
ready-mark poll, the forge's poll as a trigger, the queue reordering, the forge-asking cancel, and the forge
module's announcement files (ADR 0406, ADR 0407).
- The ready mark: the label `mesh/ready`, the `ready at` comment, `mark-ready`, the forge module's
`ready-authors` setting, and mesh-delivery's subscriptions to labeled, unlabeled and commented events
(ADR 0421 §4, §11); a recheck by hand as a step of the flow (ADR 0421 §12).
- A group's order: `after:` lines, the ranked rules and the order of the repositories' names (ADR 0419, which
supersedes ADR 0249).
- Delivery member by member, a stop of a member after a failed one, and the own path's separate batch for a
group's merges (ADR 0418 §1, §5, §6).
- The controller's tables of builds, build requests, rollouts and their tiers, batches and merges, and
assignments; the record mirrors, probe D17's audit and healer H6 for those kinds (ADR 0408).
## Milestones
Confirmed by the operator on 2026-10-17 (playbook 04). Each is one release of playbook 07; the next starts
when the previous one's delivery is done.
| # | Outcome | Acceptance tests | Repositories | Issues |
|---|---|---|---|---|
| 1 | A merge during a running delivery is never dropped and needs no person's release; the merge window lives on the bus; a rollout that shares nothing with an open one runs beside it | 8, 9, 11 | mesh-controller | 973 (with the rest of 525) |
| 2 | A ready mark is acted on within seconds, and each pull request has at most one waiting check | 5, 6, 7 | mesh-controller, mesh-catalog (mesh-delivery, gitea) | 974, 975, 976 |
| 2b | build-agent and route-proxy live whole in mesh-controller (ADR 0420, accepted 2026-10-17): the controller accepts a reviewed move, mesh-delivery announces each merge with its Approve, and the two modules move; built as two releases before the rule that a milestone is one release | — | mesh-controller, mesh-catalog (mesh-delivery, build-agent, route-proxy) | release 1: 1025; release 2: 1019 |
| 3 | A feature with a new seat verb across the controller and the catalogue is one group, one batch, one delivery plan and one Approve: no order lines, no ready mark, each module built and sent once | 1, 6 | mesh-controller, mesh-catalog (mesh-delivery, gitea) | 977, 978, 1022, 895 (with 1046 folded into it: unit and integration tests split, each level within its bound); 605 (1021 closed as its duplicate) is resolved by this build (ADR 0419 §7) |
| 4 | A delivery backs up the data it moves first, and a failed backup holds it and is said | 3 | mesh-controller, mesh-catalog (mesh-delivery, the backup seat's holder) | 980, split into 1065, 1066, 1067, 1068 |
| 3 fix | A group approved together merges in one pass and goes out in one delivery plan, also when its members' tests read each other's repositories | 1 | mesh-catalog (mesh-delivery, gitea) | 1082, 1049, 1047, 1083, 1079 (1084 closed as its duplicate) |
| 5 | One read of a module on a node names the commit, build and rollout that put it there | 10, 12 | mesh-controller, mesh-catalog (mesh-config, nats) | 981 |
| 5b | The controller makes no build or deploy plan: building and delivering a change are done outside it, driven by events on the bus, and the controller receives only what each node should run | — (its decision names them) | mesh-controller, mesh-catalog (mesh-delivery) | 1117 |
| 6 | Builds use installed toolchains with no container, builders fetch over SSH with their own key, and an SDK change ships in one group with its users | 2, 4 | mesh-catalog (build modules, build-agent, ssh-client, gitea), mesh-controller, mesh-sdk | 979, 971 |
The issues are linked `depends-on` in this order; 976 depends on 975, and 979 also on 971. Milestone 3's first live group (milestone 4's, on 2026-10-17) held one Approve but merged in two passes and went out in two delivery plans: a member's tests read the other member's repository, so the first merge made the second's verdict out of date, and a closed group never asked its composed verdict again (issues 1049, 1082). The operator put a fix of milestone 3 after milestone 4 and before milestone 5 on 2026-10-18; acceptance test 1 is proved again by it. The operator put milestone 5b after milestone 5 and before milestone 6 on 2026-10-18, as an issue rather than a research effort; where the planning goes is settled by a decision record before it is built (issue 1117), and milestone 6's build changes are read against it. Milestone 2b was
put before milestone 3 by the operator on 2026-10-17. Milestone 3 builds ADR 0418, ADR 0419 and ADR 0421
(issues 977, 978 and 1022), and splits mesh-controller's unit tests from its integration tests (issue 895, the
operator on 2026-10-17), so the unit level holds its bound of ADR 0391; the integration level's sixty seconds
is issue 1055. Acceptance test 6 spoke of a ready mark when milestone 2 was delivered; restated for
the Approve, it is proved again by milestone 3.
## What is not decided here
- A web view of deliveries beyond the pull request's comment.
- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge
module's protection tool.
- Moving the per-tier walk out of the controller (Phase C).
- Whether `mesh/delivery-group` is required on the trunks: the operator's setting, through the forge module's
protection tool.
- How a builder's public SSH key reaches the forge: the forge's and the SSH client's mechanism.
@@ -2,7 +2,7 @@
layer: to-be
status: in-progress
code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog, mesh-sdk]
updated: 2026-10-11
updated: 2026-10-14
decisions:
- "ADR 0292"
- "ADR 0283"
@@ -13,6 +13,7 @@ decisions:
- "ADR 0236"
- "ADR 0227"
- "ADR 0256"
- "ADR 0368"
---
# 48 — A module says how it is healthy
@@ -200,7 +201,7 @@ build alike, so any build of the module is excused while the wait checks out.
**Each module of a send keeps its own pass** (ADR 0254). A send is still judged as one, with one verdict.
The gate also counts, per module, the judgings in a row that found it healthy or waiting for a person;
when the send fails, a module whose count reached the passes asked keeps a pass, said "on its own", and is
neither put back nor left as a move no gate has seen for every other walk to wait on. A module found broken
neither put back nor left as a move no gate has seen for every other rollout to wait on. A module found broken
is marked failed and put back at once, and the send's verdict waits until the modules beside it have their
own, or the bound, so every module of a failed send leaves it with a verdict: its own pass, or put back.
@@ -246,6 +247,11 @@ The runtime restarts what exits, as now. The node-engine never restarts, recreat
being unhealthy. The condition reaches the operator through the conversation (to-be 46), or a healer once a
record gives one that act.
One module repairs itself, by its own record: the dnsmasq module's watch restarts the resolver's unit when
it stops answering, at most once every two minutes and three times an hour, and says through its declared
health whether the restart helped (ADR 0368). The node-engine still restarts nothing; any other restart on
health needs a record of its own.
## 8. A declaration is proved before it is trusted
- **`module check`** refuses a `health` field naming an endpoint the module does not declare under
@@ -520,7 +526,8 @@ controller.
## What is not decided here
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231).
- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231). The resolver's own
restart is decided, for that module alone, by ADR 0368.
- Health of scheduled work, and of what a module's events do (issue 276).
- A slow starter that needs more than five minutes to be ready — a change to the bound, recorded, when one
exists.
+1 -1
View File
@@ -31,7 +31,7 @@ shape — what each is for, where its records live, and how they depend on one a
| **Placement** | to decide, send and apply what each node runs, and say why | the controller's `inventory` database |
| **Provisioning** | to resolve which module serves a provision for which consumer, and wire the two | the controller |
| **Identity and access** | to issue, hold, rotate and check credentials and grants | the controller's `identity` and `licences` databases, and the vault |
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and walk |
| **Change and delivery** | to take one commit from its pull request to every node that should run it, and put it back when it fails | the `mesh-delivery` module, and the controller's planner and rollout |
| **Health and repair** | to notice what is wrong, say it once, repair what may be repaired unattended, and record what was done by hand | the controller's condition store |
| **Data** | to know every item of data a module holds, how precious it is, and keep it recoverable | the manifests' data declarations, and every node's `node-backup` seat |
| **Connectivity** | to make every node and module reachable by name where it should be, and unreachable where it should not | the controller |
@@ -1,8 +1,8 @@
---
layer: to-be
status: designed
code: []
updated: 2026-10-10
status: in-progress
code: [mesh-controller]
updated: 2026-10-17
decisions:
- "ADR 0289"
- "ADR 0201"
@@ -12,7 +12,9 @@ decisions:
# 54 — The controller's records on the bus
**What each node runs, how far each walk has got, what each build is doing and what each setting holds
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: mesh-controller's record mirrors (internal/recordmirror, cmd/mesh-controller/record_mirrors.go, internal/inventory's record mirrors), the mechanism specified here. Not yet checked mirror by mirror; not broken down into milestones.
**What each node runs, how far each rollout has got, what each build is doing and what each setting holds
can be watched on the bus, and what is watched never says something the controller's store does not**
(ADR 0289,
from gap G2 of research 036).
@@ -51,7 +53,7 @@ do not have: a check that the bucket says what the store says.
Each value is built from a named list of fields. A field not on the list is never in the value. A value
holds only what is built from its record: no epoch, no time of writing and no time of the last look,
because any lease holder must build the same bytes from the same record, and the audit must too. Times
that belong to the record itself, such as when a walk was opened or a resource's state began, are fields
that belong to the record itself, such as when a rollout was opened or a resource's state began, are fields
like any other. The epoch and the time of the write travel in the event.
### nodes — `mesh-controller_nodes`, key: the node's name
@@ -74,15 +76,15 @@ like any other. The epoch and the time of the write travel in the event.
| the build the node was last sent for it and the build its node reports running | — |
| the seats it holds there, and the provisions it is bound to | the bindings' credentials, which are secrets |
### walks — `mesh-controller_walks`, key: the walk's id
### rollouts — `mesh-controller_walks`, key: the rollout's id
| Holds | Leaves out, and why |
|---|---|
| the repository and commit, or the batch of merges it walks (ADR 0276) | — |
| the repository and commit, or the batch of merges it rollouts (ADR 0276) | — |
| its state (building, rolling, done, failed), its current tier, and each tier's modules and nodes with each step's state | the build and apply logs, which the `plans` and `builds` verbs answer |
| when it was opened, when it last moved, when it ended, and its note | — |
A walk is removed from the mirror 14 days after it ended.
A rollout is removed from the mirror 14 days after it ended.
### builds — `mesh-controller_builds`, key: the build's id
@@ -137,8 +139,8 @@ called).
caller today): a numbered migration of `inventory` and the consolidated schema, in the same change.
- **A removal is a purge** of the key, said as a `removed` event. A purge leaves one marker for the key.
Each purge gives its marker a time to live of one day (a per-message time to live, which the bus server
has offered since release 2.11 and the bucket enables), so the markers of removed walks and builds do
not pile up in a bucket that has no age. The controller's one-minute tick removes walks and builds 14
has offered since release 2.11 and the bucket enables), so the markers of removed rollouts and builds do
not pile up in a bucket that has no age. The controller's one-minute tick removes rollouts and builds 14
days after they ended.
- **A write that fails** sends no event, is logged, and is left to the audit. The writer does not retry
in a loop: a retry loop against a refusing bus is the reconnect loop of issue 327 in another place.
@@ -169,7 +171,7 @@ counts in §6 shows the gap.
**D17, the mirrors match their store**, is a probe in the self-check's registry (to-be 45 §4). Every run
does this for each mirror:
1. It reads every record the mirror should hold from the store (for walks and builds, those open or ended
1. It reads every record the mirror should hold from the store (for rollouts and builds, those open or ended
within 14 days) and builds each value as the writer would.
2. It lists the bucket's keys and reads their values.
3. It compares. A key missing, a value that differs, and a key with no record are each a candidate. A
@@ -225,10 +227,10 @@ remain open there.
| The bucket matches the store | D17 on every self-check run; its own test, which changes a key behind the writer's back and expects `mirror-drifted`, then H6's write, then the condition cleared on the next run; and a test that a change of lease alone raises nothing |
| Quiet means no writes | live: over ten minutes in which `plans`, `nodes`, `builds` and `settings` show no change, no mirror's write count rises |
| A watcher sees a change | live: a setting changed through `settings` is seen by a watch on `mesh-controller_settings` and as one `setting-changed` within a second |
| A value is under its cap, or cut and marked | the controller's test, which builds the largest walk the mesh could make (every module on every node) |
| A value is under its cap, or cut and marked | the controller's test, which builds the largest rollout the mesh could make (every module on every node) |
| Only the audit reads a mirror | a test over the controller's packages, which fails when any package but the audit's reads a mirror bucket |
| A write follows the commit | the controller's test that a rolled-back transaction makes no write and sends no event |
| Ended walks and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the walks bucket's keys including deleted ones at most its live keys plus a day's removals |
| Ended rollouts and builds leave after 14 days, and their markers after a day | D17, which treats a key past its 14 days as one with no record; the controller's test over a real bus with a short marker age; live, `nats.nats_buckets` shows the rollouts bucket's keys including deleted ones at most its live keys plus a day's removals |
| A lost event is said | the controller's test, in which the bus refuses an event after the write, expecting `mirror-event-lost` on the next D17 run |
| A former lease holder cannot overwrite | the controller's test with two writers, the second refused by compare-and-set |
| A lost acknowledgement is no conflict | the controller's test in which the bus stores a write and drops its acknowledgement: the next write to that key succeeds and no condition is raised |
@@ -236,7 +238,7 @@ remain open there.
## What it leaves out
- **History.** A mirror is current state. What happened beyond the 14 days a walk or build is kept, and
- **History.** A mirror is current state. What happened beyond the 14 days a rollout or build is kept, and
beyond EVENTS' week, is gap G8 of research 036, which remains open there.
- **People reaching the mirrors**, through a WebSocket and an auth callout, is gap G1, open in the same
effort.
+119 -11
View File
@@ -1,15 +1,19 @@
---
layer: to-be
status: designed
code: []
updated: 2026-10-12
status: in-progress
code: [mesh-catalog, hq]
updated: 2026-10-18
decisions:
- "ADR 0290"
- "ADR 0321"
- "ADR 0423"
- "ADR 0424"
---
# 55 — The knowledge base
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built and live since 2026-10-11: the knowledge base is the module mesh-knowledge in mesh-catalog, holding every decision record and research effort (issues 425 and 426, resolved and delivered). Left: the later slices, issues 428 and 429.
**The mesh keeps its decision records and research efforts whole in a module of its own, `mesh-knowledge`, one
instance on the control-node, holding the seat `knowledge-base`: each record's number, status, topic, typed links,
its clauses with their own status, and its reasoning, with every version kept. A record is proposed, checked when it
@@ -26,7 +30,8 @@ gets, in one call under 4 KiB, the clauses in force, the records they come from,
read the reasoning behind each.
This document designs the first slice in detail, tracked by issue 425. The slices after it are in
[§ 9](#9-the-slices-after-the-first).
[§ 9](#9-the-slices-after-the-first), which also designs the third: designs and the glossary's entries become records,
and the knowledge base works out each design's state (ADR 0423, ADR 0424).
## 1. The parts
@@ -39,9 +44,12 @@ This document designs the first slice in detail, tracked by issue 425. The slice
├ verbs ask, get, holds, links, history, reserve, propose, import, return
│ served by the control-node's tool runner
├ events proposed, reviewed, accepted, changed, imported, drafted, confirmed, returned
└ consumes the router's warrants: the operator's Approve for a record that needs it (§ 5)
├ consumes the router's warrants: the operator's Approve for a record that needs it (§ 5)
└ consumes the delivery module's delivery-done announcement: a design's state (§ 9)
mesh-issues (control-node) the counters for decision and research numbers (§ 6)
mesh-issues (control-node) the counters for decision and research numbers (§ 6);
the issues' design links, read for a design's state (§ 9)
mesh-delivery (its seat) announces each delivery done, on the bus (§ 9)
claude-code (same node) claude_code_judge: review a proposal (§ 5), draft and confirm clauses (§ 7)
records (any node) records_read at a commit: the import's source (§ 8)
messenger (control-node) the router: the operator's ask and its warrant
@@ -55,9 +63,11 @@ the records module, as the issue tracker's import does.
## 2. What one record holds
- `number` and `kind` (decision or research), its key together; `title`, `topic`, `status`, `date`, `deciders`;
- `number` and `kind` (decision or research; with the third slice also design and glossary entry, § 9), its key
together; `title`, `topic`, `status`, `date`, `deciders`;
- its **reasoning**, in sections: for a decision, context, the options weighed, consequences, how it is checked,
references; for a research effort, its overview and each of its documents by name; markdown, as written;
references; for a research effort, its overview and each of its documents by name; for a design, its `to-be`
section and, where something of it runs, its `as-is` section (§ 9); markdown, as written;
- for a decision, its **clauses** (§ 3);
- its **links**, typed, each stored once and read from both ends;
- its **versions**: every change is a new version with who made it, when, and why in one line; `history` lists them
@@ -130,7 +140,7 @@ rules that judge the whole record set run (an accepted decision cited by nothing
undo the acceptance. It is a fault of the records, which their authors repair, not of the module: it is told to the
operator once, as a warning naming the rule and the records, and cleared once the rule holds again. The module's
health names it and stays healthy, because health says only whether the module works, and a finding never holds a
walk at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
rollout at the module's first-node gate (the operator's decision of 2026-10-12 on issue 504, as issue 508 decided for a proposal never
reviewed).
The rule `cited` is also judged ahead, in the review, before the judgement: a decision that names no home and that no
@@ -190,11 +200,89 @@ field by field and section by section, and the result is written on issue 426.
|---|---|---|---|
| 1 | this document: the module, the import, clauses, answers, `propose`, its own review and acceptance | the acceptance sentence above, live, and a record proposed, reviewed and accepted without a pull request | issue 425 |
| 2 | the decision records and research efforts leave this repository; path mentions become "ADR NNNN" and "research NNN"; playbooks 01 and 02 changed | this repository holds no decision record, and `get` answers every one | issue 426 |
| 3 | designs, playbooks and glossary entries as kinds | `holds` answers for a design's decisions and a glossary word | issue 428 |
| 3 | designs and the glossary's entries as records, moved in one delivery, and each design's state worked out by the knowledge base (§ 9.1 to § 9.3; ADR 0423, ADR 0424) | `get` answers design 47 with its sections and links, a word's entry answers from the knowledge base, and a design's state follows its issues' deliveries without anyone writing it | issue 428 |
| 4 | meeting notes, drawings and blog posts | not decided; needs its own decision | issue 429 |
Numbers from the issue tracker, which `reserve` needs, are the tracker's own slice, tracked by issue 424. This
repository keeps no copy of the records, generated or otherwise (issue 435, answered by the operator).
repository keeps no copy of the records, generated or otherwise (issue 435, answered by the operator). Playbooks
stay in this repository as files: they are how work is done here, not records of what was decided (research 061).
### 9.1 Designs and glossary entries as records
**A design** is a record of the kind `design`, named "design NN" and keeping its number (ADR 0423). Its text is its
sections: the `to-be` section, what is being built, and the `as-is` section, what runs. Each of the as-is documents
becomes the as-is section of the design it describes, never a record of its own: the import matches 3 of the 18 by
name and 15 by topic, and shows the operator the list before it runs. Its owning repositories are its field `code`;
its decisions, and every design it names, are `cites` links. It holds no state written by anyone (§ 9.2). A new
design takes its number from `reserve` and is written with `propose`, and is reviewed as every record is (§ 5).
**A glossary entry** is a record of the kind `glossary entry`, one per entry of the glossary page, kept whole: its
head word or words (an entry may name several with one meaning, as research effort, graduation, hand-off and playbook
do), its domain, its meaning, its *Not:* line with each retired word's scope, and its identifier line. Each row of
the page's homonym table becomes an entry of its own, naming the qualified form each domain uses. The entries replace
the page as the authority on words (ADR 0423, changing ADR 0244).
**The readers change with them.** Everything that read the page or the design folders reads the records instead:
| Reader | Reads, after the move |
|---|---|
| the knowledge base's own rule on a retired word (§ 4) | the *Not:* and identifier lines of the entries, no longer the page through the records module; it judges a design when it is proposed, as it judges a decision |
| hq's word check | the entries' *Not:* and identifier lines, and its uniqueness check (a head word heading two entries, or also retired) over the entries; it covers what stays in this repository, and designs leave its coverage |
| the catalogue's repository check | the retired words of no scope or *(tools)* from the entries; its copied list is retired, so hq has no copy to compare |
| hq's cycle check | the design records, for every design naming its decisions, not files under `03-DESIGN/` |
Each of the four fails, saying so, when the knowledge base does not answer, and never passes on nothing.
**Links become links between records, and the old paths go.** At the import every file link inside a design becomes
a link between records, which the rule on a link to nothing then checks (§ 4). The links from documents that stay in
this repository are rewritten to name the design ("design 47"). No page stays at an old path: afterwards this
repository holds no file under `03-DESIGN/` and no `00-META/glossary.md`.
**It moves in one delivery**, as the records did: the import, the two kinds, the four readers, the rewritten links,
the playbooks' steps that write a design, and the instruction text sessions read, which names the glossary's place.
A reader left behind would pass on nothing, so none moves alone.
### 9.2 A design's state
**The knowledge base says a design's state, and nobody writes it** (ADR 0424): proposed, designed, in progress or
implemented, kept matching what runs. No person, session or document sets it, and `propose` refuses a design that
carries one.
It reads two things. The issues are those the issue tracker holds with a `design` link to the design. Whether one
is done comes from its deliveries: an issue is done when it is resolved and the delivery its fix went out in reached
`delivered`, whether a hotfix alone or a delivery group. The delivery module announces each delivery done on the
bus, and on each announcement the knowledge base works the state of every design that issue links out again.
```
issue tracker ── issues with a design link ──┐
├──▶ knowledge base ── works out ──▶ the design's state
mesh-delivery ── "delivery done", bus ───────┘ (served by get, ask)
```
**The rules.**
| Case | State |
|---|---|
| a decision the design names is not accepted | proposed |
| no linked issue done, or no linked issue at all | designed |
| some linked issues done | in progress |
| every linked issue done | implemented |
Three kinds of issue count apart: an issue closed won't fix is left out; an issue resolved with nothing to deliver,
a fix of documents only, is done when resolved; and a new issue linked to an implemented design makes it in progress
again, until that issue is done. Milestones are not read: a milestone is a group of a design's issues delivered
together, and the state depends only on the issues and their deliveries.
**Imported designs say what they lack.** A design imported with no linked issue shows `no issues linked yet` beside
designed, until an issue is linked to it; the import names each such design by number. Linking their issues is
tracked by issue 428.
### 9.3 Its place beside design 47
The delivery-done announcement this slice reads is design 47's
([47 — Delivery](47-delivery-from-commit-to-delivered.md)): its hotfixes and delivery groups, and each delivery's
end at `delivered`. This slice is an extension after design 47's last milestone. It changes none of design 47's
milestones and nothing of how a delivery runs; it only listens for the announcement.
## 10. How it is checked
@@ -212,6 +300,26 @@ repository keeps no copy of the records, generated or otherwise (issue 435, answ
| A record written in this repository meanwhile writes its clauses | `records.py`, check `clauses`, on every hq pull request, until the records left this repository (issue 426); since then `records.py`, check `no-records`, refuses any record written here |
| Live | the acceptance sentence, asked from a session outside this repository |
**The third slice's acceptance tests** (§ 9.1 to § 9.3; research 062's direction), what its build proves before it
is done:
1. After the import, `knowledge-base.get` of design 47 returns its sections and its links, and every link from a
design resolves to a record: none is refused by the rule on a link to nothing.
2. This repository's main holds no file under `03-DESIGN/` and no `00-META/glossary.md`, and its link check passes
with the 76 links from documents that stay here rewritten.
3. hq's cycle check and word check read the knowledge base, and fail, saying so, when it does not answer.
4. A proposal holding a retired word is refused after the move, the word read from the entries; the entries are
175 and hold 183 head words and 29 retired words, 0 unparsed, both counts checked.
5. In the module's tests, each state rule of § 9.2 gives its state from fixture issues; live, an issue linked to a
design whose hotfix is delivered turns the design implemented within a minute of the delivery being done, and
a new issue linked to it turns it in progress.
6. The import reports every design with no linked issue as `no issues linked yet`, by number.
7. The instruction text sessions read names the glossary's new place.
8. Every entry of the glossary's Config section is filed under the Config domain, and the domains the entries name
are the twelve of ADR 0354.
9. Tests 1 to 8 pass together after the one delivery, run against main and the live knowledge base, with nothing
in this repository, the catalogue or the instruction text pointing at a removed path.
## What it leaves out
Semantic or vector search; a browser view; line comments on a proposal (the judgement's findings name the section and
@@ -1,8 +1,8 @@
---
layer: to-be
status: designed
code: []
updated: 2026-10-13
status: in-progress
code: [mesh-catalog, mesh-controller]
updated: 2026-10-17
decisions:
- "ADR 0316"
- "ADR 0317"
@@ -12,6 +12,8 @@ decisions:
# 56 — The mail client, set up by the mesh
**Where it stands, 2026-10-17** (the records cleanup, issue 1072; playbook 07, "The design's state is written back"). Built on main: issue 586, this design's core, is resolved and delivered (hq#336, mesh-catalog#262, mesh-controller#286), and mesh-catalog's betterbird module has every piece named here. Not yet proven live end to end, nor given an as-is: issue 1076.
**On every node that holds a display server, the `betterbird` module installs Betterbird from a pinned
upstream release, ships the Owl add-on with it through Betterbird's enterprise policy, and declares the
operator's mail accounts without their credentials. A password or token never passes through the mesh: each
@@ -0,0 +1,363 @@
---
layer: to-be
status: in-progress
code: [mesh-controller, mesh-host, mesh-sdk, mesh-catalog]
updated: 2026-10-17
decisions:
- "ADR 0348"
- "ADR 0349"
- "ADR 0201"
- "ADR 0113"
- "ADR 0158"
- "ADR 0360"
- "ADR 0362"
---
# 57. Building config and provisioning
**The work of ADR 0348 (config is a domain of its own) and ADR 0349 (provisioning), broken into
packages small enough that each ends at something that can be seen to work, in the order their
dependencies allow.** The two decision records are the authority on *what* is built; this document holds
only the packages, their order, their sizes and their proofs, and is wrong the moment it disagrees with
them. It is the shape [design 38](38-building-the-operators-machine.md) gave the operator's machine.
## How this is built
**Each package is one issue in the issue tracker, and one pull request per repository it touches.** The
issue carries the package's number here in its title, links the packages it waits for with `depends-on`,
and links ADR 0348 or ADR 0349. At most five packages are worked at once, each by its own agent in its
own worktree; a package starts once every package it depends on is merged. Every pull request is
reviewed independently at its head and merged only with its checks green.
**It ships as one release.** The operator's decision of 2026-10-14: one delivery, not one per package.
Each of mesh-controller, `mesh-host`, mesh-sdk, mesh-catalog and hq has the branch
`feat/57-config-and-provisioning`, and every package's pull request merges into that branch, never into
main. When the last package is merged, the whole is rehearsed in the lab: the rollout, the moves of data in
their order, and every proof that is run live. Then each repository's branch reaches main in one pull
request, and the five, sharing their branch's name, are one delivery group (ADR 0239 §2), judged as one
future state. The five are merged within one merge window, so they are one batch and one rollout (ADR 0276
§1, §2), in an order declared with `after:` lines in each pull request (ADR 0239 §11): mesh-sdk, then
`mesh-host`, then mesh-controller, then mesh-catalog, then hq. The release holds every package but T6, R1 and R2:
the media catalogue and the applications live in repositories of their own and move after it, and only
then can the old forms be refused (R1) and the record say what runs (R2).
**The moves of data run after the delivery, in this order, each a verb that says what it did:**
placement out of the settings layers (F2), the settings layers into mesh-config (K5), the secrets into
the vault, nothing rotated (V3), then the providers one by one (B2, B3). Each is rehearsed in the lab
first, and each is checked by reading back through the module that owns the data before anything
points at it.
**What a release costs.** A proof that is run live (dunst on a workstation, a secret reaching its file, a
database consumer connecting) is first seen in the lab rehearsal, and a fault found then holds the
release, not one package. A merge's builds go to every machine at once (ADR 0360), so the release has no
first node to hold it: after the pull requests' checks, the lab rehearsal is the only live check before it
reaches every node. A machine that fails to apply it, or whose provisioner reports a module's files failed
(ADR 0362), is named in the rollout's end and raises apply-failing, and is fixed forward with a new pull
request into main; one that says nothing within fifteen minutes is named silent and gets its builds from
the backlog rollout once it is heard from (ADR 0360 §10).
**Nothing breaks on the way.** ADR 0349 clause 17: each file is filled the old way or the new way, never
both, by the form of its manifest, and the old forms stay accepted until the last module has moved.
Inside the release this still holds: the media catalogue and the applications kept in their own
repositories move after it, so the old forms stay accepted until they have (R1).
**Where it stands, 2026-10-17** (the Milestones section below). The packages' branches were merged into
main of mesh-controller, the node-engine (`mesh-host`) and mesh-catalog on 2026-10-16. T6 moved after them, as
planned above, and is resolved. B3 was left out as well, so B3, R1 and R2 are what is left.
## What exists already, measured
Counted 2026-10-14 at mesh-controller `1601cb1`, the node-engine (`mesh-host`) `1a26778`, mesh-sdk `a2d2af0`, mesh-catalog
`ebf7f37`.
| Piece | Today | Becomes |
|---|---|---|
| the controller's composer | `catalogue.Resolution.compose` fills every namespace but `${secret:}` in one loop (`declaration.go` ~1000–1085); the namespaces and their fields are tabled in `fillers` | fills placement only, for a module that has moved |
| settings | tables `settings`, `settings_history`; the `settings` verb (`modules.go` 407–760); proposals and asks (`proposals.go`, 975 lines); trusted files; `takes-settings` merging; about 4 100 lines and 2 500 of tests | the `mesh-config` module and its buckets; the verb `config` |
| placement kept as settings | `places`, `accesses` and the kept networks are read from settings layers | facts of an assignment, the controller's own |
| secrets | the controller mints, seals and accepts every secret (`internal/secrets`, `inventory/secrets.go` 1 125 lines; tables `secret`, `module_secret`, `secret_ask`, `operator_key`); the vault (`mesh-vault`, TypeScript) is read-only and never asked | the vault, in Go, holding and minting; handing out on request and reply |
| bindings | grant files and binding files composed by the controller (`declaration.go` 750–818); `${bound:}` 117 uses | each provider answers its consumers; the vault mints the pair's credential |
| contributions | `${contribution:}` (4 collectors), `${shell:}` (3), `${environment:}` (1); 22 contributors | collections, filled by the provisioner client |
| file text | 225 file resources in 88 modules, all inline (about 193 KB); 12 modules keep a second copy in 34 files with a test that the two agree (ADR 0349's context counted eleven; the twelfth is lemurs) | template files under `files/`, packed by the module's build |
| packing a directory | the builder's `archive` artifact packs a tree and publishes it by digest; the node-engine fetches and unpacks it | the same mechanism carries a module's templates |
| the node-engine's file writing | `applyFile`, `applyBlock`, `applyInto`, create-once, drift written back, `Keep` backups, restart-on and reload-on, container reads | the same code, reached also through a narrow local write verb |
| the node-engine's local verbs | one socket (`cli.sock`), peer-checked by uid, forwarding to the controller; no local write | a second, narrow socket for the provisioner client |
| bus identities | one principal per node for the tool runner, one per module with a process of its own | the provisioner client gets one of its own |
| the SDK | KV, watches, asks, caller proof; no sealing helper; the TypeScript `provisioner` is a database-provider harness | a seal box, provider helpers, and readers for config, bound values and secrets |
**Two measurements decide the order.** The builder can already pack a module's directory, and the
controller already fills every placeholder at registration time from text; so template files can ship
first, the controller reading the text back from the packed `files/`, and nothing on a node changes. And
the node-engine's writing code needs no change in kind, only a second way in; so the provisioner client
reuses it rather than writing as root itself.
## The order the work allows
```
package waits for package waits for
F1 — P1 —
F2 — P2 P1
F3 — P3 F3
F4 — P4 P3
T1 — P5 P2, P4, T1, K3
T2 T1 P6 P5
T3 T4 T5 T2 P7 P6
T6 T1 V1 F1
K1 T1 (same file) V2 V1
K2 K1 V3 V2
K3 F3, K2 V4 V3, P5
K4 K3 V5 V3, V4
K5 F2, K4 B1 P5, V5
K6 K1 B2 B1
K7 K5, P6 B3 B2
L1 P6 L2 L1
S1 P6 R1 T5, T6, K6, K7, P7, V4, B3, L2, S1
R2 R1
```
The issue tracker carries these as `depends-on` links (thirteen issues also link what they wait for
through another package), so the packages ready to start are those whose linked packages are all merged
into the release's branches. An issue's "Waits for: … delivered" means merged into the release's
branches: nothing is delivered before the release. The first wave touches different repositories and different files:
**T1, F1, F3, P1 and F4**. K1 waits for T1 because both change `internal/catalogue/manifest.go`; P3
waits for F3 because both change `internal/broker`; F2 joins as soon as a slot is free.
## 0 — Foundations
**F1 (issue 650) — A seal box in the SDK.** *mesh-sdk. Half a day.* The ADR 0206 box (X25519, HKDF-SHA256,
AES-256-GCM) moves from the licence manager into `mesh-sdk/go/seal`, with a TypeScript twin, and the
licence manager uses it. **Proof:** a byte-for-byte test against the licence manager's existing sealed
values; the licence manager's tests pass unchanged.
**F2 (issue 651) — Placement leaves the settings layers.** *mesh-controller. Two days.* `places`, `accesses` and the
kept networks become facts of an assignment, held by the controller beside the assignment, with a
migration moving each existing layer's keys; `Settle` stops letting a setting replace a provider's
served keys. **Proof:** composition of every node before and after the migration is byte-identical;
the settings layers hold no placement key afterwards.
**F3 (issue 652) — The assignments and the modules' config declarations on the bus.** *mesh-controller. Two days.*
The controller keeps the bucket of assignments design 54 decides (which module on which node), and a
bucket of each registered module's declared config values, written when they change, with one event per
write. mesh-config needs both to resolve. **Proof:** a test against a real bus: an assignment and a
registration each change exactly one key and raise one event.
**F4 (issue 653) — The words.** *hq. Half a day.* The glossary gains *config value*, *predefined key*, *provision*
and *provisioner client*, *collection*, and the ownership words *owned*, *seed* and *managed*. The word "setting" goes on *config
value*'s *Not:* line only in R2, once the 67 documents still using it are reworded; until then the
glossary says it is the older word. **Proof:** `words.py` passes.
## 1 — Template files (ADR 0349 clause 2)
**T1 (issue 654) — A file resource may name a template.** *mesh-controller. Three days; the largest package of the
stage.* A file resource takes `"template": "files/<name>"` in place of `content`; the module's build packs
`files/` into an archive artifact of its own (the existing `archive` packing); registration and `Resolve`
read each template's text back from the packed tree into `content`, so composition and the node-engine
are unchanged; `module check` and the merge gate read templates from the repository tree beside the
manifest; a file naming both `template` and `content` is refused. **Proof:** composition tests: a module
with its text in `files/` composes byte-identically to the same module with it inline; the merge gate
judges a change to a template alone.
**T2 (issue 655) — The twelve modules with copies.** *mesh-catalog. A day.* adwaita, clipmenu, dunst, feh,
gnome-keyring, i3, i3status-rust, lemurs, picom, rofi, screen-lock and xdg move to their templates; the
34 inline copies and the `sameAsSource` tests go. **Proof:** each module's composed declaration on its
nodes is unchanged (the merge gate's plan says no file changes).
**T3 (issue 656) — The desktop modules.** *mesh-catalog. A day.* Every remaining module assigned only to
workstations. **Proof:** as T2.
**T4 (issue 657) — The servers' modules.** *mesh-catalog. Two days.* Every remaining module with a build. **Proof:**
as T2.
**T5 (issue 658) — The ten modules without a build.** *mesh-catalog. A day.* ca-trust, de-spiegel, invoicing,
local-model-consumer, matrix, node-env, only-office, sshd, step-ca and supabase gain an archive build for
their templates. **Proof:** as T2; afterwards no manifest in the catalogue carries file text.
**T6 (issue 659) — The media catalogue and the applications.** *mesh-media-catalog and each application's
repository. Two days.* **Proof:** as T2, per repository.
## 2 — Config (ADR 0348)
**K1 (issue 660) — A module declares its config.** *mesh-controller. Two days.* The manifest takes `config` (type,
description, validation, trusted, `predefined`, `default` or `required`), `takes-config` and
`${config:<key>}`, with `settings`, `takes-settings` and `${setting:}` read as aliases; registration
refuses everything ADR 0348's "How it is checked" lists for registration: a value without type or
description; a value with neither a default nor `required`; a default failing its validation; an
object or list not trusted; a predefined key not on the list or of another type (with K2); an untrusted
value falling back to a trusted key; a value neither in a template nor read at runtime; a placeholder
naming an undeclared value; a `reload-on` or `restart-on` naming an undeclared file; a moved module's
mergeable file not naming its keys; a mesh word or `puid`/`pgid` in `takes-config`. **Proof:** the registration refusal tests, one per
rule; every manifest in the catalogue still registers.
**K2 (issue 661) — The list of predefined keys.** *mesh-controller and mesh-catalog. A day.* `predefined.json` at the
catalogue's root, with `font-size` and `agent-managed-settings` first; the build validates it and every
manifest naming a key; the merge check refuses removing a key still named or holding a value; the newest
trunk build's list is published. **Proof:** the merge check refuses a planted removal; an older build's
list does not replace a newer one.
**K3 (issue 662) — The mesh-config module.** *mesh-catalog. Four days.* A Go module holding the seat `mesh-config`:
the buckets `mesh-config_values`, `_resolved`, `_predefined`, `_proposals`; the four sources resolved from
F3's assignments and declarations; the verb `config` (set, clear, list, history, narrowed by module and
node); validation; the hand-act log; `config-changed`; only its account writes its buckets. Terminal-only
for trusted values until K4. **Proof:** its tests resolve each source in turn; a test against a real bus
refuses a put from another account.
**K4 (issue 663) — Trusted values are proposed and approved.** *mesh-catalog. Two days.* mesh-config asks under its
own name, with its own grant and cap of open asks, binds each option to the level, key and value, records
the approver beside the value, and checks the proposal on Approve (ADR 0277's rules, ADR 0348 clause 8).
**Proof:** a proposal test end to end with a stand-in channel; an Approve against a changed value is
refused.
**K5 (issue 664) — The values move, and `settings` forwards.** *mesh-controller and mesh-catalog. Two days.* Each
existing layer moves into mesh-config (a node layer to `module.<node>.<module>`; a whole-mesh layer to
mesh config, a manifest default or each node, as ADR 0348's consequences say; claude-code's managed
settings to `agent-managed-settings`), recorded with why; for a module not yet handed to the provisioner
the controller reads `mesh-config_resolved` where it read its layers (ADR 0349 clause 17's old way,
an exception to ADR 0348 §9 that R1 removes), and `settings` answers by forwarding to `config`. **Proof:** composition of
every node before and after is byte-identical; `settings` and `config` list the same values.
**K6 (issue 665) — The catalogue says config.** *mesh-catalog. A day.* The 15 modules using `${setting:}`, the 9
declaring settings and the 4 with `takes-settings` move to the new names and the full declaration.
**Proof:** the catalogue's check finds no old name in a moved module.
**K7 (issue 666) — The operator's acceptance on dunst.** *live. An hour.* With every level cleared the first
workstation's dunstrc says 10; a mesh font size, then that node's font size, then dunst's own value on
that node each appear in its dunstrc within the call's answer, and the history shows each change with who and why. Needs P6.
## 3 — The provisioner (ADR 0349 clauses 3–5, 11, 13)
**P1 (issue 667) — The node-engine knows who owns a placed file.** *`mesh-host`. A day.* `declaration.File` gains
`owner-at-runtime` (`owned`, `seed`, `managed`), mapped onto today's paths (drift written back,
create-once, `into`), and the declaration gains the list of files the provisioner fills. **Proof:** the
apply tests: each ownership behaves as its old form did.
**P2 (issue 668) — The narrow local write verb.** *`mesh-host`. Two days.* A second unix socket accepts a write only from
the provisioner client's account (`SO_PEERCRED`) and only for a path the declaration names as filled,
writes through `applyFile`, `applyBlock` or `applyInto` with `Keep`, and fires restart-on, reload-on and
container reads as an apply does. **Proof:** tests: an undeclared path and another uid are refused; a
write under `reload-on` reloads its program once.
**P3 (issue 669) — The provisioner client's own bus account.** *mesh-controller. A day.* A principal of its own per
node for `node-provisioner`, granted to read its node's keys in each domain's bucket, publish
`prov.applied.<node>.>` and ask the vault; nothing else. **Proof:** the grants test, and a test against a
real bus refusing a read of another node's keys.
**P4 (issue 670) — The provisioner's server.** *mesh-catalog. Two days; after P3.* A Go module holding `mesh-provisioner` once:
the registry of which domain answers which kind, the buckets `_registry`, `_needs`, `_applied` and
`_collect`. **Proof:** its tests; the buckets listed live.
**P5 (issue 671) — The provisioner's client.** *mesh-catalog. Four days.* A Go service on every node, outside the tool
runner: fetches a module's templates by digest, records needs, watches each owner's keys for its node,
fills only when every value is there, writes through P2, reports `prov.applied`, writes an `owned` file
back on drift, removes a module's files and needs when it leaves, fills from current values when its node
comes back. Its first owners are placement (from the declaration) and config (from
`mesh-config_resolved`). **Proof:** the client's test list in ADR 0349, against a real bus.
**P6 (issue 672) — The controller hands a module to the provisioner, and the rollout waits for it.** *mesh-controller.
Two days.* For a module marked moved, the declaration carries placement, the templates' digest and the
filled-files list, and the controller runs no other filling pass for it; a rollout counts the module applied
on a machine only after the node-engine's report and that node's `prov.applied` for the new build, and a
failed `prov.applied` is a failed apply (ADR 0362). dunst moves first. **Proof:** composition and rollout
tests; dunst on the first workstation filled by the client, live.
**P7 (issue 673) — The node-engine stops writing placed files from the declaration.** *`mesh-host`. A day.* For a moved
module the engine writes its filled files only through P2, and removes as an orphan only what the verb
placed. **Proof:** the apply tests; no placed file written twice in a day's journal on the
first workstation.
## 4 — Secrets (ADR 0349 clauses 6–11, 16; issue 647)
**V1 (issue 674) — The vault in Go, holding secrets.** *mesh-catalog. Four days.* `mesh-vault` is rewritten in Go:
its own store, a copy of each secret sealed to the operator key, `get` on request and reply sealed to the
asker's key (F1's box; the node-engine's key format is settled here and written down), `secret-changed`
without a value, no secret in any bucket. **Proof:** the vault's test list in ADR 0349.
**V2 (issue 675) — The vault's verbs.** *mesh-catalog and mesh-controller. Three days.* `secret accept`, `secret-ask`
(with ADR 0277 §24–§27's record and bounds in the vault's store), `issued` (ADR 0305, the provider's tool
answering the vault alone) and `rotate` become the vault's; the controller's verbs forward. **Proof:** the
`secret-ask` and issuing provider tests in ADR 0349.
**V3 (issue 676) — The secrets move to the vault.** *mesh-controller and mesh-catalog. Two days.* Every row of
`secret`, `module_secret`, `secret_ask` and the operator key's public half are copied into the vault as they are,
nothing rotated, and read back through the vault before anything points at it. **Proof:** for every
secret, the vault's sealed answer opens to the same value the controller's does.
**V4 (issue 647) — A node's secrets come from the vault.** *mesh-catalog, mesh-controller, `mesh-host`.
Three days.* The client fills `${secret:}` from the vault's `get`; for a moved module the controller seals
nothing; the core's credentials travel sealed by the vault to their node and the node-engine opens them.
**Proof:** a secret rotated in the vault reaches its file without a new declaration, live.
**V5 (issue 677) — The vault mints pair credentials.** *mesh-catalog. Two days.* One value per provider assignment and
provision, remade for all holders when the vault sees a consumer's need recorded or deleted (ADR 0158
§4–§6). A need recorded for a bind the vault already holds from V3 adopts that value and remakes
nothing, so the move itself rotates no credential. **Proof:** the provider test in ADR 0349, with a stand-in provider.
## 5 — Bindings (ADR 0349 clause 12)
**B1 (issue 678) — A provider's helper in the SDK.** *mesh-sdk. Two days.* Go: watch the needs for a kind, write one
entry per consumer, refuse a derived value for a multi-holder consumer, answer from the vault's pair
credential. The TypeScript `provisioner` harness is renamed so the word means one thing. **Proof:** its
tests with a stand-in bus.
**B2 (issue 679) — PostgreSQL answers its consumers.** *mesh-catalog and mesh-controller. Three days.* The first
provider moves; its grant files and binding files are retired. **Proof:** every database consumer on
every node connects after the move with the credential it had, adopted by the vault as V5 says, live.
**B3 (issue 680) — The other providers.** *mesh-catalog. A week, one pull request per provider.* **Proof:** as B2, per
provider; afterwards the controller composes no grant file.
## 6 — Collections (ADR 0349 clause 13)
**L1 (issue 684) — A collection is declared and filled by the client.** *mesh-controller and mesh-catalog. Three days.*
A collector requires a collection; a contributor provides an item from a template; the client fills the
collector's template from `_collect`. `${contribution:}`, `${shell:}` and `${environment:}` read as
collections. **Proof:** zsh's file holds every placed contributor's item and loses one whose contributor
leaves.
**L2 (issue 681) — The collectors and contributors move.** *mesh-catalog. Two days.* zsh first, then the bar, the
display session, the hotkeys, xorg, power and node-env, with their 22 contributors. **Proof:** each
collector's composed file is unchanged at the move.
## 7 — A module's code asks at runtime (ADR 0349 clause 14)
**S1 (issue 685) — The SDK's readers.** *mesh-sdk and mesh-catalog. Two days.* `config`, `bound` and their watches,
answered by the node's client for what the manifest declares. `secrets` refuses until the module runs
under a bus account of its own, which waits on issue 569 and its decision. **Proof:** the SDK's test in
ADR 0349; claude-code reads its managed settings through it.
## 8 — The old forms go (ADR 0349 clause 17)
**R1 (issue 682) — Refused and removed.** *mesh-controller, `mesh-host`, mesh-catalog. Three days.* Once every module of
every catalogue has moved: the catalogue's check refuses inline text, `settings`, `${setting:}` and the
old contribution forms; the controller's filling passes but placement, its settings, secret and binding
code and tables (a new migration) go; the node-engine keeps opening sealed values for the core's files
only; the controller stops reading `mesh-config_resolved`. **Proof:** the controller's test that its
code opens no secret outside genesis and reads no config value; the absence of the
removed code.
**R2 (issue 683) — The record says what runs.** *hq. Two days.* The as-is designs are written, designs 27 and 32 are
amended, the 67 documents that still say "setting" in running prose are reworded so the glossary can
retire the word (F4 left it off the *Not:* line for them), design 49 lists the Config domain, ADR 0348's and 0349's built status is said, and this document's status becomes `implemented`.
## Milestones
The packages above were built as one delivery, not milestone by milestone; the milestones group them by what
each makes true. Whether a milestone's issues are delivered is read from the issue tracker (playbook 07).
| # | Outcome | Packages | Issues |
|---|---|---|---|
| 1 | Config, provisioning, secrets, bindings, collections and runtime asks run the new way; the old forms are still accepted | every package but T6, B3, R1, R2 | 647, 650 to 679 but 659, 681, 684, 685 |
| 2 | The media catalogue and the applications carry their file text as templates | T6 | 659 |
| 3 | The other providers answer their consumers; the old forms are refused; the record says what runs | B3, R1, R2 | 680, 682, 683 |
## What is deliberately not here
- **Secrets in a module's own code** wait on issue 569 (a module's tools run as the tool runner's
account) and its decision; S1 ships without them.
- **Release separated from delivery** stays issue 644's, and **the bus's replicas** issue 645's.
- **Renaming "walk"** is issue 646's.
## How this list is kept true
Each package's proof that runs in its tests is run when it is merged into the release's branch, and its
line here gains the date and the commit; a proof that runs live is run in the lab rehearsal and again
after the release's delivery. A package whose proof fails is not reworded; the failure is recorded under it and the package
stays open. The issue tracker says which packages are worked now; this document says only the order.
@@ -0,0 +1,322 @@
---
layer: to-be
status: designed
code: [mesh-catalog, mesh-media-catalog, mesh-sdk, mesh-tools, mesh-controller, mesh-host, mesh-lab]
updated: 2026-10-17
decisions:
- "ADR 0383"
- "ADR 0384"
- "ADR 0385"
- "ADR 0386"
- "ADR 0387"
- "ADR 0388"
- "ADR 0389"
- "ADR 0390"
- "ADR 0411"
- "ADR 0412"
- "ADR 0413"
- "ADR 0414"
---
# 58 — Agent work as the mesh's own
**An agent's work belongs to the mesh, not to one vendor's program or one machine. A thread is kept whole on the
bus by a thread seat; a thread handler on an always-on node runs the agent harness headless for it, in a sandbox
of its own, with the thread's own branch on the forge as its workspace; the agent works through a workspace toolkit
and named tools, never a general command runner. Every person, agent and module is an identity held by one identity
seat, with a role made of grants; the bus checks each login by its signature, and the tool runner checks each call
against the role, tool by tool, by the effect every tool carries. An identity is also its logins: the mesh makes each
person's and agent's machine account on every machine it is present on, gives each its own bus password, and runs a
module for an identity only where the module is set to; everything else runs as the operator.**
(ADR 0383 to ADR 0390, from research 053; ADR 0411 to ADR 0414, from research 057.)
**Acceptance.** The operator starts a thread from the terminal client, an agent on an always-on node works it in a
sandbox, asks a question the operator answers in place, and the thread moves to another node and carries on with
nothing lost; the agent's calls outside its role are refused, saying which grant is missing.
## 1. The parts
```
any node always-on nodes the control-node
───────── ─────────────── ────────────────
front ends nox-thread-handler nox-threads (seat thread)
├ terminal client (first) ├ takes a thread's lease ├ a restricted stream per thread
├ web page (then) ├ runs the agent harness headless ├ the object store for large content
└ Telegram: questions, │ inside a microsandbox └ leases in the bus's key-value store
approvals ├ answers permission requests
│ │ through the mesh's asks identity seat
│ └ commits + pushes the workspace ├ persons, agents, modules, roles
│ branch every turn │ in the bus's key-value store
│ nox-workspace (node seat workspace) ├ signs logins (one key per role)
│ └ read, glob, search, edit, write, └ revokes over the bus, reads back
│ git, named tools; no command runner
│ claude-code
│ ├ mirrors a session into its thread
│ ├ rebuilds a session on pick-up
│ └ writes the login and licence token
│ into the sandbox
└──────────────── every call over the bus, carrying its thread and,
for a module acting for an agent, the agent's token
```
## 2. Threads (ADR 0383)
A **thread** is one piece of agent work, kept as a stream on the bus in the mesh's own format. The stream is the
source of truth: the conversation, the agent's questions and their answers, tool calls and results, usage, and
every change of state. Large content goes to the object store and is named from the stream. The stream is
restricted (ADR 0376): only modules granted it read it. A thread is never deleted; the trail is kept for good.
A thread is **open**, **assigned** (to a node and an agent harness), **released** or **stopped**. Its seat's verbs create,
assign, release, stop, move it, and take it over with a reason. The assignment is a **lease** in the bus's
key-value store, changed only by a compare-and-set on its revision. Its holder renews it every 20 s. A lease not
renewed for 60 s is said to the operator; nothing takes it over by itself.
## 3. Running a thread (ADR 0384)
The thread handler on an always-on node takes the lease and starts the agent harness (Claude Code today) headless, one
turn at a time. It answers the agent harness's permission requests through the mesh's asks, so a front end answers them in
place. At the end of every turn it commits and pushes the workspace, which is the thread's own branch on the forge;
a node that picks the thread up fetches it, so a move loses at most the turn in progress. Before a shutdown it
releases at a turn boundary; on a stop it ends the process group, since a tool's child process outlives the agent harness.
The claude-code module mirrors the session's stream into the thread and, on pick-up, rebuilds a session from the
thread. Usage per thread comes from each run's result and rate-limit messages on the stream, not the transcript, and the
licence manager records it with the thread as its session.
The front ends are clients of a thread doing the same work: the terminal client first, then the web page, and
Telegram for questions and approvals. Their first scope is the conversation as it streams, the agent's questions and
permission requests answered in place, sending a message and stopping a turn. The rest is tracked by issue 850.
## 4. The workspace toolkit (ADR 0385)
The node seat `workspace`, held by `nox-workspace` on every thread-handler node and served through the mesh's MCP
server, works in the thread's workspace only, by relative path: read, glob, regex search, edit, write, git, and named
tools such as running the repository's tests and its merge check. It is built early. There is no general command
runner in it, and no module offers one to an agent: a command an agent needs becomes a named tool in the module that
owns it. Claude Code's own built-in tools, its command tool included, are turned off once the toolkit is in.
## 5. Identities and roles (ADR 0386)
The identity seat, held once for the mesh, owns every **identity** (a person, an agent or a module) and every
**role**, in the bus's key-value store. An identity has a name, a kind, a role and whether it is switched on. They
change only through the seat's verbs, each recorded with who asked; a module or front end whose role grants the
seat's reads may watch the store live, and it holds no secret. Only the
operator creates a person or an agent, or changes its role. A module's identity is asked for by the controller when
the module is assigned to a node, named by machine and module, with a role made from what its manifest declares it
calls.
A role holds **grants**. A grant is an effect everywhere, an effect on one seat, or one named tool, and may be
limited to named machines. Two fences check them: the bus, coarsely, by seat and module, from the role's signing
key; and the tool runner, exactly, tool by tool, in the SDK. Every call carries its thread; a module acting for an
agent carries a token the identity seat signed for that agent, and the tool runner then checks the agent's role. A
refusal names the missing grant; every call is recorded with its identity and thread. A session on an operator's
machine calls as the operator's person identity or as an agent, never through the machine's tool-runner login.
### 5.1 An identity on the machines (ADR 0411, ADR 0414)
A person or agent identity also holds a **login name**, the **machines** it is present on and, for a person, the
login name kept on a machine where it differs. The seat starts with two: the operator's person identity and `nox`, the one
agent; every piece of agent work is a thread of `nox` (§2), and a second agent identity is made only when a rule needs
another role. The agent is `nox` everywhere: its login, its forge account (the forge account `nox-agent` renamed,
keys and team kept), Keycloak, mail and the bus. A login or service account the mesh makes for the operator from now on carries the operator's identity name;
the operator's existing logins and forge account are kept as they are. On a machine where agents ran as the
operator's login, that login stays the operator's and `nox` gets a login of its own; nox's bus account is `agent.nox`. `nox` is also the
operator's terminal command (`mesh-cli`), and prose says which is meant.
An identity's login on one machine is its **machine account**. The controller composes the machine account of every
identity present on a machine into that machine's declaration, and the node-engine applies it with its `user`
resource. A login that exists is adopted under its name, uid and home; the mesh never renames or deletes one. An
agent's machine account has its own home and group, no password, no root group and no sudo rule, and the node-engine
judges on every look that it cannot become root without a person, by the ways ADR 0266 §12 to §17 list. No module declares a login for a
person or an agent: claude-code's `agent-account` resource goes, and the catalogue's check refuses a manifest that
declares one. The node record's operator account and agent account are read from the identities, no longer stated
at the controller's terminal or in the mesh-config snapshot; the machine facts `account`, `agent-account` and their
homes keep their names. A machine names an agent account only where the identity seat marks an agent as its agent
account: the control node, marked `nox`, where the guard and the licence stay as they are. The other machines are
marked with none, so the guard stays off there and the operator's sessions keep the mesh's tools, while the login `nox` serves the
modules run for `nox`. Where a person or an agent is present, its kept logins and the agent account mark change only
through a verb of the identity seat that only the operator calls, on an Approve; an assignment never sets the mark,
because an agent may call the verb that changes an assignment (ADR 0266 §4). The verb, and the controller at composition,
refuse an agent whose login on a machine would be `root`, a person's login there or a well-known service account
(ADR 0266 §5). The self-check judges every agent login,
marked or not. The seed: the operator on every machine, with the operator's existing login kept where it differs from the
identity's name; `nox` on every machine, the agent account on the control node.
```
identity seat controller each machine (node-engine)
───────────── ────────── ──────────────────────────
person operator, machines, composes, per machine: the operator's login (adopted)
the login kept on each the machine accounts present nox, root: never (adopted on the control node,
agent nox, its machines the facts account, agent-account made elsewhere)
each identity's bus password ──► its secret, owned by its login, 0600
```
### 5.2 Run for an identity (ADR 0412)
Every module runs as the operator account of its machine, as today, with no setting; running for the operator is that
default run. A manifest may declare that its module can **run for** an agent identity, at assignment, at start or both: serve one identity with that identity's machine account, home and bus password.
An assignment of such a module may name the identity it runs for, and the module may let the one who starts it
choose: claude-code started for the operator is the default run with the operator's configuration, and started for `nox`
it runs for `nox` with the agent's home. A start
goes down only: a module is started for an identity by a process already running as that identity's login, or by the
operator's own login for an agent; an agent never starts one for a person, and the module refuses it, naming who asked
and for whom. A module may be assigned to one machine once per identity it runs for there, each its own process under
its own login, and the controller keys an assignment by machine, module and identity. The first two to declare it are
claude-code and ssh-client. On a machine the identity seat marks with an agent account, claude-code's assignment is
run for that agent (`nox` on the control node), so its licence and home scope stay in the agent's home, and the controller refuses
one there that is not. ssh-client keeps its default assignment for
the operator wherever `nox` is present and gains a second one run for `nox`, which makes the agent's forge key in the
agent's home and closes issue 989; gitea's provider registers that key on the forge account `nox`. A module run for an identity calls as that
identity and gets no module identity of its own (this narrows ADR 0386 §4, which holds for every other assignment). A module with an account of its
own (`runs-as`, ADR 0259 §8) cannot declare it, and a run for an identity with no machine account on that machine is
refused at composition and at start. ADR 0266's guard
stands: on the control node, a session started for the operator gets the operator's configuration and its tool calls are still
refused.
### 5.3 A bus password per identity (ADR 0413)
Each person and agent identity has a bus account of its own (`person.<name>` for the operator, `agent.nox`), whose memberships are made
from its role, and its password is a secret owned by its machine account at mode 0600 on each machine it is present
on, which no other account but root can read. A person's password is placed only on a machine that names an agent
account of its own judged unable to become root, by a verdict at most fifteen minutes old; on the other machines, where agents can still run as the operator,
it is not placed, and the controller says so. A module run for an identity reaches the bus with that password, can do
only what the identity's role allows, and every call is recorded as that identity's. Nox's role never holds an approval
grant, a trusted party's grant or a dangerous tool named in it (ADR 0387 §4 does not apply to it), and the identity seat
refuses one; each dangerous act of nox's is asked one by one. The break-glass agent (§6) keeps its own rules and gets
no login and no bus password from this. This changes ADR 0259 §8, which kept every bus credential out of an agent's account; the rest of §8 stands.
The password lives in today's list of bus accounts and is replaced by a signed login for the same role at the bus
switch (§7, §8).
### 5.4 Traced to GENESIS
Checked on 2026-10-17 (playbook 02, step 1); no conflict was found.
- [Mission](../../00-META/mission.md): agents "hold identity", and there is one kind of participant, some of them
human; persons and agents are held alike, with the same fields, and differ in what their role allows, which the
mission calls modality. "The mesh brokers credentials": the bus password per identity and the service accounts made
from it. "The mesh owns the truth": logins are derived from identities, no longer stated per node. "Dogfood": logins
go through the node-engine's existing resource.
- [Context](../../00-META/context.md): "Identity is mesh-internal", with no directory and no SSO, so the identity seat
holds it and Keycloak only receives accounts; "Nodes are personal" and the mesh is a guest, so existing logins are
adopted, never renamed; human agents are few, often one, so the seat holds two identities and one agent.
- [Effect](../../00-META/effect.md): a human agent's environment is part of the mesh, its login included.
## 6. Effects, asks and break-glass (ADR 0387)
Every seat verb and module tool is one permission and carries one **effect**, set in its module's manifest:
**reads**, **acts** or **dangerous**. The catalogue's check refuses an unmarked tool, once every module already in
the catalogue is marked. A dangerous tool is granted only by name, and naming one in a role goes to the operator as
an ask first. Every other dangerous act is an ask (ADR 0259), bound to that act and carried out on the warrant.
The **break-glass** agent holds every tool and never asks. It is sealed until the operator opens it, or a condition
the operator has named as opening it is raised (none is named at the start); it stays open until the operator closes
it, and every act it takes is said to the operator at once. A tool served without an effect, from a module built
before its mark, counts as dangerous.
## 7. Signed logins on the bus (ADR 0388)
The bus checks each login by its signature and reads no list. The root key is sealed in the vault and used only to
replace the identity seat's signing keys, on the operator's Approve. The seat signs with one scoped signing key per
role, so every login stays small and each identity gets its own inbox from the role's template. An agent's login
lives minutes, a module's and a node's hours; each is renewed by a reconnect forced before expiry, by the handler or
the node-engine.
A switched-off identity is revoked over the bus. The seat reads back every update, sends updates at least a second
apart, and resends the current state to every server id it has not seen on the system account, which every joined
server carries. This changes ADR 0106's rule that the bus is configured only through its settings file.
## 8. The switch (ADR 0389)
A new bus with signed logins runs beside today's, joined as a leaf node carrying the system account. Identities move
one at a time, each checked before the next. Streams, key-value buckets and the object store move last, each by a
copy of its whole configuration: copy, wait for equal counts, stop its writers, then one update hands the addresses
over. Today's bus is switched off when nothing is left on it.
## 9. Sandboxes (ADR 0390)
Each session runs in a microsandbox of its own, its version pinned and declared in a module. The workspace is
mounted; the claude-code module writes in the agent's login and the licence token without its refresh part, and
replaces the token as the licence manager refreshes it. Connections are allowed only to the vendor's interface, by
address range, and to the bus. A node whose KVM modules are not loaded gets them from the sandbox module.
## 10. The build's acceptance tests
The build is not done until each of these has been shown, and the milestones of the work breakdown (playbook 04)
carry them:
1. A licence token replaced under a running session keeps that session working.
2. The sandbox module loads KVM on the home server, and a sandbox runs there.
3. Every existing module's tools carry an effect, and the catalogue's check refuses an unmarked one.
4. The switch, run on the mesh's own modules in the lab from the bed test kept on the lab's branch
`research/053-signed-bus-logins`, loses nothing and leaves every identity on the new bus.
5. The acceptance sentence at the top, end to end.
From research 057 (ADR 0411 to ADR 0414):
6. The identity seat lists the operator and `nox` with their machines, kept logins and the control node's agent account mark; a
create or a presence change by any caller but the operator's Approve is refused.
7. On every other machine a login `nox` exists; on the control node the login `nox` is adopted with its uid and home unchanged;
on every machine the node-engine judges `nox` unable to become root; a login `nox` given a sudo rule in a test is left
unaltered and the self-check raises `agent-can-become-root` for it, on a machine that marks no agent account too;
the operator's existing logins are
unchanged.
8. No manifest in the catalogue declares a login for a person or an agent; the catalogue's check refuses one.
9. For every machine, the declaration before and after differs, for the modules not set to run for an identity, only
in the machine accounts' `user` resources (now the controller's, and `nox` added on the other machines); claude-code on the control node, run for `nox`, places the same
files in the same home; on the other machines the guard stays off and the operator's sessions keep the mesh's tools.
10. A session started for `nox` on the control node gets the agent's home configuration and one started for the operator the
operator's, with no tool call through; a session as `nox` that starts claude-code for the operator is refused; with
ssh-client run for `nox`, nox's key is on the forge account `nox` and the session pushes a branch over SSH as `nox`.
11. Nox's bus password is owned by `nox` at mode 0600 on every machine nox is present on; a call with it outside
nox's role is refused; a grant test finds no approval grant, trusted party's grant or dangerous tool in nox's
role; the operator's password is placed only on a machine with an agent account judged root-free within fifteen
minutes, and the controller says so for every other.
12. The forge account `nox-agent` is renamed `nox`, keeps every machine's key, stays in the agents' write team, and
every `forge-account` entry names `nox`.
## 11. Milestones
Each milestone is one feature step (playbook 07), usable as soon as its delivery plan is done, and starts when
the previous one's is. What it replaces is in the as-is documents
[15 — The agent and its licences](../00-as-is/15-the-agent-and-its-licences.md) and
[16 — Logins on the bus](../00-as-is/16-logins-on-the-bus.md), and for milestones 11 to 14
[17 — Accounts on a machine](../00-as-is/17-accounts-on-a-machine.md). The issues are opened once the operator confirms
the milestones, and their numbers written here.
**Confirmed by the operator on 2026-10-16.** One issue per milestone: 1 is issue 878, 2 is 879, 3 is 880, 4 is
881, 5 is 882, 6 is 883, 7 is 884, 8 is 885, 9 is 886, 10 is 887. A milestone's issue is split into one issue per
repository part when its feature step starts (playbook 07): several of the parts are modules that do not exist yet,
and the issue tracker takes only registered owners. The building waits until the delivery under way of
[57 — Building config and provisioning](57-building-config-and-provisioning.md) is done; the status here stays
`designed` until milestone 1 starts.
| # | Outcome, in one sentence | Proves | Repositories and their parts | Depends on |
|---|---|---|---|---|
| 1 | Every tool on the mesh says whether it reads, acts or is dangerous, a tool's description shows it, and the catalogue refuses a module that leaves one unmarked | test 3 | mesh-sdk: the effect in a tool's declaration, an unmarked tool served as dangerous; mesh-controller: the manifest field; mesh-catalog and mesh-media-catalog: every module marked, the check | — |
| 2 | A session works on a repository's workspace through the mesh's tools alone: read, glob, search, edit, write, git, its tests and its merge check, with no command runner | — | mesh-catalog: the module `nox-workspace` on the node seat `workspace`, on the always-on nodes; mesh-controller: the seat | 1 |
| 3 | A thread created through the thread seat runs headless on an always-on node, its conversation readable through the seat, its usage counted per thread, and a move to another node loses at most the turn in progress | — | mesh-catalog: `nox-threads` (seat `thread`, restricted streams, leases), `nox-thread-handler` (seat `thread-handler`), claude-code (mirror and rebuild), claude-licence-manager (usage per thread) | 2 |
| 4 | The operator starts, follows, answers and stops a thread from a terminal client, and answers its questions on Telegram | — | mesh-catalog: the terminal client, a Go program of `nox-threads`; claude-code: questions and permission requests through the asks | 3 |
| 5 | Every session runs in a sandbox of its own that reaches only the vendor and the bus, and the home server runs one too | tests 1 and 2 | mesh-catalog: the sandbox module (microsandbox pinned, KVM loaded where off); `nox-thread-handler`: runs sessions in it; claude-code: writes the login and the licence token without its refresh part | 3 |
| 6 | Every person, agent and module is an identity with a role, and an agent's call outside its role is refused, saying which grant is missing | — | mesh-catalog: the identity seat's module; mesh-sdk: the tool runner's exact check, calls carrying thread and agent token; mesh-controller: an identity asked for each assignment; mesh-tools: sessions call as a person or an agent, never through the tool runner's login | 1 |
| 7 | A dangerous act not named in a role comes to the operator as an ask, and the sealed break-glass agent opens only on the operator's word or a named condition | test 5 | mesh-sdk: dangerous calls asked; identity module: dangerous tools granted by name on an ask, break-glass and its telling | 6, 5, 4 |
| 8 | The bus checks signed logins on a second bus beside today's, the identity seat signs and revokes them, and the switch, run in the lab on the mesh's own modules, loses nothing | test 4 | identity module: signing keys, renewal, revocation read back and resent to new servers; the node-engine (`mesh-host`): node and module logins renewed; mesh-controller: the second bus as a leaf with the system account; mesh-lab: the bed test from the branch `research/053-signed-bus-logins` | 6 |
| 9 | Every identity is on the new bus, its data moved by copy, and today's bus is switched off | — | mesh-controller and the identity module: one identity at a time, then streams, buckets and the object store; mesh-catalog: the old bus's module retired | 8 |
| 10 | The operator follows and answers threads from the web page as from the terminal client, logged in as a person identity | — | mesh-catalog: the web page, a client of the thread seat over the bus's WebSocket listener | 9, 4 |
The front ends' later parts, the inference seat and the other agent harnesses are tracked by issue 850; the
seat-sense "role" in older designs by issue 872.
**Milestones 11 to 14, from research 057, confirmed by the operator on 2026-10-17.** One issue per repository
part, opened at the confirmation, so a fixing agent per repository can start when its step does. Milestone 11
waits for nothing; 12 waits for milestone 6.
| # | Outcome, in one sentence | Proves | Repositories and their parts (issues) | Depends on |
|---|---|---|---|---|
| 11 | The agent's forge account is `nox`, with its keys and its place in the agents' write team kept | test 12 | mesh-catalog gitea: the rename, keys registered on `nox`, every `forge-account` entry (993) | — |
| 12 | The mesh makes the operator's logins and nox's on every machine from the identities, and judges nox unable to become root everywhere | tests 6 to 9 | identity module: logins, machines, kept logins, the agent account mark, the operator's verb (994); the node-engine (`mesh-host`): existing logins adopted, every agent login judged (995); mesh-controller: machine accounts composed, node accounts read from the identities, the terminal lines retired (996); mesh-catalog: claude-code declares no login, the catalogue's check refuses one (997) | 6 |
| 13 | The operator and nox each reach the bus with their own password, held to their role; the operator's only where no agent runs as the operator | test 11 | mesh-controller: the operator's and nox's bus accounts, secrets at 0600, the fifteen-minute verdict (999); identity module: nox's role refuses approvals and dangerous tools (998); until milestone 7 builds the asks, a dangerous act of nox's is refused, not asked | 12 |
| 14 | The operator starts a claude-code session for nox or for themself, and nox pushes over SSH with its own key | test 10 | mesh-controller: run for, assignments keyed by identity, the refusals (1001); mesh-sdk: calls as the identity, a start goes down only (1002); claude-code: started for nox or for the operator (1003); ssh-client and gitea: nox's own key, closing issue 989 (1004) | 11, 13 |
The agent's and the operator's accounts in Keycloak and mail (ADR 0414 §2 and §3) follow milestone 13 and are
tracked by issue 1000, which needs a design section first. A second agent identity is made only when a rule
needs another role (ADR 0414 §1); no milestone makes one.
+4 -2
View File
@@ -47,15 +47,17 @@ document is written and this one's status becomes `implemented`.
| [`42-the-machines-modules-in-order.md`](42-the-machines-modules-in-order.md) | **In progress.** The order the machines' modules of research 026 and 027 are built and rolled out: every machine's first (sudo, localization, time sync, pacman, logrotate, avahi, systemd, docker, `~/.ssh`, scripts, kernel), then both workstations', then one machine model's; each proven on one workstation before the rest | ADR 0173, ADR 0182, ADR 0205 |
| [`45-a-core-that-cannot-fail-silently.md`](45-a-core-that-cannot-fail-silently.md) | **Designed.** The core says when it is wrong, refuses what is stale or unreadable, heals what it knows, upgrades one machine at a time with a witness that rolls it back, and is checked against the real mesh before merge: the writers and signals tables, the condition store, `doctor`, the minimal output channel, healers, the lease and report order, staged upgrades, the facts snapshot and replays, in six phases | ADR 0227, ADR 0224, ADR 0218 |
| [`46-the-conversation-with-the-operator.md`](46-the-conversation-with-the-operator.md) | **Designed.** The mesh tells and asks its operator over channels that are holders of two kinded benches, `channel` and `intake`, declaring capabilities from a fixed vocabulary; the router orders them by work context and never lowers the bar; an answer that performs an action is checked and performed by the controller, on a TOTP code or a verified Telegram sender, never on a desk click alone; operator messages addressed or answered by the mesh's responder, untrusted input kept as data, references for what words may not carry, and a recoverable factor; Telegram first, in six phases | ADR 0234, ADR 0227 |
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the walk, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
| [`47-delivery-from-commit-to-delivered.md`](47-delivery-from-commit-to-delivered.md) | **Designed.** A delivery is one commit in one repository, from its pull request's head to every machine; a delivery group is deliveries sharing a branch name, ordered and checked as one future state; both owned by the `mesh-delivery` module with one state table, its state on the bus, every transition said, noted on the commit and shown on the pull request; the controller keeps the planner, the gate, sending and the rollout, and the core's own updates never wait for the module | ADR 0239, ADR 0238, ADR 0236 |
| [`49-the-mesh-in-domains.md`](49-the-mesh-in-domains.md) | **In progress.** Every concept belongs to one of eleven domains, which owns its one word; the glossary is organised by them and is the authority, a retired word is named on its replacement's line with its scope, and two checks hold this repository's documents and the catalogue's tool descriptions to it | ADR 0244, ADR 0006, ADR 0008, ADR 0302 |
| [`50-split-dns-on-a-machine-with-a-vpn-client.md`](50-split-dns-on-a-machine-with-a-vpn-client.md) | **In progress.** A machine whose VPN client pushes servers of its own runs a resolver of its own, on a node seat, that routes the VPN's domains over its link and every other name to the mesh's resolvers; the VPN client's module carries the adapter | ADR 0247, ADR 0223 |
| [`51-the-registries-keep-what-is-named.md`](51-the-registries-keep-what-is-named.md) | **In progress.** The artifact store, the package registry and each machine's container runtime get tools that say what they hold, what names each thing and what could go; every removal is a dry run unless a person asks and says why, and what decides is always a record | ADR 0251, ADR 0189 |
| [`52-shares-and-mounts.md`](52-shares-and-mounts.md) | **In progress.** A node shares one folder at a time through `nfs-server`, which provides it as a provision to the private network only, mapping every client to the folder's owner; a node that asks mounts it through `mounts`, a mount unit and an automount per share, so nothing mounts or fails at boot; a device that comes and goes is an occasional source whose absence is a state | ADR 0263, ADR 0051, ADR 0092 |
| [`53-the-issue-tracker.md`](53-the-issue-tracker.md) | **Designed.** The mesh keeps its issues in one module, `issues`, on the control-node: every number from one counter advanced by compare-and-set, the cycle's issue rules refused at the write, an event per change; an issue lives whole in the tracker, hq keeping no issue content (ADR 0291) | ADR 0275, ADR 0291 |
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, walks, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
| [`54-the-controllers-records-on-the-bus.md`](54-the-controllers-records-on-the-bus.md) | **Designed.** The controller keeps five record mirrors on the bus (nodes, assignments, rollouts, builds, settings): each a bucket written from its store only when the record changed, with one event per write, and audited against the store every self-check run, a difference raised as `mirror-drifted` before the healer H6 repairs it | ADR 0289 |
| [`55-the-knowledge-base.md`](55-the-knowledge-base.md) | **Designed.** The mesh keeps its decision records and research efforts whole in one module, `mesh-knowledge`, on the control-node: number, status, topic, typed links both ways, each decision's clauses with their own status, and the reasoning, with every version; a record is proposed, checked at the write, reviewed by the knowledge base's own judgement and accepted there, with the operator's Approve when it needs it; agents ask what holds in one bounded call | ADR 0290 |
| [`56-the-mail-client-set-up-by-the-mesh.md`](56-the-mail-client-set-up-by-the-mesh.md) | **Designed.** On every node with a display server, the `betterbird` module installs Betterbird from a pinned upstream release mirrored into the artifact store, ships the Owl add-on pinned and installed by Betterbird's enterprise policy, and declares the operator's mail accounts in a trusted setting without credentials; each account signs in once at a person's first use, and one the profile already holds is adopted by its address | ADR 0316, ADR 0317 |
| [`57-building-config-and-provisioning.md`](57-building-config-and-provisioning.md) | **Designed.** The work of ADR 0348 and ADR 0349 as 37 packages, each an issue with its proof: foundations, template files, config in its own module, the provisioner and its client, secrets in the vault, bindings from providers, collections, a module's runtime readers, and removing the old forms; at most five worked at once | ADR 0348, ADR 0349 |
| [`58-agent-work-as-the-meshs-own.md`](58-agent-work-as-the-meshs-own.md) | **Designed.** Agent work as the mesh's own: a thread seat keeps every thread whole on the bus; a thread handler runs the harness headless in a sandbox per session, the workspace on the thread's own forge branch; a workspace toolkit and named tools, no general command runner; one identity seat for persons, agents and modules, roles of grants, every tool an effect, signed bus logins, and the switch to them one identity at a time | ADR 0383, ADR 0384, ADR 0385, ADR 0386, ADR 0387, ADR 0388, ADR 0389, ADR 0390 |
## Not yet written
@@ -28,7 +28,7 @@ for _, n := range nodes {
```
So a provider on the same machine is never a `Provider` in `world.Offered`, never becomes a
`Needed`, and the credential loop — which walks `resolved.Needs` — has nothing to walk. Every step
`Needed`, and the credential loop — which traverses `resolved.Needs` — has nothing to traverse. Every step
is individually reasonable and the sum is a silent gap.
## Why it was not noticed
@@ -92,7 +92,7 @@ node that looks like a real one.
## Fixed
Needs fan out per consuming module in one place, after the resolution walk. The credential's key
Needs fan out per consuming module in one place, after the resolution rollout. The credential's key
gains the consuming module, the grant file is named after both halves of the consumer, needs are
matched by provision *and* module, and the provisioners name the role and the access key after the
module rather than the machine. The refusal is gone because there is nothing left to refuse.
@@ -87,7 +87,7 @@ too new to have anybody watching it.
## Fixed
**The registry module names its image and is added, not built.** The lab's artifact-store test now
walks the only path open to a real first mesh — the manifest goes in directly, no builder and no
rollouts the only path open to a real first mesh — the manifest goes in directly, no builder and no
store involved — and passes.
**And the cycle is refused where it is written.** A manifest that provides `artifact-store` and
@@ -45,7 +45,7 @@ field it does not know — whole. That was
mesh now records which host each machine reports and `status` names every machine running an older one
than another does.
So the flag day is visible before it is walked into, which it was not when this was filed. It does not
So the flag day is visible before the mesh runs into it, which it was not when this was filed. It does not
make the field free: **the oldest host in the mesh is still what the mesh may send**, and one machine of
four is behind today. A sequence that an old host refuses takes that machine out of the mesh's reach
entirely — worse than the replay it prevents, which has never been observed.
@@ -18,7 +18,7 @@ half-understood declaration off a machine, and exactly what makes this expensive
[Issue 087](../087-the-controller-cannot-tell-a-host-is-too-old/00-report.md) is resolved: the mesh now
records the host version each machine reports and `status` names every machine running an older host
than another does. The flag day is visible before it is walked into, which it was not on 2026-09-23.
than another does. The flag day is visible before the mesh runs into it, which it was not on 2026-09-23.
That makes the cost measurable rather than hypothetical, and the measurement is the reason this is not
being built today.
@@ -72,7 +72,7 @@ what the sidecar is has nowhere in the design layer to look, which is how this w
resource type no decision records and contradicting an accepted one. The rule is "no design
without a decision"; the check is "no design without *a* decision." An unenforced rule is
indistinguishable from a wrong one, and these two documents are what that gap looks like when
something walks through it.
something rollouts through it.
## Open questions
@@ -92,7 +92,7 @@ built from a repository elsewhere, or a mesh whose forge module is down — and
event performs, so the two cannot disagree about what "moved" means.
**Bases first needed edges, and there were none.** The order this report asked for was written and
walked a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
traversed a graph that no build had ever recorded: a recipe reads its base from a build argument, so the
digest was never in the file the builder read edges from. A build now reports what it was handed, the
control plane records it by artifact path, and the order is read from each module's newest build.
@@ -38,7 +38,7 @@ Every node's resolution is read first, and the names are attributed across them
terminus serves the name**: among the providers a name reaches, the one that is not itself published
under a labelled name through another provider. An identity provider is routed through the proxy and
so is a consumer of names, not their end; the proxy contributes no label to anyone and is. The rule
knows nothing of what "route" means — it reads the graph the modules declared — and it walks nodes,
knows nothing of what "route" means — it reads the graph the modules declared — and it rollouts nodes,
requirements and contributions in order, so one mesh yields one region. Every shape of contribution
is counted, so a module routed under several names has every one of them resolved.
@@ -73,5 +73,5 @@ give.
All three faults are closed on the controller's main: one ask in flight (8d4e940), a merge rebuilds what
stands on it (7e701c0), and a merge produces a tiered plan the mesh keeps, read with `plans` (d2ed6d5,
76f2756, ADR 0162). Plans were since carried into walks and deliveries (ADR 0236, ADR 0239); the order a
76f2756, ADR 0162). Plans were since carried into rollouts and deliveries (ADR 0236, ADR 0239); the order a
person kept is the order the tiers give.
@@ -86,6 +86,6 @@ machine. That is the path the console exists to replace, so it is a workaround a
Both pull requests named above are merged and live. Read through the console on 2026-10-08:
`mesh_describe mesh-controller.plan` declares `node` (required) and `files`; `mesh-controller.push`
declares `node`, `behind`, `cause` and `why`, and its description says that a push naming no machine
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` walks every verb it serves:
is a push of the whole mesh. The controller's `seatverbs_schema_test.go` exercises every verb it serves:
an argument a verb does not declare is refused, and every flag of a verb's command is in its schema
or accounted for.
@@ -46,7 +46,7 @@ Pull requests: novox/mesh-tools#14 and novox/mesh-controller#71 — open, not me
first line of its answer, that it is a push of the whole mesh and which machines it is sending, and
its last line names the machines told. `plan` gains `files`, `push` gains `behind` (the whole mesh
said outright, refused beside `node`), and the listing verbs a `limit`.
- **Checked by tests that walk every verb the controller announces:** for every combination of a
- **Checked by tests that exercise every verb the controller announces:** for every combination of a
verb's declared arguments, the verb either refuses or every argument given changes the command it
runs; every verb refuses an argument it does not declare, and `node` where it takes none; and every
flag of the command a verb runs — read from the command's own source — is an argument of the verb's
@@ -26,7 +26,7 @@ list alone. The flush has no such narrowing.
## What it costs
`record` is the policy for a change that must be walked through the mesh by hand. A named push cannot
`record` is the policy for a change that must be rolled out through the mesh by hand. A named push cannot
do that, so the policy does not hold the change back from anything but the merge. The command's name
says one machine, and the command's output is the only place that says otherwise.
@@ -61,7 +61,7 @@ The mesh-delivery tests `checkask_test.go` cover:
- An announcement that carries a decided gate status uses it.
- H2 re-asks once, a second re-ask is refused by the table, and a late verdict still settles the delivery.
The table tests walk the new row, and the pairs the table does not hold are refused by name. Live: once
The table tests exercise the new row, and the pairs the table does not hold are refused by name. Live: once
the fix is rolled out, the stalled delivery has its check asked at the owner's first tick. The controller
journal then shows its verdict, and `show` moves from proposed to checked to ready without a person
acting.
@@ -39,7 +39,7 @@ The mesh's own record shows it:
- **What each machine reports.** `postgres.server`, `mongodb.server` and nine `mailu.*` containers
have run only since 18:57 on the control node, and `postgres.server` since 18:56 on the anchor.
No walk held back for later was involved. The sends were the plan's own sends to its first machines.
No rollout held back for later was involved. The sends were the plan's own sends to its first machines.
## Why it happened
@@ -53,7 +53,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
push". So the gate did not judge the recorded builds, the refusal of ungated sends did not see
them, and the gated send carried them. `record` held only against the end of a person's push to
another machine (issue 259's `heldBack`) and against the bus (the bus step). It did not hold
against a plan's walk, a rollback, a healer or a rotation.
against a plan's rollout, a rollback, a healer or a rotation.
3. **Nothing tells a person, before or during a send, that it will recreate every container of a
module at once.** A change that only alters how containers are declared reads as harmless. A
health check, an environment variable or a label each recreates the container all the same,
@@ -64,7 +64,7 @@ No walk held back for later was involved. The sends were the plan's own sends to
## What is at stake
`record` is how the mesh keeps the providers, the network path and the irreplaceable data out of
unattended walks. Tonight showed that it holds only until some other module on the same machine
unattended rollouts. Tonight showed that it holds only until some other module on the same machine
rolls out. This was still true after the incident: plex (`record`, irreplaceable data) was rebuilt
by a media-catalogue merge at 19:01 and sent nowhere. The next gated send to the anchor, for any
module, would recreate it.
@@ -13,8 +13,8 @@ the tier durations measured for the repository: three times their ninetieth perc
than 30 minutes. So the watchdog and the line a person reads used different start times and could use
different bounds. The line was the one that was wrong.
The line for a walk waiting on its delivery's word had the same flaw. It said "for" the time since the
last save. The watchdog for that wait (S16) counts from when the walk was opened.
The line for a rollout waiting on its delivery's word had the same flaw. It said "for" the time since the
last save. The watchdog for that wait (S16) counts from when the rollout was opened.
Ruled out: a clock difference between the machine running the verb and the store. The count was small
and kept growing again from zero, which only a column being rewritten explains.
@@ -44,8 +44,8 @@ nothing.
## Fix (mesh-controller)
- The line, the `since` and `late` fields of `status --json`, and the count of late plans all start from
when the plan entered its tier. If the delivery let the walk go after that, they start from the go. A
walk still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
when the plan entered its tier. If the delivery let the rollout go after that, they start from the go. A
rollout still waiting for its delivery's word counts from when it was opened, as S16 does, and is never
called late. A plan read without the tier stamp falls back to its last save.
- LATE uses the bound the watchdog uses: the measured bound for the plan's repository, or the 30-minute
floor when nothing has been measured. The watchdog's start time and bound now come from the same two
@@ -53,4 +53,4 @@ nothing.
- The loop compares the plan before and after each step and saves only when they differ. The fields the
store stamps itself are left out of that comparison.
- A refused step is logged and saved only when its note is new. The refusal's wording called the gated
walk a "release plan", a name the glossary retires; it now says "walk".
rollout a "release plan", a name the glossary retires; it now says "walk" (since renamed rollout, issue 646).
@@ -23,7 +23,7 @@ another, with nothing on the mesh noting the difference.
ADR 0238 makes the delivery plan the planner's one answer for a pull request's check, the merge gate and
the merge. The plan counts a new module's directory as something the merge builds. The merge handler
read the same answer (`touchedBy`'s `added`) and dropped it: it only walks modules the catalogue already
read the same answer (`touchedBy`'s `added`) and dropped it: it only rollouts modules the catalogue already
holds, and a new one is not held until it is built. The code said so outright: "a new module, which a
check judges before it merges and a merge does not build". The take-in refuses a build off the trunk
with "merge it, and the merge builds it". So one part of the controller promised the build and another
@@ -29,7 +29,7 @@ already built from it are, and without waiting (`askNewModules`). The build's ta
module, as it does for a hand `build`. Nothing is sent: no machine is assigned a new module, and
assigning one stays a person's act.
The new module is outside the merge's plan. A plan walks modules the catalogue holds, tier by tier, and
The new module is outside the merge's plan. A plan rollouts modules the catalogue holds, tier by tier, and
sends each built one to the machines running it. A new module has no machine and nothing standing on it,
and the plan's ask for a module the catalogue does not hold fails the plan.
@@ -7,16 +7,16 @@ opened: 2026-10-08
## Symptom
On 2026-10-08 an agent proving the tunnel join (novox/mesh-lab PR #61) on the laptop could not run the
two-node walk, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
two-node rollout, the lab's oldest end-to-end bed. Four things stood in the way, and none of them was
reported by any check before someone tried to run a bed:
1. **The walk's scenario stocks a packet-filter runtime image that cannot be built.**
1. **The rollout's scenario stocks a packet-filter runtime image that cannot be built.**
`scripts/build-module-runtime.sh` copied the tool runner's compiled TypeScript and its `node_modules`
out of mesh-tools. mesh-tools has had no package at its root since the runtime became `node-tools`, a
Go binary that launches every bundle (ADR 0175,
ADR 0193),
so the script failed at its first step.
2. **The walk's builder dialled the old broker.** It was started with the old bus's guest account on
2. **The rollout's builder dialled the old broker.** It was started with the old bus's guest account on
loopback. The mesh moved its bus to NATS (ADR 0106,
ADR 0127,
ADR 0131), and the builder
@@ -27,8 +27,8 @@ reported by any check before someone tried to run a bed:
4. **The replays prover could not run a test in mesh-tools**, because that repository's Go module lives
in a subdirectory (`node-tools/`), and the prover ran `go test` only at a repository's root.
The same day, while trying to run the walk again, two more gaps surfaced: the object store's
provisioner image, which the walk also stocks, cannot be built, because the client image it copies
The same day, while trying to run the rollout again, two more gaps surfaced: the object store's
provisioner image, which the rollout also stocks, cannot be built, because the client image it copies
from is gone upstream; and the laptop could not reach the public container registry, which the lab's
egress check needs (an outside condition, not the lab's fault).
@@ -56,9 +56,9 @@ In novox/mesh-lab PR #63, stacked on #61:
builds `node-tools` from mesh-tools' Go module, and runs it as the image's entrypoint. The image serves
on the node's runtime credential: the Go runtime refuses to serve a module on that module's own
credential.
- **The walk no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
- **The rollout no longer stocks a packet-filter runtime.** None of the catalogue's nftables containers
names an image (its verbs are a tools bundle the node's runtime launches), so the image was never run.
- **The walk's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
- **The rollout's builder holds the build seat over the bus.** A lab module claims `node-build-agent` with
an own `broker` secret, as the catalogue's build agent does. Assigning it to the anchor issues the
account ([issue 203](../203-a-fresh-assignment-is-pushed-before-its-credential-exists/00-report.md)).
The bus's user list is placed, the push writes the sealed credential, and the builder starts on it.
@@ -71,12 +71,12 @@ In novox/mesh-lab PR #63, stacked on #61:
Still open:
- `builds.test.ts` still dials the old broker. Its bed enrols no machine, so the mesh has nowhere to
deliver a builder's credential. It needs the walk's enrolment, or it should be retired into the walk.
deliver a builder's credential. It needs the rollout's enrolment, or it should be retired into the rollout.
- Beds that run a per-module runtime container on the module's own credential (the tools beds for the
wiki and the code forge, and the audit bed) run into the runtime's refusal. They need the node's
runtime credential instead.
- The object store provisioner's image needs a client source that still exists.
- The walk has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
- The rollout has not been run end to end since these changes: on 2026-10-08 the laptop could not reach the
public container registry.
## How it is checked
@@ -14,25 +14,25 @@ script now does those steps by hand. Verified: the image for the packet filter b
answers `initialize` and `tools/list`; `node-tools` in it, started against a real bus with a plain URL,
launches the bundle and serves nothing until the mesh issues a membership, which is correct.
**Why the walk stocked that image at all.** The tunnel branch passed the held images to
**Why the rollout stocked that image at all.** The tunnel branch passed the held images to
`deriveTheFilterOn`, which turns the catalogue's manifest into the lab's form: build section gone, each
container's artifact replaced by a stocked image. The catalogue's nftables manifest has no container,
so no image is referenced. Removing the build section is what the bed needed; the stocked image was
not.
**The builder.** mesh-controller's `cmd/mesh-builder` reads `MESH_BROKER_FILE` and nothing else for the
bus. Without it, it says it has no credential. The walk set the old broker's variable, which the builder
bus. Without it, it says it has no credential. The rollout set the old broker's variable, which the builder
no longer reads. A builder's account is a module's account (`builder issue` and `module issue` both mint
through the inventory), composed into the bus's user list and sealed to the machine as the module's
`broker` own secret. Assigning a module that declares that secret now issues it in the same act (issue
203). The catalogue's build agent is that module, but it runs the builder as a container from an
artifact the mesh builds, and it needs the artifact store and package registry bound. The walk has
artifact the mesh builds, and it needs the artifact store and package registry bound. The rollout has
neither before it can build, so it declares a lab module with the same claim and the same secret. In
this bed the bus comes from the bundle, so its user list is placed by hand, the step the walk already
this bed the bus comes from the bundle, so its user list is placed by hand, the step the rollout already
takes at enrolment.
**The uplink and the prover** were each fixed by the agent that met them (mesh-lab #61 and #62), and are
listed here only so the lab's state on that day is in one place.
**Ruled out.** That the walk failed because of the tunnel changes. The builder and runtime faults are in
**Ruled out.** That the rollout failed because of the tunnel changes. The builder and runtime faults are in
code that #61 does not touch, and they reproduce on main.
@@ -21,7 +21,7 @@
the planner's sort stalled. Every two-way pair was a cycle, whatever the rules.
- The person's order, the controller first, was right for this change. The new node-engine calls a verb
the new controller adds, which is ADR 0246's case: a verb is promised before it is required.
*Engine before controller* comes from ADR 0236's order for its walks, where the node-engine had to
*Engine before controller* comes from ADR 0236's order for its rollouts, where the node-engine had to
accept something new first. The rule reads only module names, so it cannot tell the two cases apart. *Built
by* is read from the graph.
@@ -59,7 +59,7 @@ and fails only when it cannot read.
**Merged.** mesh-controller#135 (the build agent judges a pull request by the base branch's
`merge-check.sh`) merged on 2026-10-08 at 09:10 UTC, and mesh-lab#65 (R310) at 09:10 UTC. Every later
controller delivery carries it; the last, of mesh-controller#170, finished its walk on 2026-10-09 at
controller delivery carries it; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at
00:11 UTC, and `mesh-controller.status` named no machine behind.
**The open half is closed.** `gitea_branch_protection_get` for the media catalogue's main, read on
@@ -32,11 +32,11 @@ was asked of a new, unrelated pull request.
## Why it is a design issue
ADR 0239
decision 10, *the switch*, adopts the walks open when mesh-delivery is assigned. It says nothing of the merges the
decision 10, *the switch*, adopts the rollouts open when mesh-delivery is assigned. It says nothing of the merges the
bus still holds from before: the forge's `pull.merged` is kept on the bus and heard again by a new consumer.
Each was taken as a merge just made whose head was never heard, so the table's row *merged unchecked → held*
held each for a person. The table had no state for "this already happened", and `held`'s only ways on were
a release, which would walk it, and a stop, which says a person stopped a delivery that was never running.
a release, which would roll it out, and a stop, which says a person stopped a delivery that was never running.
Healer H2's `close` has no transition from `held`, rightly: a held delivery is the operator's.
Decision 5 says what a transition owes outside is "retried until done". An effect the forge answers can never
@@ -45,14 +45,14 @@ be done was retried without end, every ten seconds, and nothing said so.
## Fix
- **A new final state, `retired`: history.** A delivery made from the forge's word at its merge, whose merge
was heard fifteen minutes or more after it happened and whose walk, if the controller keeps one, is not
was heard fifteen minutes or more after it happened and whose rollout, if the controller keeps one, is not
open, is history. At adoption it goes straight to `retired` by the observed row *history*, before any check,
hold or walk. Its pull request is not told: the forge may no longer hold it, or holds another under its
hold or rollout. Its pull request is not told: the forge may no longer hold it, or holds another under its
number. Its transitions are still said on the bus and kept on the commit's note.
- **A person's verb for the ones already held: `retire-history`**, with why. One delivery by id, refused when
the rule does not take it, or every held one (of one repository when named), answering how many it retired
and naming each one it left held. `dry` says what it would do and changes nothing. It is the act
*held → retired* of the table, never a release: nothing is walked, delivered or put back.
*held → retired* of the table, never a release: nothing is rolled out, delivered or put back.
- **Owed work ends.** The forge's answer that what an effect is for does not exist (a 404, or "does not
exist") is final: the effect is given up with one line, kept among the owner's refusals. Any other failure is
tried again later each time, from ten seconds doubling to an hour, and given up after sixty tries, saying so.
@@ -5,20 +5,20 @@
- 699 deliveries were `held`. 688 say "merged without a passing check", all made between 23:05:27 and
23:05:29 on 2026-10-06, the owner's first start. A sample across repositories shows merges from
2026-10-02 to two hours before that start, each with the transition "its head was not heard by this owner,
so it is made from the forge's word", then *merged unchecked → held*. None has a walk.
- The other 11 are held for other reasons: a merge with no walk opened within ten minutes (10), and a group
so it is made from the forge's word", then *merged unchecked → held*. None has a rollout.
- The other 11 are held for other reasons: a merge with no rollout opened within ten minutes (10), and a group
whose composed check did not pass (1). They are not history, and this issue leaves them alone. One of them
was past its bound, which makes the 689th finding.
**Where it comes from.** The holder subscribes to the forge's `pull.merged` on the bus after reading back its
state and the controller's walks. The bus hands a new consumer the merges it still holds. For each, the holder
state and the controller's rollouts. The bus hands a new consumer the merges it still holds. For each, the holder
finds no delivery for the head, so it makes one from the forge's word. The pull request's statuses at its
merge do not include a passing `mesh/merge-gate` (the check did not exist for most of them), so the table's
*merged unchecked* row holds it.
**What would close them, and why not.**
- `release` walks a held delivery: wrong, these reached the machines already.
- `release` rollouts a held delivery: wrong, these reached the machines already.
- `stop` takes it to `stopped`, which says a person stopped a delivery and sets `mesh/delivery` to error on
the commit. Wrong in meaning, and it would ask the forge for a view on a pull request that is gone.
- Healer H2's `close` reads `held`'s bound, which names no H2 transition, and refuses: "the state is the
@@ -43,6 +43,6 @@ mesh-catalog#121 (the delivery's owner retires history and ends owed work the fo
under ADR 0250.
**Seen live.** `mesh-delivery.deliveries` on 2026-10-09 lists the held history as `retired`, all at
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The walk of
2026-10-08 09:40:40 UTC, and `mesh-delivery.retire-history` answers as a verb of the seat. The rollout of
mesh-catalog#121's own delivery reads `stopped`; the holder that answers today serves the fix all the
same, and `mesh-controller.status` names no machine behind.
@@ -53,6 +53,6 @@ lift.
every other state in the table moves on its facts?
3. Should `mesh/delivery-group` be required on the trunks, so a group cannot merge before its composed
check passed? To-be 47 leaves this to the operator; this issue is the case that setting decides.
4. The group's order put the catalogue's member first, and the node-engine's member walked at once on the
controller's own path. Is a group's order meant to hold for a member whose walk never waits for
4. The group's order put the catalogue's member first, and the node-engine's member rolled out at once on the
controller's own path. Is a group's order meant to hold for a member whose rollout never waits for
mesh-delivery?
@@ -3,7 +3,7 @@
**2026-10-08.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`),
from `mesh-delivery.show` of both members, and from the holder's journal on the control node.
**Where the hold is set.** `groupTurns` (holder.go) walks a closed group's members in order. For a member
**Where the hold is set.** `groupTurns` (holder.go) rolls out a closed group's members in order. For a member
in `published` it computes `composed`: the group's check exists, its verdict is `pass` or `warning`, and it
names the members' current heads. When `composed` is false it writes the member's `HeldWhy` ("its group's
composed check did not pass for the heads that merged") and settles it, and the table's `hold` row takes it
@@ -20,7 +20,7 @@ to `held`. `composed` is false in three different cases, and the code does not t
look at the members.
- `groupTurns` skips a member in `held` (`default: return nil`), so the tick after the verdict sees the
passing check and does nothing with it.
- The table has two rows out of `held`: `go` (the walk started on a word that was not this owner's) and
- The table has two rows out of `held`: `go` (the rollout started on a word that was not this owner's) and
`release` (a person's act). There is no observed row that takes a held delivery back when the reason it
was held is gone, and `HeldWhy` is never cleared once set.
@@ -41,7 +41,7 @@ check was asked. To-be 47 lists whether to require it as not decided.
- A lost verdict. The holder logged it at 10:10:21 and the group's record holds it.
- The order. The group's order was asked and kept before the check; it plays no part in the hold.
**Not this issue, noted.** The node-engine's member walked at once (`published` → `delivering` at the
**Not this issue, noted.** The node-engine's member rolled out at once (`published` → `delivering` at the
same instant), because the controller's own path does not wait for mesh-delivery (ADR 0239 decision 8).
The group's order put it second. Open question 4 in the report asks whether that is intended.
@@ -57,9 +57,9 @@ ADR 0270
its group's composed check is asked for the current heads and has no verdict.
- **A hold whose reason is gone lifts, by observation.** A member held because its group's composed check had not
passed goes back to `published` (the event *lifted*) once the check passes or warns for the heads that merged,
while its own walk is still open. It then takes its turn in order.
- A held member whose own walk was since taken over by a later walk is not lifted, since its walk can no longer
deliver it. It is `covered` when that later walk was delivered and sent its module (issue 319).
while its own rollout is still open. It then takes its turn in order.
- A held member whose own rollout was since taken over by a later rollout is not lifted, since its rollout can no longer
deliver it. It is `covered` when that later rollout was delivered and sent its module (issue 319).
Open question 3, whether `mesh/delivery-group` is required on the trunks, stays the operator's setting. Open
question 4 is not touched.
@@ -2,7 +2,7 @@
opened: 2026-10-08
---
# 317. A held delivery's module was walked by a later merge's plan
# 317. A held delivery's module was rolled out by a later merge's plan
## Symptom
@@ -22,7 +22,7 @@ took `openrazer` along. `mesh-controller.plans`:
| c78b5fc9 (#124's merge) | 11:07:02 | superseded at tier 0 by the plan at 86077780; docker, gitea, openrazer planned there again |
| 86077780 (#123's merge) | 11:07:03 | tier 0: distribution, docker, gitea and openrazer built from c78b5fc9; openrazer's gate on the laptop judging |
The controller's walk of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
The controller's rollout of the builds waiting for a gate then sent the laptop `openrazer` 88135ad0 →
c78b5fc9, beside `docker` and `node-tools`. From 11:10:58 UTC the laptop's condition for openrazer said
its account was *relogin needed*. That verdict exists only in the build #125 added, so the held change was
running on the laptop while its delivery said it waited for a person.
@@ -41,10 +41,10 @@ gone out. A person who releases it later releases nothing; a person who stops it
## Open questions
1. When a newer merge supersedes an older plan, does a module the older plan's walk was holding for its
1. When a newer merge supersedes an older plan, does a module the older plan's rollout was holding for its
delivery's word come along, or stay held with that delivery?
2. If it comes along, what does the held delivery become: superseded by the newer delivery, with the hold
carried over to it, or something else? Today the table has no row from `held` for a superseded walk.
carried over to it, or something else? Today the table has no row from `held` for a superseded rollout.
3. A trunk commit holds every earlier commit, so a later build of a module carries the held change in
any case. Is a hold on one delivery meaningful once a later merge of the same repository is walked,
any case. Is a hold on one delivery meaningful once a later merge of the same repository is rolled out,
or does a hold have to hold the module on that trunk until it is released?
@@ -3,7 +3,7 @@
**2026-10-08.** Read from `mesh-controller.plans` (the list, and each plan by id), `mesh-delivery.show`
and `deliveries`, and the code on the main branches of the controller and the catalogue.
**The held delivery's walk.** #125's merge made the plan at a6385479. Its walk waited for mesh-delivery's
**The held delivery's rollout.** #125's merge made the plan at a6385479. Its rollout waited for mesh-delivery's
word (`waited_for: mesh-delivery`, `waits: true`) and was never let go, because its delivery was held. Its
one module, `openrazer`, was "not yet asked": nothing was built.
@@ -11,32 +11,32 @@ one module, `openrazer`, was "not yet asked": nothing was built.
plan. For every open plan of the same repository and branch made before it, it takes each module the
older plan had not built (not asked, asked and not answered, or built and not yet sent), adds it to the
newer plan, and closes the older plan as superseded with the note "… planned there again". It does not
ask why the older plan had not built them. A walk waiting for a held delivery's word is open, so it is
ask why the older plan had not built them. A rollout waiting for a held delivery's word is open, so it is
superseded like any slow plan. This happened three times over (3f552274, c78b5fc9, 86077780), each newer
plan folding in what the one before it held.
The newest plan's own delivery (#123) was not held, so mesh-delivery let its walk go, and `openrazer` was
built from c78b5fc9. The controller's walk of the builds waiting for a gate then carried it to the laptop,
The newest plan's own delivery (#123) was not held, so mesh-delivery let its rollout go, and `openrazer` was
built from c78b5fc9. The controller's rollout of the builds waiting for a gate then carried it to the laptop,
since a send carries every move waiting on its machine (ADR 0236).
**Why the held delivery did not notice.** mesh-delivery's table has `superseded` rows only from
`published` and `delivering` ("a newer delivery to the same trunk took over its walk"). From `held` there
are only `go` and `release`. When the walk record said `superseded`, no row applied, so the delivery stayed
`held` with `held_why` unchanged, and its walk shows "superseded … openrazer planned there again". Two of
are only `go` and `release`. When the rollout record said `superseded`, no row applied, so the delivery stayed
`held` with `held_why` unchanged, and its rollout shows "superseded … openrazer planned there again". Two of
the three later deliveries (#122, #124) went to `superseded` correctly, through the `published` row.
**What the design says.** ADR 0218 decision 3 makes a newer merge take over an older plan's unfinished
work, and was written before deliveries could be held. ADR 0239 makes a held delivery wait for a person,
and to-be 47 says that nothing of a published delivery is registered before its walk starts, so that a
and to-be 47 says that nothing of a published delivery is registered before its rollout starts, so that a
send cannot carry it early. Neither says what a supersession does to a module a hold is keeping back.
**Ruled out.**
- A person's act. #125's transitions end at `hold`, with no `release`, and its own walk was superseded
- A person's act. #125's transitions end at `hold`, with no `release`, and its own rollout was superseded
while still waiting (`let_go` empty), so no `plans go` started it.
- Issue 295's path, a recorded build carried by a send: here the build was asked by a plan, not left
recorded.
**Located** in the controller's `supersededBy`, which folds a held walk's modules into a newer plan, and in
**Located** in the controller's `supersededBy`, which folds a held rollout's modules into a newer plan, and in
mesh-delivery's table, which has no way out of `held` when that happens, so the delivery's state stops
being true.
@@ -2,11 +2,11 @@
opened: 2026-10-08
---
# 318. One module's wait for a new login held every other module's walk
# 318. One module's wait for a new login held every other module's rollout
## Symptom
On 2026-10-08 at 11:08 UTC the controller opened its walk of the builds waiting for a gate: the laptop
On 2026-10-08 at 11:08 UTC the controller opened its rollout of the builds waiting for a gate: the laptop
first, then the workstation, then the control node. On the laptop it sent three moves in one send and
judged them together:
@@ -23,14 +23,14 @@ account was *relogin needed* from 11:10:58 UTC, which
ADR 0252
expects until the operator logs out and in again.
Meanwhile three other walks could not send to the rest of the mesh. `mesh-controller.plans` said of the
Meanwhile three other rollouts could not send to the rest of the mesh. `mesh-controller.plans` said of the
tool runner's (`novox/mesh-tools` d4845f7c, `node-tools`), the catalogue's (86077780, `docker`) and the
controller's (950562af, `build-agent`): "a build waits there for its gate: … the workstation (…, openrazer
88135ad0 → c78b5fc9) — a walk sends them, one machine at a time, each judged".
88135ad0 → c78b5fc9) — a rollout sends them, one machine at a time, each judged".
At 11:21:06 UTC the laptop's gate failed: "not healthy within 10m0s of its apply: its unit
openrazer.daemon … is unhealthy", and `openrazer` was put back to 88135ad0 on the laptop
(`build.openrazer.<laptop>.rolled-back`). The walk ended `failed`. After it, all three walks still said
(`build.openrazer.<laptop>.rolled-back`). The rollout ended `failed`. After it, all three rollouts still said
the same: `openrazer` c78b5fc9 waits on the workstation for a gate, so nothing is sent there or to the
machines after it.
@@ -40,14 +40,14 @@ The wait was known, and it was a person's: ADR 0252 says a new login holds a del
that machine. It held every module on every machine after it instead, for changes that had nothing to do
with it: the tool runner, the container engine and the build agent. The end was worse than the wait. The
gate's bound turned "waiting for a person" into "failed", the build that added the account group was put
back, and the walk that carries the other modules failed with it. A failed walk of the waiting builds
back, and the rollout that carries the other modules failed with it. A failed rollout of the waiting builds
does not open again on its own: a person releases it.
Three things meet here, each sound alone:
- a send carries every move waiting on its machine, and the gate judges them all at once
(ADR 0236);
- no other send may go where a move waits for a gate, so every walk waits for the one that judges it;
- no other send may go where a move waits for a gate, so every rollout waits for the one that judges it;
- a module's health that waits for a person reads as *not yet healthy*, which fails at the gate's bound.
## Open questions
@@ -61,5 +61,5 @@ Three things meet here, each sound alone:
back the account group it added, so the login the gate waited for can no longer help.
4. The daemon's unit had failed since 10:13 UTC, almost an hour before the new build reached the laptop.
Should a gate fail a build for a failure that was there before it was sent?
5. While a move waits for a gate on one machine, should walks of unrelated modules still be refused there,
5. While a move waits for a gate on one machine, should rollouts of unrelated modules still be refused there,
or only those whose send would carry it?
@@ -1,6 +1,6 @@
# 318 — diagnosis
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the walk of the waiting builds by id) between
**2026-10-08.** Read from `mesh-controller.plans` (the list, and the rollout of the waiting builds by id) between
11:17 and 11:22 UTC, `mesh-controller.conditions` and `status`, and the controller's code on its main
branch (`cmd/mesh-controller`).
@@ -9,15 +9,15 @@ gate all of them as carried moves. `judgeMoves` (gate.go) reads each module on e
worst reading. Any module *not yet healthy* resets the passes to zero for the whole send: `docker` and
`node-tools` were healthy, and the gate still said "0 of 3 passes" because `openrazer` was not. Past
`gateBound` (ten minutes after the apply) a *not yet healthy* gate fails. `g.Failing` names only the
modules found wanting, so only `openrazer` was put back, but the walk itself ended `failed`.
modules found wanting, so only `openrazer` was put back, but the rollout itself ended `failed`.
**Everything else waits for that verdict.** `ungatedIn` (release.go) refuses any send that is not a person's
and would carry a move no gate has seen, outside the send's own scope. A plan's sends to the rest of its
machines carry the machine's whole declaration, so while `openrazer` c78b5fc9 waited for a gate on the
workstation, the tool runner's, the catalogue's and the controller's walks were each refused there, and
they say so on every retry. They do not fail; they wait for the walk of the waiting builds, which goes one
workstation, the tool runner's, the catalogue's and the controller's rollouts were each refused there, and
they say so on every retry. They do not fail; they wait for the rollout of the waiting builds, which goes one
machine at a time and had stopped on the laptop. After it failed, the release code's rule applies: a
failed walk of the waiting builds stops the next one opening on its own until a person releases it.
failed rollout of the waiting builds stops the next one opening on its own until a person releases it.
**How *relogin needed* reached the gate.** ADR 0252 decision 5 says the node-engine states the account's
health as the module's resource of kind `account`, unhealthy with *relogin needed*, and that "the
@@ -33,8 +33,8 @@ running session lacks the group the daemon checks for. That is a guess, not chec
readings existed before or because of the person's step, and neither is a fault the build brought.
**What the design says.** ADR 0252's consequences say "a new login holds a delivery of that module on that
machine" and accept that a walk that starts on a workstation waits for the person. They do not consider the
other modules carried in the same send, the walks refused behind it, or the gate's ten-minute bound turning
machine" and accept that a rollout that starts on a workstation waits for the person. They do not consider the
other modules carried in the same send, the rollouts refused behind it, or the gate's ten-minute bound turning
the wait into a rollback. ADR 0236 makes one send judge everything waiting on a machine, to stop unjudged
builds from moving; it does not distinguish a module waiting for a person from one that is broken.
@@ -50,15 +50,15 @@ Whether the account actually left the group was not checked.
- A machine-wide fault on the laptop: its node-engine reported `applied` and current, and the gate's
wanting named one module's unit.
**Located** in the controller's walk and first-node gate: one verdict per send (`judgeMoves`), sends refused
**Located** in the controller's rollout and first-node gate: one verdict per send (`judgeMoves`), sends refused
on any machine where a move waits (`ungatedIn`), and no reading for a health that waits for a person, so a
known wait fails at the bound and stops every walk behind it.
known wait fails at the bound and stops every rollout behind it.
**Decided and fixed, 2026-10-08**, by ADR 0254.
Open questions 1 to 3: *relogin needed*, and a unit failing in that same account's own manager, read *waits
for a person*, which passes with the wait carried and is said as the module's condition `relogin-needed`;
nothing is put back for it. Question 2's other half: each module of a send counts its own passes, and, after
the settle time, in the walk path, keeps a pass when another fails beside it. Question 5: other walks are
the settle time, in the rollout path, keeps a pass when another fails beside it. Question 5: other rollouts are
still refused while a move waits for a gate (ADR 0236 §4a stands). With the wait a pass, 318's case leaves
nothing waiting for them. A fault decided before the settle time, and a plan's own tier path, still leave
the healthy modules of a failed send without a verdict (ADR 0254, consequences). Question 4, a unit that
@@ -71,4 +71,4 @@ between 12:41 and 13:03 UTC; the node-engine's and the controller's deliveries r
**Seen live.** On 2026-10-09 `mesh-controller.conditions` holds `module.openrazer.<machine>.relogin-needed`
for the laptop and the workstation, raised at 21:02 UTC: "Its update is in place and nothing was undone",
and no walk waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
and no rollout waits behind it (`mesh-controller.status`: nothing waiting, no machine behind).
@@ -15,8 +15,8 @@ What they are, read with `mesh-delivery.deliveries` and `show`:
| how many | repository | why held | what it builds |
|---|---|---|---|
| 4 | hq | merged without a passing check | nothing: "the change touches no module of the mesh's graph" |
| 2 | catalogue | no walk opened within ten minutes of the merge | a new module, sent nowhere |
| 8 | controller | no walk opened within ten minutes of the merge | the controller, the build agent, the route proxy |
| 2 | catalogue | no rollout opened within ten minutes of the merge | a new module, sent nowhere |
| 8 | controller | no rollout opened within ten minutes of the merge | the controller, the build agent, the route proxy |
| 2 | catalogue | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | one module each |
Two examples:
@@ -26,7 +26,7 @@ Two examples:
`proposed` to `held`: "merged without a passing check". Its plan builds nothing.
- `novox/mesh-controller@ec3769a8a669`, controller #105. Ready at 2026-10-07 00:26 UTC, merged at 00:27,
held at 00:37: "merged, and the controller opened no walk for it within 10m0s". The controller's modules
have been walked since from later commits of the same main, for example the group
have been rolled out since from later commits of the same main, for example the group
`feat/delivery-checks-verb`, whose controller member is `delivered`.
Five of them (the four hq merges and controller #105) are past the held bound of one day. The self-check's
@@ -39,7 +39,7 @@ A held delivery asks a person "does this go on?". For these sixteen the question
a later commit of the same trunk has already delivered what they held, or they hold nothing to deliver. The
state table offers a person four ways out of `held`, and none fits:
- `release` starts a walk of the old commit: wrong for a commit a later one superseded, and nothing for one
- `release` starts a rollout of the old commit: wrong for a commit a later one superseded, and nothing for one
that builds nothing.
- `retire-history` is for history made from the forge's word (ADR 0250),
and refuses every one of these because their heads were heard.
@@ -57,7 +57,7 @@ The issue states the need; the decision is for graduation.
- **An end for a held delivery that was superseded**, as `published` and `delivering` already have
(`superseded`, "a newer delivery to the same trunk took over its walk"): either because a later commit's
delivered walk carried the same modules on the same trunk, or because it builds nothing at all.
delivered rollout carried the same modules on the same trunk, or because it builds nothing at all.
- **Automatic where nothing is lost**: a held delivery whose plan builds nothing has nothing for a person to
decide.
- **A person's act, with why, otherwise**: a held delivery whose modules a later commit delivered is ended
@@ -70,5 +70,5 @@ The issue states the need; the decision is for graduation.
2. The four hq merges were heard at the owner's first start from the bus's history, as announcements rather
than merges. Are they history in the sense of ADR 0250, missed by its rule because their head was
replayed too?
3. Why did the controller open no walk for eight of its own merges within ten minutes? The controller's own
path never waits, so its walk may have been missed rather than absent. Not diagnosed here.
3. Why did the controller open no rollout for eight of its own merges within ten minutes? The controller's own
path never waits, so its rollout may have been missed rather than absent. Not diagnosed here.
@@ -7,14 +7,14 @@
| event | kind | needs |
|---|---|---|
| `go` | observed | its walk started on a word not this owner's (the controller's own path, or a person's `plans go`) |
| `release` | a person's act | why and who; it walks the delivery's own commit |
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no walk open |
| `go` | observed | its rollout started on a word not this owner's (the controller's own path, or a person's `plans go`) |
| `release` | a person's act | why and who; it rolls out the delivery's own commit |
| `retire` | a person's act | `isHistory`: made from the forge's word, merge heard 15 minutes or more after it happened, no rollout open |
| `stop` | a person's act | why and who; from any state not final |
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the walk's
record saying `superseded`. A held delivery whose walk never opened has no walk record to read, and one
whose walk was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
There is no `superseded` row from `held`. `published` and `delivering` have one, guarded by the rollout's
record saying `superseded`. A held delivery whose rollout never opened has no rollout record to read, and one
whose rollout was superseded ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md))
does not match it either.
**Why `retire-history` refuses them.** `isHistory` requires `FromForgesWord()`: the delivery was made from
@@ -41,7 +41,7 @@ Neither considers a held delivery whose content a later commit delivered.
**Ruled out.**
- A missing release. Releasing any of these would walk an old commit over a newer one, or walk nothing.
- A missing release. Releasing any of these would roll out an old commit over a newer one, or roll out nothing.
- D14 itself. It reports the state truthfully; the state has no way to change.
**Located** in mesh-delivery's state table (no end for a superseded or empty held delivery, and history
@@ -52,14 +52,14 @@ read by how a delivery was made) and in to-be 47, whose state table carries the
**Eleven held deliveries, read again.** On 2026-10-09 eleven deliveries were `held`, each raising D14's
`delivery.<id>.stalled` and held for 24 to 48 hours. For each, `mesh-delivery.show` gave its state, why it was
held, its merge commit and its plan. A fresh clone of the repository then showed whether the merge commit is an
ancestor of the commit a finished walk delivered. `mesh-controller.status` listed none of the modules involved
ancestor of the commit a finished rollout delivered. `mesh-controller.status` listed none of the modules involved
as behind. All eleven were covered: none held a change missing where it should run.
**Why the controller opened no walk (open question 3).** This part comes from reading the code, and part of it
**Why the controller opened no rollout (open question 3).** This part comes from reading the code, and part of it
is still a guess.
- *A new module.* When a merge only adds a module, the controller's `SourceMoved`
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no walk ("built,
(`cmd/mesh-controller/upgrades.go`) builds and registers it, sends it nowhere and opens no rollout ("built,
registered when the build lands, and sent nowhere", issue 300). mesh-delivery's `MovesNothing()`
(`model.go`) counts `Plan.New` as something to move, though. So a delivery whose plan is "builds new: …, sent
nowhere" is not let through to `published` → `delivered`. Instead the unwalked rule in `holder.go` holds it
@@ -67,25 +67,25 @@ is still a guess.
read from the code, and it matches `novox/mesh-catalog@60641176abaf`.
- *The controller's own merges (a guess, not proved).* `SourceMoved` also returns without a plan in two cases:
when the modules the merge touches are "already built from it" (`already > 0`), and when the merge is read as
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no walk
of its own when another walk, or a person's push, had already built from a later commit. Which of the two hit
history (`isHistory`: merged before the source was last looked at). Either would leave a merge with no rollout
of its own when another rollout, or a person's push, had already built from a later commit. Which of the two hit
these merges is not yet known. The controller's journal for the minutes after each merge would say.
- *The link from a walk to a delivery.* `deliveryWalking()` (`holder.go`) links a walk to the delivery whose
`merged_as` equals the walk's commit, and to no other delivery. A later walk that carried an earlier merge's
change therefore never reaches that earlier delivery, and nothing reads "this walk's commit contains your
- *The link from a rollout to a delivery.* `deliveryWalking()` (`holder.go`) links a rollout to the delivery whose
`merged_as` equals the rollout's commit, and to no other delivery. A later rollout that carried an earlier merge's
change therefore never reaches that earlier delivery, and nothing reads "this rollout's commit contains your
merge". That is the missing input the first open question asks about. mesh-delivery could answer it from the
forge (is the merge commit an ancestor of the delivered walk's commit) together with the walk's moved modules,
forge (is the merge commit an ancestor of the delivered rollout's commit) together with the rollout's moved modules,
without asking the controller what each machine runs.
**Four were closed with `stop`, before this issue was found.** On 2026-10-09 the operator agreed to close the
held deliveries. Four were stopped with why "covered: its merge … is contained in … which is delivered (walk …)":
| delivery | merge | contained in | walk |
| delivery | merge | contained in | rollout |
|---|---|---|---|
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier walk of the controller |
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same walk |
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same walk |
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue walk |
| `novox/mesh-controller@ec3769a8a669` | 099c176 | cec797e (the controller's main) | a finished three-tier rollout of the controller |
| `novox/mesh-controller@ea09f074db01` | e7dca6c | cec797e | the same rollout |
| `novox/mesh-controller@98ef7bac6e6a` | cdbbd9e | cec797e | the same rollout |
| `novox/mesh-catalog@60641176abaf` | 3da80a4 | a1406e7 (the catalogue's main) | a new module sent nowhere; a later finished catalogue rollout |
Their stalled conditions cleared at the next D14 observation. Open D14 conditions went from 11 to 7. As this
issue already said, `stop` is the wrong record for them: their merged pull requests now carry `mesh/delivery`
@@ -101,12 +101,12 @@ remaining seven were left `held` for this issue's fix.
| `novox/hq@2221be11a6ef` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/hq@897bcce50252` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/hq@ef6ab814b053` | merged without a passing check | merge on hq's main; builds nothing | the same |
| `novox/mesh-controller@b98fd0f396c8` | no walk opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
| `novox/mesh-controller@ea92af2a7d77` | no walk opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release walk that is done; the step looks stale but is not final |
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished walk of the same module | its group is still `delivering`, and its own walk is `superseded` |
| `novox/mesh-controller@b98fd0f396c8` | no rollout opened within ten minutes | 7f25666, contained in cec797e | a member of a group still `delivering` (2 of 3); `stop` stops the group's later members too |
| `novox/mesh-controller@ea92af2a7d77` | no rollout opened within ten minutes | 3808634, contained in cec797e | `show` lists a step still `judging` on the control-node, carried by a release rollout that is done; the step looks stale but is not final |
| `novox/mesh-catalog@66106d93accd` | its group's composed check did not pass for the heads that merged ([issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) | 7cd989b, contained in a later finished rollout of the same module | its group is still `delivering`, and its own rollout is `superseded` |
The seven show the two ends this issue asks for: four hold nothing to deliver, and three are covered by a
later walk.
later rollout.
## 2026-10-09 — the fix, proposed
@@ -115,12 +115,12 @@ ADR 0270
mesh-delivery:
- **A merge that moves nothing is delivered at once.** `MovesNothing()` no longer counts `Plan.New`: a module
the controller builds and sends nowhere opens no walk, so there is nothing to wait for. A merge whose plan moves
the controller builds and sends nowhere opens no rollout, so there is nothing to wait for. A merge whose plan moves
nothing goes from `proposed`, `checked`, `rejected` or `held` to `delivered`, before the row *merged unchecked
→ held*.
- **`covered`, a new final state: a later merge that contains it was delivered.** For every module its plan
moves, a delivered delivery of the same repository and trunk, merged after it, delivered on its own passing
check by the walk of its own merge commit, whose machine steps say the module reached every machine running
check by the rollout of its own merge commit, whose machine steps say the module reached every machine running
it. The forge then says, through a new read-only tool `gitea_contains` (its compare of the two commits),
whether the later merge commit contains this one; merge order is only the first filter. A forge that cannot
answer leaves it held. `mesh/delivery` says `covered: a later merge that contains it was delivered: …` as a
@@ -128,7 +128,7 @@ mesh-delivery:
- **Read on every tick**, so what is held now ends without an event.
This answers open question 1: mesh-delivery reads it from its own records and asks the forge for ancestry; the
controller is not asked. Open question 3, why the controller's own merges opened no walk, is not answered by this fix; it no
controller is not asked. Open question 3, why the controller's own merges opened no rollout, is not answered by this fix; it no
longer leaves anything held.
**How to verify after mesh-delivery is delivered with it.** Within a minute of its start:
@@ -137,13 +137,13 @@ longer leaves anything held.
merges (`builds new: …, sent nowhere`), and none of the five controller merges held for "no walk opened".
The hq and module-adding ones are `delivered`; the controller ones are `covered`.
2. `mesh-delivery.show` of one of them, for example `novox/mesh-controller@b98fd0f396c8`, ends with a
transition `held → covered (covered)` whose why names the later delivery and its walk.
transition `held → covered (covered)` whose why names the later delivery and its rollout.
3. `mesh-delivery.checks` for its pull request shows `mesh/delivery` as `success covered: …`.
4. The forge module is delivered in the same pull request. Until its `gitea_contains` answers, nothing is
covered, and the held controller merges stay held.
5. At the self-check's next reading, `mesh-controller.conditions` no longer lists their
`delivery.<id>.stalled`.
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered walk sent
6. The two catalogue merges held for their group (issue 316) are covered only if a later delivered rollout sent
their module. One that stays held is a real question: its change may not run anywhere.
**Accepted 2026-10-09.** The operator accepted the record named above; it is no longer proposed.
@@ -52,7 +52,7 @@ not addresses, paths or secrets — so they reached the operator untouched.
**Nothing said whether the operator was needed, or let them answer.** No field of a condition said whether
the operator was needed, so every warning was shown, and an explanation could imply an instruction. The
verbs that answer the common cases existed (release or stop a delivery, start a waiting walk, restart a
verbs that answer the common cases existed (release or stop a delivery, start a waiting rollout, restart a
service, silence a condition), but the desk could not take an answer: `node-notifier.send` had no actions,
and the conversation design that adds them (to-be 46) was not built. The one way to act was a session.
@@ -70,7 +70,7 @@ nothing is not sent. The desk offers the actions, and the operator channel's hol
through its seat verb. Every channel says the time in the operator's zone; a clearance is one line.
- novox/mesh-controller: the condition's three new fields, the plain rule, a wording for every kind the
controller raises, the richer words at the producers that know more (a walk's modules, a unit's name, a
controller raises, the richer words at the producers that know more (a rollout's modules, a unit's name, a
hand-act's cause, a delivery's repository), and a lint over every condition the suite raises.
- novox/mesh-catalog: the messenger shows the words, keeps quiet what needs nothing, offers and performs
the answers, says since when in the operator's zone (the new setting `time-zone`), says a clearance in
@@ -88,5 +88,5 @@ To-be 45 §2 is amended: what `status` lists, and the new condition.
## Resolved, 2026-10-09
mesh-controller#150 and mesh-lab#68 (R325) merged on 2026-10-08 at 14:58 UTC. Every later controller
delivery carries #150; the last, of mesh-controller#170, finished its walk on 2026-10-09 at 00:11 UTC, and
delivery carries #150; the last, of mesh-controller#170, finished its rollout on 2026-10-09 at 00:11 UTC, and
`mesh-controller.status` named no machine behind.
@@ -20,7 +20,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
1.4 a second, or about 43 every 30 seconds.
- Over the server's whole uptime (39 hours 49 minutes at 14:13 UTC) the average is 39.7 writes a second.
In that window one walk was in progress, so a few of those writes were real changes. The diagnosis
In that window one rollout was in progress, so a few of those writes were real changes. The diagnosis
shows why most were not.
## Why it is an issue
@@ -40,7 +40,7 @@ watcher, and the write rate cannot be used to see that deliveries are moving.
## How it is checked
- **In mesh-delivery's tests:** a reconcile pass over walks that did not change performs no write. The
- **In mesh-delivery's tests:** a reconcile pass over rollouts that did not change performs no write. The
test's fake store counts writes, and the test fails on any write.
- **Live:** over ten minutes in which no delivery changes state, the write count on
`$KV.mesh-delivery_deliveries.>` (`nats.nats_subscriptions`) rises by no more than the number of
@@ -56,6 +56,6 @@ build agent. So it resolves without a replay. The test above is still the proof.
- `keep` writes only when the delivery differs from the value it last wrote. Either compare the encoded
value with the last one written, or set a changed flag wherever the delivery is changed.
- `Reconcile` skips a walk whose revision the delivery already holds.
- `Reconcile` skips a rollout whose revision the delivery already holds.
- A retry of an owed effect that fails again changes only its try count and next time. Whether that
needs a write at every try is a choice for the fix: the count may stay in memory between writes.
@@ -12,16 +12,16 @@ runtime. `keep` does not compare the delivery with what it wrote before. Every c
of transitions is empty.
- `settleAndKeep` calls `settle` and passes the result to `moved`. A delivery with nothing to move is
still written.
- `walkMoved` takes one walk from the controller, records it on its delivery (`linkWalk`) and calls
`settleAndKeep`. A walk that has not changed since the last pass leaves the delivery as it was, and
- `walkMoved` takes one rollout from the controller, records it on its delivery (`linkWalk`) and calls
`settleAndKeep`. A rollout that has not changed since the last pass leaves the delivery as it was, and
the delivery is written anyway.
- `Reconcile` asks the controller's `delivery-walks` for the walks it keeps and calls `walkMoved` for each.
The answer holds every open walk and the last 50 that ended. The ticker in main.go runs `Reconcile`
- `Reconcile` asks the controller's `delivery-walks` for the rollouts it keeps and calls `walkMoved` for each.
The answer holds every open rollout and the last 50 that ended. The ticker in main.go runs `Reconcile`
every 30 seconds.
So each pass writes about one delivery per walk the controller returns. On 2026-10-08 that was 50 walks
(32 done, 14 superseded, 3 failed, 1 walking), against about 43 writes per pass measured. *I think* the
difference is walks that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
So each pass writes about one delivery per rollout the controller returns. On 2026-10-08 that was 50 rollouts
(32 done, 14 superseded, 3 failed, 1 `walking`), against about 43 writes per pass measured. *I think* the
difference is rollouts that match no delivery: `walkMoved` keeps those aside in memory and writes nothing.
**What made it 40 a second before today.** Before mesh-catalog #121 (the fix of
[issue 313](../313-history-was-held-for-a-person-and-its-views-retried-for-ever/00-report.md), merged
@@ -39,5 +39,5 @@ doubling to an hour) and ends them after 60 tries. A retry still writes its deli
- *The groups bucket.* `mesh-delivery_groups` has 26 keys, and its subscription counted 719 writes in the
same 39 hours.
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every walk the
**Located** in mesh-delivery: `keep` writes without comparing, and `Reconcile` passes every rollout the
controller keeps through `settleAndKeep` every 30 seconds.
@@ -24,7 +24,7 @@ Measured again on 2026-10-08 with the `nats` module's tools:
The long-lived connections were young too. At 14:15 UTC the controller's two were 4 minutes old, the
build agents' 2 to 19 minutes, the route proxies' 13 to 17 minutes and the tool runners' 10 minutes to 3
hours. Two walks were sending new builds of the controller, the build agent and catalogue modules
hours. Two rollouts were sending new builds of the controller, the build agent and catalogue modules
between 14:00 and 14:20 UTC, and the build agent's container on the control node started at 14:19 UTC.
*I think* those young connections are restarts by deliveries; they are tens, not thousands.
@@ -17,7 +17,7 @@ controller's, so each place that needs the bus opens one, uses it and closes it:
command.
- `sayingOnTheBus` (delivery.go): a delivery's `check`, `go` and `stop`, which mesh-delivery asks for.
The serving controller dials too. Its watchdogs run every 30 seconds, and while a walk waits for a
The serving controller dials too. Its watchdogs run every 30 seconds, and while a rollout waits for a
build, `buildSeatPause` (plan_retry.go) opens a fresh connection to read whether the build seat's
holders paused, and closes it. `status` does the same.
@@ -41,7 +41,7 @@ its long-lived connections are named `mesh-controller` too and are told apart on
minute. The research's guess came from their age alone.
**Located** in the controller: one connection per command process that touches the bus, and one per
watchdog tick while a walk waits for a build.
watchdog tick while a rollout waits for a build.
## 2026-10-09, resolved
@@ -17,7 +17,7 @@ the fix is the attribution tool; with it, this diagnosis continues.
- **A holder that never said it paused.** `PausedSaid` (controller, internal/link/queue.go) asks for the
last message on each build holder's pause subject. A holder that never said one gets a not-found
answer, which the server counts as an API error. The serving controller's watchdog reads it every 30
seconds while a walk waits for a build (issue 327).
seconds while a rollout waits for a build (issue 327).
- **Short-lived consumers.** The controller heard 495 "consumer deleted" advisories in the first 8
minutes of its current connection, about one a second. They are short-lived consumers: the controller's
advisory reader (internal/link/advisories.go) ignores them and says why, "every watch of a bucket and
@@ -7,12 +7,12 @@ opened: 2026-10-08
## What was observed
On 2026-10-08 a person released a delivery of the catalogue that the group check (issue 316) had held. Its
walk had one tier with one module, `dunst`, which runs on the laptop and the workstation. The walk sent the
rollout had one tier with one module, `dunst`, which runs on the laptop and the workstation. The rollout sent the
new build to the laptop first, and the gate passed it there:
> dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent
Twenty-eight minutes later the same walk said the opposite, about the same gate:
Twenty-eight minutes later the same rollout said the opposite, about the same gate:
> dunst failed its gate on the laptop in tier 0: healthy 3 times over 2m5s; put back dunst to ea462666 on
> the laptop, in one send; the workstation left as it was
@@ -31,22 +31,22 @@ build that worked, recorded it as failed, and so blocked its delivery until a ne
## Fix
The controller judged the first machine's report again on every step of the walk, also after the gate had
The controller judged the first machine's report again on every step of the rollout, also after the gate had
passed (see `01-diagnosis.md`). The fix merged in mesh-controller pull request 165 (8170fc5), with its
replay in mesh-lab pull request 72.
- Once a module's gate has passed, the first machine's later reports no longer decide anything about it:
the walk sends the rest.
the rollout sends the rest.
- A module carried by another module's send takes over that pass before its step is read.
- A guard, not reached with the two changes above: should a later path still stop a walk at a module whose
gate passed, that module is not put back and its build is not marked failed. The walk stops with the
- A guard, not reached with the two changes above: should a later path still stop a rollout at a module whose
gate passed, that module is not put back and its build is not marked failed. The rollout stops with the
reason, and the module is left sent to its first node only, which `plans retry` takes up again.
## How it is checked
`TestReplay335` in the controller replays the live case with only what the controller had before the fix:
a passed gate on the laptop, the laptop's later report not about this build (or a failure of another
send), and the walk 30 minutes and 9 seconds after the first send. A second test drives the walk over a
send), and the rollout 30 minutes and 9 seconds after the first send. A second test drives the rollout over a
tier of two modules, the second carried by the first's send, and fails if the carried module's step is read
before it takes over the pass. On the commit before the fix it fails
with "laptop did not report it applied within 30m0s", and the passed build reaches the put-back. It is R335
@@ -6,15 +6,15 @@
found ADR 0236 (what a rollback is), issue 318 (a wait for a login failing a gate) and issue 279, none of
them about a gate that passed and was then failed.
**The walk's own lines.** The controller's journal on the control-node said, for this walk (times UTC):
**The rollout's own lines.** The controller's journal on the control-node said, for this rollout (times UTC):
| time | what the controller said |
|---|---|
| 16:44:45 | tier 0 built; sent dunst to the laptop first, the rest once its gate passes |
| 16:44:55 | the laptop applied what it was sent |
| 16:46:52 | dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent |
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another walk sends |
| 16:49:12 | the other walk (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another rollout sends |
| 16:49:12 | the other rollout (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation |
| 17:14:54 | dunst failed its gate on the laptop: healthy 3 times over 2m5s; put back on the laptop; the workstation left as it was |
The laptop's node-engine logged `applied` at 16:44:55 and nothing more until 18:01:58. It did not report
@@ -25,14 +25,14 @@ on the 16:49:12 send for over an hour. Why it did not is not yet known; the lapt
waits 30 minutes (`planWaitBound`) for a first machine to report what it was sent. The failure came at that
bound and not at the gate's.
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a walk reads
**The code path.** In `advanceOnce` (`release_plan.go`), every step of a rollout reads
`nextRollout(state, …)` before anything else, for every module in the tier not yet sent to the rest. For a
module sent first, `nextRollout` looks at the first machine's last report. If that report is not about the
declaration the machine was last sent (`Current` false), it waits; past the bound it answers
`failed: <machine> did not report it applied within 30m0s`. Nothing in it looked at the gate's verdict.
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the walk tries again on
the next step). At 16:49:12 the other walk's send made the laptop's last report no longer current. From
After the pass at 16:46:52, the send to the rest was refused (the error returns, and the rollout tries again on
the next step). At 16:49:12 the other rollout's send made the laptop's last report no longer current. From
then on `nextRollout` waited for a report it would never get about this build. At the bound it failed, and
`advanceOnce` called `failFirstSend`.
@@ -47,7 +47,7 @@ laptop. That is the line seen at 17:14:54.
- The person-wait of issue 318 (ADR 0254), `putBackBroken` and the carried-module paths: the plan had one
module, nothing was carried, and nothing was found broken.
- Sleep: the laptop did not sleep until 18:01, and the failure was at 17:14:54.
- The release itself (`mesh-delivery.release`): it let the walk go at 16:44, and the walk ran as any other.
- The release itself (`mesh-delivery.release`): it let the rollout go at 16:44, and the rollout ran as any other.
**Located.** mesh-controller, `cmd/mesh-controller/release_plan.go`: `nextRollout` judged a passed gate's
first machine again, and `failFirstSend` put back a module whose gate had passed. The path has been there
@@ -57,4 +57,4 @@ step as the pass and another send reaches the first machine before the 30-minute
**What it needs to deliver dunst.** The build is recorded as failed at its gate, and the controller does
not send a build marked that way again on its own. The registered build is back at the one before. Once the
fix is delivered, a new build of dunst (`rebuild dunst`, or a newer merge of it) is judged at the gate
again and walked to both machines.
again and rolled out to both machines.
@@ -20,10 +20,10 @@ From the merge on, every send to the control-node was held. `mesh-controller.pla
> planned step: sending control-node would replace the bus (nats 88135ad0 → 32307bd1), which is a planned
> step ... — tried again for 27m45s
The same refusal held the controller's own walk and the release of builds kept back for a gate.
The same refusal held the controller's own rollout and the release of builds kept back for a gate.
For those 28 minutes `mesh-controller.conditions` listed **no condition** naming the hold or saying it
waited for a person. The operator was not told. The hold was found only because a walk looked stuck, and
waited for a person. The operator was not told. The hold was found only because a rollout looked stuck, and
the operator asked: "where would you be waiting then? Where is my approve button?"
The operator then gave the word, the bus upgrade started at 18:52 and the bus answered healthy at 18:54.
@@ -33,7 +33,7 @@ The held sends then went through.
The design makes the bus wait for a person, and that part worked: nothing replaced the bus unasked. But a
wait that only a person can end is said nowhere a person looks. The refusal is kept as the note on each
held walk, which is read only by somebody already asking `plans`. Rule 4 of
held rollout, which is read only by somebody already asking `plans`. Rule 4 of
ADR 0227
(no failure is silent) is met in the letter and missed in effect: the mesh held every send to its
most important machine and said nothing until asked.
@@ -47,10 +47,10 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
## Impact
- Every send to the control-node stops: the controller's own walk, the node-engine's, every module's walk
- Every send to the control-node stops: the controller's own rollout, the node-engine's, every module's rollout
that includes the control-node, and gated builds waiting for release. Other machines in the same tier
wait with it.
- The operator learns of it only by noticing a stuck walk, which can take hours when nobody is watching.
- The operator learns of it only by noticing a stuck rollout, which can take hours when nobody is watching.
- The only condition that could have fired, a tier past its bound (watchdog S3, "stalled"), fires no
sooner than 30 minutes in, and its words ("stuck halfway ... longer than usual. Nothing is lost") name
neither the bus nor the verb a person runs. It never fired in this case, because the step was taken
@@ -62,9 +62,9 @@ The diagnosis is in [01-diagnosis.md](01-diagnosis.md).
| Rule | Check |
|---|---|
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a walk wants to send raises a condition on the next watchdog tick, before any tier bound |
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held walks or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a walk whose only refusal is the bus's planned step |
| a send held for the bus's planned step is a condition at once | the controller's test: a pending bus build whose machine a rollout wants to send raises a condition on the next watchdog tick, before any tier bound |
| its words say what waits, behind what, and the exact verb | the same test reads the headline and explanation: the held rollouts or modules, the bus build from and to, and `mesh-controller.bus upgrade` |
| the hold is not also reported as lateness | the same test: S3 raises no `stalled` for a rollout whose only refusal is the bus's planned step |
| it clears when the step starts | the test ends with a bus step started: the condition clears, `bus-maintenance` opens |
| the merge says it before it happens | the merge gate's test: a change that moves the bus's build reports that a bus upgrade by a person is needed and which machine waits until then |
@@ -81,8 +81,8 @@ why none is possible.
with the operator as its resolver). Its headline says, in plain words, what waits ("sends to the
control-node: the controller, the node-engine, …"), behind what (the bus build, from and to), since
when, and the verb that ends it (`mesh-controller.bus upgrade --why … --reversible|--irreversible`). It
is raised on the first refusal, not after a bound. A walk held only by it is excluded from S3, as a
walk held by a paused build seat is.
is raised on the first refusal, not after a bound. A rollout held only by it is excluded from S3, as a
rollout held by a paused build seat is.
2. **An ask, once asks exist.** When the decision flow of ADR 0259 (an ask is a message on the bus, any
channel answers it; on an open pull request, not yet merged) is built, the condition becomes an ask
to the operator: "start the bus upgrade now?", whose yes runs the step with the operator's word
@@ -13,13 +13,13 @@ The send is refused whole. `busHeld` (`bus_step.go`) gives the words "sending X
(nats from → to), which is a planned step: `bus upgrade --why …` …". So the remedy is in the words, but
only in the error.
**Where the error goes.** In `release_plan.go`, the walk's step loop catches the error, keeps it as the
**Where the error goes.** In `release_plan.go`, the rollout's step loop catches the error, keeps it as the
plan's note ("tier N: … — tried again"), saves the plan when the note changed, prints it to the
controller's log once, and tries again on the next tick. Nothing raises a condition from it. That is why
`plans` showed it and `conditions` did not.
**Which watchdog could have said it.** Only S3 (`watchPlans`, `signals.go`), "plan tier progress", raises
`plan.<id>.stalled` when a walk has been in one tier past its bound. The bound is
`plan.<id>.stalled` when a rollout has been in one tier past its bound. The bound is
`max(30 min, 3 × the p90 of that repository's measured tiers)` (`tierAtLeast = planWaitBound = 30 min`,
`watchdogs.go` `boundsOf`). So:
@@ -29,14 +29,14 @@ controller's log once, and tries again on the next tick. Nothing raises a condit
mesh keeps it where it is."). Its summary carries the plan's note, so the bus's reason would have
appeared there, but the headline a person reads names neither the bus nor the verb, and its resolver is
not the operator;
- a walk held by a paused build seat is excluded from S3 (`pausedWaiting`), and a walk waiting for its
delivery's word has its own watchdog (S16). A walk held for the bus step has neither.
- a rollout held by a paused build seat is excluded from S3 (`pausedWaiting`), and a rollout waiting for its
delivery's word has its own watchdog (S16). A rollout held for the bus step has neither.
**What the bus step does raise.** Probe DB (`probeBusStep`, `bus_step.go`) raises `bus-maintenance` only
while a step runs, and `bus-upgrade-failed` when one ended unhealthy. It reads only the latest bus step.
Before a step is started there is nothing for it to read, so a pending bus build raises nothing.
**Before the merge.** `delivery.go` lists the bus among the modules whose walk starts at the merge
**Before the merge.** `delivery.go` lists the bus among the modules whose rollout starts at the merge
(`onTheControllersPath`), and the merge gate (`merge_gate.go`) composes the machines with the change. Neither
reads `pendingBus` or says that the change will hold the bus's machine until a person runs the step. So a
merge that moves the bus's build passes green and blocks every later send to the control-node.
@@ -64,12 +64,12 @@ person, and the person must be told.
ADR 0271
(proposed) takes suggested fixes 1 and 3. mesh-controller #173 builds them:
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a walk's send was refused
- **Row S17**, `bus.<bus module>.step-waiting`: raised on the first watchdog tick after a rollout's send was refused
for the bus's planned step, while the new bus build would still replace the bus on its machine. It is a
warning, for the operator. It names the machines, each waiting walk and the modules of its tier, the bus build
warning, for the operator. It names the machines, each waiting rollout and the modules of its tier, the bus build
from and to, since when, and the `mesh-controller.bus` call with `upgrade`. It clears once the bus's machine has
been sent the build the mesh holds.
- **S3 leaves out a walk held only by it.**
- **S3 leaves out a rollout held only by it.**
- **The delivery plan says it before the merge**: "merging this needs a person's bus upgrade: … nothing else is
sent to <machine> either", and its summary, which `mesh/merge-gate` carries, says the same.
@@ -20,7 +20,7 @@ The plan rebuilt and moved them whatever the merge changed. Two examples:
- Mesh-controller PR #159, merged as `ca09a07f`, changed files under `cmd/mesh-controller`,
`internal/broker`, `internal/link`, `internal/catalogue/verbs.go` and the root `module.json`. Nothing
under `examples/route-proxy` changed. Its walk said, on the home server, "recreates 1 of route-proxy's
under `examples/route-proxy` changed. Its rollout said, on the home server, "recreates 1 of route-proxy's
3 container(s) (with a new image: server)" and "recreates 1 of build-agent's 1 container(s)".
- Mesh-controller PR #163, merged as `9b028b42`, did the same. The route proxy's program at that commit
is byte for byte the program at the commit before it (measured in the diagnosis), and it was still
@@ -43,8 +43,8 @@ The operator stated the rule the delivery plan should follow:
## Why it is an issue
- **Deliveries are slow.** A controller merge walked for about fifteen minutes instead of about five,
and the walks of one repository must not overlap (issue 335), so every merge after it waits too. The
- **Deliveries are slow.** A controller merge rolled out for about fifteen minutes instead of about five,
and the rollouts of one repository must not overlap (issue 335), so every merge after it waits too. The
operator considers this a major bug and a main reason deliveries are slow.
- **A public service restarts for nothing.** The route proxy on the home server was recreated at
merges that did not change its program.
@@ -31,7 +31,7 @@ Read in mesh-controller at `8170fc58`.
4. `mesh-controller` itself is built from the repository's root (its manifest's bundle compiles
`cmd/mesh-controller`; its source path is empty), so every changed file in the repository touches
it as well.
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the walk serial here:
5. The tiers come from `inventory.Dependencies`. Two of its edge kinds make the rollout serial here:
- a `packages` edge, from a module whose build read a repository to the module whose source that
repository is. So `route-proxy` and `build-agent` both depend on `mesh-controller`, because they
share its repository, not because either is built from the controller's artifacts;
@@ -108,9 +108,9 @@ change. The build source of the route proxy is `examples/route-proxy`, `internal
## Measured: what the extra tiers cost
From `mesh-controller.plans` (read only), the first-node gate of each tier of tonight's controller
walks:
rollouts:
| walk | mesh-controller | build-agent | route-proxy | three gates |
| rollout | mesh-controller | build-agent | route-proxy | three gates |
|---|---|---|---|---|
| `3f68a495` | 3m40s | 3m43s | 3m46s | 11m09s |
| `0cf5a3a5` | 3m39s | 3m26s | 3m34s | 10m39s |
@@ -120,12 +120,12 @@ walks:
| `ca09a07f` | 3m43s | 3m46s | 3m40s | 11m09s |
| `efcdd5dd` | 3m38s | 3m40s | 3m37s | 10m55s |
Two more walks (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still walking at
21:50. The gates alone are 10 to 11 minutes per walk, of which 7 minutes are the two packaging tiers.
Two more rollouts (`a5f53ef9`, `ac91357a`) were superseded part-way, and `8170fc58` was still rolling out at
21:50. The gates alone are 10 to 11 minutes per rollout, of which 7 minutes are the two packaging tiers.
Each tier also builds before it sends: the build seat's image took 53 seconds at 21:36. Estimated from
the plan's creation time and the build records, `ca09a07f` walked from 21:07 to about 21:23 and
the plan's creation time and the build records, `ca09a07f` rolled out from 21:07 to about 21:23 and
`efcdd5dd` from 21:32 to about 21:47, about 15 minutes each. The controller's own tier, alone, is about
4 to 5 minutes. Of the seven completed walks, five recreated `route-proxy` on the home server
4 to 5 minutes. Of the seven completed rollouts, five recreated `route-proxy` on the home server
(`3f68a495`, `e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`), and its program had changed in one of
them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07f`).
@@ -133,7 +133,7 @@ them. Four recreated `build-agent` (`e3ec15f7`, `7f65e627`, `9b028b42`, `ca09a07
The deploy plan names a build of a packaging module by the module's own source commit, which is the
catalogue's. `moveFacts.identical` (cmd/mesh-controller/release.go) calls two builds identical when
their commits are equal. In `efcdd5dd`'s walk both modules went `7b307a06 → 7b307a06`, nothing was
their commits are equal. In `efcdd5dd`'s rollout both modules went `7b307a06 → 7b307a06`, nothing was
said to be recreated, and the gate still took about 3.5 minutes for each. Yet the route proxy's
program did change at `efcdd5dd` (table above). **I do not know whether that program reached the home
server.** Either it was sent and the plan said nothing, or it was not sent and the route proxy runs the
@@ -146,7 +146,7 @@ commit would also make issue 280's "repeat" rule register a different program as
- **The changed paths not reaching the planner.** PR #159's delivery record carries its paths and
`module_dirs_said: true`. They are read, and `touchedBy` uses them for the controller's own module.
They are never consulted for a packaging module.
- **The catalogue moving at the same time.** In some walks the catalogue's commit also changed
- **The catalogue moving at the same time.** In some rollouts the catalogue's commit also changed
(`32307bd1 → 7b307a06`), but the README-only what-if shows the controller's repository alone is enough
to plan both modules.
@@ -181,7 +181,7 @@ the commit into the program.
with the build; that is exact for Go and needs a reader per language.
2. **Reproducible builds**: `-buildvcs=false` (or a `.dockerignore` that leaves `.git` out of the
context) in both recipes, so an unchanged source gives an identical program. This alone does not
stop the walk: the image digest still differs (issue 280), and the planner would still plan and gate
stop the rollout: the image digest still differs (issue 280), and the planner would still plan and gate
three tiers. It is cheap and worth doing beside 1, so the program's identity no longer depends on the
commit.
3. **Split the route proxy and the build seat into repositories of their own.** Then the existing
@@ -191,5 +191,5 @@ the commit into the program.
and a change to that library then moves all three again, through a real build edge.
Recommendation: 1 with its check, and 2 beside it. A further question for the fix: whether `built-by`
should order a walk at all. It is a running-order edge for the build seat's worker (issue 206); the
should order a rollout at all. It is a running-order edge for the build seat's worker (issue 206); the
operator's rule 2 counts only modules built *from* a changed module.
@@ -23,7 +23,7 @@ What happened, in order (UTC):
**under the same JSON key `declared`**. The two edits touched different lines, so git merged them
without a conflict. The result does not compile:
`internal/store/store.go:145:2: Declared redeclared`.
4. **00:55.** The walk the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
4. **00:55.** The rollout the merge started (`plan-1791507264215942736`) failed: "`mesh-host` failed to build
in tier 0". `mesh/delivery` on the head reads `failure`. Nothing was sent to a node, and every node kept
`d28ca95`.
5. **01:09.** Node-engine #67 repaired the trunk. It renamed #66's field to `DeclaredUnit`, key
@@ -33,7 +33,7 @@ The facts behind the "different lines" claim, read from the node-engine's reposi
contain `d28ca95`. Their common ancestor is `a2c2c35` (#63's merge). `dfb1415` holds one `Declared`
(store.go line 107), `d28ca95` holds the other (line 139), and `690966d` holds both (lines 107 and 145).
No node was harmed, because a walk builds before it sends. But the merge check said "this may merge" of
No node was harmed, because a rollout builds before it sends. But the merge check said "this may merge" of
a merge that did not build, and both required statuses were green when it merged.
## The cost before this: checks that went stale
@@ -33,7 +33,7 @@ merging made, with two, was never built before the merge.
or ready delivery back to proposed. Its comment says "a person asked for it again, or its group
changed".
- `holder.go`, `PullMerged` (line 347): a merge records the merge commit on that pull request's own
delivery and links its walk. It does not touch the repository's other open deliveries into the same
delivery and links its rollout. It does not touch the repository's other open deliveries into the same
trunk.
- The forge marks a head pending only when it announces a new head (issue 305). A merge into the trunk is
not a new head of any other pull request.
@@ -60,14 +60,14 @@ note. Each gap says what is open, why it matters, and what would close it.
9. **On a large data pool the root search may never finish.** The node-engine's search for programs that run as
root now has one bound of fifteen minutes (`mesh-host` #61; the inner two-minute limit is gone). On a machine with
a very large data pool the walk can still run past it, and the verdict then stays "not judged". That is loud,
a very large data pool the traversal can still run past it, and the verdict then stays "not judged". That is loud,
not silent.
*Closes with:* setuid turned off on the data datasets, so the walk skips them.
*Closes with:* setuid turned off on the data datasets, so the traversal skips them.
10. **An agent can merge its own pull request.** No mesh repository's main requires an approval on the forge
(`required_approvals: 0`), and the forge tool merges as the forge's administrator once the required checks are
green. An agent can therefore change a module that runs on every node, open the pull request, wait for the
checks and merge it, and the controller walks the change out. The checks judge whether every machine still
checks and merge it, and the controller rolls the change out. The checks judge whether every machine still
composes, not what the change is for. The operator's yes in a session is a convention, and nothing checks it.
*Closes with:* a merge that needs the operator's own approval: a required approval on the forge from the
operator's personal account only, or an approval through ADR 0259's asks. That holds only when the forge's
@@ -98,7 +98,7 @@ note. Each gap says what is open, why it matters, and what would close it.
14. **Ways to root the node-engine's judge does not see.** ADR 0266 §3 lists them under *not judged, so not seen*:
among them a root-run unit or script the agent account can write, a directory on root's `PATH` it can write,
root's ssh keys readable by it, and a setuid program on a filesystem the search does not walk. None is known
root's ssh keys readable by it, and a setuid program on a filesystem the search does not traverse. None is known
on the control-node, but a healthy verdict says nothing about them. The record left them "for a later record"
with nothing tracking it; added here on 2026-10-10.
*Closes with:* each way judged by the node-engine beside the others, or a record that decides against judging
@@ -16,14 +16,14 @@ stops answering while it runs.
| 10:57:57 | the route proxy's first "lookup … on <private address>:53: … connection refused" | the route proxy's log |
| 10:58:36 | `machine.<home server>.network` raised: one of its two resolvers does not answer | conditions |
| 10:58:45 | `machine.<control node>.network` raised; the home server's cleared, held under it | conditions |
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the walks |
| 10:59:34 | the controller 5438d7ff and the node-engine 8f763ee9 sent to their first machines | the rollouts |
| 10:59:37 | `module.step-ca.<control node>.unhealthy` raised | conditions |
| 10:59:42 | `machine.<control node>.network` **cleared**, its newest evidence (10:59:37) still "connection refused" | conditions |
| 11:00:23 | `machine.<control node>.network` reopened, raised time 11:00:23 | conditions |
| 11:06:00 | `seat.mesh-dns-resolver.<control node>.wrong` raised by D2 | conditions |
| 11:10:29 | both builds failed their gate: "<control node> as a whole: raised since it was sent: machine.<control node>.network" — put back | conditions |
| 11:18:12 | `dnsmasq.service` restarted by hand; every question answered again | the operator |
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release walked the node-engine | conditions |
| 11:21:46 | `machine.<each>.core-behind` raised: "… and no plan is rolling them out", while a release rolled out the node-engine | conditions |
### What is measured, and what is guessed
@@ -54,12 +54,12 @@ answered none of its questions for the next run (`Confirm`, issue 277), because
question can be lost. Its first run after 10:57:57 held it; the next, at 11:06:00, raised it. Every one
of those questions had been refused, which no load explains.
**Measured: why the second walk of the node-engine waited on its own first machine.** A release keeps no
**Measured: why the second rollout of the node-engine waited on its own first machine.** A release keeps no
record per module — its modules are in its tier only — and D10 read only the records to know what a plan
is rolling out. So while a release walked the node-engine at 11:21, D10 raised `core-behind` on every
is rolling out. So while a release rolled out the node-engine at 11:21, D10 raised `core-behind` on every
machine, "and no plan is rolling them out", and the gate on the home server, its first machine, waited
on it (issue 281's settle window) until D10's next run five minutes later found the new engine there.
The walk then passed. It did not fail, but it waited five minutes for a condition its own send caused.
The rollout then passed. It did not fail, but it waited five minutes for a condition its own send caused.
**Not explained: why the second resolver did not save the lookups.** Every machine's resolver file lists
both resolvers (`options timeout:1 attempts:2`), and the control node's own file lists its private address
@@ -153,7 +153,7 @@ registry, is the operator's decision.
- An undecided part of a network statement — `unknown` or `starting`, as a node-engine just started says
— holds the open network conditions whose newest evidence names that part, and only those. A statement
undecided as a whole holds every part. The next statement decides.
- D10 counts what a release's tier names as rolling out, so a walked node-engine is not `core-behind` on
- D10 counts what a release's tier names as rolling out, so a rolled out node-engine is not `core-behind` on
its own first machine.
- D2 is unchanged: a resolver that answered nothing, refused or silent, is held for the next run, because
a few hundred milliseconds of refusals is a resolver restarting as well as one that stopped. D2 says it
@@ -61,8 +61,8 @@ A replay in mesh-lab's register is still owed before this core issue resolves (A
## A sequel (2026-10-10)
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a walk:
mesh-delivery matches a walk to the delivery merged as its commit, so two catalogue merges 18 seconds apart
left the earlier one held for a person and the walk carrying it waiting for a word nothing gave, for 58
Planning an earlier merge at the later merge's commit left that earlier merge's delivery without a rollout:
mesh-delivery matches a rollout to the delivery merged as its commit, so two catalogue merges 18 seconds apart
left the earlier one held for a person and the rollout carrying it waiting for a word nothing gave, for 58
minutes, with no condition raised. See
[issue 362](../362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md).
@@ -2,7 +2,7 @@
opened: 2026-10-09
---
# 351. A held delivery whose module a backlog walk carried to every machine is never covered
# 351. A held delivery whose module a backlog rollout carried to every machine is never covered
## Symptom
@@ -19,12 +19,12 @@ What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `ha
| time | what happened |
|---|---|
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word |
| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded |
| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours |
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its rollout waited for its word |
| 10-08 11:06 | three later catalogue merges took its rollout over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the rollout ended `failed`, then superseded |
| 10-08 14:27 | #127 merged as bf542128; its rollout built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog rollout sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog rollout, as it had for 24 hours |
No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever
move it again (ADR 0267):
@@ -32,22 +32,22 @@ left as it is, the delivery stays held for ever.
## Why it is an issue
ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running
ADR 0270 covers a held merge when a later delivered merge's rollout "sent the module to every machine running
it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge
commit, which the forge says contains a6385479. The decision's question — does this change go on? — was
answered by the operator's own release of the build. Yet the record cannot end:
- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what
- **A backlog rollout's end is read by nothing.** While it judges, the controller's record of it names what
its gate carried, and mesh-delivery credits the delivery that published each carried build one step per
machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing
from that: the steps stay `judging` for ever.
- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running
it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the
- **`covered` reads only a rollout's own aggregate steps.** A rollout's own record says "every machine running
it" or "the rest running it"; a step per machine, which is all a backlog rollout gives, never counts as the
module sent everywhere.
ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module
whose build reached its machines on the operator's word, after the mesh held it back, rather than by the
walk of the merge that built it. A held delivery in that case has no end but a person's wrong act
rollout of the merge that built it. A held delivery in that case has no end but a person's wrong act
([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and
the condition it raises every hour is one nobody can clear correctly.
@@ -56,7 +56,7 @@ the condition it raises every hour is one nobody can clear correctly.
- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and
`gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128.
- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps.
- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
- The rollout that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
rightly no coverer.
- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module
- A person's `release` of #125 (the verb): it would roll out a6385479, an older commit than the one the module
was built from, and is what ADR 0270 ruled out.
@@ -2,56 +2,56 @@
**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`,
at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127,
`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`,
`mesh-controller.delivery-walks` for #127's rollout and for the backlog rollout, `plans`, `hand-acts`, `module list`,
the forge's `gitea_contains`, and the holder's journal on the control node.
**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a
**Where a backlog rollout's steps come from.** `walkMoved` (holder.go) hands a rollout whose record carries a
`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives
the delivery whose merge commit each carried build was made from one step per machine: `judging` while the
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog rollout
holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`.
`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43
and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the
module was not in that walk's tiers, and the backlog walk is a walk of its own.)
and 14:46 UTC were never moved on. (The controller's own record of #127's rollout never named the module: the
module was not in that rollout's tiers, and the backlog rollout is a rollout of its own.)
**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a
later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only
on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running
on a `sent` step whose machine is the rollout's own aggregate, "every machine running it" or "the rest running
it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is
found, and the forge is never asked — which is why the journal has no line.
**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a
backlog walk that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
**What "done on a machine" means for a backlog rollout.** Read from the controller's records, not its code: a
backlog rollout that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done,
with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of
the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis
is wrong there.
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb —
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed rollout, and the `release` verb —
see the report.
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog rollout's `done` machines for
nothing, and `carries`, which counts no per-machine step.
## The fix
In mesh-catalog #163, one pull request, in `modules/mesh-delivery`:
- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that
walk (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
step it failed on, and a step another walk carried, are left as they are.
- **A finished backlog rollout passes the steps it carried.** `creditReleased`: every step credited to that
rollout (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
step it failed on, and a step another rollout carried, are left as they are.
- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when
the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
the rollout's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still
judging, one failed or one never sent leave the delivery held.
- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its
open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was
- **A rollout that carried a step still judging is asked for by its id.** The controller lists back only its
open rollouts and the last fifty ended at a start; the backlog rollout of this case, ended the day before, was
no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for
such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
such a rollout by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
its record as said.
Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own
walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
rollout of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
The record's consequence that named the one-machine gap gains a progressive insight naming this one.
**How it is checked**, in `covered_test.go`:
@@ -59,15 +59,15 @@ The record's consequence that named the one-machine gap gains a progressive insi
- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered;
still judging on one, failed on one, never sent to one, or a plan naming no machines stays held.
- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one
and leaves the other judging; a failed step stays failed; a step another walk carried is not touched.
and leaves the other judging; a failed step stays failed; a step another rollout carried is not touched.
- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held
while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the
while the backlog rollout judges, covered once it is done and read by its id, naming the later merge and the
module.
All three fail on the catalogue's main before the change and pass after it.
**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are
asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once
asked for by the backlog rollout's id and pass; #127 then carries the module for #125; the forge is asked once
whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads
delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the
acceptance of this issue, read with `mesh-delivery.show` of #125.
@@ -21,7 +21,7 @@ On 2026-10-09, local times on the control node (its controller's and node-engine
migrations than the store had applied, and served what it could.
- 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3
times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent,
which the put-back had removed from the machine. The plan walked on, believing the controller delivered.
which the put-back had removed from the machine. The plan rolled out on, believing the controller delivered.
## Why it is an issue
@@ -35,8 +35,8 @@ Four rules did not hold.
gate of the build it had replaced.
- **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than
itself"). A controller older than the store's schema starts behind its own records. Nothing refused it.
- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to
the merge plan's send: a release keeps no module records, and the check for a walk in progress read
- **One rollout of a module on a machine at a time** (issue 281, ADR 0236). A release's rollout was invisible to
the merge plan's send: a release keeps no module records, and the check for a rollout in progress read
only those.
The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md).

Some files were not shown because too many files have changed in this diff Show More