Author SHA1 Message Date
jschoubben cb4b3c2398 Prove a push over SSH from g14 (issue 955) 2026-10-17 02:30:30 +02:00
mesh-tools 375e77690b Merge pull request 'Feature agents-write-git, step 2: design 29 cites ADR 0399 and ADR 0400 (issue 955)' (#390) from feat/agents-write-git-ssh into main 2026-10-16 22:31:07 +00:00
jschoubben 24701df23e Merge branch 'main' into feat/agents-write-git-ssh
mesh/merge-gate head 24701df23e8a merged into trunk dd5a06529556 beside mesh-catalog#340@ae9cda85 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 24701df23e8a merged into trunk dd5a06529556 beside mesh-catalog#340@ae9cda85 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/agents-write-git-ssh delivered: every member is delivered
2026-10-16 22:27:02 +00:00
mesh-tools dd5a065295 Merge pull request 'Design 47's milestones and issues, and the as-is delivery as it runs today' (#393) from handoff/47-milestones into main 2026-10-16 22:09:55 +00:00
jschoubben 748641dd22 Review of #393: rollouts run one at a time today (issue 525); Approve on Telegram (issue 953 for the terminal); 976's place in the order; milestone 1 takes the rest of 525 and acceptance test 9
mesh/merge-gate head 748641dd22ae merged into trunk 73b9cc3257a1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 748641dd22ae merged into trunk 73b9cc3257a1 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-17 00:06:59 +02:00
jschoubben 75d3ac0f82 Playbooks 04 and 07 after research 055: a milestone's shape from what-if; the Approve on every merge into main; when the two-step exception ends (issue 922)
mesh/merge-gate head 75d3ac0f8258 merged into trunk 73b9cc3257a1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 75d3ac0f8258 merged into trunk 73b9cc3257a1 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-17 00:04:48 +02:00
jschoubben b6a191d0a6 Design 47's milestones and issues (playbook 04), and the as-is delivery rewritten from research 055's findings
mesh/merge-gate head b6a191d0a6dc merged into trunk 73b9cc3257a1 — pass: the change touches no module of the mesh's graph
mesh/repo-check head b6a191d0a6dc merged into trunk 73b9cc3257a1 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
The operator confirmed six milestones on 2026-10-17; issues 973-981 and 971
carry their repository parts, linked depends-on in order. The as-is described
the predecessor's pipeline; it now says what runs today, which design 47
replaces.
2026-10-17 00:04:00 +02:00
mesh-tools 73b9cc3257 Merge pull request 'Playbook 08: a fix that moves a file to a new filler compares the values first (issue 957)' (#391) from fix/957-playbook-08-first-fill into main 2026-10-16 21:56:23 +00:00
mesh-tools f795356ae1 Merge pull request 'Design 47 rewritten from research 055's decisions: hotfix, feature and epic from the trunk, delivery on the bus' (#392) from graduate/055-nats-native-delivery into main 2026-10-16 21:50:05 +00:00
jschoubben aa4bfe289e Design 47 and the glossary: the reviewer's findings (three work queues, a feature in one repository, what-if, ADR 0406 clause numbers, backups by protection, no suite-time claim, the Go module's judge account)
mesh/merge-gate head aa4bfe289ee4 merged into trunk f9c240a957b8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head aa4bfe289ee4 merged into trunk f9c240a957b8 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 23:44:59 +02:00
jschoubben 6f0db74ee4 Design 47: the SDK commits the catalogue accepts (ADR 0402 §4)
mesh/merge-gate head 6f0db74ee47f merged into trunk f9c240a957b8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 6f0db74ee47f merged into trunk f9c240a957b8 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 23:42:56 +02:00
jschoubben e7854a31a6 Design 47: a member naming the SDK member's head is proven by its own merge check (ADR 0402's review)
mesh/merge-gate head e7854a31a6a9 merged into trunk f9c240a957b8 — pass: the change touches no module of the mesh's graph
mesh/repo-check head e7854a31a6a9 merged into trunk f9c240a957b8 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 23:41:37 +02:00
jschoubben 8bfb11da2c Design 47 and the glossary: the batch's record in the chain is the rollout (ADR 0408's review) 2026-10-16 23:40:11 +02:00
jschoubben 96f2ae72c1 Rewrite design 47 from research 055's decisions: hotfix, feature and epic from the trunk, delivery on the bus
ADR 0401-0408 graduate research 055: no release branch, two more ordering
rules, the composed check as each member's gate, a backup before the send,
installed toolchains, events and work queues on the bus, the merge window
as keys, and one chain kept on the bus alone. Design 47 is rewritten from
them (one design per thing); the glossary takes hotfix, feature, toolchain,
milestone and epic and retires release branch; design 18 loses the
toolchain image; playbooks 07 and 08 drop the release-branch wait.
2026-10-16 23:39:12 +02:00
nox 78353ad272 Playbook 08: a fix that moves a file to a new filler compares the values first (issue 957)
mesh/merge-gate head 78353ad272a3 merged into trunk f795356ae1d6 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 78353ad272a3 merged into trunk f795356ae1d6 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 22:37:59 +02:00
nox fa5fd9fc69 Cite ADR 0399 and 0400 in design 29: the account's forge key lives in ~/.ssh (issue 955)
mesh/merge-gate out of date: judged against trunk f9c240a957b8, the trunk is now f795356ae1d6; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk f9c240a957b8, the trunk is now f795356ae1d6; judged again on a recheck or when it is merged
mesh/delivery-group group feat/agents-write-git-ssh checking: 0 of 2 member(s) ready
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 22:37:15 +02:00
mesh-tools f9c240a957 Merge pull request 'Playbook 07: report the whole issue tree every five minutes, every two while a delivery runs' (#389) from fix/pb07-overview-every-five into main 2026-10-16 19:34:36 +00:00
mesh-tools fff89b7c62 Playbook 07: report the whole issue tree every five minutes, every two while a delivery runs
mesh/merge-gate head fff89b7c6218 merged into trunk 80e36c75957f — pass: the change touches no module of the mesh's graph
mesh/repo-check head fff89b7c6218 merged into trunk 80e36c75957f — pass: its merge-check.sh passed
mesh/delivery delivered
The operator asked for a status update more often, with an overview of the issues worked on.
2026-10-16 19:18:42 +00:00
mesh-tools 80e36c7595 Merge pull request 'Playbook 09 and the hq-cleanup skill: a cleanup that deletes only what is proven done' (#388) from process/09-cleanup into main 2026-10-16 19:08:33 +00:00
jschoubben 3c49b389d1 Playbook 09: a deletion is approved as an ask on the operator's warrant once design 58 milestone 7 is built; until then the operator runs the approved list
mesh/merge-gate head 3c49b389d1e7 merged into trunk b29008ad36f4 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 3c49b389d1e7 merged into trunk b29008ad36f4 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 20:51:00 +02:00
jschoubben 222fbe95f4 Playbook 09 and the hq-cleanup skill: a cleanup reads first, deletes only what the operator approves, and reports what it kept (the operator, 2026-10-16)
mesh/merge-gate head 222fbe95f4bb merged into trunk b29008ad36f4 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 222fbe95f4bb merged into trunk b29008ad36f4 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 20:01:13 +02:00
mesh-tools b29008ad36 Merge pull request 'Playbook 02: one design per thing' (#387) from process/02-one-design-per-thing into main 2026-10-16 17:41:22 +00:00
jschoubben 101d7f30f7 Playbook 02: one design per thing; a graduation rewrites the design that covers it instead of writing a second (the operator, 2026-10-16)
mesh/merge-gate head 101d7f30f7e2 merged into trunk 225eee14d0bf — pass: the change touches no module of the mesh's graph
mesh/repo-check head 101d7f30f7e2 merged into trunk 225eee14d0bf — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 19:40:48 +02:00
mesh-tools 225eee14d0 Merge pull request 'Design 47: a move of the trunk asks every active check again (ADR 0395)' (#381) from design/47-recheck-on-trunk-move into main 2026-10-16 17:35:52 +00:00
jschoubben 12e75cdfa2 Design 47: the transition paragraph, the recheck verb and the section's bullets say ADR 0395 whole (review)
mesh/merge-gate head 12e75cdfa215 merged into trunk f1081c4f9768 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 12e75cdfa215 merged into trunk f1081c4f9768 — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 19:27:52 +02:00
jschoubben 377fa3f88a Design 47: say ADR 0395 as accepted — a conflict is asked again at every move, an ended check's builds are not reused, a group's composed check reruns through its members, and the cost bound 2026-10-16 19:26:47 +02:00
jschoubben 11e32ce5be Merge remote-tracking branch 'origin/main' into design/47-recheck-on-trunk-move 2026-10-16 19:26:35 +02:00
mesh-tools f1081c4f97 Merge pull request 'Playbook 08: a second fault the delivery shows joins the issue only in the same files' (#386) from fix/pb08-second-fault into main 2026-10-16 17:09:53 +00:00
mesh-tools 152cc0c0e7 Playbook 08: a second fault the delivery shows joins the issue only in the same files
mesh/merge-gate head 152cc0c0e77c merged into trunk a8b951f3b488 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 152cc0c0e77c merged into trunk a8b951f3b488 — pass: its merge-check.sh passed
mesh/delivery delivered
Taken from issue 937, where mesh-catalog #335 was taken into the issue
and issue 943 was opened for a fault in other code.
2026-10-16 17:06:25 +00:00
mesh-tools a8b951f3b4 Merge pull request 'Playbook 08: needing a decision keeps a fix small' (#385) from process/08-a-decision-keeps-a-fix-small into main 2026-10-16 15:58:01 +00:00
jschoubben 0eadc27fc7 Playbook 08: needing a decision keeps a fix small; a feature is several modules with research or a design before it (the operator, 2026-10-16)
mesh/merge-gate head 0eadc27fc722 merged into trunk 4e437eb1488f — pass: the change touches no module of the mesh's graph
mesh/repo-check head 0eadc27fc722 merged into trunk 4e437eb1488f — pass: its merge-check.sh passed
mesh/delivery delivered
2026-10-16 17:46:23 +02:00
jschoubben 91f82cd38e Design 47: a group's own merge in order asks nothing again (ADR 0395 §1, review)
mesh/merge-gate out of date: judged against trunk b9af4e373dc0, the trunk is now 6dcfa2226e63; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk b9af4e373dc0, the trunk is now 6dcfa2226e63; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 13:32:39 +02:00
jschoubben a65e1076a2 Design 47: a pull request that no longer merges cleanly is told, not checked (ADR 0395 §2)
mesh/merge-gate head a65e1076a2e5 merged into trunk b9af4e373dc0 — pass: the change touches no module of the mesh's graph
mesh/repo-check head a65e1076a2e5 merged into trunk b9af4e373dc0 — pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-16 11:42:03 +02:00
jschoubben 09ec150bd6 Design 47: a move of the trunk asks every active check again, one at a time per pull request (ADR 0395, issue 888)
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate error: the check could not run: killed by hand
mesh/repo-check error: the check could not run: killed by hand
2026-10-16 11:40:14 +02:00
13 changed files with 644 additions and 514 deletions
+44
View File
@@ -0,0 +1,44 @@
---
name: hq-cleanup
description: Use when the operator asks to clean up the forge and the working copies — stale or merged branches, old pull requests, finished worktrees, leftover folders, stashes. Triggers on "cleanup", "clean up our repos", "stale branches", "old worktrees", "tidy the forge". Reads first, deletes only what the operator approves, and reports what it cleaned, kept and left.
---
# hq-cleanup
Runs playbook [09 — Cleanup](../../../00-META/process/09-cleanup.md); the playbook is authoritative. This skill
adds the mechanics.
## 1. Take stock (one read-only agent, smaller model)
Brief it to change nothing and to write one report to the session's scratchpad:
- **Forge.** `gitea.gitea_list_repos`; per repository its default branch and **every branch from the git refs**
(`gitea.gitea_api` GET `/repos/{owner}/{repo}/git/refs/heads`; the list tool misses branches, issue 949);
`gitea.gitea_branch_protection_get`; open pull requests (`gitea.gitea_open_pulls`, titles and update times
from `gitea.gitea_api` `/repos/{owner}/{repo}/pulls?state=open`). Per branch: contained in the default branch
(`gitea.gitea_contains`, with the default branch's sha), patch-equal (`git cherry` in a local clone), tip date.
- **Local.** Under the projects folder: each clone's `git worktree list --porcelain`, `git branch -vv`,
`git branch --merged origin/<default>`, `git stash list`, `git status --porcelain` per worktree; folders and
loose files beside the clones with `du -sh` and newest file. A worktree under another session's scratchpad is
Kept.
- A **Missing tools** section: each read it did without a mesh tool.
## 2. Classify and ask
Classes as playbook 09 step 2: Safe, Likely safe, Kept, Unknown. Tell the operator the counts and the report's
place, and ask one question: delete the Safe list? Likely-safe and Unknown items are asked separately.
## 3. Delete the approved list only
- Forge branch: `gitea.gitea_delete_branch`.
- Worktree: `git worktree remove <path>` (never `--force`), then `git worktree prune`.
- Local branch: `git branch -d <name>` (never `-D`).
Once design 58 milestone 7 is built (issue 884), each deletion is a call held as an ask: the operator approves
the exact list on Telegram and the mesh carries it out on the warrant. Until then a session's permission check
refuses these; hand the operator the approved list as a file, with the commands, and do not work around the
refusal.
## 4. Report
Four lists: **cleaned** (with its evidence), **kept** (why), **unknown** (what it looks like), **the operator
kept** (their word). Open each missing tool with `issue-tracker.open`, or note it on an existing issue.
+24 -11
View File
@@ -219,10 +219,11 @@ every other domain has something to run on. **Recorded by** the controller and t
(ADR 0229).
- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its
records (nodes, assignments, rollouts, builds, settings), after each change commits in its store and only
records (nodes, settings, declared config, predefined keys), after each change commits in its store and only
when the value differs, with one event per write. The store stays the record; the self-check compares
the two and raises `mirror-drifted` on any difference
(ADR 0289). Shortened to *mirror* where the
(ADR 0289, ADR 0356). Builds, rollouts and assignments are no mirrors: they are
the chain, kept on the bus as their only source (ADR 0408). Shortened to *mirror* where the
text is about the controller's records; the builder's *mirrored* `upstream` image (Change and delivery) is
a different thing, a copy of an outside image in the artifact store.
@@ -393,7 +394,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
**Purpose.** To take one commit from its pull request to every node that should run it, and to put it
back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the
planner, the rollout). **Downstream of** Module, Core, Health and repair and Data.
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323.
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323, 0401–0408.
**Uses:** module, node, declaration and send (Placement), health (Health and repair), person.
- **merge check** — the two statuses a pull request carries before it may merge: the **merge gate**
@@ -408,7 +409,10 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat
**builder**. Any node may hold the seat, so there is no build node by nature
(ADR 0237).
*Not:* ~~build machine~~
- **build queue** — the controller's queue of builds to run; one entry in it is a **build request**.
- **build queue** — the builds waiting to run, kept by the bus as three work queues a builder takes in order:
delivery builds, then checks of pull requests marked ready, then every other check; each check queue keeps one
request per pull request, the newest (ADR 0406); one entry in it is a
**build request**.
The queue's verbs still say *ask* for an entry; the conversation's ask is another thing
(see [Homonyms](#homonyms)).
- **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by **version**.
@@ -450,9 +454,14 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat
- **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's
repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One
level: a group holds deliveries, never groups. Its state is derived from its members, never set.
- **release branch** — a branch other than `main`, shared by name across repositories, that collects a feature's
merged work and is brought into `main` as one delivery (design 57's `feat/57-config-and-provisioning`). While it
is in flight, a change that touches its files waits for it (playbook 07, *Is the coast clear?*).
- **hotfix** — one pull request in one repository, delivered at its merge (ADR 0401).
- **feature** — one branch name across its repositories: in two or more, its pull requests are one delivery group,
merged together and delivered member by member in the group's order; in one, a single pull request delivered as a
lone delivery (ADR 0401). Work is never held back on a branch other than a trunk: the trunk is what runs.
*Not:* ~~release branch~~ (hq)
- **toolchain** — a language's compiler and tools, installed by that language's build module on every node holding
the build seat, pinned to one version that is the same on every builder; its version is part of a build's reuse
key (ADR 0405). No longer an image.
- **delivery plan** — what a delivery does to the mesh: its **build plan** (the modules it moves and
their dependents, in tiers), its **deploy plan** (per node, what it receives and what waits for a
person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a
@@ -460,15 +469,17 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat
*Not:* ~~change plan~~, ~~release plan~~
- **merge window** — the rolling wait in which the controller collects merges before it plans them: each
merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest
10 minutes after its first merge (ADR 0276).
10 minutes after its first merge (ADR 0276). Kept as two keys with a time to live on the bus; the first to
expire closes the batch (ADR 0407).
- **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more
than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*,
which is a delivery group.
- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat
of the same name. It records and decides; the controller sends, judges and rolls back when it is asked.
- **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236,
ADR 0276). One rollout is open at a time; it names every merge it answers. A primitive the delivering stage
repository, tier by tier, to every node that runs each module at once (ADR 0276, ADR 0360). Rollouts that share a
repository, a module or a graph edge run one after another; others run side by side (ADR 0407). It names every
merge it answers, and its record is one key of the bus's rollouts bucket, a link of the chain (ADR 0408). A primitive the delivering stage
asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules
built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll`
policy.
@@ -732,6 +743,9 @@ Not a domain of the mesh but of the way it is built, listed because its words me
([55 — The knowledge base](../03-DESIGN/01-to-be/55-the-knowledge-base.md)). Not **the record**, the literal reader
of this repository.
- **design** — a document in `03-DESIGN/`: **as-is** (what runs) or **to-be** (what is being built).
- **milestone** — one step of a design's work breakdown, usable on its own once delivered and named by what the
operator or the mesh can then do; delivered as one feature (playbook 04, ADR 0401).
- **epic** — a run of milestones in one design, released milestone by milestone (ADR 0401).
- **issue** — a question about the mesh that someone must answer, numbered and tracked by the issue tracker, which
holds it whole: its report, diagnosis and resolution included (ADR 0291);
the folders in `04-ISSUES/` are imported there and removed.
@@ -784,7 +798,6 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
| release | a person letting a held delivery go on | **release** | Change and delivery |
| | taking a consumer off a licence | **unbind** (verb `release`, to rename) | Identity and access |
| | a copy of a work an engine can fetch | **media release** | Media |
| | a branch brought into `main` as one delivery | **release branch** | Change and delivery |
| gate | the judgement on a delivery's first node | **first-node gate** | Change and delivery |
| | the pull request status | **merge gate** | Change and delivery |
| | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement |
+1
View File
@@ -58,6 +58,7 @@ the expensive half.
| [06](06-writing-a-module.md) | Writing a module | Something that runs today must run on the mesh |
| [07](07-feature-branches.md) | Features | Work that changes code across issues or repositories: prepared context, local development, one review, one check, one delivery |
| [08](08-small-fixes.md) | Small fixes | One issue in one repository, about an hour of work: one review, one push, one delivery |
| [09](09-cleanup.md) | Cleanup | Merged branches, stale pull requests, finished worktrees and folders: read first, delete only what the operator approves, report the rest |
## The cycle is checked
+4 -1
View File
@@ -33,7 +33,10 @@ about an hour, and said to the operator if it runs past.
- **Pre-check each record before proposing it**, as for research (playbook [01](01-research.md), step
5): every record, issue or design it names is in `links`; each deferral names its issue; each figure
is given as measured; no word on a glossary *Not:* line.
3. **Write the design.** Create the document under `03-DESIGN/01-to-be/` with frontmatter:
3. **Write the design.** **One design per thing** (the operator, 2026-10-16): when a to-be design already
covers what the decisions change, they rewrite that design, as *Amending an existing design* below says,
and no second design is written beside it that refers to it. Only for something no design covers, create
the document under `03-DESIGN/01-to-be/` with frontmatter:
```yaml
---
+2 -1
View File
@@ -30,7 +30,8 @@ A design becomes **milestones**, and each milestone becomes **issues**:
4. **Break it into milestones.** Write them into the design, in a section *Milestones*: for each, its number,
its one-sentence outcome, the acceptance tests of the design it proves, the repositories it touches, and
what it depends on. Order them so each is usable as soon as it is delivered, the smallest useful one
first.
first. A milestone is named by what it lets someone do; how it ships is shaped by the plan its changes would
have, asked through `mesh-delivery.what-if`: the modules it moves and their fan-out (ADR 0401 §6).
5. **Ask the operator to confirm the milestones**, in a short table: number, outcome, the acceptance tests it
proves, repositories, the parts each repository builds. Change what they change.
6. **Open the issues.** For each confirmed milestone, `issue-tracker.open` one issue per repository part, its
+16 -43
View File
@@ -121,32 +121,10 @@ never into the running one, unless it blocks the running one.
report, and the round does not wait for it. What a round asks is only what makes the change wrong against
its issues and its decisions.
## Is the coast clear?
Before the push of phase 3 (and of a small fix, playbook [08](08-small-fixes.md) step 5), the lead checks, for every
repository the change touches, whether a **release branch** is in flight there: a branch other than `main` that
collects a feature's merged work to bring it into `main` as one delivery.
1. **Look for one.** A pull request into `main` whose head is a release branch (`gitea.gitea_open_pulls`), a
release branch holding merged work that `main` does not have yet (`git log origin/main..origin/<branch>`), or
a delivery plan of that release branch still running (`mesh-controller.plans`).
2. **Compare the files.** `git diff --name-only origin/main...origin/<release branch>` against the files the
change touches.
3. **Overlap: hold.** If the release branch touches a file the change touches, nothing is pushed into that repository
until the release branch's delivery plan is done. Then the feature branch is brought up to the new `main`, the unit
tests run again, and the push follows. The hold is on the wait list (its bound: the release branch's delivery plan),
and is said to the operator with what the change waits for.
4. **No overlap: go on.** A release branch in another repository, or one that shares no file with the change, does not
hold it; its merge joins the merge window as any other (ADR 0276).
Merging beside a release branch still in flight would make its lead resolve someone else's conflicts in the
middle of it, and could send part of the change with its delivery (2026-10-16: the file module's
controller part touched four files feature 57's release branch also changed).
## Phase 3 — Check (no review)
1. **Push every feature branch and open one pull request per repository, together, once the coast is clear**
(above). The integration tests
1. **Push every feature branch and open one pull request per repository, together.**
The integration tests
run only in the merge check (the operator, 2026-10-16), never before. Pull requests in two or
more repositories that share the branch name are one delivery group (ADR 0239 §2).
Where a session pushes through the forge tool `gitea_commit` (one commit on the forge's current base), the
@@ -162,12 +140,16 @@ controller part touched four files feature 57's release branch also changed).
5. **The reviewer marks each pull request ready as soon as the acceptance is shown** with
`mesh-delivery.mark-ready` (the delivery of the head it approved, who reviewed it and why), which writes the
label `mesh/ready` and then `ready at <full sha>`, the order the forge counts (issue 913); mesh-delivery merges it when its checks pass
(ADR 0364). The operator's word is waited for only when the step changes what the operator sees, removes
and the operator approved the merge into main on Telegram, the channel that proves who answered today (ADR 0364,
ADR 0398 §3; an Approve at the terminal is issue 953). Beyond that
Approve, the operator's word is waited for only when the step changes what the operator sees, removes
data, or needs a step only the operator's terminal can take; the lead says which before the step starts.
**One exception: a catalogue change that uses a new manifest word.** The catalogue's merge check reads
manifests with the controller the mesh runs, so the controller's pull request is merged and delivered first;
then `mesh-delivery.recheck` the catalogue's and merge it on the new verdict. Plan such a feature as two steps.
The exception ends when milestone 3 of design 47 is delivered (issue 977, ADR 0402, ADR 0403): the group's composed
check, judged by the group's controller, is then each member's gate, and the feature is one group.
## Phase 4 — Deliver
@@ -177,14 +159,6 @@ delivery: no machine receives it (issue 889 while it still starts one). The lead
naming its pull requests, and removes what is left: no `feat/<slug>` branch and no `.work/<slug>` worktree
survive.
**Removing what is left means everything the feature made, wherever it lives**: every worktree the lead or its
agents made (under `.work/<slug>`, in a session's scratchpad, a reviewer's copy), every per-issue or per-package
branch, and every branch the feature pushed to the forge. Before one goes, check it holds nothing that is not on
`main`. Compare content, not hashes: a commit put on the forge with `gitea_commit` lands with a new hash, so a
merged branch still shows commits "not on main" (`git diff origin/main <branch> -- <the files it changed>` is
empty when its content is there). What is not on `main` is delivered in the next step or written on its issue
before it goes. Feature 57 left over a hundred worktrees and per-issue branches in one scratchpad (2026-10-16).
## Models
| Who | Model | Why |
@@ -210,13 +184,13 @@ on, with when it started and its bound.
| the review | 15 minutes a round | ask the reviewer; stop at round 3 |
| a merge check | 15 minutes | read the build queue (`mesh-controller.queue`) and the check's log |
| the ready mark | 15 minutes after the acceptance is shown | the reviewer marks it, or the lead does |
| a release branch in flight that shares files | its delivery plan done | report it every ten minutes; bring the branch up to `main` after |
| a merge after ready | 5 minutes after the checks pass | read `mesh-delivery.show` for the delivery |
| a delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
**Every ten minutes, and at once when something lands, stops or needs the operator,** the lead reports, in
one of the two layouts below. Times are local and absolute; every number has its unit and its bound or
baseline; an issue is named by its number and a few words, never by its number alone.
**Every five minutes, every two minutes while a delivery plan runs, and at once when something lands, stops or
needs the operator,** the lead reports, in one of the two layouts below (the operator, 2026-10-16: an overview
of the issues, often). Times are local and absolute, from the clock (`date`), never guessed; every number has its
unit and its bound or baseline; an issue is named by its number and a few words, never by its number alone.
**The full layout** (in the session), the issues as a tree: grouped by repository, each issue under the one it
waits for, marked ✓ landed, ▶ fixing, ⏸ waiting, ✗ stopped:
@@ -233,6 +207,7 @@ mesh-controller (agent: fixing, 2 of 3 landed)
mesh-catalog (agent: waiting)
└── ⏸ 751 (above, under 736)
Opened along the way: <each issue opened during the feature, its number and a few words, and where it goes> — or "none"
Past its bound: <the wait, how long, what is being done> — or "none"
Needs you: <one line per question, with my recommendation> — or "nothing"
Allowance: week <n> % (<+d> since the last report), session <n> %
@@ -245,8 +220,9 @@ Next: <the next action, one line>
<slug> step <n>/<m> · <phase> <used>/<bound> min · <landed>/<total> landed · past bound: <none | what> · needs you: <nothing | what> · week <n> %
```
A report says what changed since the last one first; an unchanged table is not repeated, the short layout
says it instead.
A report says what changed since the last one first. In the session every report is the full layout, the whole
tree of the step's issues included, also when only time passed: the operator reads the state from it without
asking. The short layout is for a channel.
## The context of the lead's own session
@@ -268,16 +244,13 @@ issues in `fixing` that nobody works on reads to the operator as stuck work.
request into a feature branch exists. A pull request per issue, or an agent waiting on a merge check in
phase 1, is the failure this playbook prevents.
- A context file exists per touched repository under `.work/<slug>/` while the feature runs.
- After a feature's delivery, `git branch -r | grep feat/<slug>` and `git worktree list` return nothing for it, in every clone the
feature used, its session scratchpads included.
- After a feature's delivery, `git branch -r | grep feat/<slug>` and `git worktree list` return nothing for it.
- More than one open pull request in a repository that share no feature name, or a feature step whose issues
stay in `fixing` for more than a working day, is the smell: stop and bring it to the operator.
- A review report that asks a change outside the issues' done-when, or a fourth round, is the failure of the
review rules; `issue-tracker.list` with `flag: abandoned` lists issues left in `fixing` with nobody on them.
- A pull request reviewed and green but not marked ready for more than 15 minutes is a lead that did not
mark it.
- A pull request opened into a repository whose release branch was in flight and shared a file with it is the failure of
*Is the coast clear?*; the lead's report names the check it made before the push.
- A step that adds or changes a module marked ready with no integration test that starts the module whole is
the failure of phase 3's fourth step.
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it cannot find is a
+22 -7
View File
@@ -2,7 +2,10 @@
**Trigger.** One issue in one repository, small enough to build in about an hour: a bug with a known cause,
a missing check, a wording, a playbook or design edit. Anything larger, or anything that needs a change in a
second repository first, is a feature and follows [playbook 07](07-feature-branches.md).
second repository first, is a feature and follows [playbook 07](07-feature-branches.md). A feature is the
larger kind: it touches several modules and has a research effort or a design before it. **Needing a decision
does not make a fix a feature**: the decision is settled first (step 1) and the fix stays small (the operator,
2026-10-16).
**Related issues in one fix.** A small fix may cover two or three related issues when they are in the same
repository, mostly in the same module, and together still fit in about an hour: they touch the same files, or
@@ -23,7 +26,8 @@ one delivery**. Everything before the push is local and fast.
1. **Read the issue and the code it names** (the session's own model, or one read-only agent on the smaller
model answering exact questions with file:line). If a design question is open, it is settled before any
code is written: on the issue, through the knowledge base, or by the operator. A fix that turns out to
need a decision or a second repository stops here and becomes a feature.
need a second repository, or more than about an hour, stops here and becomes a feature; one that needs a
decision stays a small fix.
2. **Move the issue to `fixing`** with `by` as who@node.
3. **Build on a local branch** `fix/<issue>-<slug>`, in a worktree of its own branched from `main`. Write the
test that fails today first; make the change; keep `gofmt`, `go vet` and the unit tests of what changed green
@@ -35,8 +39,8 @@ one delivery**. Everything before the push is local and fast.
done-when is opened as an issue of its own, never asked as a change). The reviewer's findings go back to the same agent that wrote the fix, continued by
its id, so the fix keeps its context. A playbook or design edit of a few lines may skip the reviewer: the
operator's word on the pull request is its review, and the lead marks it ready on that word.
5. **Push once and open the pull request, once the coast is clear** (playbook [07](07-feature-branches.md), *Is
the coast clear?*: no release branch in flight in the repository that shares a file with the fix). The forge's merge
5. **Push once and open the pull request.** The trunk is what runs and nothing holds merged work back
(ADR 0401), so a fix waits for no other change's delivery. The forge's merge
check runs the integration tests, the only
place they run (the operator, 2026-10-16). A failed check
goes back to the same agent with the failure, and the reviewer reads only the change it makes.
@@ -49,12 +53,24 @@ one delivery**. Everything before the push is local and fast.
or needs the operator's terminal.
7. **Follow its delivery plan to done** (`mesh-controller.plans` with `wait` and `until: done`), prove the fix on
the mesh (the symptom gone, measured), and resolve the issue with `located-in` and `fixed-by` the same hour.
8. **Remove the fix's worktree and its branch**, local and on the forge, as playbook [07](07-feature-branches.md)
phase 4 says: content checked on `main` first, not hashes.
A fix that stops before its delivery moves its issue back to `open` with a note saying how far it got
(branch, last commit, what is left), as playbook [07](07-feature-branches.md) says.
**A second fault found by the delivery.** When a fix's delivery shows a second fault that keeps the issue's
done-when from holding, and that fault is in the same files, it is taken into the same issue as a second pull
request with a review round of its own. A fault in other code is opened as an issue of its own, and the first
issue waits for it (issue 937: mesh-catalog #335 was taken in; issue 943 was opened).
**A fix that moves a file to a new filler compares the values first.** When a fix makes a module's file filled a
new way for the first time (the provisioner client in place of the controller, a provider's entry in place of a
composed value), the fix lists every value the old way put into that file on each machine it reaches and shows,
in its review, that the new way gives the same values, or says which differ and why. A difference nobody looked
at is an outage: on 2026-10-16 the forge and the identity provider on the control node were down for 17 minutes
because postgres's consumers were served the port inside its container instead of the port it is published on
(issue 957). The comparison is read from the machines (the file as it is today) before the merge, not from the
code.
## Bounds
| Waiting on | Bound | Past the bound |
@@ -78,7 +94,6 @@ reading itself and uses one subagent only for the review.
- A small fix has one pull request with one or few commits naming its issue, opened after its review: a pull
request with many heads is a fix reviewed after its push.
- Its issue moves to `resolved` within the hour its delivery plan ends.
- After its delivery, no worktree and no branch named `fix/<issue>-…` is left, locally or on the forge.
- A pull request reviewed and green but unmarked
for more than 15 minutes, a fix to a module's running marked ready with no integration test of the module whole, or a review report
asking a change outside the issue's done-when, is the failure of steps 4 to 6.
+49
View File
@@ -0,0 +1,49 @@
# Playbook 09 — Cleanup
**Trigger.** The operator asks for a cleanup, or the forge and the working copies have gathered what nobody
uses: merged branches, stale pull requests, worktrees and folders of finished work.
**Who runs it.** A session on the operator's word, with the operator deciding every deletion. Agents read and
propose; a deletion is the operator's act or is done on the operator's word for the list it was shown.
## The aim
Remove what is provably done, and nothing else. A cleanup that deletes one piece of unfinished work costs more
than every stale branch it removed. So every item is shown with its evidence, and anything not provably safe is
reported and left.
## Steps
1. **Take stock, reading only.** One agent on the smaller model reads, changing nothing:
- the forge: every repository, every branch **as git refs** (the branch list tool misses some, issue 949),
every open pull request with its head, title and last update;
- each working copy on the machine: its worktrees, local branches, stashes, and the folders beside the
clones, with size and newest file.
It writes one report with each item, its class and its evidence, and a list of the tools it lacked.
2. **Classify each item**, by evidence, never by name or age alone:
- **Safe** — a forge branch whose tip is contained in its repository's default branch, with no open pull
request, not protected and not a long-lived branch the mesh treats as a trunk; a worktree that is clean,
whose branch is contained in the remote default branch, untouched for two hours and not in another
session's working folder; a local branch contained in the remote default branch.
- **Likely safe, not proven** — a branch whose every commit is patch-equal to one on the default branch
(the forge lands some commits under new hashes): listed, deleted only on the operator's word for it.
- **Kept** — unmerged commits, a dirty worktree, a stash, an open pull request, a protected or trunk-like
branch, a repository of another owner, another session's worktree.
- **Unknown** — a folder or file outside git: described, never opened where a secret may be, left.
Stale open pull requests (no update for three days) are reported with their author's issue, never closed.
3. **Ask the operator** with the counts per class and the report's place. One question: delete the safe
items as listed? The likely-safe and unknown items are separate questions, item by item when few.
4. **Delete only the approved list**, each with the command that refuses an unsafe case (a worktree removal
that refuses a dirty one, a branch deletion that refuses an unmerged one). A refusal is reported, never
forced. The operator approves a deletion on a channel that proves who answered, as an ask bound to that
exact list and carried out on the warrant (ADR 0387, design 58 milestone 7, issue 884). Until that is built,
an agent session's permission check refuses these deletions, and the approved list is handed to the
operator to run.
5. **Report**: what was cleaned, what was kept and why, what was unknown, what the operator kept, and the
missing tools, each opened as an issue (`issue-tracker.open`) or added to an existing one.
## How it is checked
- The report names, for every deleted item, the evidence it was safe (contained, clean, no pull request).
- Nothing classed Kept or Unknown is deleted without the operator's word for that item.
- A cleanup's missing tools are issues, not notes in a session.
+71 -81
View File
@@ -1,100 +1,90 @@
---
layer: as-is
status: implemented
code: [hal]
updated: 2026-08-23
code: [mesh-controller, mesh-catalog modules/mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/build-agent]
updated: 2026-10-17
decisions:
- "ADR 0010"
- "ADR 0010"
- "ADR 0010"
- "ADR 0239"
- "ADR 0267"
- "ADR 0276"
- "ADR 0282"
- "ADR 0289"
- "ADR 0304"
- "ADR 0350"
- "ADR 0358"
- "ADR 0360"
- "ADR 0364"
- "ADR 0382"
- "ADR 0395"
- "ADR 0398"
---
# Delivery — from a push to a running node
# Delivery — from a merge to the nodes, as it runs
One trigger, three silos, and a fan-out point that is the most consequential boundary in the
mesh.
What runs on the mesh's main branches on 2026-10-16, read from the code and the live record by research 055
(findings 1 to 12). It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) replaces, milestone
by milestone. Until 2026-10-16 this document described the predecessor's pipeline, which no longer runs.
## The trigger
## Who does what
A push to the forge. The forge calls a webhook; the receiving module verifies its signature
and emits an event. The coordinator resolves which modules the pushed commits affect, orders
them into dependency levels, and creates a pipeline per level.
- **mesh-delivery** owns deliveries and delivery groups (ADR 0239): one delivery per commit, a group per branch
name shared across repositories. It records each transition in its state on the bus (key-value buckets
`deliveries` and `groups`), marks pull requests, asks the controller to deliver, and merges a pull request marked
ready at its head once both required checks pass against the trunk's head (ADR 0364) and the operator approved
the merge into main on a channel that proves who answered (ADR 0398 §3).
- **The controller** plans, orders, sends and judges. It computes a delivery plan for a repository and the files a
merge changes, moves only modules whose build source changed with their build dependents (ADR 0267), and
reuses a merge check's build of the same reuse key (ADR 0350).
- **The builders** (holders of the build seat) take build requests and checks from a work-queue stream on the bus
and build inside a toolchain image another module provides, with the network off; on 2026-10-13 no node had Go
or npm installed (issue 626).
- **The forge module** publishes every forge event on the bus through the forge's system webhook since 2026-10-14
(issue 729), and still polls the forge every 30 s, or every 300 s once it hears events. It keeps files of what it
announced so it does not announce twice.
There is no second path. A manual trigger exists for a module the push detection missed, and
using it routinely is a sign the detection is wrong rather than a workflow.
## From a merge to a rollout
Detection is the pipeline's most fragile input. It has failed for reasons that have nothing to
do with the change: a webhook truncating its commit list on a large merge, a forge address
whose port broke the module-path matching. The characteristic outcome is the bad one — **a
merge that created no pipeline, and nothing said so**.
- **The merge window.** The controller collects merges into an open batch; each merge restarts the window, which
closes after 90 seconds without a merge or 10 minutes after the batch's first merge (ADR 0276). A lone merge is
cut at once (ADR 0282 §2, as ADR 0304 §1 changed it). The batch and its merges are rows in the controller's
store, cut by a timer every 5 s, with a catch-up timer every 5 minutes for merges no batch holds.
- **The rollout.** A closed batch becomes one rollout, at most one commit per repository, tiered along the module
graph. Rollouts run one at a time: a batch cut while a rollout is open is queued behind it, whatever it shares
with it. Running unrelated rollouts side by side (ADR 0282 §3, ADR 0304 §2) is decided and not built (issue 525). A rollout's builds go to every machine at once, with no first-machine gate, and the rollout ends on
the machines' reports or after fifteen minutes of silence (ADR 0360). A controller timer advances rollouts every
30 s.
- **A merge during a rollout.** Its place in the next batch is a store row the open rollout's holder writes. On
2026-10-16, when another controller worked the plans, a merge was not recorded and waited until a person released
it (issue 921).
- **"Release branches".** A pull request into a branch other than a trunk is judged by the controller built from that
branch (ADR 0358); feature 57 shipped about 37 packages that way over about 2.5 days and held the controller for
55.7 h.
## Three silos
## Checks
Cardinality is the whole point, and the three differ
(ADR 0010):
- Every pull request's head gets `mesh/merge-gate` (the composed machines) and `mesh/repo-check` (the repository's
`merge-check.sh`), each judged against one trunk commit (ADR 0323). A trunk move asks the check of every active,
cleanly merging pull request again (ADR 0395).
- A delivery group's members are each checked beside their siblings' heads, and the group also gets a composed
check, `mesh/delivery-group`; each member keeps its own gate beside it (ADR 0382).
- The queue's order (delivery builds first, then ready checks, then the rest) is kept by pausing the worker and
deleting and republishing a request (issue 558); an old head's check is ended after asking the forge for the
current head (issue 726).
- mesh-delivery ticks every 10 s, reconciles with the controller every 30 s and polls the forge for ready marks
every minute, although the forge module already publishes labeled, commented and synchronized events.
| Silo | Runs | Where | Does |
|---|---|---|---|
| **build** | once per module feature | the build node | compile and bundle into a self-contained output |
| **publish** | once per module feature | the build node | package that output and upload it; a package-registry feature publishes here |
| **deploy** | once per module feature **per node** | every assigned node | install, configure, start, verify |
## Where the record of what runs lives
Commands and events are addressed per feature, not per module.
Builds and build requests, rollouts with their tiers, batches and merges, and assignments are rows in the
controller's store; artifacts are blobs in the artifact store by digest (ADR 0251); packages are versions in the
package registry. The controller writes three record mirrors to the bus: assignments, declared config and
predefined keys; of the five ADR 0289 §7 decided, the nodes, rollouts, builds and settings mirrors are not in the
code. An assignment names its module and node but not the build, artifact or rollout that put it there: which commit
runs where, and which delivery brought it, takes three verbs and a join by hand.
Build hands over a **staged tree**, not a package. Packaging belongs to publish, so a failed
upload retries by re-packaging rather than by re-sending something stale, and build never needs
to know how each module composes its artifact. The handover is a tree in a known location,
because the build's own working directory is reference-counted and may be gone by the time a
later stage runs.
## The artifact
The artifact is **build output** — compiled and bundled with its dependency graph inlined —
never a filtered copy of source (ADR 0010).
A deploy is extract-and-run and touches no network.
The consequence is the whole cost of the decision: **anything not in the build output does not
ship.** Every file kind had to be brought into that rule separately, and each was discovered by
something silently not happening after deploy — migrations reading a source layout, provisioning
scripts reading a source layout, selection files never packaged at all.
## The fan-out, and the build node
After publish, work fans out to every assigned node. The build node is the only node that has
already passed through two silos when this happens, and that asymmetry has its own defect
class: anything advancing a node's stage must account for **both** pre-fan-out stages. Code
that knew only about the first parked the build node forever while every other node deployed
cleanly — and the recovery sweeper, which knew the same subset, reported nothing to recover.
A recovery mechanism that knows less than the thing it guards is worse than none, because it
reports success over a stall it cannot see.
## Levels
A level completes before the next begins, so a module builds against its dependencies as they
were just published. The shared library is at level zero, which means anything that breaks it
breaks the first module of every cascade.
## What green proves
**A green pipeline proves transport, not effect.** The stages report that a message was
dispatched and accepted, which is not the same as the thing being running, correct, or present.
This is the mesh's most consistent failure shape and it is not incidental to the design — it is
what the stage reporting currently measures. Documented instances include a service reported
started when the container command merely returned, an image pull failure that did not fail the
deploy, a package install that 404ed from every mirror while the job went green (see
[`04-ISSUES/001`](../../04-ISSUES/001-failed-package-install-reports-success/00-report.md)),
and a node left on old code after a failed artifact download with a version marker that had
already advanced.
A verify stage exists to close this gap. It has been built and, for a period, was never
scheduled, because the coordinator's stage list did not include it.
No backup is taken before a delivery moves a module that holds data; backups run nightly and on demand (ADR 0214).
## What is not covered
The end-to-end harness for this pipeline has not built since the workspace was removed, and
nothing reports that nothing runs it
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). The
pipeline's coverage is currently assumed rather than checked — which is the same class of claim
this repository exists to make people stop making.
The integration tests run only in each repository's merge check; there are no end-to-end tests of delivery yet,
and the lab is not used (the operator, 2026-10-16).
+3 -2
View File
@@ -5,8 +5,9 @@ code:
- mesh-controller cmd/mesh-builder
- mesh-controller internal/builder
- mesh-catalog modules/build-agent
updated: 2026-10-08
updated: 2026-10-16
decisions:
- "ADR 0405"
- "ADR 0262"
- "ADR 0233"
- "ADR 0219"
@@ -138,7 +139,7 @@ need to run. Third-party software still arrives as an image, because that is how
## What this costs, stated before it is chosen
**Every language is permanent.** It needs an SDK — broker client, sealed-credential reading, the
event envelope, tool serving — a toolchain image, and a bundler the builder understands. And the
event envelope, tool serving — a build module that installs its toolchain on every builder (ADR 0405), and a bundler the builder understands. And the
spine changes rarely but cascades when it does
(ADR 0039): with one language a contract
change is one edit; with four it is four that must land together, and a mesh whose SDKs disagree
@@ -5,7 +5,7 @@ code:
- mesh-controller internal/inventory
- mesh-controller internal/catalogue
- mesh-controller cmd/mesh-controller
updated: 2026-10-02
updated: 2026-10-16
decisions:
- "ADR 0181"
- "ADR 0182"
@@ -14,6 +14,8 @@ decisions:
- "ADR 0051"
- "ADR 0113"
- "ADR 0117"
- "ADR 0399"
- "ADR 0400"
---
# 29 — A node has operator accounts, and the mesh owns what lives under a home
@@ -220,6 +222,17 @@ each need a decision before the modules that replace the generators can be writt
The modules §2 calls *a family* — the shell, the terminal, the desktop, everything under a home that is not `~/.ssh` — are designed in [37 — The operator's machine](37-the-operators-machine.md), under ADR 0173 to ADR 0177. This document keeps `~/.ssh`, the CA and the roster files. Two things it listed as not built are decided there: user-scoped services (ADR 0177) and the one-off steps a hook used to run (declared state, or a seat's verb).
## The forge over SSH — 2026-10-16
`~/.ssh` gains the account's own key for the forge. ADR 0399 decides that each machine's account has an SSH
key made on the machine, that the forge registers its public half on the machine's agent user, and that
`ssh-client` writes the forge's SSH key into `known_hosts`, ssh's configuration entry for the forge and git's
rule that rewrites the forge's `https://` address to its SSH address. ADR 0400 decides how the public half
travels: `ssh-client` writes it to its own state, keyed by node and account, and the forge's provider reads it
there, since a manifest's contribution cannot carry a value made at run time. The forge's SSH address is its
machine's name on the mesh, where its SSH port is open; the key's file is `~/.ssh/mesh_ed25519`, beside the
operator's own keys, which stay theirs (§3). Built under issue 955.
## References
- The gap was found generating `~/.ssh/config` from the *HAL* registry (`hal/terminal`'s
@@ -236,3 +249,4 @@ The modules §2 calls *a family* — the shell, the terminal, the desktop, every
- ADR 0117,
ADR 0118 —
the never-sever-the-channel rule and the found-vs-owned semantics, applied here to `~/.ssh`.
- ADR 0399, ADR 0400 — git over SSH with a key made on each machine, and how its public half reaches the forge.
@@ -4,68 +4,112 @@ status: in-progress
code: [mesh-controller, mesh-catalog modules/mesh-delivery]
updated: 2026-10-16
decisions:
- "ADR 0408"
- "ADR 0407"
- "ADR 0406"
- "ADR 0405"
- "ADR 0404"
- "ADR 0403"
- "ADR 0402"
- "ADR 0401"
- "ADR 0398"
- "ADR 0395"
- "ADR 0382"
- "ADR 0377"
- "ADR 0364"
- "ADR 0362"
- "ADR 0360"
- "ADR 0350"
- "ADR 0335"
- "ADR 0323"
- "ADR 0313"
- "ADR 0306"
- "ADR 0304"
- "ADR 0299"
- "ADR 0298"
- "ADR 0282"
- "ADR 0276"
- "ADR 0270"
- "ADR 0267"
- "ADR 0250"
- "ADR 0249"
- "ADR 0239"
- "ADR 0238"
- "ADR 0236"
- "ADR 0233"
- "ADR 0214"
- "ADR 0201"
---
# 47 — Delivery: from a commit to delivered
**A delivery is one commit in one repository, from the moment its pull request's head is announced to the
moment every machine of its deploy plan runs it, or to the moment it fails, is superseded or is stopped. A
delivery group is two or more deliveries that share a head branch name, delivered as one unit in order. Both
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
(ADR 0239.)
**A change reaches the machines in one of three ways, all owned by mesh-delivery: a hotfix (one pull request,
delivered at its merge), a feature (one branch name across its repositories: in two or more, one delivery
group merged together and delivered member by member in its order; in one, a lone delivery), or an epic (a run of milestones, each delivered as a feature). The trunk is what runs; there
is no "release branch"** (ADR 0401).
**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a
rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout
at a time. Every delivery the rollout answers ends with it
(ADR 0276).
**Every step starts on an event, and the state lives on the bus.** The forge's events, the controller's
events and the nodes' reports move a delivery; the bus's own rules keep the queues, the merge window and the
order; and one chain, from module to commit to build to rollout to assignment, is kept in the bus's
key-value store as its only source (ADR 0406, ADR 0407, ADR 0408). A running delivery never touches merging:
every merge is recorded at once for the next delivery plan of what it touches.
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
of source and artifacts decides, and nothing here changes that. This design says who answers for one
commit's journey and in what order the steps are allowed.
commit's journey, in what order the steps are allowed, and where each fact of it is kept. It was rewritten on
2026-10-16 from the decisions of research 055, which replaced the "release branches", the toolchain image, the
controller's delivery tables and the timers that polled them.
## The parts
```
forge's holder ──pull.updated / .merged / .closed──► mesh-delivery ◄──checked / plan-moved── controller
(statuses, the view, (deliveries, groups, (planner, gate,
the notes) the table, its state) registration,
▲ │ │ sending, walk)
└── its tools: note, view, status ◄───────────────┘ └── its verbs: delivery-plan, -order,
-check, deliver, delivery-stop,
delivery-walks
forge (system webhook) ──forge.pull-request.* / merged / status──► the bus ◄──built / checked / plan-moved── controller
│ ▲ (planner, order,
│ │ registration,
▼ │ sending)
mesh-delivery │
(deliveries, groups, a key per │
pull request, the state table) ▼
node-engine on every node
builders (build seat) ◄── work queues: delivery builds, ready checks, other checks ───────────────────────── (reports; backups first)
```
- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants,
registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk
rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers
every pull request's head with its check itself, so a check never depends on mesh-delivery.
- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the
group, the state table and their state, and nothing else.
- **The forge's holder** sets a commit's statuses, keeps the delivery's view as one comment on the pull
request, and appends to the commit's note under `refs/notes/mesh-plan`.
- **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350):
a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a
reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build
of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and
says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull
request closes unmerged, and after seven days.
- **mesh-delivery** (the seat `mesh-delivery`) owns hotfixes, features and milestones as deliveries and delivery
groups, decides each step on events, keeps one key per pull request, and asks the controller for each act
and the nodes' backup seats for the backups a delivery needs. It never sends to a machine, writes a
declaration or registers a build (ADR 0239 §25).
- **The controller** keeps planning, ordering, registration, sending and judging. It writes the chain's
links it owns — builds, rollouts with their tiers, assignments — and the merge window's
keys, all on the bus, by compare-and-set, while it holds the lease.
- **The builders**, the holders of the build seat, build and check with toolchains installed on their node, as
the build seat's own account, and take their jobs from the bus's work queues, delivery builds first
(ADR 0405, ADR 0406).
- **The forge module** (`gitea`) publishes every forge event on the bus through the forge's system webhook
(issue 729): pull requests updated, synchronized, labeled, unlabeled, commented, merged and closed, and
commit statuses. It sets statuses, keeps the delivery's view as one comment on the pull request, and appends
to the commit's note under `refs/notes/mesh-plan`.
- **Each node's backup seat** (`node-backup`) backs up the data a delivery plan moves there before the plan
sends anything (ADR 0404).
## Three ways a change reaches the machines
*ADR 0401.*
| Way | What it is | How it is delivered |
|---|---|---|
| **hotfix** | one pull request in one repository | at its merge, as a lone delivery; a lone merge is cut at once (ADR 0282 §2, as ADR 0304 §1 changed it) |
| **feature** | one branch name across its repositories; in two or more, its pull requests are one delivery group; in one, a single pull request | a group is merged together once its composed check passes and every member is marked, reviewed and approved, then delivered member by member in its order (ADR 0239 §16); in one repository, as a lone delivery like a hotfix |
| **epic** | a run of milestones in one design (playbook 04) | each milestone is a feature, delivered when its own group is ready; the next starts when the previous one's delivery plan is done |
- **The trunk is what runs.** Every delivery is cut from a repository's trunk. A merge into any other branch
starts no delivery, and the controller says so for that merge (issue 889). Holding merged work back, on a
branch beside the trunk or per module, was dropped: from 2026-10-13 23:52 to 2026-10-16 09:32 feature 57's
branches held the controller for 55.7 h, and 28 of 68 other merges touched a held module (research 055,
finding 3).
- **A milestone is functional, its delivery shaped technically.** A milestone is named by what the operator or
the mesh can do once it is delivered; how it ships is shaped by the delivery plan its changes would have, asked of
the controller's planner through mesh-delivery's `what-if` (ADR 0239 §3): the modules it moves and their fan-out.
- **What a merge waits for.** mesh-delivery merges a group's members once the group's composed check passed at
their heads, the independent reviewer marked each member ready (ADR 0364 §1, §2), each member carries the
review ADR 0382 §6 asks, and the operator approved each merge into main (ADR 0398 §3). The operator's Approve is
asked for every merge into main, a hotfix's too (the operator, 2026-10-16 23:40).
## A delivery
@@ -74,12 +118,17 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
- the pull request (number, base, head branch, description), or none for a commit that reached the trunk
without one;
- its **delivery plan**: build plan, deploy plan and verdict, as the controller's planner computed them for
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
its diffset (ADR 0238 §2, as ADR 0323 §1 changed it);
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
**machine steps** while delivering;
- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
window and the batch*).
- once merged, **the commit it landed on the trunk as**, and the delivery plan that delivers it: the plan of the
batch that answers its merge, which may carry a later commit of its repository that contains it.
**One key per pull request.** Beside the delivery, mesh-delivery keeps one key per open pull request on the
bus: its head, the trunk commit its current check is asked against, and its verdict, written by
compare-and-set (ADR 0406 §4). It writes a new head when the forge says one, and a new trunk commit only when it
asks the pull request's check again (ADR 0395 §1). A running check watches that key and stops when the head or
the trunk commit in it moves.
## The state table
@@ -94,388 +143,365 @@ allows.
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
| ready | it passed; it waits to be merged | none | — |
| rejected | it failed its check | none | — |
| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started |
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says |
| published | merged on its trunk, recorded in a batch; it waits for its plan to start | 30 minutes | `superseded` when a plan that does not answer its merge took over its plan; `go` when its plan started |
| held | it waits for a person: a failed backup (ADR 0404 §4), a policy of record; it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
| delivering | its plan runs: backups, then builds asked and sent tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the plan's key says |
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
**A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head
at the moment the build seat clones it, the **trunk commit**, and the verdict names both. When another merge
moves the trunk, a `ready` delivery goes back to `proposed` on the observed event `trunk moved`, with both
commits in the transition's reason; a `rejected` one stays `rejected`, its statuses saying the verdict is out
of date. Nothing is asked for it: a recheck, a new head or the merge tool asks the check again. The row from
`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for
an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout.
when the builder clones it, and the verdict names both. When another merge moves the trunk, a `ready`
delivery goes back to `proposed` on the event `trunk moved`, and mesh-delivery asks the check of every active,
cleanly merging open pull request into that trunk at once (ADR 0395, *A move of the trunk asks every check
again* below). The row from `proposed` to `checked` takes only a verdict that names the trunk's head.
What this leaves as it is, said so nobody reads more into it:
**What needs no person ends by itself** (ADR 0270): a merge that moves nothing is `delivered` at its merge; a
merge a later delivered merge contains is `covered`, by the forge's word that the later merge commit contains
it; a merge whose plan said its modules unchanged is `delivered` when that plan ends (issue 611); a stop whose
plan ended without building its modules is read as covered once a later merge covers it (issues 474, 480).
- **mesh-delivery's trunk head is the last merge it heard** into that trunk while the delivery was open. A push
to the trunk that is not a merge it hears does not move it; the forge's holder, which reads the branch, and
the merge tool, which compares commits when asked, still catch it.
- **A verdict that names no trunk commit**, from a build seat older than ADR 0323, counts in mesh-delivery as it
did: such a delivery stays `ready`. The forge reads the same verdict out of date once its trunk moves, and the
merge tool refuses it and asks the check. The two agree once the head is judged again.
- **A verdict no build judges** (a change that touches no module, in a repository with no `merge-check.sh` or
not the mesh's) names the base branch's head the forge announced with the head as its trunk commit.
- **In a busy repository the merge tool can refuse, ask, and refuse again**: another merge may move the trunk
while the asked check runs, and the new verdict is then out of date too. Each refusal asks once more; the
merge goes through once a check finishes before the next merge.
- **A check that was asked and never answers** is listed by `stalled` an hour after the ask, and is asked again
with `mesh-delivery.recheck`, which the merge tool's refusal names.
**History is retired, not held** (ADR 0250). A delivery made from the forge's word at its merge, heard fifteen
minutes or more after it, with no plan of it open, goes to `retired` at adoption. **Nothing of a published
delivery is registered before its plan starts.**
The transitions are those of ADR 0239 decision 2, ADR 0270 and ADR 0323. A delivery that is
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by
`stalled` at once. Nothing waits silently.
**The machine steps.** While a delivery is `delivering`, each machine its plan reaches has a row with the
module and the build: `backed-up`, `sent`, `applied` or `failed`, and `silent` when no report came within
ApplySilentAfter (fifteen minutes). They are read from the plan's key, written by the controller from the
nodes' reports, never inferred from time.
**What needs no person ends by itself** (ADR 0270):
## Order without a person
- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and
sends nowhere) is `delivered` as soon as it is merged, whatever its check said;
- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine
running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the
forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge
order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery`
says `covered: a later merge that contains it was delivered` as a success. A module that runs on one
machine only is covered too: its rollout records the empty rest as sent once its first machine passes
(ADR 0270, as corrected 2026-10-12);
- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends,
whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads
that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and
may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such
module with why, "its build source holds none of the changed files", and not in its modules, which keep
meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered,
saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below.
Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's
`TestAMergeWhoseModulesMatchWhatRunsIsDelivered`;
- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and
is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after
its stop` with the covering merge, once, as a success. No state or transition is added (issue 480);
- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes.
*ADR 0239 §11 and §12, ADR 0249, ADR 0402.*
**History is retired, not held** (ADR 0250).
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout
of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes
to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout
starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a
build registered before its turn would be carried by somebody else's send.
A delivery group's order is a table of pairs, each with the rule that won (ADR 0249 §3). The controller's
`delivery-order` infers it from the manifests and the module graph; a person writes an `after: <repository>`
line only where the rules cannot see the order.
**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the
module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy
readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the
controller's `plan-moved` event, never inferred from time.
| Rank (highest first) | Rule | Read from |
|---|---|---|
| 1 | *declared*: an `after:` line | the member's description |
| 2 | *built by* and *version skew* | the module graph and the change |
| 2 | *verb before caller*: a member adding a verb to a seat's served verbs before a member adding that verb to an invoked verb | the manifests (ADR 0402) |
| 2 | *SDK before its users*: a member changing `mesh-sdk` before every member whose modules are built on it | the module graph (ADR 0402) |
| 3 | *engine before controller* | module names |
| 4 | the repositories' names | — |
Rules of one rank in opposite directions reject the group, naming both members, both rules and the line that
would settle it (ADR 0249 §4).
**The SDK by version.** A Go module names the SDK by its released version and fetches it from the mesh's forge,
checked against its recorded checksum; vendored copies go. Inside a group, a member may name the SDK member's
head: its merge check resolves that commit from the forge and builds against it, and the head stays fetchable
after the group merges because the forge keeps every pull request's head. The catalogue's check accepts an SDK
commit on the SDK's trunk or the head of an SDK pull request the forge holds, open or merged (ADR 0402 §2 to §4).
## Checks
### One check per pull request, kept by the bus
*ADR 0323, ADR 0350, ADR 0395, ADR 0406.*
- **Three work queues, taken in order.** Delivery builds go on a work-queue stream of their own; checks of pull
requests marked ready on a second; every other check on a third. A builder takes from the first, then the
second, then the third. Each check stream has one subject per pull request and keeps only the newest request:
a newer request replaces one still waiting, by the stream's rule. A ready mark publishes its pull request's
check request on the ready stream; a request left behind on the third for a head and trunk commit already
judged or running is acknowledged and dropped when taken, by reading the pull request's key. Nothing pauses the
worker or deletes and republishes a request (issue 558).
- **A running check stops itself.** It watches its pull request's key and stops when the head or the trunk
commit in it moves, without asking the forge (issue 726). An ended check delivers no verdict, so none of its
builds is reused.
- **A failed builder loses nothing.** A job a builder took and never acknowledged goes to another builder
after its wait.
- **A change is built once** (ADR 0350). A check that passes also builds what its merge would move into the
artifact store's scratch namespace under a reuse key, and the plan of the merge commit copies the build of the
same key, named by the check's verdict, instead of building.
### A move of the trunk asks every check again
*ADR 0395.*
- A trunk move asks the check of every active open pull request into that trunk that merges cleanly, against the
new head; a conflict is told once on the pull request, not checked.
- A pull request with no push and no ready mark for 7 days is left out; a push, a ready mark or a recheck asks
its check.
- For a group, a move asks each member's own check, and the composed check runs again through those rechecks; a
move made by a member merging in the group's order asks the other members nothing.
- Its cost is at most one check per active, cleanly merging pull request per trunk move; issue 944 measures it.
### A delivery group's checks
*ADR 0239 §14, ADR 0323 §6, ADR 0382, ADR 0403.*
- **The composed check is each member's gate.** The group's composed check composes every machine with all
members' heads, each merged into its own trunk's head first, judged by the controller built from the group's
controller member when it has one (ADR 0382 §3). Its verdict sets `mesh/delivery-group` and each member's
`mesh/merge-gate`, and names every member it read. No gate runs for a member alone.
- **Each member's own repository check stays required.** Each member's `mesh/repo-check` runs its repository's
suite with the siblings read beside it (ADR 0382 §2).
- **No composed verdict, no gate.** A member's gate is pending while no composed verdict at the members' current
heads exists. A member's new head, close or base move puts the composed verdict out of date, and with it every
member's gate. When mesh-delivery cannot say the groups, the pull request is checked alone and its verdict
says so (ADR 0382 §1).
- **A member merges only with its group passed and reviewed** (ADR 0382 §6), and in the group's order
(ADR 0364 §8).
## Ready marks and the merge
*ADR 0364, ADR 0406 §1.*
The independent reviewer marks a pull request ready at the reviewed head with the label `mesh/ready` and a
comment `ready at <full sha>`, through `mesh-delivery.mark-ready`. The forge's `labeled` and `commented` events
reach mesh-delivery in seconds; once both required checks passed against the trunk's head, it asks the operator's
Approve for a merge into main, with the pull request, its issues, the reviewer's verdict and the delivery plan, and
merges the marked head through the forge's merge tool on that Approve (ADR 0398 §3). When the trunk moved, the merge tool asks the fresh check, and the event
of its passing verdict asks the merge again; no timer asks it. A push clears the mark (the `synchronized`
event), and a failed check is said once and listed by `stalled`.
## The merge window and the batch
*2026-10-10, by ADR 0276.*
*ADR 0276, ADR 0282, ADR 0299, ADR 0304, ADR 0407.*
- **The window.** The controller puts every merge into a branch a module follows into the open **batch**.
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
merges and its times are kept in the controller's store and survive a restart.
- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for
every commit in it, tiered along the module graph across repositories, with at most one commit per
repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the
earlier merges of a repository that its later commit contains.
- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started
is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new
rollout, which answers every merge the folded one answered.
- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller,
the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for
mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its
rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again
behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10).
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is
answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends
`covered` at once.
- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for
an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own
commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own
commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose
merge it rollouts again nowhere (below).
- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's
tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is
at its turn.
- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a
folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting
rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
delivery it holds.
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch
waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout
answers within ten minutes is held with the true reason (the open rollouts of its repository and that none
names its merge, or that the controller holds none) and listed by `stalled` at once.
- **Every merge is recorded at once.** The forge's merged event records a merge into the open batch of what it
touches, or opens one, by whichever controller holds the plans. A merge is never refused, held or dropped,
and needs no person's release (issue 921). The bus's durable consumer keeps every merge event until it is
recorded; no catch-up timer looks for missed merges.
- **The window is two keys.** An open batch's window is a key whose time to live is `merge-window` (90 s),
refreshed by each merge into the batch, beside a key set once when the batch opens whose time to live is
`merge-window-at-most` (10 minutes). The first to expire closes the batch, heard by a watcher; no cutter
polls. A stop that records merges again into the open batch closes its window too (ADR 0299 §3).
- **A lone merge is cut at once**: no batch it shares a repository, module, graph edge or delivery group member
with is assembling (ADR 0282 §2, ADR 0304 §1, ADR 0306).
- **The plan.** A closed batch becomes one delivery plan: every commit in it, tiered along the module graph
across repositories, at most one commit per repository, the latest of its trunk. It answers every merge in the
batch, its own commits and the earlier merges a later commit contains (ADR 0276 §2, as ADR 0299 §7 changed it).
- **A plan fixes its commits when it starts.** A plan starts at its first send, after mesh-delivery's word, and
from then nothing that merges changes it. A plan still waiting for its word is folded into the next batch
(ADR 0276 §4).
- **Plans that share something run one after another.** A batch that shares a repository, a module or a graph
edge with an open plan waits for it, and starts with every merge recorded since when that plan ends. A batch
that shares none runs beside the open ones, so a hotfix is never held by an unrelated feature's delivery.
- **The ends.** A plan done ends every delivery it answers: `delivered` for its own commits, `covered` for an
earlier merge it carried by a later commit. A plan that failed fails its own commits' deliveries; each earlier
merge it answered is then delivered on its own commit, the newest first, until one is delivered (ADR 0276 §3).
- **Shown.** `plans` lists the open batch first, with its window's time left and its merges, and a batch waiting
behind an open plan names that plan; the controller says the batch as `plan-moved` in the state `assembling`.
## A backup before the send
## A stop on a shared rollout
*ADR 0214 §7, ADR 0233, ADR 0404.*
*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.*
When a delivery plan is ready to send, each node it reaches backs up, through its `node-backup` seat, every item
the plan's moved modules keep there whose protection is a backup (a copy or a dump). An item protected by
`redundancy`, one that says `backup: none`, and a cache are never asked, whatever their class. A node with no backup holder backs up
nothing, and the plan says so for that node.
One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other
delivery on it. A stop therefore ends only the change it is for:
- **The send waits for every backup.** The controller sends nothing of the plan to any node until every backup
it asked has answered done.
- **A failed backup holds the delivery.** A backup that fails, or does not answer within ten minutes, holds the
delivery with the node, the module and the backup's own words as the reason, and raises a condition for the
operator. A person's `release` asks the backups again, and the send goes on when they pass.
- **What it costs.** About 85% of data modules back up in under 30 s; the slowest measured, postgres on the home
server, about 165 s (research 055, finding 5). A plan moving no backed-up item costs nothing more.
## The send and the rollout
*ADR 0360, ADR 0362, ADR 0377, ADR 0406 §5.*
- **Every machine at once.** A plan's builds go to every machine that runs them in one step, tier by tier; a tier
is asked once the tier before it is built and sent (ADR 0360 §1, §6). No machine goes first and nothing waits
for a health reading. A module of policy `record` that would recreate a container or move data waits for a
person (ADR 0377).
- **Every move is said.** The controller says every move of a plan as `plan-moved`, a move a command makes among
them, so mesh-delivery needs no periodic reconcile; it reads `delivery-walks` once when it starts.
- **A rollout advances on events.** The `built` and sent events of a tier ask the next; the nodes' reports end
the rollout (ADR 0360 §7, with ADR 0362's provisioning report). Each send puts a key whose time to live is
ApplySilentAfter (fifteen minutes); its expiry names the machine silent. There is no rollout ticker.
- **A failed apply is loud and fixed forward** (ADR 0360 §8); the node-engine's witness still puts back a
controller or tool runner that does not come up (ADR 0360 §9).
## A stop on a shared plan
*ADR 0299.*
One plan carries every merge of its batch, so a stop ends only the change it is for:
- **A group's stop.** When a member after a failed one is stopped and its plan is open, mesh-delivery asks
`delivery-stop` for that plan with the stopped members' merges to leave out. The controller ends the plan and
records every other merge it answered again in the open batch of its kind, whose window the stop closes.
- **What is left out with it.** A later merge of the stopped merge's repository, and a merge that moves a
module a left-out merge moves, would deliver the stopped change; each is left out too, its delivery `stopped`,
naming why.
- **A person's stop.** `stop` of a delivery whose plan carries others first answers what it would take down, and
acts on `way`: `again` (the others delivered again without it) or `walk` (the whole plan stopped).
- **What it does not do.** What the stopped plan already sent stays on the machines it reached; putting it back
is issue 509's.
- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks
`delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass
stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge
it answered again.
- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A
module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too.
Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped
merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is
rolled out again.
- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran
already wait, so an older commit never rollouts after a newer one; only with none open do they form a new
batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop
before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules
where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes
to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every
merge moved are written in one transaction.
- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left
out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery
whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays
`published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every
delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop.
- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped
delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it
stops the whole rollout.
- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it
answers with each of them, which of them contain this merge and end either way, and two ways given with
`way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout
stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is
marked stopped only once its rollout is; a stop the controller refuses leaves it as it was.
- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its
asked builds still register, so a later send may carry them further; the stop names the machines, and
putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the
stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it
can see, and holding a merged change back for good is a revert's job.
## The delivery budget
*Decided 2026-10-10 by ADR 0282.
Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by
slice as each slice below is built; until a slice is built, the text above is what runs.*
*ADR 0282, ADR 0304 §4, ADR 0313, ADR 0360 §12, ADR 0362 §5.*
**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard
from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes;
a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is
shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes
(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers
14.2 (issue 338).
A delivery's **delivery time** runs from its merge until every machine its plan sent has reported the send
applied, and for a provisioned module also placed. A **leaf** delivery's budget is three minutes, a **core**
delivery's ten. A delivery held for a person, a failed backup among them, is shown and held to no budget. Each
phase is kept — the window, the wait behind a plan that shares something, the backups, the build, the send, the
reports — and a delivery over its budget raises `delivery.<class>.over-budget`, naming its longest phase. The
**fix time** (ADR 0313) runs from an issue's first move to `fixing` to its last fix delivered, and contains the
delivery time.
*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget
above runs from the merge; the operator's own target is wider, the **fix time**, which runs from an issue's
first move to `fixing` to the moment its last fix is delivered. Each issue's class carries a **fix budget** too
(five minutes leaf, ten core, the delivery budget's own figures), and the tracker splits a fix time into four
phases — implementing, review, checks, and merge to running (the delivery time above) — so a fix over its
budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget.
The delivery budget stays as written: a measure inside the fix budget, not replaced by it.
## Builds without a toolchain image
**What changes in the rollout.**
*ADR 0267, ADR 0350, ADR 0405.*
- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged.
- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One
node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf
build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged.
- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf
module with a long-running resource; one for a leaf module whose every resource is a tools bundle.
- **A tier's first nodes are judged at once.**
- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new
tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before
anything is judged; a step that would carry anything else is refused unsent and said on the gate line.
- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy
report; the plan shows an expected landing time from the last seven days' medians.
- **One build module per language** (Go; Node and TypeScript) installs its toolchain as a declared package on
every node holding the build seat, pinned to one version that is the same on every builder. The toolchain
image and its module retire.
- **Every builder runs the same toolchain, and says so.** Each builder names its toolchain versions with every
build outcome, and the controller's self-check raises a condition while two builders differ.
- **Builds and tests run directly**, as the build seat's own account on its node, in no container. A
repository's `# mesh-check-toolchain` line selects which installed toolchain runs its check.
- **A judge has an account of its own.** A judge built from a sibling's merge (ADR 0382 §3) is built and run as a
second account of the Go build module, whose home, configuration and cache the build seat's account cannot write,
so a check's script cannot change it.
- **Every dependency comes by the version its lock file names**, checked against its checksum: the mesh's own
code from the mesh's forge, its own packages from its package registry, any other from its public source.
- **A builder fetches from the forge over SSH with a key of its own**, as the build seat's account, under a forge
account for that builder that reads the mesh's repositories and pushes nothing.
- **A build's reuse key** holds the installed toolchain's version and the build module's recipe version in place
of the toolchain image's digest, so a toolchain update rebuilds only the modules of that language.
*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it:
## One chain, on the bus alone
- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open
rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open
rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
for builds an open rollout owns).
- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the
merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order.
- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or
send, the dependents of what its merges moved included; a change whose modules are not known yet is related to
every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node.
- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a
warning naming the rollout and the core rollouts that set it aside.
- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an
unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its
own builds.
- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by
issue 528.
*ADR 0408.*
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out
of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every
open judgement on its node.
Module, commit, build, published package or artifact, rollout and assignment are one chain, each link
answerable from the next. A **rollout** here is the controller's record of one batch's delivery; a delivery's own
delivery plan (*A delivery* above) stays the delivery's, and names the rollout that delivers it.
*A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`,
a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge
into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace
such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once
it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy
container there, its first-node gate starts at the report that says the container replaced, and the wait is a
phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget
is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a
rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the
node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the
judgement.
| Link | Names | Written by | Bucket |
|---|---|---|---|
| build | its module, repository and commit, its reuse key, what it published: an artifact by digest or a package release by name and version | the controller | `mesh-controller_builds` |
| rollout | the commit it fixed per repository, its tiers, the builds it carried, the machine steps, and every assignment it changed with the build before and after; its states and transitions | the controller | `mesh-controller_rollouts` |
| assignment | the module and node, the build, artifact or package and commit it runs, and the rollout that put it there | the controller | `mesh-controller_assignments` |
**Who builds what.**
Beside the chain, on the bus too: deliveries and groups (mesh-delivery's `deliveries` and `groups`, ADR 0239 §37),
one key per open pull request (mesh-delivery), and each open batch with its merges and its window's two keys (the
controller, ADR 0407).
| Part | Owner | What it adds |
|---|---|---|
| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` |
| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery.<class>.over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` |
| the lone merge | mesh-controller | the cut at once when nothing waits to join |
| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch |
| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once |
| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` |
**Slices**, each one pull request per repository, each with its acceptance sentence:
| Slice | Delivers | Done when |
|---|---|---|
| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases |
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout |
| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median |
Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges
only once it is built.
## A delivery group
- **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull
requests. Two or more make a group. A group that has started delivering takes no new member.
- **Order**: `after: <repository>` lines in a member's description, and the controller's `delivery-order`
over the graph (ADR 0239 decision 4). The plan shows the order and why each pair is ordered: *declared*,
*built by*, *version skew*, *engine before controller*, *by name*.
- **Precedence** (ADR 0249): when rules order one pair both ways, the higher one decides. *Declared* comes
first, then *built by* and *version skew*, then *engine before controller*. The pair shows the rule that
won and the rules it won over. Rules of one rank both ways are a contradiction. The group is `rejected`,
and the reason names both members, both rules and the `after:` line that would settle it.
- **Check**: the members' heads composed together, one gate over every machine with all their definitions,
each head merged into its own trunk's head first and named with that trunk commit (ADR 0323). It is asked
by mesh-delivery through `delivery-check` when the group forms or a member's head moves, and posted on
every member's head as `mesh/delivery-group`. A member rebased or pushed is a new head with its own check:
it takes its old head's place in the group, even once the group is delivering, and the composed check runs
again for the new heads instead of holding the group. A merge into a member's trunk that is not one of the
group's members puts the composed check out of date: the group reads `checking` with the reason, and the
composed check runs again once a member is judged again. Checked by mesh-delivery's
`TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember` and `TestARebasedMemberIsComposedAgainNotHeld`.
- **Each member's own check reads its siblings** (ADR 0382): the group's other open members into a base of
the same name are read beside a member's own check at their head merged onto their base, and a mesh-controller
sibling's merge judges it, so changes that are only consistent together pass their own checks. The controller
learns the siblings from mesh-delivery's `groups`; the verdict names each one. A sibling's new head, close or
base move puts that verdict out of date, and the merge tool merges such a member only when
`mesh/delivery-group` passed at the heads read and every sibling head read is reviewed.
- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`.
A member published while its group's composed check is asked and not answered waits there for the answer;
one held because that check had not passed is *lifted* back to `published` once it passes or warns for the
heads that merged and its own rollout is still open (ADR 0270).
- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its
rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered.
Its own builds were put back at its gate.
A group's state is derived from its members and its composed check, and is shown, never stored apart from
them.
- **The bus is the only source.** Each bucket is written by the controller that holds the lease, by
compare-and-set, and the controller reads its own buckets to decide; a rollout resumes from its key after a
restart. No table of the controller's store holds builds, rollouts, batches or assignments, so no record mirror,
audit or healer exists for them; the nodes, settings, declared-config and predefined-keys mirrors stay as
ADR 0289 and ADR 0356 have them. A module reads a bucket by naming it in `reads`; mesh-config reads the
assignments bucket.
- **No key of the chain is removed.** Each bucket keeps up to 64 earlier revisions of each key as its history. A
value is never cut to a summary and stays under the bus's largest message. The bus's storage is declared as data
protected by a backup, so it is backed up nightly and before a delivery moves the bus.
- **One read answers each question.** For any module on any node, one read of its assignment names the commit,
the build, the artifact or package and the rollout that put it there; one read of a rollout lists every
assignment it changed.
- **The tables go by a numbered migration**: builds, rollouts and assignments are written once into the buckets,
waiting build requests onto the work queues and an open batch into its keys; then the tables are dropped from the
controller's store and its consolidated schema.
- **The artifact store and the package registry keep the bytes**; the chain names them (ADR 0251).
## What every transition does
In this order, each by its owner:
1. **kept**: mesh-delivery puts the delivery in its state (`deliveries`, one key per delivery; `groups`, one
key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried, later
each time, from ten seconds doubling to an hour. The forge's answer that what it is for does not exist ends
it at once, and sixty failed tries end it too, each said among the owner's refusals (ADR 0250);
1. **kept**: mesh-delivery puts the delivery in its state by compare-and-set before anything else; what it owes
outside is kept with it and retried from ten seconds doubling to an hour, at most sixty times (ADR 0250);
2. **said**: the event `mesh-delivery.transition` (and `mesh-delivery.group` when a group's derived state
changes), with the delivery's id, from, to, event, why and plan summary, and no secret or address;
3. **noted**: mesh-delivery asks the forge's holder (`gitea_note_append`) to append a line to the commit's
note: the head while the delivery is off the trunk, the commit it landed as once it is on it, and at its
end a line of what was executed there. The forge's holder writes it in its own repository as the forge's
own account, and a line already there is not appended again;
4. **reflected** (not for history, whose pull request the forge may no longer hold): the delivery's view, one comment on the pull request with the state, the plan, the machine
steps and the group, kept current (`gitea_delivery_view`); the status `mesh/delivery` on the head, and
`mesh/delivery-group` on every member's head (`gitea_commit_status`). Every status links to the pull
request, `mesh/merge-gate` among them.
changes), with no secret or address;
3. **noted**: a line on the commit's note, appended by the forge module as the forge's own account;
4. **reflected**: the delivery's view on the pull request, `mesh/delivery` on the head and `mesh/delivery-group`
on every member's head.
A doubled step is refused by compare-and-set and by the bus's message de-duplication, which also replaces the
forge module's files of what it announced (ADR 0406 §6).
## The verbs
**mesh-delivery's seat** (served by its holder; the first five read):
**mesh-delivery's seat**:
| Verb | Answers or does |
|---|---|
| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group |
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout |
| `deliveries` | every delivery not final, the final ones of the last day, and every one `stalled` still lists |
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the chain's links |
| `groups` | every group with its members in order and its derived state |
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner |
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds |
| `recheck` | a rejected or ready delivery back to proposed, with why |
| `release` | a held delivery to delivering, with why: a person's word |
| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise |
| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing |
| `stalled` | every delivery held past its bound, with the transition H2 may take |
| `mark-ready` | a reviewer's mark at a head: the label and the `ready at` comment, by the mesh's own forge account |
| `recheck` | a check asked again for a delivery whose verdict is out of date (ADR 0395 §6) |
| `release` | a held delivery goes on, with why; for a failed backup it asks the backups again |
| `stop` | a delivery to stopped, with why; on a shared plan it first answers what it would take down (ADR 0299) |
| `close` | H2's only verb: the transition the table names for a stalled delivery |
| `retire-history` | held history to retired (ADR 0250) |
| `times` | each class's budget, the week's median and worst delivery time, and the deliveries over budget |
**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`,
`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which
takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299),
`delivery-walks`. A person has `plans go <plan> --why`, which does what `deliver` does under their name and
is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command
too.
**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It
also announces `pull.closed`, and a merge's head and the statuses the forge holds on it.
## The store
Key-value state on the bus, declared by the module (ADR 0201): `deliveries` and `groups`. One writer, the
seat's holder, whose code changes state from a single goroutine. A final delivery is kept for thirty days and
then removed. Its note on the commit keeps its record after that. The bus's own snapshot (ADR 0235) backs it
up.
**The controller's verbs for it**: `delivery-plan`, `delivery-order`, `delivery-check`, `deliver`,
`delivery-stop`, `delivery-walks`, and `plans` (the open batch, the open plans, and a what-if for a repository
and its files). A person has `plans go <plan> --why`.
## The bootstrap
- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
controller's own, started by the merge and witnessed on the machine. mesh-delivery records it.
- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the
controller starts it as it always did.
- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and
`plans go <walk> --why` as the way on.
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
- A plan that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
controller's own, started by the merge; mesh-delivery records it (ADR 0239 §31).
- Any other plan waits for `deliver` while the seat has a holder on record; with none, the controller starts it.
- A replaced controller or mesh-delivery carries on from the bus: the keys, the queues and the durable
consumers hold every step, so nothing is lost or repeated (ADR 0406 §6).
## The switch
## Acceptance tests
1. The controller's change (the seat, its verbs, `plan-moved`, the wait for a holder's word, `plans go`) is
merged and rolls out as the core does. With no holder on record it behaves exactly as before.
2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built
and published.
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start
from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in
`delivering`.
4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass.
What the build proves before this design is implemented (research 055's direction):
## Phases
1. A feature across the controller and the catalogue with a new seat verb is delivered as one group, with no
`after:` line, no two steps and one composed check.
2. An SDK change and a Go module using it are delivered as one group.
3. A data module is backed up before a delivery moves it, and a failed backup holds the delivery.
4. A build on any builder uses the installed toolchain with no container, and a toolchain update rebuilds only
modules of that language when its version changes.
5. Two merges into one repository within a minute leave each open pull request with one queued check, by the
stream's rule.
6. A ready mark is acted on within seconds of its event.
7. A controller replaced in the middle of a rollout resumes it with no step lost or repeated.
8. The merge window closes by its key's expiry.
9. A hotfix to a module the running feature's delivery does not touch reaches the machines in minutes, beside it.
10. For any module on any node, one read of its assignment names the commit, build, artifact or package and the
rollout that put it there, and one read of that rollout lists every assignment it changed.
11. Two merges into one repository while a delivery of that repository runs are both delivered by the next plan,
with no person's release and nothing held in mesh-delivery.
12. No table in the controller's store holds builds, rollouts or assignments.
| Phase | Delivers | Done when |
|---|---|---|
| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` |
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 |
| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
## What retires
- The "release branches" and the controller's judge of them (ADR 0358, superseded by ADR 0401).
- The toolchain image and its module (ADR 0405).
- The window cutter, the merge catch-up timer, the rollout ticker, mesh-delivery's tick, reconcile and
ready-mark poll, the forge's poll as a trigger, the queue reordering, the forge-asking cancel, and the forge
module's announcement files (ADR 0406, ADR 0407).
- The controller's tables of builds, build requests, rollouts and their tiers, batches and merges, and
assignments; the record mirrors, probe D17's audit and healer H6 for those kinds (ADR 0408).
## Milestones
Confirmed by the operator on 2026-10-17 (playbook 04). Each is one feature step of playbook 07; the next starts
when the previous one's delivery is done. Milestones 5 and 6 are split into steps when they come up.
| # | Outcome | Acceptance tests | Repositories | Issues |
|---|---|---|---|---|
| 1 | A merge during a running delivery is never dropped and needs no person's release; the merge window lives on the bus; a rollout that shares nothing with an open one runs beside it | 8, 9, 11 | mesh-controller | 973 (with the rest of 525) |
| 2 | A ready mark is acted on within seconds, and each pull request has at most one waiting check | 5, 6, 7 | mesh-controller, mesh-catalog (mesh-delivery, gitea) | 974, 975, 976 |
| 3 | A feature with a new seat verb across the controller and the catalogue is delivered as one group: no `after:` line, one composed check, no "release branches" | 1 | mesh-controller, mesh-catalog (mesh-delivery) | 977, 978 |
| 4 | A delivery backs up the data it moves first, and a failed backup holds it and is said | 3 | mesh-controller, mesh-catalog (mesh-delivery, the backup seat's holder) | 980 |
| 5 | One read of a module on a node names the commit, build and rollout that put it there | 10, 12 | mesh-controller, mesh-catalog (mesh-config, nats) | 981 |
| 6 | Builds use installed toolchains with no container, builders fetch over SSH with their own key, and an SDK change ships in one group with its users | 2, 4 | mesh-catalog (build modules, build-agent, ssh-client, gitea), mesh-controller, mesh-sdk | 979, 971 |
The issues are linked `depends-on` in this order; 976 depends on 975, and 979 also on 971.
## What is not decided here
- A web view of deliveries beyond the pull request's comment.
- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge
module's protection tool.
- Moving the per-tier rollout out of the controller (Phase C).
- Whether `mesh/delivery-group` is required on the trunks: the operator's setting, through the forge module's
protection tool.
- How a builder's public SSH key reaches the forge: the forge's and the SSH client's mechanism.
+1 -1
View File
@@ -10,7 +10,7 @@ Before changing anything here, read the playbooks in
amendment, issues, build handoff, constitution sync) is documented there, and agents operate
through them. Thin skills in `.claude/skills/` wrap these playbooks for invocation
(`hq-new-research`, `hq-graduate`, `hq-new-issue`, `hq-diagnose`, `hq-amend-design`,
`hq-handoff`, `hq-sync-constitution`, `hq-status`); each defers to its playbook as
`hq-handoff`, `hq-sync-constitution`, `hq-status`, `hq-cleanup`); each defers to its playbook as
authoritative and adds only the mechanical scaffolding.
## The development cycle