From 96f2ae72c17d5c23821a4fd656a4216dea6185e3 Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 16 Oct 2026 23:39:12 +0200 Subject: [PATCH] Rewrite design 47 from research 055's decisions: hotfix, feature and epic from the trunk, delivery on the bus ADR 0401-0408 graduate research 055: no release branch, two more ordering rules, the composed check as each member's gate, a backup before the send, installed toolchains, events and work queues on the bus, the merge window as keys, and one chain kept on the bus alone. Design 47 is rewritten from them (one design per thing); the glossary takes hotfix, feature, toolchain, milestone and epic and retires release branch; design 18 loses the toolchain image; playbooks 07 and 08 drop the release-branch wait. --- 00-META/glossary.md | 33 +- 00-META/process/07-feature-branches.md | 29 +- 00-META/process/08-small-fixes.md | 4 +- 03-DESIGN/01-to-be/18-building-a-module.md | 5 +- .../47-delivery-from-commit-to-delivered.md | 770 +++++++++--------- 5 files changed, 402 insertions(+), 439 deletions(-) diff --git a/00-META/glossary.md b/00-META/glossary.md index 49487e40..53ff703d 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -219,10 +219,11 @@ every other domain has something to run on. **Recorded by** the controller and t (ADR 0229). - **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its - records (nodes, assignments, rollouts, builds, settings), after each change commits in its store and only + records (nodes, settings, declared config, predefined keys), after each change commits in its store and only when the value differs, with one event per write. The store stays the record; the self-check compares the two and raises `mirror-drifted` on any difference - (ADR 0289). Shortened to *mirror* where the + (ADR 0289, ADR 0356). Builds, delivery plans with their rollouts, and assignments are no mirrors: they are + the chain, kept on the bus as their only source (ADR 0408). Shortened to *mirror* where the text is about the controller's records; the builder's *mirrored* `upstream` image (Change and delivery) is a different thing, a copy of an outside image in the artifact store. @@ -393,7 +394,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences` **Purpose.** To take one commit from its pull request to every node that should run it, and to put it back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the planner, the rollout). **Downstream of** Module, Core, Health and repair and Data. -**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323. +**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323, 0401–0408. **Uses:** module, node, declaration and send (Placement), health (Health and repair), person. - **merge check** — the two statuses a pull request carries before it may merge: the **merge gate** @@ -408,7 +409,9 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat **builder**. Any node may hold the seat, so there is no build node by nature (ADR 0237). *Not:* ~~build machine~~ -- **build queue** — the controller's queue of builds to run; one entry in it is a **build request**. +- **build queue** — the builds waiting to run, kept by the bus as two work queues: delivery builds, which every + builder takes first, and checks, one per pull request, the newest kept (ADR 0406); one entry in it is a + **build request**. The queue's verbs still say *ask* for an entry; the conversation's ask is another thing (see [Homonyms](#homonyms)). - **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by **version**. @@ -450,9 +453,13 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat - **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One level: a group holds deliveries, never groups. Its state is derived from its members, never set. -- **release branch** — a branch other than `main`, shared by name across repositories, that collects a feature's - merged work and is brought into `main` as one delivery (design 57's `feat/57-config-and-provisioning`). While it - is in flight, a change that touches its files waits for it (playbook 07, *Is the coast clear?*). +- **hotfix** — one pull request in one repository, delivered at its merge (ADR 0401). +- **feature** — one branch name across its repositories, whose pull requests are one delivery group, merged and + delivered as one (ADR 0401). Work is never held back on a branch other than a trunk: the trunk is what runs. + *Not:* ~~release branch~~ (hq) +- **toolchain** — a language's compiler and tools, installed by that language's build module on every node holding + the build seat, pinned to one version that is the same on every builder; its version is part of a build's reuse + key (ADR 0405). No longer an image. - **delivery plan** — what a delivery does to the mesh: its **build plan** (the modules it moves and their dependents, in tiers), its **deploy plan** (per node, what it receives and what waits for a person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a @@ -460,15 +467,17 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat *Not:* ~~change plan~~, ~~release plan~~ - **merge window** — the rolling wait in which the controller collects merges before it plans them: each merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest - 10 minutes after its first merge (ADR 0276). + 10 minutes after its first merge (ADR 0276). Kept as two keys with a time to live on the bus; the first to + expire closes the batch (ADR 0407). - **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*, which is a delivery group. - **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat of the same name. It records and decides; the controller sends, judges and rolls back when it is asked. - **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per - repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236, - ADR 0276). One rollout is open at a time; it names every merge it answers. A primitive the delivering stage + repository, tier by tier, to every node that runs each module at once (ADR 0276, ADR 0360). Rollouts that share a + repository, a module or a graph edge run one after another; others run side by side (ADR 0407). It names every + merge it answers, and its record is part of its delivery plan's key on the bus (ADR 0408). A primitive the delivering stage asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll` policy. @@ -732,6 +741,9 @@ Not a domain of the mesh but of the way it is built, listed because its words me ([55 — The knowledge base](../03-DESIGN/01-to-be/55-the-knowledge-base.md)). Not **the record**, the literal reader of this repository. - **design** — a document in `03-DESIGN/`: **as-is** (what runs) or **to-be** (what is being built). +- **milestone** — one step of a design's work breakdown, usable on its own once delivered and named by what the + operator or the mesh can then do; delivered as one feature (playbook 04, ADR 0401). +- **epic** — a run of milestones in one design, released milestone by milestone (ADR 0401). - **issue** — a question about the mesh that someone must answer, numbered and tracked by the issue tracker, which holds it whole: its report, diagnosis and resolution included (ADR 0291); the folders in `04-ISSUES/` are imported there and removed. @@ -784,7 +796,6 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical. | release | a person letting a held delivery go on | **release** | Change and delivery | | | taking a consumer off a licence | **unbind** (verb `release`, to rename) | Identity and access | | | a copy of a work an engine can fetch | **media release** | Media | -| | a branch brought into `main` as one delivery | **release branch** | Change and delivery | | gate | the judgement on a delivery's first node | **first-node gate** | Change and delivery | | | the pull request status | **merge gate** | Change and delivery | | | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement | diff --git a/00-META/process/07-feature-branches.md b/00-META/process/07-feature-branches.md index 5d9dad8d..1e5a09d2 100644 --- a/00-META/process/07-feature-branches.md +++ b/00-META/process/07-feature-branches.md @@ -121,32 +121,10 @@ never into the running one, unless it blocks the running one. report, and the round does not wait for it. What a round asks is only what makes the change wrong against its issues and its decisions. -## Is the coast clear? - -Before the push of phase 3 (and of a small fix, playbook [08](08-small-fixes.md) step 5), the lead checks, for every -repository the change touches, whether a **release branch** is in flight there: a branch other than `main` that -collects a feature's merged work to bring it into `main` as one delivery. - -1. **Look for one.** A pull request into `main` whose head is a release branch (`gitea.gitea_open_pulls`), a - release branch holding merged work that `main` does not have yet (`git log origin/main..origin/`), or - a delivery plan of that release branch still running (`mesh-controller.plans`). -2. **Compare the files.** `git diff --name-only origin/main...origin/` against the files the - change touches. -3. **Overlap: hold.** If the release branch touches a file the change touches, nothing is pushed into that repository - until the release branch's delivery plan is done. Then the feature branch is brought up to the new `main`, the unit - tests run again, and the push follows. The hold is on the wait list (its bound: the release branch's delivery plan), - and is said to the operator with what the change waits for. -4. **No overlap: go on.** A release branch in another repository, or one that shares no file with the change, does not - hold it; its merge joins the merge window as any other (ADR 0276). - -Merging beside a release branch still in flight would make its lead resolve someone else's conflicts in the -middle of it, and could send part of the change with its delivery (2026-10-16: the file module's -controller part touched four files feature 57's release branch also changed). - ## Phase 3 — Check (no review) -1. **Push every feature branch and open one pull request per repository, together, once the coast is clear** - (above). The integration tests +1. **Push every feature branch and open one pull request per repository, together.** + The integration tests run only in the merge check (the operator, 2026-10-16), never before. Pull requests in two or more repositories that share the branch name are one delivery group (ADR 0239 §2). Where a session pushes through the forge tool `gitea_commit` (one commit on the forge's current base), the @@ -202,7 +180,6 @@ on, with when it started and its bound. | the review | 15 minutes a round | ask the reviewer; stop at round 3 | | a merge check | 15 minutes | read the build queue (`mesh-controller.queue`) and the check's log | | the ready mark | 15 minutes after the acceptance is shown | the reviewer marks it, or the lead does | -| a release branch in flight that shares files | its delivery plan done | report it every ten minutes; bring the branch up to `main` after | | a merge after ready | 5 minutes after the checks pass | read `mesh-delivery.show` for the delivery | | a delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` | @@ -270,8 +247,6 @@ issues in `fixing` that nobody works on reads to the operator as stuck work. review rules; `issue-tracker.list` with `flag: abandoned` lists issues left in `fixing` with nobody on them. - A pull request reviewed and green but not marked ready for more than 15 minutes is a lead that did not mark it. -- A pull request opened into a repository whose release branch was in flight and shared a file with it is the failure of - *Is the coast clear?*; the lead's report names the check it made before the push. - A step that adds or changes a module marked ready with no integration test that starts the module whole is the failure of phase 3's fourth step. - A bed run from `.work//mesh-lab` that fails naming a `.work///…` path it cannot find is a diff --git a/00-META/process/08-small-fixes.md b/00-META/process/08-small-fixes.md index 149a2612..f9e35baf 100644 --- a/00-META/process/08-small-fixes.md +++ b/00-META/process/08-small-fixes.md @@ -39,8 +39,8 @@ one delivery**. Everything before the push is local and fast. done-when is opened as an issue of its own, never asked as a change). The reviewer's findings go back to the same agent that wrote the fix, continued by its id, so the fix keeps its context. A playbook or design edit of a few lines may skip the reviewer: the operator's word on the pull request is its review, and the lead marks it ready on that word. -5. **Push once and open the pull request, once the coast is clear** (playbook [07](07-feature-branches.md), *Is - the coast clear?*: no release branch in flight in the repository that shares a file with the fix). The forge's merge +5. **Push once and open the pull request.** The trunk is what runs and nothing holds merged work back + (ADR 0401), so a fix waits for no other change's delivery. The forge's merge check runs the integration tests, the only place they run (the operator, 2026-10-16). A failed check goes back to the same agent with the failure, and the reviewer reads only the change it makes. diff --git a/03-DESIGN/01-to-be/18-building-a-module.md b/03-DESIGN/01-to-be/18-building-a-module.md index a63c64fd..cc139a23 100644 --- a/03-DESIGN/01-to-be/18-building-a-module.md +++ b/03-DESIGN/01-to-be/18-building-a-module.md @@ -5,8 +5,9 @@ code: - mesh-controller cmd/mesh-builder - mesh-controller internal/builder - mesh-catalog modules/build-agent -updated: 2026-10-08 +updated: 2026-10-16 decisions: + - "ADR 0405" - "ADR 0262" - "ADR 0233" - "ADR 0219" @@ -138,7 +139,7 @@ need to run. Third-party software still arrives as an image, because that is how ## What this costs, stated before it is chosen **Every language is permanent.** It needs an SDK — broker client, sealed-credential reading, the -event envelope, tool serving — a toolchain image, and a bundler the builder understands. And the +event envelope, tool serving — a build module that installs its toolchain on every builder (ADR 0405), and a bundler the builder understands. And the spine changes rarely but cascades when it does (ADR 0039): with one language a contract change is one edit; with four it is four that must land together, and a mesh whose SDKs disagree diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index df084f83..187c139b 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -4,69 +4,112 @@ status: in-progress code: [mesh-controller, mesh-catalog modules/mesh-delivery] updated: 2026-10-16 decisions: + - "ADR 0408" + - "ADR 0407" + - "ADR 0406" + - "ADR 0405" + - "ADR 0404" + - "ADR 0403" + - "ADR 0402" + - "ADR 0401" + - "ADR 0398" - "ADR 0395" - "ADR 0382" + - "ADR 0377" + - "ADR 0364" + - "ADR 0362" + - "ADR 0360" - "ADR 0350" - - "ADR 0335" - "ADR 0323" - "ADR 0313" - "ADR 0306" + - "ADR 0304" - "ADR 0299" - - "ADR 0298" - "ADR 0282" - "ADR 0276" - "ADR 0270" + - "ADR 0267" - "ADR 0250" - "ADR 0249" - "ADR 0239" - "ADR 0238" - - "ADR 0236" + - "ADR 0233" + - "ADR 0214" - "ADR 0201" --- # 47 — Delivery: from a commit to delivered -**A delivery is one commit in one repository, from the moment its pull request's head is announced to the -moment every machine of its deploy plan runs it, or to the moment it fails, is superseded or is stopped. A -delivery group is two or more deliveries that share a head branch name, delivered as one unit in order. Both -are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.** -(ADR 0239.) +**A change reaches the machines in one of three ways, all owned by mesh-delivery: a hotfix (one pull request, +delivered at its merge), a feature (one branch name across its repositories, merged and delivered as one +delivery group), or an epic (a run of milestones, each delivered as a feature). The trunk is what runs; there +is no "release branch"** (ADR 0401). -**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a -rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout -at a time. Every delivery the rollout answers ends with it -(ADR 0276). +**Every step starts on an event, and the state lives on the bus.** The forge's events, the controller's +events and the nodes' reports move a delivery; the bus's own rules keep the queues, the merge window and the +order; and one chain, from module to commit to build to delivery plan to assignment, is kept in the bus's +key-value store as its only source (ADR 0406, ADR 0407, ADR 0408). A running delivery never touches merging: +every merge is recorded at once for the next delivery plan of what it touches. This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison of source and artifacts decides, and nothing here changes that. This design says who answers for one -commit's journey and in what order the steps are allowed. +commit's journey, in what order the steps are allowed, and where each fact of it is kept. It was rewritten on +2026-10-16 from the decisions of research 055, which replaced the "release branches", the toolchain image, the +controller's delivery tables and the timers that polled them. ## The parts ``` - forge's holder ──pull.updated / .merged / .closed──► mesh-delivery ◄──checked / plan-moved── controller - (statuses, the view, (deliveries, groups, (planner, gate, - the notes) the table, its state) registration, - ▲ │ │ sending, walk) - └── its tools: note, view, status ◄───────────────┘ └── its verbs: delivery-plan, -order, - -check, deliver, delivery-stop, - delivery-walks + forge (system webhook) ──forge.pull-request.* / merged / status──► the bus ◄──built / checked / plan-moved── controller + │ ▲ (planner, order, + │ │ registration, + ▼ │ sending) + mesh-delivery │ + (deliveries, groups, a key per │ + pull request, the state table) ▼ + node-engine on every node + builders (build seat) ◄── work queues: delivery builds, ready checks, other checks ───────────────────────── (reports; backups first) ``` -- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants, - registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk - rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers - every pull request's head with its check itself, so a check never depends on mesh-delivery. -- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the - group, the state table and their state, and nothing else. -- **The forge's holder** sets a commit's statuses, keeps the delivery's view as one comment on the pull - request, and appends to the commit's note under `refs/notes/mesh-plan`. -- **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350): - a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a - reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build - of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and - says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull - request closes unmerged, and after seven days. +- **mesh-delivery** (the seat `mesh-delivery`) owns hotfixes, features and milestones as deliveries and delivery + groups, decides each step on events, keeps one key per pull request, and asks the controller for each act + and the nodes' backup seats for the backups a delivery needs. It never sends to a machine, writes a + declaration or registers a build (ADR 0239 §25). +- **The controller** keeps planning, ordering, registration, sending and judging. It writes the chain's + links it owns — builds, delivery plans with their rollouts and tiers, assignments — and the merge window's + keys, all on the bus, by compare-and-set, while it holds the lease. +- **The builders**, the holders of the build seat, build and check with toolchains installed on their node, as + the build seat's own account, and take their jobs from the bus's work queues, delivery builds first + (ADR 0405, ADR 0406). +- **The forge module** (`gitea`) publishes every forge event on the bus through the forge's system webhook + (issue 729): pull requests updated, synchronized, labeled, unlabeled, commented, merged and closed, and + commit statuses. It sets statuses, keeps the delivery's view as one comment on the pull request, and appends + to the commit's note under `refs/notes/mesh-plan`. +- **Each node's backup seat** (`node-backup`) backs up the data a delivery plan moves there before the plan + sends anything (ADR 0404). + +## Three ways a change reaches the machines + +*ADR 0401.* + +| Way | What it is | How it is delivered | +|---|---|---| +| **hotfix** | one pull request in one repository | at its merge, as a lone delivery; a lone merge is cut at once (ADR 0282 §2, as ADR 0304 §1 changed it) | +| **feature** | one branch name across its repositories; its pull requests are one delivery group | merged together once the group's composed check passes and every member is marked ready, then delivered in the group's order | +| **epic** | a run of milestones in one design (playbook 04) | each milestone is a feature, delivered when its own group is ready; the next starts when the previous one's delivery plan is done | + +- **The trunk is what runs.** Every delivery is cut from a repository's trunk. A merge into any other branch + starts no delivery, and the controller says so for that merge (issue 889). Holding merged work back, on a + branch beside the trunk or per module, was dropped: from 2026-10-13 23:52 to 2026-10-16 09:32 feature 57's + branches held the controller for 55.7 h, and 28 of 68 other merges touched a held module (research 055, + finding 3). +- **A milestone is functional, its delivery shaped technically.** A milestone is named by what the operator or + the mesh can do once it is delivered; how it ships is shaped by the controller's what-if plan (`plans` for its + repositories and files): the modules it moves and their fan-out. +- **What a merge waits for.** mesh-delivery merges a group's members once the group's composed check passed at + their heads, the independent reviewer marked each member ready (ADR 0364 §1, §2), each member carries the + review ADR 0382 §6 asks, and the operator approved each merge into main (ADR 0398 §3). The operator's Approve is + asked for every merge into main, a hotfix's too (the operator, 2026-10-16 23:40). ## A delivery @@ -75,12 +118,17 @@ Its id is `/@`. It holds: - the pull request (number, base, head branch, description), or none for a commit that reached the trunk without one; - its **delivery plan**: build plan, deploy plan and verdict, as the controller's planner computed them for - its diffset (what ADR 0238 called the "change plan", under the glossary's name); + its diffset (ADR 0238 §2, as ADR 0323 §1 changed it); - its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its **machine steps** while delivering; -- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch - that answers its merge, which may carry a later commit of its repository that contains it (see *The merge - window and the batch*). +- once merged, **the commit it landed on the trunk as**, and the delivery plan that delivers it: the plan of the + batch that answers its merge, which may carry a later commit of its repository that contains it. + +**One key per pull request.** Beside the delivery, mesh-delivery keeps one key per open pull request on the +bus: its head, the trunk commit its current check is asked against, and its verdict, written by +compare-and-set (ADR 0406 §3). It writes a new head when the forge says one, and a new trunk commit only when it +asks the pull request's check again (ADR 0395 §1). A running check watches that key and stops when the head or +the trunk commit in it moves. ## The state table @@ -95,418 +143,346 @@ allows. | checked | a verdict arrived | 1 minute | nothing: it is decided at once | | ready | it passed; it waits to be merged | none | — | | rejected | it failed its check | none | — | -| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started | -| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's | -| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says | +| published | merged on its trunk, recorded in a batch; it waits for its plan to start | 30 minutes | `superseded` when a plan that does not answer its merge took over its plan; `go` when its plan started | +| held | it waits for a person: a failed backup, a merge not ready, a policy of record; it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's | +| delivering | its plan runs: backups, then builds asked and sent tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the plan's key says | | delivered, failed, superseded, stopped, retired, covered | final | — | — | **A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head -at the moment the build seat clones it, the **trunk commit**, and the verdict names both. When another merge -moves the trunk, a `ready` delivery goes back to `proposed` on the observed event `trunk moved`, with both -commits in the transition's reason; a `rejected` one stays `rejected`, its statuses saying the verdict is out -of date. The move itself asks the check again of every active open pull request into that trunk that merges -cleanly (ADR 0395, *A move of the trunk asks every check again* below); a recheck, a new head or the merge tool -asks it too. The row from -`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for -an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by -mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout. +when the builder clones it, and the verdict names both. When another merge moves the trunk, a `ready` +delivery goes back to `proposed` on the event `trunk moved`, and mesh-delivery asks the check of every active, +cleanly merging open pull request into that trunk at once (ADR 0395, *A move of the trunk asks every check +again* below). The row from `proposed` to `checked` takes only a verdict that names the trunk's head. -What this leaves as it is, said so nobody reads more into it: +**What needs no person ends by itself** (ADR 0270): a merge that moves nothing is `delivered` at its merge; a +merge a later delivered merge contains is `covered`, by the forge's word that the later merge commit contains +it; a merge whose plan said its modules unchanged is `delivered` when that plan ends (issue 611); a stop whose +plan ended without building its modules is read as covered once a later merge covers it (issues 474, 480). -- **mesh-delivery's trunk head is the last merge it heard** into that trunk while the delivery was open. A push - to the trunk that is not a merge it hears does not move it; the forge's holder, which reads the branch, and - the merge tool, which compares commits when asked, still catch it. -- **A verdict that names no trunk commit**, from a build seat older than ADR 0323, counts in mesh-delivery as it - did: such a delivery stays `ready`. The forge reads the same verdict out of date once its trunk moves, and the - merge tool refuses it and asks the check. The two agree once the head is judged again. -- **A verdict no build judges** (a change that touches no module, in a repository with no `merge-check.sh` or - not the mesh's) names the base branch's head the forge announced with the head as its trunk commit. -- **In a busy repository the merge tool can refuse, ask, and refuse again**: another merge may move the trunk - while the asked check runs, and the new verdict is then out of date too. Each refusal asks once more; the - merge goes through once a check finishes before the next merge. -- **A check that was asked and never answers** is listed by `stalled` an hour after the ask, and is asked again - with `mesh-delivery.recheck`, which the merge tool's refusal names. +**History is retired, not held** (ADR 0250). A delivery made from the forge's word at its merge, heard fifteen +minutes or more after it, with no plan of it open, goes to `retired` at adoption. **Nothing of a published +delivery is registered before its plan starts.** -The transitions are those of ADR 0239 decision 2, ADR 0270, ADR 0323 and ADR 0395. A delivery that is -merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A -delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by -`stalled` at once. Nothing waits silently. +**The machine steps.** While a delivery is `delivering`, each machine its plan reaches has a row with the +module and the build: `backed-up`, `sent`, `applied` or `failed`, and `silent` when no report came within +ApplySilentAfter (fifteen minutes). They are read from the plan's key, written by the controller from the +nodes' reports, never inferred from time. -**What needs no person ends by itself** (ADR 0270): +## Order without a person -- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and - sends nowhere) is `delivered` as soon as it is merged, whatever its check said; -- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine - running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the - forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge - order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery` - says `covered: a later merge that contains it was delivered` as a success. A module that runs on one - machine only is covered too: its rollout records the empty rest as sent once its first machine passes - (ADR 0270, as corrected 2026-10-12); -- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends, - whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads - that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and - may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such - module with why, "its build source holds none of the changed files", and not in its modules, which keep - meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered, - saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below. - Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's - `TestAMergeWhoseModulesMatchWhatRunsIsDelivered`; -- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and - is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after - its stop` with the covering merge, once, as a success. No state or transition is added (issue 480); -- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes. +*ADR 0239 §11 and §12, ADR 0249, ADR 0402.* -**History is retired, not held** (ADR 0250). -A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout -of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes -to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that -fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout -starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a -build registered before its turn would be carried by somebody else's send. +A delivery group's order is a table of pairs, each with the rule that won (ADR 0249 §3). The controller's +`delivery-order` infers it from the manifests and the module graph; a person writes an `after: ` +line only where the rules cannot see the order. -**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the -module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy -readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the -controller's `plan-moved` event, never inferred from time. +| Rank (highest first) | Rule | Read from | +|---|---|---| +| 1 | *declared*: an `after:` line | the member's description | +| 2 | *built by* and *version skew* | the module graph and the change | +| 2 | *verb before caller*: a member adding a verb to a seat's served verbs before a member adding that verb to an invoked verb | the manifests (ADR 0402) | +| 2 | *SDK before its users*: a member changing `mesh-sdk` before every member whose modules are built on it | the module graph (ADR 0402) | +| 3 | *engine before controller* | module names | +| 4 | the repositories' names | — | + +Rules of one rank in opposite directions reject the group, naming both members, both rules and the line that +would settle it (ADR 0249 §4). + +**The SDK by version.** A Go module names the SDK by its released version and fetches it from the mesh's forge, +checked against its recorded checksum; vendored copies go. Inside a group, a member may name the SDK member's +head, and the group's composed check proves they build and compose together (ADR 0402 §2). + +## Checks + +### One check per pull request, kept by the bus + +*ADR 0323, ADR 0350, ADR 0395, ADR 0406.* + +- **Three work queues, taken in order.** Delivery builds go on a work-queue stream of their own; checks of pull + requests marked ready on a second; every other check on a third. A builder takes from the first, then the + second, then the third. Each check stream has one subject per pull request and keeps only the newest request: + a newer request replaces one still waiting, by the stream's rule. A ready mark publishes its pull request's + check request on the ready stream; a request left behind on the third for a head and trunk commit already + judged or running is acknowledged and dropped when taken, by reading the pull request's key. Nothing pauses the + worker or deletes and republishes a request (issue 558). +- **A running check stops itself.** It watches its pull request's key and stops when the head or the trunk + commit in it moves, without asking the forge (issue 726). An ended check delivers no verdict, so none of its + builds is reused. +- **A failed builder loses nothing.** A job a builder took and never acknowledged goes to another builder + after its wait. +- **A change is built once** (ADR 0350). A check that passes also builds what its merge would move into the + artifact store's scratch namespace under a reuse key, and the plan of the merge commit copies the build of the + same key, named by the check's verdict, instead of building. + +### A move of the trunk asks every check again + +*ADR 0395.* + +- A trunk move asks the check of every active open pull request into that trunk that merges cleanly, against the + new head; a conflict is told once on the pull request, not checked. +- A pull request with no push and no ready mark for 7 days is left out; a push, a ready mark or a recheck asks + its check. +- For a group, a move asks each member's own check, and the composed check runs again through those rechecks; a + move made by a member merging in the group's order asks the other members nothing. +- Its cost is at most one check per active, cleanly merging pull request per trunk move; issue 944 measures it. + +### A delivery group's checks + +*ADR 0239 §14, ADR 0323 §6, ADR 0382, ADR 0403.* + +- **The composed check is each member's gate.** The group's composed check composes every machine with all + members' heads, each merged into its own trunk's head first, judged by the controller built from the group's + controller member when it has one (ADR 0382 §3). Its verdict sets `mesh/delivery-group` and each member's + `mesh/merge-gate`, and names every member it read. No gate runs for a member alone. +- **Each member's own repository check stays required.** Each member's `mesh/repo-check` runs its repository's + suite with the siblings read beside it (ADR 0382 §2), in under a minute (ADR 0391, issue 890). +- **No composed verdict, no gate.** A member's gate is pending while no composed verdict at the members' current + heads exists. A member's new head, close or base move puts the composed verdict out of date, and with it every + member's gate. When mesh-delivery cannot say the groups, the pull request is checked alone and its verdict + says so (ADR 0382 §1). +- **A member merges only with its group passed and reviewed** (ADR 0382 §6), and in the group's order + (ADR 0364 §8). + +## Ready marks and the merge + +*ADR 0364, ADR 0406 §1.* + +The independent reviewer marks a pull request ready at the reviewed head with the label `mesh/ready` and a +comment `ready at `, through `mesh-delivery.mark-ready`. The forge's `labeled` and `commented` events +reach mesh-delivery in seconds; once both required checks passed against the trunk's head, it asks the operator's +Approve for a merge into main, with the pull request, its issues, the reviewer's verdict and the delivery plan, and +merges the marked head through the forge's merge tool on that Approve (ADR 0398 §3). When the trunk moved, the merge tool asks the fresh check, and the event +of its passing verdict asks the merge again; no timer asks it. A push clears the mark (the `synchronized` +event), and a failed check is said once and listed by `stalled`. ## The merge window and the batch -*2026-10-10, by ADR 0276.* +*ADR 0276, ADR 0282, ADR 0299, ADR 0304, ADR 0407.* -- **The window.** The controller puts every merge into a branch a module follows into the open **batch**. - Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by - default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its - merges and its times are kept in the controller's store and survive a restart. -- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for - every commit in it, tiered along the module graph across repositories, with at most one commit per - repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the - earlier merges of a repository that its later commit contains. -- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started - is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new - rollout, which answers every merge the folded one answered. -- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller, - the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for - mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its - rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again - behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10). -- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is - answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends - `covered` at once. -- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for - an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own - commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own - commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose - merge it rollouts again nowhere (below). -- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's - tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is - at its turn. -- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a - folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting - rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once. -- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped: - novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated", - and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as - `plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged - delivery it holds. -- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch - waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout - answers within ten minutes is held with the true reason (the open rollouts of its repository and that none - names its merge, or that the controller holds none) and listed by `stalled` at once. +- **Every merge is recorded at once.** The forge's merged event records a merge into the open batch of what it + touches, or opens one, by whichever controller holds the plans. A merge is never refused, held or dropped, + and needs no person's release (issue 921). The bus's durable consumer keeps every merge event until it is + recorded; no catch-up timer looks for missed merges. +- **The window is two keys.** An open batch's window is a key whose time to live is `merge-window` (90 s), + refreshed by each merge into the batch, beside a key set once when the batch opens whose time to live is + `merge-window-at-most` (10 minutes). The first to expire closes the batch, heard by a watcher; no cutter + polls. A stop that records merges again into the open batch closes its window too (ADR 0299 §3). +- **A lone merge is cut at once**: no batch it shares a repository, module, graph edge or delivery group member + with is assembling (ADR 0282 §2, ADR 0304 §1, ADR 0306). +- **The plan.** A closed batch becomes one delivery plan: every commit in it, tiered along the module graph + across repositories, at most one commit per repository, the latest of its trunk. It answers every merge in the + batch, its own commits and the earlier merges a later commit contains (ADR 0276 §2, as ADR 0299 §7 changed it). +- **A plan fixes its commits when it starts.** A plan starts at its first send, after mesh-delivery's word, and + from then nothing that merges changes it. A plan still waiting for its word is folded into the next batch + (ADR 0276 §4). +- **Plans that share something run one after another.** A batch that shares a repository, a module or a graph + edge with an open plan waits for it, and starts with every merge recorded since when that plan ends. A batch + that shares none runs beside the open ones, so a hotfix is never held by an unrelated feature's delivery. +- **The ends.** A plan done ends every delivery it answers: `delivered` for its own commits, `covered` for an + earlier merge it carried by a later commit. A plan that failed fails its own commits' deliveries; each earlier + merge it answered is then delivered on its own commit, the newest first, until one is delivered (ADR 0276 §3). +- **Shown.** `plans` lists the open batch first, with its window's time left and its merges, and a batch waiting + behind an open plan names that plan; the controller says the batch as `plan-moved` in the state `assembling`. +## A backup before the send -## A stop on a shared rollout +*ADR 0214 §7, ADR 0233, ADR 0404.* -*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.* +When a delivery plan is ready to send, each node it reaches backs up, through its `node-backup` seat, every item +the plan's moved modules keep there whose protection is a backup: an irreplaceable or valuable item, and a +rebuildable one unless it says `backup: none`. A cache is never backed up. A node with no backup holder backs up +nothing, and the plan says so for that node. -One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other -delivery on it. A stop therefore ends only the change it is for: +- **The send waits for every backup.** The controller sends nothing of the plan to any node until every backup + it asked has answered done. +- **A failed backup holds the delivery.** A backup that fails, or does not answer within ten minutes, holds the + delivery with the node, the module and the backup's own words as the reason, and raises a condition for the + operator. A person's `release` asks the backups again, and the send goes on when they pass. +- **What it costs.** About 85% of data modules back up in under 30 s; the slowest measured, postgres on the home + server, about 165 s (research 055, finding 5). A plan moving no backed-up item costs nothing more. + +## The send and the rollout + +*ADR 0360, ADR 0362, ADR 0377, ADR 0406 §4.* + +- **Every machine at once.** A plan's builds go to every machine that runs them in one step, tier by tier; a tier + is asked once the tier before it is built and sent (ADR 0360 §1, §6). No machine goes first and nothing waits + for a health reading. A module of policy `record` that would recreate a container or move data waits for a + person (ADR 0377). +- **Every move is said.** The controller says every move of a plan as `plan-moved`, a move a command makes among + them, so mesh-delivery needs no periodic reconcile; it reads `delivery-walks` once when it starts. +- **A rollout advances on events.** The `built` and sent events of a tier ask the next; the nodes' reports end + the rollout (ADR 0360 §7, with ADR 0362's provisioning report). Each send puts a key whose time to live is + ApplySilentAfter (fifteen minutes); its expiry names the machine silent. There is no rollout ticker. +- **A failed apply is loud and fixed forward** (ADR 0360 §8); the node-engine's witness still puts back a + controller or tool runner that does not come up (ADR 0360 §9). + +## A stop on a shared plan + +*ADR 0299.* + +One plan carries every merge of its batch, so a stop ends only the change it is for: + +- **A group's stop.** When a member after a failed one is stopped and its plan is open, mesh-delivery asks + `delivery-stop` for that plan with the stopped members' merges to leave out. The controller ends the plan and + records every other merge it answered again in the open batch of its kind, whose window the stop closes. +- **What is left out with it.** A later merge of the stopped merge's repository, and a merge that moves a + module a left-out merge moves, would deliver the stopped change; each is left out too, its delivery `stopped`, + naming why. +- **A person's stop.** `stop` of a delivery whose plan carries others first answers what it would take down, and + acts on `way`: `again` (the others delivered again without it) or `walk` (the whole plan stopped). +- **What it does not do.** What the stopped plan already sent stays on the machines it reached; putting it back + is issue 509's. -- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks - `delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass - stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge - it answered again. -- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A - module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too. - Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped - merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is - rolled out again. -- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran - already wait, so an older commit never rollouts after a newer one; only with none open do they form a new - batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop - before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules - where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes - to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every - merge moved are written in one transaction. -- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left - out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery - whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays - `published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every - delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop. -- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped - delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it - stops the whole rollout. -- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it - answers with each of them, which of them contain this merge and end either way, and two ways given with - `way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout - stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is - marked stopped only once its rollout is; a stop the controller refuses leaves it as it was. -- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its - asked builds still register, so a later send may carry them further; the stop names the machines, and - putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the - stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it - can see, and holding a merged change back for good is a revert's job. ## The delivery budget -*Decided 2026-10-10 by ADR 0282. -Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by -slice as each slice below is built; until a slice is built, the text above is what runs.* +*ADR 0282, ADR 0304 §4, ADR 0313, ADR 0360 §12, ADR 0362 §5.* -**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard -from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes; -a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is -shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes -(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers -14.2 (issue 338). +A delivery's **delivery time** runs from its merge until every machine its plan sent has reported the send +applied, and for a provisioned module also placed. A **leaf** delivery's budget is three minutes, a **core** +delivery's ten. A delivery held for a person, a failed backup among them, is shown and held to no budget. Each +phase is kept — the window, the wait behind a plan that shares something, the backups, the build, the send, the +reports — and a delivery over its budget raises `delivery..over-budget`, naming its longest phase. The +**fix time** (ADR 0313) runs from an issue's first move to `fixing` to its last fix delivered, and contains the +delivery time. -*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget -above runs from the merge; the operator's own target is wider, the **fix time**, which runs from an issue's -first move to `fixing` to the moment its last fix is delivered. Each issue's class carries a **fix budget** too -(five minutes leaf, ten core, the delivery budget's own figures), and the tracker splits a fix time into four -phases — implementing, review, checks, and merge to running (the delivery time above) — so a fix over its -budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget. -The delivery budget stays as written: a measure inside the fix budget, not replaced by it. +## Builds without a toolchain image -**What changes in the rollout.** +*ADR 0267, ADR 0350, ADR 0405.* -- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged. -- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One - node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf - build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged. -- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf - module with a long-running resource; one for a leaf module whose every resource is a tools bundle. -- **A tier's first nodes are judged at once.** -- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new - tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before - anything is judged; a step that would carry anything else is refused unsent and said on the gate line. -- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy - report; the plan shows an expected landing time from the last seven days' medians. +- **One build module per language** (Go; Node and TypeScript) installs its toolchain as a declared package on + every node holding the build seat, pinned to one version that is the same on every builder. The toolchain + image and its module retire. +- **Every builder runs the same toolchain, and says so.** Each builder names its toolchain versions with every + build outcome, and the controller's self-check raises a condition while two builders differ. +- **Builds and tests run directly**, as the build seat's own account on its node, in no container. A + repository's `# mesh-check-toolchain` line selects which installed toolchain runs its check. +- **A judge has an account of its own.** A judge built from a sibling's merge (ADR 0382 §3) is built and run as a + second account of the build module, whose home, configuration and cache the build seat's account cannot write, + so a check's script cannot change it. +- **Every dependency comes by the version its lock file names**, checked against its checksum: the mesh's own + code from the mesh's forge, its own packages from its package registry, any other from its public source. +- **A builder fetches from the forge over SSH with a key of its own**, as the build seat's account, under a forge + account for that builder that reads the mesh's repositories and pushes nothing. +- **A build's reuse key** holds the installed toolchain's version and the build module's recipe version in place + of the toolchain image's digest, so a toolchain update rebuilds only the modules of that language. -*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what -a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it: +## One chain, on the bus alone -- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open - rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and - judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open - rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17 - for builds an open rollout owns). -- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the - merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order. -- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or - send, the dependents of what its merges moved included; a change whose modules are not known yet is related to - every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node. -- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a - warning naming the rollout and the core rollouts that set it aside. -- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an - unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its - own builds. -- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by - issue 528. +*ADR 0408.* -*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out -of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch; -overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every -open judgement on its node. +Module, commit, build, published package or artifact, delivery plan and assignment are one chain, each link +answerable from the next. -*A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`, -a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge -into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace -such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once -it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy -container there, its first-node gate starts at the report that says the container replaced, and the wait is a -phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget -is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a -rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the -node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the -judgement. +| Link | Names | Written by | Bucket | +|---|---|---|---| +| build | its module, repository and commit, its reuse key, what it published: an artifact by digest or a package release by name and version | the controller | one key per build | +| delivery plan | the commit it fixed per repository, the builds it carried, its rollout and tiers, the machine steps, and every assignment it changed with the build before and after | the controller | one key per plan | +| assignment | the module and node, the build, artifact or package and commit it runs, and the plan that put it there | the controller | one key per module on a node | +| delivery, group | as *A delivery* says | mesh-delivery | `deliveries`, `groups` | +| pull request | head, trunk commit asked against, verdict | mesh-delivery | one key per open pull request | +| batch | its merges, and the window's two keys | the controller | one key per batch, two per window | -**Who builds what.** - -| Part | Owner | What it adds | -|---|---|---| -| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration | -| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` | -| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery..over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` | -| the lone merge | mesh-controller | the cut at once when nothing waits to join | -| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch | -| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once | -| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` | - -**Slices**, each one pull request per repository, each with its acceptance sentence: - -| Slice | Delivers | Done when | -|---|---|---| -| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase | -| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases | -| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout | -| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other | -| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median | - -Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges -only once it is built. - -## A delivery group - -- **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull - requests. Two or more make a group. A group that has started delivering takes no new member. -- **Order**: `after: ` lines in a member's description, and the controller's `delivery-order` - over the graph (ADR 0239 decision 4). The plan shows the order and why each pair is ordered: *declared*, - *built by*, *version skew*, *engine before controller*, *by name*. -- **Precedence** (ADR 0249): when rules order one pair both ways, the higher one decides. *Declared* comes - first, then *built by* and *version skew*, then *engine before controller*. The pair shows the rule that - won and the rules it won over. Rules of one rank both ways are a contradiction. The group is `rejected`, - and the reason names both members, both rules and the `after:` line that would settle it. -- **Check**: the members' heads composed together, one gate over every machine with all their definitions, - each head merged into its own trunk's head first and named with that trunk commit (ADR 0323). It is asked - by mesh-delivery through `delivery-check` when the group forms or a member's head moves, and posted on - every member's head as `mesh/delivery-group`. A member rebased or pushed is a new head with its own check: - it takes its old head's place in the group, even once the group is delivering, and the composed check runs - again for the new heads instead of holding the group. A merge into a member's trunk that is not one of the - group's members puts the composed check out of date: the group reads `checking` with the reason, and the - composed check runs again once a member is judged again. Checked by mesh-delivery's - `TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember` and `TestARebasedMemberIsComposedAgainNotHeld`. -- **Each member's own check reads its siblings** (ADR 0382): the group's other open members into a base of - the same name are read beside a member's own check at their head merged onto their base, and a mesh-controller - sibling's merge judges it, so changes that are only consistent together pass their own checks. The controller - learns the siblings from mesh-delivery's `groups`; the verdict names each one. A sibling's new head, close or - base move puts that verdict out of date, and the merge tool merges such a member only when - `mesh/delivery-group` passed at the heads read and every sibling head read is reviewed. -- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`. - A member published while its group's composed check is asked and not answered waits there for the answer; - one held because that check had not passed is *lifted* back to `published` once it passes or warns for the - heads that merged and its own rollout is still open (ADR 0270). -- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its - rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered. - Its own builds were put back at its gate. - -A group's state is derived from its members and its composed check, and is shown, never stored apart from -them. - -## A move of the trunk asks every check again - -*ADR 0395, changing ADR 0323 §4.* When the forge's holder hears a repository's trunk move, mesh-delivery asks the -check of every active open pull request into that trunk against its new head at once, so a verdict is current by -the time anyone looks. - -- **A conflict is told, not checked.** Before asking a check, mesh-delivery asks the forge whether the head merges - cleanly into the new head, which needs no builder. When it does not, no check is asked: its statuses fail naming the trunk commit and the files in - conflict where the forge gives them, and it is told once on the pull request, not again while the same conflict - stands. Every later trunk move and its next push ask the forge again; once it merges cleanly its check is asked. -- **One check per pull request.** When a check against a newer trunk commit is asked for a pull request, its check - against an older trunk commit is dropped if it waits and ended if it runs, said in the build log and on the - queue, as a new head already ends its old head's check (issue 726). An ended check delivers no verdict, so none of its builds is reused (ADR 0350 §7 keeps only builds a verdict - names); its scratch records wait for ADR 0350 §5's seven-day sweep. A try and a delivery's build are never touched. -- **The queue's order.** Delivery builds first; then checks of pull requests marked ready; then the rest, the most - recently pushed first. -- **Inactive pull requests are left out.** One with no push and no ready mark for 7 days is not asked again on a - trunk move; its statuses say out of date and that a push, a ready mark or a recheck asks its check. A recheck - takes a delivery that is still proposed and whose verdict is out of date, and asks its check (issue 888). -- **Groups.** For a group's members a move asks only each member's own check; the group's composed check runs again - through those rechecks (ADR 0323 §6) and is never asked by the move itself. A move made by a member merging in - the group's order asks the other members nothing, since they merge moments later; their verdicts are out of date - until then, and one that does not merge is asked again at the next move, its next push or a recheck. - -**Its cost** is at most one check asked per active, cleanly merging pull request per trunk move: about 550 a day at -ADR 0323's measured load, fewer once conflicting and inactive ones are left out; ending an older check saves builder -time, not checks asked. Issue 944 measures it once built. +- **The bus is the only source.** Each bucket is written by its owner by compare-and-set, and the controller + reads its own buckets to decide. No table of the controller's store holds builds, rollouts, batches or + assignments, so no record mirror, audit or healer exists for them; the nodes, settings, declared-config and + predefined-keys mirrors stay as ADR 0289 and ADR 0356 have them. +- **What ended stays answerable.** A key of an ended plan or build stays while another link names it; each + bucket keeps up to 64 earlier revisions of each key as its history. A value is never cut to a summary and stays under the + bus's largest message. The bus's storage is declared as data and backed up like any data module's. +- **One read answers each question.** For any module on any node, one read of its assignment names the commit, + the build, the artifact or package and the plan that put it there; one read of a plan lists every assignment + it changed. +- **The tables go by a numbered migration**: their rows are written once into the buckets, then dropped from the + controller's store and its consolidated schema. +- **The artifact store and the package registry keep the bytes**; the chain names them (ADR 0251). ## What every transition does In this order, each by its owner: -1. **kept**: mesh-delivery puts the delivery in its state (`deliveries`, one key per delivery; `groups`, one - key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried, later - each time, from ten seconds doubling to an hour. The forge's answer that what it is for does not exist ends - it at once, and sixty failed tries end it too, each said among the owner's refusals (ADR 0250); +1. **kept**: mesh-delivery puts the delivery in its state by compare-and-set before anything else; what it owes + outside is kept with it and retried from ten seconds doubling to an hour, at most sixty times (ADR 0250); 2. **said**: the event `mesh-delivery.transition` (and `mesh-delivery.group` when a group's derived state - changes), with the delivery's id, from, to, event, why and plan summary, and no secret or address; -3. **noted**: mesh-delivery asks the forge's holder (`gitea_note_append`) to append a line to the commit's - note: the head while the delivery is off the trunk, the commit it landed as once it is on it, and at its - end a line of what was executed there. The forge's holder writes it in its own repository as the forge's - own account, and a line already there is not appended again; -4. **reflected** (not for history, whose pull request the forge may no longer hold): the delivery's view, one comment on the pull request with the state, the plan, the machine - steps and the group, kept current (`gitea_delivery_view`); the status `mesh/delivery` on the head, and - `mesh/delivery-group` on every member's head (`gitea_commit_status`). Every status links to the pull - request, `mesh/merge-gate` among them. + changes), with no secret or address; +3. **noted**: a line on the commit's note, appended by the forge module as the forge's own account; +4. **reflected**: the delivery's view on the pull request, `mesh/delivery` on the head and `mesh/delivery-group` + on every member's head. + +A doubled step is refused by compare-and-set and by the bus's message de-duplication, which also replaces the +forge module's files of what it announced (ADR 0406 §5). ## The verbs -**mesh-delivery's seat** (served by its holder; the first five read): +**mesh-delivery's seat**: | Verb | Answers or does | |---|---| -| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group | -| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout | +| `deliveries` | every delivery not final, the final ones of the last day, and every one `stalled` still lists | +| `show` | one delivery or group whole: plan, transitions, machine steps, order, the chain's links | | `groups` | every group with its members in order and its derived state | -| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere | +| `what-if` | the delivery plan a diffset would have, asked of the controller's planner | | `table` | the state table and the machine-step table, with bounds and H2's repairs | -| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds | -| `recheck` | a rejected or ready delivery back to proposed, with why; a delivery still proposed whose verdict is out of date has its check asked (ADR 0395) | -| `release` | a held delivery to delivering, with why: a person's word | -| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) | -| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise | -| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing | +| `stalled` | every delivery held past its bound, with the transition H2 may take | +| `mark-ready` | a reviewer's mark at a head: the label and the `ready at` comment, by the mesh's own forge account | +| `recheck` | a check asked again for a delivery whose verdict is out of date (ADR 0395 §6) | +| `release` | a held delivery goes on, with why; for a failed backup it asks the backups again | +| `stop` | a delivery to stopped, with why; on a shared plan it first answers what it would take down (ADR 0299) | +| `close` | H2's only verb: the transition the table names for a stalled delivery | +| `retire-history` | held history to retired (ADR 0250) | +| `times` | each class's budget, the week's median and worst delivery time, and the deliveries over budget | -**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`, -`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which -takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299), -`delivery-walks`. A person has `plans go --why`, which does what `deliver` does under their name and -is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command -too. - -**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It -also announces `pull.closed`, and a merge's head and the statuses the forge holds on it. - -## The store - -Key-value state on the bus, declared by the module (ADR 0201): `deliveries` and `groups`. One writer, the -seat's holder, whose code changes state from a single goroutine. A final delivery is kept for thirty days and -then removed. Its note on the commit keeps its record after that. The bus's own snapshot (ADR 0235) backs it -up. +**The controller's verbs for it**: `delivery-plan`, `delivery-order`, `delivery-check`, `deliver`, +`delivery-stop`, `delivery-walks`, and `plans` (the open batch, the open plans, and a what-if for a repository +and its files). A person has `plans go --why`. ## The bootstrap -- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the - controller's own, started by the merge and witnessed on the machine. mesh-delivery records it. -- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the - controller starts it as it always did. -- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises - `plan..waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and - `plans go --why` as the way on. - A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone. +- A plan that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the + controller's own, started by the merge; mesh-delivery records it (ADR 0239 §31). +- Any other plan waits for `deliver` while the seat has a holder on record; with none, the controller starts it. +- A replaced controller or mesh-delivery carries on from the bus: the keys, the queues and the durable + consumers hold every step, so nothing is lost or repeated (ADR 0406 §4). -## The switch +## Acceptance tests -1. The controller's change (the seat, its verbs, `plan-moved`, the wait for a holder's word, `plans go`) is - merged and rolls out as the core does. With no holder on record it behaves exactly as before. -2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built - and published. -3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start - from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in - `delivering`. -4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass. +What the build proves before this design is implemented (research 055's direction): -## Phases +1. A feature across the controller and the catalogue with a new seat verb is delivered as one group, with no + `after:` line, no two steps and one composed check. +2. An SDK change and a Go module using it are delivered as one group. +3. A data module is backed up before a delivery moves it, and a failed backup holds the delivery. +4. A build on any builder uses the installed toolchain with no container, and a toolchain update rebuilds only + modules of that language when its version changes. +5. Two merges into one repository within a minute leave each open pull request with one queued check, by the + stream's rule. +6. A ready mark is acted on within seconds of its event. +7. A controller replaced in the middle of a rollout resumes it with no step lost or repeated. +8. The merge window closes by its key's expiry. +9. A hotfix to a module the running feature's delivery does not touch reaches the machines in minutes, beside it. +10. For any module on any node, one read of its assignment names the commit, build, artifact or package and the + delivery plan that put it there, and one read of that plan lists every assignment it changed. +11. Two merges into one repository while a delivery of that repository runs are both delivered by the next plan, + with no person's release and nothing held in mesh-delivery. +12. No table in the controller's store holds builds, rollouts or assignments. -| Phase | Delivers | Done when | -|---|---|---| -| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` | -| B — the conditions | S16, the controller's own `plan..waiting`; probe D14 reads `stalled` and raises `delivery..stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 | -| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while | +## What retires + +- The "release branches" and the controller's judge of them (ADR 0358, superseded by ADR 0401). +- The toolchain image and its module (ADR 0405). +- The window cutter, the merge catch-up timer, the rollout ticker, mesh-delivery's tick, reconcile and + ready-mark poll, the forge's poll as a trigger, the queue reordering, the forge-asking cancel, and the forge + module's announcement files (ADR 0406, ADR 0407). +- The controller's tables of builds, build requests, rollouts and their tiers, batches and merges, and + assignments; the record mirrors, probe D17's audit and healer H6 for those kinds (ADR 0408). + +## Milestones + +Set by playbook 04 at the handoff, after the operator confirms them. ## What is not decided here - A web view of deliveries beyond the pull request's comment. -- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge - module's protection tool. -- Moving the per-tier rollout out of the controller (Phase C). +- Whether `mesh/delivery-group` is required on the trunks: the operator's setting, through the forge module's + protection tool. +- How a builder's public SSH key reaches the forge: the forge's and the SSH client's mechanism.