diff --git a/00-META/glossary.md b/00-META/glossary.md index 49487e40..8363b63d 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -219,10 +219,11 @@ every other domain has something to run on. **Recorded by** the controller and t (ADR 0229). - **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its - records (nodes, assignments, rollouts, builds, settings), after each change commits in its store and only + records (nodes, settings, declared config, predefined keys), after each change commits in its store and only when the value differs, with one event per write. The store stays the record; the self-check compares the two and raises `mirror-drifted` on any difference - (ADR 0289). Shortened to *mirror* where the + (ADR 0289, ADR 0356). Builds, rollouts and assignments are no mirrors: they are + the chain, kept on the bus as their only source (ADR 0408). Shortened to *mirror* where the text is about the controller's records; the builder's *mirrored* `upstream` image (Change and delivery) is a different thing, a copy of an outside image in the artifact store. @@ -393,7 +394,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences` **Purpose.** To take one commit from its pull request to every node that should run it, and to put it back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the planner, the rollout). **Downstream of** Module, Core, Health and repair and Data. -**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323. +**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323, 0401–0408. **Uses:** module, node, declaration and send (Placement), health (Health and repair), person. - **merge check** — the two statuses a pull request carries before it may merge: the **merge gate** @@ -408,7 +409,10 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat **builder**. Any node may hold the seat, so there is no build node by nature (ADR 0237). *Not:* ~~build machine~~ -- **build queue** — the controller's queue of builds to run; one entry in it is a **build request**. +- **build queue** — the builds waiting to run, kept by the bus as three work queues a builder takes in order: + delivery builds, then checks of pull requests marked ready, then every other check; each check queue keeps one + request per pull request, the newest (ADR 0406); one entry in it is a + **build request**. The queue's verbs still say *ask* for an entry; the conversation's ask is another thing (see [Homonyms](#homonyms)). - **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by **version**. @@ -450,9 +454,14 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat - **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One level: a group holds deliveries, never groups. Its state is derived from its members, never set. -- **release branch** — a branch other than `main`, shared by name across repositories, that collects a feature's - merged work and is brought into `main` as one delivery (design 57's `feat/57-config-and-provisioning`). While it - is in flight, a change that touches its files waits for it (playbook 07, *Is the coast clear?*). +- **hotfix** — one pull request in one repository, delivered at its merge (ADR 0401). +- **feature** — one branch name across its repositories: in two or more, its pull requests are one delivery group, + merged together and delivered member by member in the group's order; in one, a single pull request delivered as a + lone delivery (ADR 0401). Work is never held back on a branch other than a trunk: the trunk is what runs. + *Not:* ~~release branch~~ (hq) +- **toolchain** — a language's compiler and tools, installed by that language's build module on every node holding + the build seat, pinned to one version that is the same on every builder; its version is part of a build's reuse + key (ADR 0405). No longer an image. - **delivery plan** — what a delivery does to the mesh: its **build plan** (the modules it moves and their dependents, in tiers), its **deploy plan** (per node, what it receives and what waits for a person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a @@ -460,15 +469,17 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat *Not:* ~~change plan~~, ~~release plan~~ - **merge window** — the rolling wait in which the controller collects merges before it plans them: each merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest - 10 minutes after its first merge (ADR 0276). + 10 minutes after its first merge (ADR 0276). Kept as two keys with a time to live on the bus; the first to + expire closes the batch (ADR 0407). - **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*, which is a delivery group. - **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat of the same name. It records and decides; the controller sends, judges and rolls back when it is asked. - **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per - repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236, - ADR 0276). One rollout is open at a time; it names every merge it answers. A primitive the delivering stage + repository, tier by tier, to every node that runs each module at once (ADR 0276, ADR 0360). Rollouts that share a + repository, a module or a graph edge run one after another; others run side by side (ADR 0407). It names every + merge it answers, and its record is one key of the bus's rollouts bucket, a link of the chain (ADR 0408). A primitive the delivering stage asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll` policy. @@ -732,6 +743,9 @@ Not a domain of the mesh but of the way it is built, listed because its words me ([55 — The knowledge base](../03-DESIGN/01-to-be/55-the-knowledge-base.md)). Not **the record**, the literal reader of this repository. - **design** — a document in `03-DESIGN/`: **as-is** (what runs) or **to-be** (what is being built). +- **milestone** — one step of a design's work breakdown, usable on its own once delivered and named by what the + operator or the mesh can then do; delivered as one feature (playbook 04, ADR 0401). +- **epic** — a run of milestones in one design, released milestone by milestone (ADR 0401). - **issue** — a question about the mesh that someone must answer, numbered and tracked by the issue tracker, which holds it whole: its report, diagnosis and resolution included (ADR 0291); the folders in `04-ISSUES/` are imported there and removed. @@ -784,7 +798,6 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical. | release | a person letting a held delivery go on | **release** | Change and delivery | | | taking a consumer off a licence | **unbind** (verb `release`, to rename) | Identity and access | | | a copy of a work an engine can fetch | **media release** | Media | -| | a branch brought into `main` as one delivery | **release branch** | Change and delivery | | gate | the judgement on a delivery's first node | **first-node gate** | Change and delivery | | | the pull request status | **merge gate** | Change and delivery | | | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement | diff --git a/00-META/process/07-feature-branches.md b/00-META/process/07-feature-branches.md index 5d9dad8d..1e5a09d2 100644 --- a/00-META/process/07-feature-branches.md +++ b/00-META/process/07-feature-branches.md @@ -121,32 +121,10 @@ never into the running one, unless it blocks the running one. report, and the round does not wait for it. What a round asks is only what makes the change wrong against its issues and its decisions. -## Is the coast clear? - -Before the push of phase 3 (and of a small fix, playbook [08](08-small-fixes.md) step 5), the lead checks, for every -repository the change touches, whether a **release branch** is in flight there: a branch other than `main` that -collects a feature's merged work to bring it into `main` as one delivery. - -1. **Look for one.** A pull request into `main` whose head is a release branch (`gitea.gitea_open_pulls`), a - release branch holding merged work that `main` does not have yet (`git log origin/main..origin/`), or - a delivery plan of that release branch still running (`mesh-controller.plans`). -2. **Compare the files.** `git diff --name-only origin/main...origin/` against the files the - change touches. -3. **Overlap: hold.** If the release branch touches a file the change touches, nothing is pushed into that repository - until the release branch's delivery plan is done. Then the feature branch is brought up to the new `main`, the unit - tests run again, and the push follows. The hold is on the wait list (its bound: the release branch's delivery plan), - and is said to the operator with what the change waits for. -4. **No overlap: go on.** A release branch in another repository, or one that shares no file with the change, does not - hold it; its merge joins the merge window as any other (ADR 0276). - -Merging beside a release branch still in flight would make its lead resolve someone else's conflicts in the -middle of it, and could send part of the change with its delivery (2026-10-16: the file module's -controller part touched four files feature 57's release branch also changed). - ## Phase 3 — Check (no review) -1. **Push every feature branch and open one pull request per repository, together, once the coast is clear** - (above). The integration tests +1. **Push every feature branch and open one pull request per repository, together.** + The integration tests run only in the merge check (the operator, 2026-10-16), never before. Pull requests in two or more repositories that share the branch name are one delivery group (ADR 0239 §2). Where a session pushes through the forge tool `gitea_commit` (one commit on the forge's current base), the @@ -202,7 +180,6 @@ on, with when it started and its bound. | the review | 15 minutes a round | ask the reviewer; stop at round 3 | | a merge check | 15 minutes | read the build queue (`mesh-controller.queue`) and the check's log | | the ready mark | 15 minutes after the acceptance is shown | the reviewer marks it, or the lead does | -| a release branch in flight that shares files | its delivery plan done | report it every ten minutes; bring the branch up to `main` after | | a merge after ready | 5 minutes after the checks pass | read `mesh-delivery.show` for the delivery | | a delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` | @@ -270,8 +247,6 @@ issues in `fixing` that nobody works on reads to the operator as stuck work. review rules; `issue-tracker.list` with `flag: abandoned` lists issues left in `fixing` with nobody on them. - A pull request reviewed and green but not marked ready for more than 15 minutes is a lead that did not mark it. -- A pull request opened into a repository whose release branch was in flight and shared a file with it is the failure of - *Is the coast clear?*; the lead's report names the check it made before the push. - A step that adds or changes a module marked ready with no integration test that starts the module whole is the failure of phase 3's fourth step. - A bed run from `.work//mesh-lab` that fails naming a `.work///…` path it cannot find is a diff --git a/00-META/process/08-small-fixes.md b/00-META/process/08-small-fixes.md index 149a2612..f9e35baf 100644 --- a/00-META/process/08-small-fixes.md +++ b/00-META/process/08-small-fixes.md @@ -39,8 +39,8 @@ one delivery**. Everything before the push is local and fast. done-when is opened as an issue of its own, never asked as a change). The reviewer's findings go back to the same agent that wrote the fix, continued by its id, so the fix keeps its context. A playbook or design edit of a few lines may skip the reviewer: the operator's word on the pull request is its review, and the lead marks it ready on that word. -5. **Push once and open the pull request, once the coast is clear** (playbook [07](07-feature-branches.md), *Is - the coast clear?*: no release branch in flight in the repository that shares a file with the fix). The forge's merge +5. **Push once and open the pull request.** The trunk is what runs and nothing holds merged work back + (ADR 0401), so a fix waits for no other change's delivery. The forge's merge check runs the integration tests, the only place they run (the operator, 2026-10-16). A failed check goes back to the same agent with the failure, and the reviewer reads only the change it makes. diff --git a/03-DESIGN/01-to-be/18-building-a-module.md b/03-DESIGN/01-to-be/18-building-a-module.md index a63c64fd..cc139a23 100644 --- a/03-DESIGN/01-to-be/18-building-a-module.md +++ b/03-DESIGN/01-to-be/18-building-a-module.md @@ -5,8 +5,9 @@ code: - mesh-controller cmd/mesh-builder - mesh-controller internal/builder - mesh-catalog modules/build-agent -updated: 2026-10-08 +updated: 2026-10-16 decisions: + - "ADR 0405" - "ADR 0262" - "ADR 0233" - "ADR 0219" @@ -138,7 +139,7 @@ need to run. Third-party software still arrives as an image, because that is how ## What this costs, stated before it is chosen **Every language is permanent.** It needs an SDK — broker client, sealed-credential reading, the -event envelope, tool serving — a toolchain image, and a bundler the builder understands. And the +event envelope, tool serving — a build module that installs its toolchain on every builder (ADR 0405), and a bundler the builder understands. And the spine changes rarely but cascades when it does (ADR 0039): with one language a contract change is one edit; with four it is four that must land together, and a mesh whose SDKs disagree diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index df084f83..7af5b4cd 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -4,69 +4,112 @@ status: in-progress code: [mesh-controller, mesh-catalog modules/mesh-delivery] updated: 2026-10-16 decisions: + - "ADR 0408" + - "ADR 0407" + - "ADR 0406" + - "ADR 0405" + - "ADR 0404" + - "ADR 0403" + - "ADR 0402" + - "ADR 0401" + - "ADR 0398" - "ADR 0395" - "ADR 0382" + - "ADR 0377" + - "ADR 0364" + - "ADR 0362" + - "ADR 0360" - "ADR 0350" - - "ADR 0335" - "ADR 0323" - "ADR 0313" - "ADR 0306" + - "ADR 0304" - "ADR 0299" - - "ADR 0298" - "ADR 0282" - "ADR 0276" - "ADR 0270" + - "ADR 0267" - "ADR 0250" - "ADR 0249" - "ADR 0239" - "ADR 0238" - - "ADR 0236" + - "ADR 0233" + - "ADR 0214" - "ADR 0201" --- # 47 — Delivery: from a commit to delivered -**A delivery is one commit in one repository, from the moment its pull request's head is announced to the -moment every machine of its deploy plan runs it, or to the moment it fails, is superseded or is stopped. A -delivery group is two or more deliveries that share a head branch name, delivered as one unit in order. Both -are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.** -(ADR 0239.) +**A change reaches the machines in one of three ways, all owned by mesh-delivery: a hotfix (one pull request, +delivered at its merge), a feature (one branch name across its repositories: in two or more, one delivery +group merged together and delivered member by member in its order; in one, a lone delivery), or an epic (a run of milestones, each delivered as a feature). The trunk is what runs; there +is no "release branch"** (ADR 0401). -**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a -rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout -at a time. Every delivery the rollout answers ends with it -(ADR 0276). +**Every step starts on an event, and the state lives on the bus.** The forge's events, the controller's +events and the nodes' reports move a delivery; the bus's own rules keep the queues, the merge window and the +order; and one chain, from module to commit to build to rollout to assignment, is kept in the bus's +key-value store as its only source (ADR 0406, ADR 0407, ADR 0408). A running delivery never touches merging: +every merge is recorded at once for the next delivery plan of what it touches. This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison of source and artifacts decides, and nothing here changes that. This design says who answers for one -commit's journey and in what order the steps are allowed. +commit's journey, in what order the steps are allowed, and where each fact of it is kept. It was rewritten on +2026-10-16 from the decisions of research 055, which replaced the "release branches", the toolchain image, the +controller's delivery tables and the timers that polled them. ## The parts ``` - forge's holder ──pull.updated / .merged / .closed──► mesh-delivery ◄──checked / plan-moved── controller - (statuses, the view, (deliveries, groups, (planner, gate, - the notes) the table, its state) registration, - ▲ │ │ sending, walk) - └── its tools: note, view, status ◄───────────────┘ └── its verbs: delivery-plan, -order, - -check, deliver, delivery-stop, - delivery-walks + forge (system webhook) ──forge.pull-request.* / merged / status──► the bus ◄──built / checked / plan-moved── controller + │ ▲ (planner, order, + │ │ registration, + ▼ │ sending) + mesh-delivery │ + (deliveries, groups, a key per │ + pull request, the state table) ▼ + node-engine on every node + builders (build seat) ◄── work queues: delivery builds, ready checks, other checks ───────────────────────── (reports; backups first) ``` -- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants, - registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk - rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers - every pull request's head with its check itself, so a check never depends on mesh-delivery. -- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the - group, the state table and their state, and nothing else. -- **The forge's holder** sets a commit's statuses, keeps the delivery's view as one comment on the pull - request, and appends to the commit's note under `refs/notes/mesh-plan`. -- **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350): - a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a - reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build - of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and - says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull - request closes unmerged, and after seven days. +- **mesh-delivery** (the seat `mesh-delivery`) owns hotfixes, features and milestones as deliveries and delivery + groups, decides each step on events, keeps one key per pull request, and asks the controller for each act + and the nodes' backup seats for the backups a delivery needs. It never sends to a machine, writes a + declaration or registers a build (ADR 0239 §25). +- **The controller** keeps planning, ordering, registration, sending and judging. It writes the chain's + links it owns — builds, rollouts with their tiers, assignments — and the merge window's + keys, all on the bus, by compare-and-set, while it holds the lease. +- **The builders**, the holders of the build seat, build and check with toolchains installed on their node, as + the build seat's own account, and take their jobs from the bus's work queues, delivery builds first + (ADR 0405, ADR 0406). +- **The forge module** (`gitea`) publishes every forge event on the bus through the forge's system webhook + (issue 729): pull requests updated, synchronized, labeled, unlabeled, commented, merged and closed, and + commit statuses. It sets statuses, keeps the delivery's view as one comment on the pull request, and appends + to the commit's note under `refs/notes/mesh-plan`. +- **Each node's backup seat** (`node-backup`) backs up the data a delivery plan moves there before the plan + sends anything (ADR 0404). + +## Three ways a change reaches the machines + +*ADR 0401.* + +| Way | What it is | How it is delivered | +|---|---|---| +| **hotfix** | one pull request in one repository | at its merge, as a lone delivery; a lone merge is cut at once (ADR 0282 §2, as ADR 0304 §1 changed it) | +| **feature** | one branch name across its repositories; in two or more, its pull requests are one delivery group; in one, a single pull request | a group is merged together once its composed check passes and every member is marked, reviewed and approved, then delivered member by member in its order (ADR 0239 §16); in one repository, as a lone delivery like a hotfix | +| **epic** | a run of milestones in one design (playbook 04) | each milestone is a feature, delivered when its own group is ready; the next starts when the previous one's delivery plan is done | + +- **The trunk is what runs.** Every delivery is cut from a repository's trunk. A merge into any other branch + starts no delivery, and the controller says so for that merge (issue 889). Holding merged work back, on a + branch beside the trunk or per module, was dropped: from 2026-10-13 23:52 to 2026-10-16 09:32 feature 57's + branches held the controller for 55.7 h, and 28 of 68 other merges touched a held module (research 055, + finding 3). +- **A milestone is functional, its delivery shaped technically.** A milestone is named by what the operator or + the mesh can do once it is delivered; how it ships is shaped by the delivery plan its changes would have, asked of + the controller's planner through mesh-delivery's `what-if` (ADR 0239 §3): the modules it moves and their fan-out. +- **What a merge waits for.** mesh-delivery merges a group's members once the group's composed check passed at + their heads, the independent reviewer marked each member ready (ADR 0364 §1, §2), each member carries the + review ADR 0382 §6 asks, and the operator approved each merge into main (ADR 0398 §3). The operator's Approve is + asked for every merge into main, a hotfix's too (the operator, 2026-10-16 23:40). ## A delivery @@ -75,12 +118,17 @@ Its id is `/@`. It holds: - the pull request (number, base, head branch, description), or none for a commit that reached the trunk without one; - its **delivery plan**: build plan, deploy plan and verdict, as the controller's planner computed them for - its diffset (what ADR 0238 called the "change plan", under the glossary's name); + its diffset (ADR 0238 §2, as ADR 0323 §1 changed it); - its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its **machine steps** while delivering; -- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch - that answers its merge, which may carry a later commit of its repository that contains it (see *The merge - window and the batch*). +- once merged, **the commit it landed on the trunk as**, and the delivery plan that delivers it: the plan of the + batch that answers its merge, which may carry a later commit of its repository that contains it. + +**One key per pull request.** Beside the delivery, mesh-delivery keeps one key per open pull request on the +bus: its head, the trunk commit its current check is asked against, and its verdict, written by +compare-and-set (ADR 0406 §4). It writes a new head when the forge says one, and a new trunk commit only when it +asks the pull request's check again (ADR 0395 §1). A running check watches that key and stops when the head or +the trunk commit in it moves. ## The state table @@ -95,418 +143,353 @@ allows. | checked | a verdict arrived | 1 minute | nothing: it is decided at once | | ready | it passed; it waits to be merged | none | — | | rejected | it failed its check | none | — | -| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started | -| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's | -| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says | +| published | merged on its trunk, recorded in a batch; it waits for its plan to start | 30 minutes | `superseded` when a plan that does not answer its merge took over its plan; `go` when its plan started | +| held | it waits for a person: a failed backup (ADR 0404 §4), a policy of record; it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's | +| delivering | its plan runs: backups, then builds asked and sent tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the plan's key says | | delivered, failed, superseded, stopped, retired, covered | final | — | — | **A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head -at the moment the build seat clones it, the **trunk commit**, and the verdict names both. When another merge -moves the trunk, a `ready` delivery goes back to `proposed` on the observed event `trunk moved`, with both -commits in the transition's reason; a `rejected` one stays `rejected`, its statuses saying the verdict is out -of date. The move itself asks the check again of every active open pull request into that trunk that merges -cleanly (ADR 0395, *A move of the trunk asks every check again* below); a recheck, a new head or the merge tool -asks it too. The row from -`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for -an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by -mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout. +when the builder clones it, and the verdict names both. When another merge moves the trunk, a `ready` +delivery goes back to `proposed` on the event `trunk moved`, and mesh-delivery asks the check of every active, +cleanly merging open pull request into that trunk at once (ADR 0395, *A move of the trunk asks every check +again* below). The row from `proposed` to `checked` takes only a verdict that names the trunk's head. -What this leaves as it is, said so nobody reads more into it: +**What needs no person ends by itself** (ADR 0270): a merge that moves nothing is `delivered` at its merge; a +merge a later delivered merge contains is `covered`, by the forge's word that the later merge commit contains +it; a merge whose plan said its modules unchanged is `delivered` when that plan ends (issue 611); a stop whose +plan ended without building its modules is read as covered once a later merge covers it (issues 474, 480). -- **mesh-delivery's trunk head is the last merge it heard** into that trunk while the delivery was open. A push - to the trunk that is not a merge it hears does not move it; the forge's holder, which reads the branch, and - the merge tool, which compares commits when asked, still catch it. -- **A verdict that names no trunk commit**, from a build seat older than ADR 0323, counts in mesh-delivery as it - did: such a delivery stays `ready`. The forge reads the same verdict out of date once its trunk moves, and the - merge tool refuses it and asks the check. The two agree once the head is judged again. -- **A verdict no build judges** (a change that touches no module, in a repository with no `merge-check.sh` or - not the mesh's) names the base branch's head the forge announced with the head as its trunk commit. -- **In a busy repository the merge tool can refuse, ask, and refuse again**: another merge may move the trunk - while the asked check runs, and the new verdict is then out of date too. Each refusal asks once more; the - merge goes through once a check finishes before the next merge. -- **A check that was asked and never answers** is listed by `stalled` an hour after the ask, and is asked again - with `mesh-delivery.recheck`, which the merge tool's refusal names. +**History is retired, not held** (ADR 0250). A delivery made from the forge's word at its merge, heard fifteen +minutes or more after it, with no plan of it open, goes to `retired` at adoption. **Nothing of a published +delivery is registered before its plan starts.** -The transitions are those of ADR 0239 decision 2, ADR 0270, ADR 0323 and ADR 0395. A delivery that is -merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A -delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by -`stalled` at once. Nothing waits silently. +**The machine steps.** While a delivery is `delivering`, each machine its plan reaches has a row with the +module and the build: `backed-up`, `sent`, `applied` or `failed`, and `silent` when no report came within +ApplySilentAfter (fifteen minutes). They are read from the plan's key, written by the controller from the +nodes' reports, never inferred from time. -**What needs no person ends by itself** (ADR 0270): +## Order without a person -- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and - sends nowhere) is `delivered` as soon as it is merged, whatever its check said; -- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine - running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the - forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge - order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery` - says `covered: a later merge that contains it was delivered` as a success. A module that runs on one - machine only is covered too: its rollout records the empty rest as sent once its first machine passes - (ADR 0270, as corrected 2026-10-12); -- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends, - whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads - that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and - may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such - module with why, "its build source holds none of the changed files", and not in its modules, which keep - meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered, - saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below. - Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's - `TestAMergeWhoseModulesMatchWhatRunsIsDelivered`; -- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and - is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after - its stop` with the covering merge, once, as a success. No state or transition is added (issue 480); -- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes. +*ADR 0239 §11 and §12, ADR 0249, ADR 0402.* -**History is retired, not held** (ADR 0250). -A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout -of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes -to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that -fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout -starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a -build registered before its turn would be carried by somebody else's send. +A delivery group's order is a table of pairs, each with the rule that won (ADR 0249 §3). The controller's +`delivery-order` infers it from the manifests and the module graph; a person writes an `after: ` +line only where the rules cannot see the order. -**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the -module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy -readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the -controller's `plan-moved` event, never inferred from time. +| Rank (highest first) | Rule | Read from | +|---|---|---| +| 1 | *declared*: an `after:` line | the member's description | +| 2 | *built by* and *version skew* | the module graph and the change | +| 2 | *verb before caller*: a member adding a verb to a seat's served verbs before a member adding that verb to an invoked verb | the manifests (ADR 0402) | +| 2 | *SDK before its users*: a member changing `mesh-sdk` before every member whose modules are built on it | the module graph (ADR 0402) | +| 3 | *engine before controller* | module names | +| 4 | the repositories' names | — | + +Rules of one rank in opposite directions reject the group, naming both members, both rules and the line that +would settle it (ADR 0249 §4). + +**The SDK by version.** A Go module names the SDK by its released version and fetches it from the mesh's forge, +checked against its recorded checksum; vendored copies go. Inside a group, a member may name the SDK member's +head: its merge check resolves that commit from the forge and builds against it, and the head stays fetchable +after the group merges because the forge keeps every pull request's head. The catalogue's check accepts an SDK +commit on the SDK's trunk or the head of an SDK pull request the forge holds, open or merged (ADR 0402 §2 to §4). + +## Checks + +### One check per pull request, kept by the bus + +*ADR 0323, ADR 0350, ADR 0395, ADR 0406.* + +- **Three work queues, taken in order.** Delivery builds go on a work-queue stream of their own; checks of pull + requests marked ready on a second; every other check on a third. A builder takes from the first, then the + second, then the third. Each check stream has one subject per pull request and keeps only the newest request: + a newer request replaces one still waiting, by the stream's rule. A ready mark publishes its pull request's + check request on the ready stream; a request left behind on the third for a head and trunk commit already + judged or running is acknowledged and dropped when taken, by reading the pull request's key. Nothing pauses the + worker or deletes and republishes a request (issue 558). +- **A running check stops itself.** It watches its pull request's key and stops when the head or the trunk + commit in it moves, without asking the forge (issue 726). An ended check delivers no verdict, so none of its + builds is reused. +- **A failed builder loses nothing.** A job a builder took and never acknowledged goes to another builder + after its wait. +- **A change is built once** (ADR 0350). A check that passes also builds what its merge would move into the + artifact store's scratch namespace under a reuse key, and the plan of the merge commit copies the build of the + same key, named by the check's verdict, instead of building. + +### A move of the trunk asks every check again + +*ADR 0395.* + +- A trunk move asks the check of every active open pull request into that trunk that merges cleanly, against the + new head; a conflict is told once on the pull request, not checked. +- A pull request with no push and no ready mark for 7 days is left out; a push, a ready mark or a recheck asks + its check. +- For a group, a move asks each member's own check, and the composed check runs again through those rechecks; a + move made by a member merging in the group's order asks the other members nothing. +- Its cost is at most one check per active, cleanly merging pull request per trunk move; issue 944 measures it. + +### A delivery group's checks + +*ADR 0239 §14, ADR 0323 §6, ADR 0382, ADR 0403.* + +- **The composed check is each member's gate.** The group's composed check composes every machine with all + members' heads, each merged into its own trunk's head first, judged by the controller built from the group's + controller member when it has one (ADR 0382 §3). Its verdict sets `mesh/delivery-group` and each member's + `mesh/merge-gate`, and names every member it read. No gate runs for a member alone. +- **Each member's own repository check stays required.** Each member's `mesh/repo-check` runs its repository's + suite with the siblings read beside it (ADR 0382 §2). +- **No composed verdict, no gate.** A member's gate is pending while no composed verdict at the members' current + heads exists. A member's new head, close or base move puts the composed verdict out of date, and with it every + member's gate. When mesh-delivery cannot say the groups, the pull request is checked alone and its verdict + says so (ADR 0382 §1). +- **A member merges only with its group passed and reviewed** (ADR 0382 §6), and in the group's order + (ADR 0364 §8). + +## Ready marks and the merge + +*ADR 0364, ADR 0406 §1.* + +The independent reviewer marks a pull request ready at the reviewed head with the label `mesh/ready` and a +comment `ready at `, through `mesh-delivery.mark-ready`. The forge's `labeled` and `commented` events +reach mesh-delivery in seconds; once both required checks passed against the trunk's head, it asks the operator's +Approve for a merge into main, with the pull request, its issues, the reviewer's verdict and the delivery plan, and +merges the marked head through the forge's merge tool on that Approve (ADR 0398 §3). When the trunk moved, the merge tool asks the fresh check, and the event +of its passing verdict asks the merge again; no timer asks it. A push clears the mark (the `synchronized` +event), and a failed check is said once and listed by `stalled`. ## The merge window and the batch -*2026-10-10, by ADR 0276.* +*ADR 0276, ADR 0282, ADR 0299, ADR 0304, ADR 0407.* -- **The window.** The controller puts every merge into a branch a module follows into the open **batch**. - Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by - default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its - merges and its times are kept in the controller's store and survive a restart. -- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for - every commit in it, tiered along the module graph across repositories, with at most one commit per - repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the - earlier merges of a repository that its later commit contains. -- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started - is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new - rollout, which answers every merge the folded one answered. -- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller, - the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for - mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its - rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again - behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10). -- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is - answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends - `covered` at once. -- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for - an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own - commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own - commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose - merge it rollouts again nowhere (below). -- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's - tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is - at its turn. -- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a - folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting - rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once. -- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped: - novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated", - and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as - `plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged - delivery it holds. -- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch - waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout - answers within ten minutes is held with the true reason (the open rollouts of its repository and that none - names its merge, or that the controller holds none) and listed by `stalled` at once. +- **Every merge is recorded at once.** The forge's merged event records a merge into the open batch of what it + touches, or opens one, by whichever controller holds the plans. A merge is never refused, held or dropped, + and needs no person's release (issue 921). The bus's durable consumer keeps every merge event until it is + recorded; no catch-up timer looks for missed merges. +- **The window is two keys.** An open batch's window is a key whose time to live is `merge-window` (90 s), + refreshed by each merge into the batch, beside a key set once when the batch opens whose time to live is + `merge-window-at-most` (10 minutes). The first to expire closes the batch, heard by a watcher; no cutter + polls. A stop that records merges again into the open batch closes its window too (ADR 0299 §3). +- **A lone merge is cut at once**: no batch it shares a repository, module, graph edge or delivery group member + with is assembling (ADR 0282 §2, ADR 0304 §1, ADR 0306). +- **The plan.** A closed batch becomes one delivery plan: every commit in it, tiered along the module graph + across repositories, at most one commit per repository, the latest of its trunk. It answers every merge in the + batch, its own commits and the earlier merges a later commit contains (ADR 0276 §2, as ADR 0299 §7 changed it). +- **A plan fixes its commits when it starts.** A plan starts at its first send, after mesh-delivery's word, and + from then nothing that merges changes it. A plan still waiting for its word is folded into the next batch + (ADR 0276 §4). +- **Plans that share something run one after another.** A batch that shares a repository, a module or a graph + edge with an open plan waits for it, and starts with every merge recorded since when that plan ends. A batch + that shares none runs beside the open ones, so a hotfix is never held by an unrelated feature's delivery. +- **The ends.** A plan done ends every delivery it answers: `delivered` for its own commits, `covered` for an + earlier merge it carried by a later commit. A plan that failed fails its own commits' deliveries; each earlier + merge it answered is then delivered on its own commit, the newest first, until one is delivered (ADR 0276 §3). +- **Shown.** `plans` lists the open batch first, with its window's time left and its merges, and a batch waiting + behind an open plan names that plan; the controller says the batch as `plan-moved` in the state `assembling`. +## A backup before the send -## A stop on a shared rollout +*ADR 0214 §7, ADR 0233, ADR 0404.* -*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.* +When a delivery plan is ready to send, each node it reaches backs up, through its `node-backup` seat, every item +the plan's moved modules keep there whose protection is a backup (a copy or a dump). An item protected by +`redundancy`, one that says `backup: none`, and a cache are never asked, whatever their class. A node with no backup holder backs up +nothing, and the plan says so for that node. -One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other -delivery on it. A stop therefore ends only the change it is for: +- **The send waits for every backup.** The controller sends nothing of the plan to any node until every backup + it asked has answered done. +- **A failed backup holds the delivery.** A backup that fails, or does not answer within ten minutes, holds the + delivery with the node, the module and the backup's own words as the reason, and raises a condition for the + operator. A person's `release` asks the backups again, and the send goes on when they pass. +- **What it costs.** About 85% of data modules back up in under 30 s; the slowest measured, postgres on the home + server, about 165 s (research 055, finding 5). A plan moving no backed-up item costs nothing more. + +## The send and the rollout + +*ADR 0360, ADR 0362, ADR 0377, ADR 0406 §5.* + +- **Every machine at once.** A plan's builds go to every machine that runs them in one step, tier by tier; a tier + is asked once the tier before it is built and sent (ADR 0360 §1, §6). No machine goes first and nothing waits + for a health reading. A module of policy `record` that would recreate a container or move data waits for a + person (ADR 0377). +- **Every move is said.** The controller says every move of a plan as `plan-moved`, a move a command makes among + them, so mesh-delivery needs no periodic reconcile; it reads `delivery-walks` once when it starts. +- **A rollout advances on events.** The `built` and sent events of a tier ask the next; the nodes' reports end + the rollout (ADR 0360 §7, with ADR 0362's provisioning report). Each send puts a key whose time to live is + ApplySilentAfter (fifteen minutes); its expiry names the machine silent. There is no rollout ticker. +- **A failed apply is loud and fixed forward** (ADR 0360 §8); the node-engine's witness still puts back a + controller or tool runner that does not come up (ADR 0360 §9). + +## A stop on a shared plan + +*ADR 0299.* + +One plan carries every merge of its batch, so a stop ends only the change it is for: + +- **A group's stop.** When a member after a failed one is stopped and its plan is open, mesh-delivery asks + `delivery-stop` for that plan with the stopped members' merges to leave out. The controller ends the plan and + records every other merge it answered again in the open batch of its kind, whose window the stop closes. +- **What is left out with it.** A later merge of the stopped merge's repository, and a merge that moves a + module a left-out merge moves, would deliver the stopped change; each is left out too, its delivery `stopped`, + naming why. +- **A person's stop.** `stop` of a delivery whose plan carries others first answers what it would take down, and + acts on `way`: `again` (the others delivered again without it) or `walk` (the whole plan stopped). +- **What it does not do.** What the stopped plan already sent stays on the machines it reached; putting it back + is issue 509's. -- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks - `delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass - stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge - it answered again. -- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A - module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too. - Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped - merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is - rolled out again. -- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran - already wait, so an older commit never rollouts after a newer one; only with none open do they form a new - batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop - before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules - where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes - to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every - merge moved are written in one transaction. -- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left - out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery - whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays - `published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every - delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop. -- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped - delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it - stops the whole rollout. -- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it - answers with each of them, which of them contain this merge and end either way, and two ways given with - `way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout - stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is - marked stopped only once its rollout is; a stop the controller refuses leaves it as it was. -- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its - asked builds still register, so a later send may carry them further; the stop names the machines, and - putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the - stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it - can see, and holding a merged change back for good is a revert's job. ## The delivery budget -*Decided 2026-10-10 by ADR 0282. -Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by -slice as each slice below is built; until a slice is built, the text above is what runs.* +*ADR 0282, ADR 0304 §4, ADR 0313, ADR 0360 §12, ADR 0362 §5.* -**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard -from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes; -a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is -shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes -(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers -14.2 (issue 338). +A delivery's **delivery time** runs from its merge until every machine its plan sent has reported the send +applied, and for a provisioned module also placed. A **leaf** delivery's budget is three minutes, a **core** +delivery's ten. A delivery held for a person, a failed backup among them, is shown and held to no budget. Each +phase is kept — the window, the wait behind a plan that shares something, the backups, the build, the send, the +reports — and a delivery over its budget raises `delivery..over-budget`, naming its longest phase. The +**fix time** (ADR 0313) runs from an issue's first move to `fixing` to its last fix delivered, and contains the +delivery time. -*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget -above runs from the merge; the operator's own target is wider, the **fix time**, which runs from an issue's -first move to `fixing` to the moment its last fix is delivered. Each issue's class carries a **fix budget** too -(five minutes leaf, ten core, the delivery budget's own figures), and the tracker splits a fix time into four -phases — implementing, review, checks, and merge to running (the delivery time above) — so a fix over its -budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget. -The delivery budget stays as written: a measure inside the fix budget, not replaced by it. +## Builds without a toolchain image -**What changes in the rollout.** +*ADR 0267, ADR 0350, ADR 0405.* -- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged. -- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One - node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf - build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged. -- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf - module with a long-running resource; one for a leaf module whose every resource is a tools bundle. -- **A tier's first nodes are judged at once.** -- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new - tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before - anything is judged; a step that would carry anything else is refused unsent and said on the gate line. -- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy - report; the plan shows an expected landing time from the last seven days' medians. +- **One build module per language** (Go; Node and TypeScript) installs its toolchain as a declared package on + every node holding the build seat, pinned to one version that is the same on every builder. The toolchain + image and its module retire. +- **Every builder runs the same toolchain, and says so.** Each builder names its toolchain versions with every + build outcome, and the controller's self-check raises a condition while two builders differ. +- **Builds and tests run directly**, as the build seat's own account on its node, in no container. A + repository's `# mesh-check-toolchain` line selects which installed toolchain runs its check. +- **A judge has an account of its own.** A judge built from a sibling's merge (ADR 0382 §3) is built and run as a + second account of the Go build module, whose home, configuration and cache the build seat's account cannot write, + so a check's script cannot change it. +- **Every dependency comes by the version its lock file names**, checked against its checksum: the mesh's own + code from the mesh's forge, its own packages from its package registry, any other from its public source. +- **A builder fetches from the forge over SSH with a key of its own**, as the build seat's account, under a forge + account for that builder that reads the mesh's repositories and pushes nothing. +- **A build's reuse key** holds the installed toolchain's version and the build module's recipe version in place + of the toolchain image's digest, so a toolchain update rebuilds only the modules of that language. -*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what -a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it: +## One chain, on the bus alone -- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open - rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and - judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open - rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17 - for builds an open rollout owns). -- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the - merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order. -- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or - send, the dependents of what its merges moved included; a change whose modules are not known yet is related to - every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node. -- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a - warning naming the rollout and the core rollouts that set it aside. -- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an - unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its - own builds. -- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by - issue 528. +*ADR 0408.* -*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out -of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch; -overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every -open judgement on its node. +Module, commit, build, published package or artifact, rollout and assignment are one chain, each link +answerable from the next. A **rollout** here is the controller's record of one batch's delivery; a delivery's own +delivery plan (*A delivery* above) stays the delivery's, and names the rollout that delivers it. -*A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`, -a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge -into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace -such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once -it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy -container there, its first-node gate starts at the report that says the container replaced, and the wait is a -phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget -is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a -rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the -node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the -judgement. +| Link | Names | Written by | Bucket | +|---|---|---|---| +| build | its module, repository and commit, its reuse key, what it published: an artifact by digest or a package release by name and version | the controller | `mesh-controller_builds` | +| rollout | the commit it fixed per repository, its tiers, the builds it carried, the machine steps, and every assignment it changed with the build before and after; its states and transitions | the controller | `mesh-controller_rollouts` | +| assignment | the module and node, the build, artifact or package and commit it runs, and the rollout that put it there | the controller | `mesh-controller_assignments` | -**Who builds what.** +Beside the chain, on the bus too: deliveries and groups (mesh-delivery's `deliveries` and `groups`, ADR 0239 §37), +one key per open pull request (mesh-delivery), and each open batch with its merges and its window's two keys (the +controller, ADR 0407). -| Part | Owner | What it adds | -|---|---|---| -| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration | -| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` | -| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery..over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` | -| the lone merge | mesh-controller | the cut at once when nothing waits to join | -| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch | -| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once | -| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` | - -**Slices**, each one pull request per repository, each with its acceptance sentence: - -| Slice | Delivers | Done when | -|---|---|---| -| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase | -| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases | -| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout | -| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other | -| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median | - -Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges -only once it is built. - -## A delivery group - -- **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull - requests. Two or more make a group. A group that has started delivering takes no new member. -- **Order**: `after: ` lines in a member's description, and the controller's `delivery-order` - over the graph (ADR 0239 decision 4). The plan shows the order and why each pair is ordered: *declared*, - *built by*, *version skew*, *engine before controller*, *by name*. -- **Precedence** (ADR 0249): when rules order one pair both ways, the higher one decides. *Declared* comes - first, then *built by* and *version skew*, then *engine before controller*. The pair shows the rule that - won and the rules it won over. Rules of one rank both ways are a contradiction. The group is `rejected`, - and the reason names both members, both rules and the `after:` line that would settle it. -- **Check**: the members' heads composed together, one gate over every machine with all their definitions, - each head merged into its own trunk's head first and named with that trunk commit (ADR 0323). It is asked - by mesh-delivery through `delivery-check` when the group forms or a member's head moves, and posted on - every member's head as `mesh/delivery-group`. A member rebased or pushed is a new head with its own check: - it takes its old head's place in the group, even once the group is delivering, and the composed check runs - again for the new heads instead of holding the group. A merge into a member's trunk that is not one of the - group's members puts the composed check out of date: the group reads `checking` with the reason, and the - composed check runs again once a member is judged again. Checked by mesh-delivery's - `TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember` and `TestARebasedMemberIsComposedAgainNotHeld`. -- **Each member's own check reads its siblings** (ADR 0382): the group's other open members into a base of - the same name are read beside a member's own check at their head merged onto their base, and a mesh-controller - sibling's merge judges it, so changes that are only consistent together pass their own checks. The controller - learns the siblings from mesh-delivery's `groups`; the verdict names each one. A sibling's new head, close or - base move puts that verdict out of date, and the merge tool merges such a member only when - `mesh/delivery-group` passed at the heads read and every sibling head read is reviewed. -- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`. - A member published while its group's composed check is asked and not answered waits there for the answer; - one held because that check had not passed is *lifted* back to `published` once it passes or warns for the - heads that merged and its own rollout is still open (ADR 0270). -- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its - rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered. - Its own builds were put back at its gate. - -A group's state is derived from its members and its composed check, and is shown, never stored apart from -them. - -## A move of the trunk asks every check again - -*ADR 0395, changing ADR 0323 §4.* When the forge's holder hears a repository's trunk move, mesh-delivery asks the -check of every active open pull request into that trunk against its new head at once, so a verdict is current by -the time anyone looks. - -- **A conflict is told, not checked.** Before asking a check, mesh-delivery asks the forge whether the head merges - cleanly into the new head, which needs no builder. When it does not, no check is asked: its statuses fail naming the trunk commit and the files in - conflict where the forge gives them, and it is told once on the pull request, not again while the same conflict - stands. Every later trunk move and its next push ask the forge again; once it merges cleanly its check is asked. -- **One check per pull request.** When a check against a newer trunk commit is asked for a pull request, its check - against an older trunk commit is dropped if it waits and ended if it runs, said in the build log and on the - queue, as a new head already ends its old head's check (issue 726). An ended check delivers no verdict, so none of its builds is reused (ADR 0350 §7 keeps only builds a verdict - names); its scratch records wait for ADR 0350 §5's seven-day sweep. A try and a delivery's build are never touched. -- **The queue's order.** Delivery builds first; then checks of pull requests marked ready; then the rest, the most - recently pushed first. -- **Inactive pull requests are left out.** One with no push and no ready mark for 7 days is not asked again on a - trunk move; its statuses say out of date and that a push, a ready mark or a recheck asks its check. A recheck - takes a delivery that is still proposed and whose verdict is out of date, and asks its check (issue 888). -- **Groups.** For a group's members a move asks only each member's own check; the group's composed check runs again - through those rechecks (ADR 0323 §6) and is never asked by the move itself. A move made by a member merging in - the group's order asks the other members nothing, since they merge moments later; their verdicts are out of date - until then, and one that does not merge is asked again at the next move, its next push or a recheck. - -**Its cost** is at most one check asked per active, cleanly merging pull request per trunk move: about 550 a day at -ADR 0323's measured load, fewer once conflicting and inactive ones are left out; ending an older check saves builder -time, not checks asked. Issue 944 measures it once built. +- **The bus is the only source.** Each bucket is written by the controller that holds the lease, by + compare-and-set, and the controller reads its own buckets to decide; a rollout resumes from its key after a + restart. No table of the controller's store holds builds, rollouts, batches or assignments, so no record mirror, + audit or healer exists for them; the nodes, settings, declared-config and predefined-keys mirrors stay as + ADR 0289 and ADR 0356 have them. A module reads a bucket by naming it in `reads`; mesh-config reads the + assignments bucket. +- **No key of the chain is removed.** Each bucket keeps up to 64 earlier revisions of each key as its history. A + value is never cut to a summary and stays under the bus's largest message. The bus's storage is declared as data + protected by a backup, so it is backed up nightly and before a delivery moves the bus. +- **One read answers each question.** For any module on any node, one read of its assignment names the commit, + the build, the artifact or package and the rollout that put it there; one read of a rollout lists every + assignment it changed. +- **The tables go by a numbered migration**: builds, rollouts and assignments are written once into the buckets, + waiting build requests onto the work queues and an open batch into its keys; then the tables are dropped from the + controller's store and its consolidated schema. +- **The artifact store and the package registry keep the bytes**; the chain names them (ADR 0251). ## What every transition does In this order, each by its owner: -1. **kept**: mesh-delivery puts the delivery in its state (`deliveries`, one key per delivery; `groups`, one - key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried, later - each time, from ten seconds doubling to an hour. The forge's answer that what it is for does not exist ends - it at once, and sixty failed tries end it too, each said among the owner's refusals (ADR 0250); +1. **kept**: mesh-delivery puts the delivery in its state by compare-and-set before anything else; what it owes + outside is kept with it and retried from ten seconds doubling to an hour, at most sixty times (ADR 0250); 2. **said**: the event `mesh-delivery.transition` (and `mesh-delivery.group` when a group's derived state - changes), with the delivery's id, from, to, event, why and plan summary, and no secret or address; -3. **noted**: mesh-delivery asks the forge's holder (`gitea_note_append`) to append a line to the commit's - note: the head while the delivery is off the trunk, the commit it landed as once it is on it, and at its - end a line of what was executed there. The forge's holder writes it in its own repository as the forge's - own account, and a line already there is not appended again; -4. **reflected** (not for history, whose pull request the forge may no longer hold): the delivery's view, one comment on the pull request with the state, the plan, the machine - steps and the group, kept current (`gitea_delivery_view`); the status `mesh/delivery` on the head, and - `mesh/delivery-group` on every member's head (`gitea_commit_status`). Every status links to the pull - request, `mesh/merge-gate` among them. + changes), with no secret or address; +3. **noted**: a line on the commit's note, appended by the forge module as the forge's own account; +4. **reflected**: the delivery's view on the pull request, `mesh/delivery` on the head and `mesh/delivery-group` + on every member's head. + +A doubled step is refused by compare-and-set and by the bus's message de-duplication, which also replaces the +forge module's files of what it announced (ADR 0406 §6). ## The verbs -**mesh-delivery's seat** (served by its holder; the first five read): +**mesh-delivery's seat**: | Verb | Answers or does | |---|---| -| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group | -| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout | +| `deliveries` | every delivery not final, the final ones of the last day, and every one `stalled` still lists | +| `show` | one delivery or group whole: plan, transitions, machine steps, order, the chain's links | | `groups` | every group with its members in order and its derived state | -| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere | +| `what-if` | the delivery plan a diffset would have, asked of the controller's planner | | `table` | the state table and the machine-step table, with bounds and H2's repairs | -| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds | -| `recheck` | a rejected or ready delivery back to proposed, with why; a delivery still proposed whose verdict is out of date has its check asked (ADR 0395) | -| `release` | a held delivery to delivering, with why: a person's word | -| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) | -| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise | -| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing | +| `stalled` | every delivery held past its bound, with the transition H2 may take | +| `mark-ready` | a reviewer's mark at a head: the label and the `ready at` comment, by the mesh's own forge account | +| `recheck` | a check asked again for a delivery whose verdict is out of date (ADR 0395 §6) | +| `release` | a held delivery goes on, with why; for a failed backup it asks the backups again | +| `stop` | a delivery to stopped, with why; on a shared plan it first answers what it would take down (ADR 0299) | +| `close` | H2's only verb: the transition the table names for a stalled delivery | +| `retire-history` | held history to retired (ADR 0250) | +| `times` | each class's budget, the week's median and worst delivery time, and the deliveries over budget | -**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`, -`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which -takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299), -`delivery-walks`. A person has `plans go --why`, which does what `deliver` does under their name and -is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command -too. - -**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It -also announces `pull.closed`, and a merge's head and the statuses the forge holds on it. - -## The store - -Key-value state on the bus, declared by the module (ADR 0201): `deliveries` and `groups`. One writer, the -seat's holder, whose code changes state from a single goroutine. A final delivery is kept for thirty days and -then removed. Its note on the commit keeps its record after that. The bus's own snapshot (ADR 0235) backs it -up. +**The controller's verbs for it**: `delivery-plan`, `delivery-order`, `delivery-check`, `deliver`, +`delivery-stop`, `delivery-walks`, and `plans` (the open batch, the open plans, and a what-if for a repository +and its files). A person has `plans go --why`. ## The bootstrap -- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the - controller's own, started by the merge and witnessed on the machine. mesh-delivery records it. -- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the - controller starts it as it always did. -- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises - `plan..waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and - `plans go --why` as the way on. - A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone. +- A plan that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the + controller's own, started by the merge; mesh-delivery records it (ADR 0239 §31). +- Any other plan waits for `deliver` while the seat has a holder on record; with none, the controller starts it. +- A replaced controller or mesh-delivery carries on from the bus: the keys, the queues and the durable + consumers hold every step, so nothing is lost or repeated (ADR 0406 §6). -## The switch +## Acceptance tests -1. The controller's change (the seat, its verbs, `plan-moved`, the wait for a holder's word, `plans go`) is - merged and rolls out as the core does. With no holder on record it behaves exactly as before. -2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built - and published. -3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start - from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in - `delivering`. -4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass. +What the build proves before this design is implemented (research 055's direction): -## Phases +1. A feature across the controller and the catalogue with a new seat verb is delivered as one group, with no + `after:` line, no two steps and one composed check. +2. An SDK change and a Go module using it are delivered as one group. +3. A data module is backed up before a delivery moves it, and a failed backup holds the delivery. +4. A build on any builder uses the installed toolchain with no container, and a toolchain update rebuilds only + modules of that language when its version changes. +5. Two merges into one repository within a minute leave each open pull request with one queued check, by the + stream's rule. +6. A ready mark is acted on within seconds of its event. +7. A controller replaced in the middle of a rollout resumes it with no step lost or repeated. +8. The merge window closes by its key's expiry. +9. A hotfix to a module the running feature's delivery does not touch reaches the machines in minutes, beside it. +10. For any module on any node, one read of its assignment names the commit, build, artifact or package and the + rollout that put it there, and one read of that rollout lists every assignment it changed. +11. Two merges into one repository while a delivery of that repository runs are both delivered by the next plan, + with no person's release and nothing held in mesh-delivery. +12. No table in the controller's store holds builds, rollouts or assignments. -| Phase | Delivers | Done when | -|---|---|---| -| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` | -| B — the conditions | S16, the controller's own `plan..waiting`; probe D14 reads `stalled` and raises `delivery..stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 | -| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while | +## What retires + +- The "release branches" and the controller's judge of them (ADR 0358, superseded by ADR 0401). +- The toolchain image and its module (ADR 0405). +- The window cutter, the merge catch-up timer, the rollout ticker, mesh-delivery's tick, reconcile and + ready-mark poll, the forge's poll as a trigger, the queue reordering, the forge-asking cancel, and the forge + module's announcement files (ADR 0406, ADR 0407). +- The controller's tables of builds, build requests, rollouts and their tiers, batches and merges, and + assignments; the record mirrors, probe D17's audit and healer H6 for those kinds (ADR 0408). + +## Milestones + +Set by playbook 04 at the handoff, after the operator confirms them. ## What is not decided here - A web view of deliveries beyond the pull request's comment. -- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge - module's protection tool. -- Moving the per-tier rollout out of the controller (Phase C). +- Whether `mesh/delivery-group` is required on the trunks: the operator's setting, through the forge module's + protection tool. +- How a builder's public SSH key reaches the forge: the forge's and the SSH client's mechanism.