|
|
|
@@ -4,69 +4,112 @@ status: in-progress
|
|
|
|
|
code: [mesh-controller, mesh-catalog modules/mesh-delivery]
|
|
|
|
|
updated: 2026-10-16
|
|
|
|
|
decisions:
|
|
|
|
|
- "ADR 0408"
|
|
|
|
|
- "ADR 0407"
|
|
|
|
|
- "ADR 0406"
|
|
|
|
|
- "ADR 0405"
|
|
|
|
|
- "ADR 0404"
|
|
|
|
|
- "ADR 0403"
|
|
|
|
|
- "ADR 0402"
|
|
|
|
|
- "ADR 0401"
|
|
|
|
|
- "ADR 0398"
|
|
|
|
|
- "ADR 0395"
|
|
|
|
|
- "ADR 0382"
|
|
|
|
|
- "ADR 0377"
|
|
|
|
|
- "ADR 0364"
|
|
|
|
|
- "ADR 0362"
|
|
|
|
|
- "ADR 0360"
|
|
|
|
|
- "ADR 0350"
|
|
|
|
|
- "ADR 0335"
|
|
|
|
|
- "ADR 0323"
|
|
|
|
|
- "ADR 0313"
|
|
|
|
|
- "ADR 0306"
|
|
|
|
|
- "ADR 0304"
|
|
|
|
|
- "ADR 0299"
|
|
|
|
|
- "ADR 0298"
|
|
|
|
|
- "ADR 0282"
|
|
|
|
|
- "ADR 0276"
|
|
|
|
|
- "ADR 0270"
|
|
|
|
|
- "ADR 0267"
|
|
|
|
|
- "ADR 0250"
|
|
|
|
|
- "ADR 0249"
|
|
|
|
|
- "ADR 0239"
|
|
|
|
|
- "ADR 0238"
|
|
|
|
|
- "ADR 0236"
|
|
|
|
|
- "ADR 0233"
|
|
|
|
|
- "ADR 0214"
|
|
|
|
|
- "ADR 0201"
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# 47 — Delivery: from a commit to delivered
|
|
|
|
|
|
|
|
|
|
**A delivery is one commit in one repository, from the moment its pull request's head is announced to the
|
|
|
|
|
moment every machine of its deploy plan runs it, or to the moment it fails, is superseded or is stopped. A
|
|
|
|
|
delivery group is two or more deliveries that share a head branch name, delivered as one unit in order. Both
|
|
|
|
|
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
|
|
|
|
|
(ADR 0239.)
|
|
|
|
|
**A change reaches the machines in one of three ways, all owned by mesh-delivery: a hotfix (one pull request,
|
|
|
|
|
delivered at its merge), a feature (one branch name across its repositories: in two or more, one delivery
|
|
|
|
|
group merged together and delivered member by member in its order; in one, a lone delivery), or an epic (a run of milestones, each delivered as a feature). The trunk is what runs; there
|
|
|
|
|
is no "release branch"** (ADR 0401).
|
|
|
|
|
|
|
|
|
|
**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a
|
|
|
|
|
rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout
|
|
|
|
|
at a time. Every delivery the rollout answers ends with it
|
|
|
|
|
(ADR 0276).
|
|
|
|
|
**Every step starts on an event, and the state lives on the bus.** The forge's events, the controller's
|
|
|
|
|
events and the nodes' reports move a delivery; the bus's own rules keep the queues, the merge window and the
|
|
|
|
|
order; and one chain, from module to commit to build to rollout to assignment, is kept in the bus's
|
|
|
|
|
key-value store as its only source (ADR 0406, ADR 0407, ADR 0408). A running delivery never touches merging:
|
|
|
|
|
every merge is recorded at once for the next delivery plan of what it touches.
|
|
|
|
|
|
|
|
|
|
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
|
|
|
|
|
of source and artifacts decides, and nothing here changes that. This design says who answers for one
|
|
|
|
|
commit's journey and in what order the steps are allowed.
|
|
|
|
|
commit's journey, in what order the steps are allowed, and where each fact of it is kept. It was rewritten on
|
|
|
|
|
2026-10-16 from the decisions of research 055, which replaced the "release branches", the toolchain image, the
|
|
|
|
|
controller's delivery tables and the timers that polled them.
|
|
|
|
|
|
|
|
|
|
## The parts
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
forge's holder ──pull.updated / .merged / .closed──► mesh-delivery ◄──checked / plan-moved── controller
|
|
|
|
|
(statuses, the view, (deliveries, groups, (planner, gate,
|
|
|
|
|
the notes) the table, its state) registration,
|
|
|
|
|
▲ │ │ sending, walk)
|
|
|
|
|
└── its tools: note, view, status ◄───────────────┘ └── its verbs: delivery-plan, -order,
|
|
|
|
|
-check, deliver, delivery-stop,
|
|
|
|
|
delivery-walks
|
|
|
|
|
forge (system webhook) ──forge.pull-request.* / merged / status──► the bus ◄──built / checked / plan-moved── controller
|
|
|
|
|
│ ▲ (planner, order,
|
|
|
|
|
│ │ registration,
|
|
|
|
|
▼ │ sending)
|
|
|
|
|
mesh-delivery │
|
|
|
|
|
(deliveries, groups, a key per │
|
|
|
|
|
pull request, the state table) ▼
|
|
|
|
|
node-engine on every node
|
|
|
|
|
builders (build seat) ◄── work queues: delivery builds, ready checks, other checks ───────────────────────── (reports; backups first)
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants,
|
|
|
|
|
registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk
|
|
|
|
|
rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers
|
|
|
|
|
every pull request's head with its check itself, so a check never depends on mesh-delivery.
|
|
|
|
|
- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the
|
|
|
|
|
group, the state table and their state, and nothing else.
|
|
|
|
|
- **The forge's holder** sets a commit's statuses, keeps the delivery's view as one comment on the pull
|
|
|
|
|
request, and appends to the commit's note under `refs/notes/mesh-plan`.
|
|
|
|
|
- **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350):
|
|
|
|
|
a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a
|
|
|
|
|
reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build
|
|
|
|
|
of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and
|
|
|
|
|
says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull
|
|
|
|
|
request closes unmerged, and after seven days.
|
|
|
|
|
- **mesh-delivery** (the seat `mesh-delivery`) owns hotfixes, features and milestones as deliveries and delivery
|
|
|
|
|
groups, decides each step on events, keeps one key per pull request, and asks the controller for each act
|
|
|
|
|
and the nodes' backup seats for the backups a delivery needs. It never sends to a machine, writes a
|
|
|
|
|
declaration or registers a build (ADR 0239 §25).
|
|
|
|
|
- **The controller** keeps planning, ordering, registration, sending and judging. It writes the chain's
|
|
|
|
|
links it owns — builds, rollouts with their tiers, assignments — and the merge window's
|
|
|
|
|
keys, all on the bus, by compare-and-set, while it holds the lease.
|
|
|
|
|
- **The builders**, the holders of the build seat, build and check with toolchains installed on their node, as
|
|
|
|
|
the build seat's own account, and take their jobs from the bus's work queues, delivery builds first
|
|
|
|
|
(ADR 0405, ADR 0406).
|
|
|
|
|
- **The forge module** (`gitea`) publishes every forge event on the bus through the forge's system webhook
|
|
|
|
|
(issue 729): pull requests updated, synchronized, labeled, unlabeled, commented, merged and closed, and
|
|
|
|
|
commit statuses. It sets statuses, keeps the delivery's view as one comment on the pull request, and appends
|
|
|
|
|
to the commit's note under `refs/notes/mesh-plan`.
|
|
|
|
|
- **Each node's backup seat** (`node-backup`) backs up the data a delivery plan moves there before the plan
|
|
|
|
|
sends anything (ADR 0404).
|
|
|
|
|
|
|
|
|
|
## Three ways a change reaches the machines
|
|
|
|
|
|
|
|
|
|
*ADR 0401.*
|
|
|
|
|
|
|
|
|
|
| Way | What it is | How it is delivered |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| **hotfix** | one pull request in one repository | at its merge, as a lone delivery; a lone merge is cut at once (ADR 0282 §2, as ADR 0304 §1 changed it) |
|
|
|
|
|
| **feature** | one branch name across its repositories; in two or more, its pull requests are one delivery group; in one, a single pull request | a group is merged together once its composed check passes and every member is marked, reviewed and approved, then delivered member by member in its order (ADR 0239 §16); in one repository, as a lone delivery like a hotfix |
|
|
|
|
|
| **epic** | a run of milestones in one design (playbook 04) | each milestone is a feature, delivered when its own group is ready; the next starts when the previous one's delivery plan is done |
|
|
|
|
|
|
|
|
|
|
- **The trunk is what runs.** Every delivery is cut from a repository's trunk. A merge into any other branch
|
|
|
|
|
starts no delivery, and the controller says so for that merge (issue 889). Holding merged work back, on a
|
|
|
|
|
branch beside the trunk or per module, was dropped: from 2026-10-13 23:52 to 2026-10-16 09:32 feature 57's
|
|
|
|
|
branches held the controller for 55.7 h, and 28 of 68 other merges touched a held module (research 055,
|
|
|
|
|
finding 3).
|
|
|
|
|
- **A milestone is functional, its delivery shaped technically.** A milestone is named by what the operator or
|
|
|
|
|
the mesh can do once it is delivered; how it ships is shaped by the delivery plan its changes would have, asked of
|
|
|
|
|
the controller's planner through mesh-delivery's `what-if` (ADR 0239 §3): the modules it moves and their fan-out.
|
|
|
|
|
- **What a merge waits for.** mesh-delivery merges a group's members once the group's composed check passed at
|
|
|
|
|
their heads, the independent reviewer marked each member ready (ADR 0364 §1, §2), each member carries the
|
|
|
|
|
review ADR 0382 §6 asks, and the operator approved each merge into main (ADR 0398 §3). The operator's Approve is
|
|
|
|
|
asked for every merge into main, a hotfix's too (the operator, 2026-10-16 23:40).
|
|
|
|
|
|
|
|
|
|
## A delivery
|
|
|
|
|
|
|
|
|
@@ -75,12 +118,17 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
|
|
|
|
|
- the pull request (number, base, head branch, description), or none for a commit that reached the trunk
|
|
|
|
|
without one;
|
|
|
|
|
- its **delivery plan**: build plan, deploy plan and verdict, as the controller's planner computed them for
|
|
|
|
|
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
|
|
|
|
|
its diffset (ADR 0238 §2, as ADR 0323 §1 changed it);
|
|
|
|
|
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
|
|
|
|
|
**machine steps** while delivering;
|
|
|
|
|
- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch
|
|
|
|
|
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
|
|
|
|
|
window and the batch*).
|
|
|
|
|
- once merged, **the commit it landed on the trunk as**, and the delivery plan that delivers it: the plan of the
|
|
|
|
|
batch that answers its merge, which may carry a later commit of its repository that contains it.
|
|
|
|
|
|
|
|
|
|
**One key per pull request.** Beside the delivery, mesh-delivery keeps one key per open pull request on the
|
|
|
|
|
bus: its head, the trunk commit its current check is asked against, and its verdict, written by
|
|
|
|
|
compare-and-set (ADR 0406 §4). It writes a new head when the forge says one, and a new trunk commit only when it
|
|
|
|
|
asks the pull request's check again (ADR 0395 §1). A running check watches that key and stops when the head or
|
|
|
|
|
the trunk commit in it moves.
|
|
|
|
|
|
|
|
|
|
## The state table
|
|
|
|
|
|
|
|
|
@@ -95,418 +143,353 @@ allows.
|
|
|
|
|
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
|
|
|
|
|
| ready | it passed; it waits to be merged | none | — |
|
|
|
|
|
| rejected | it failed its check | none | — |
|
|
|
|
|
| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started |
|
|
|
|
|
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
|
|
|
|
|
| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says |
|
|
|
|
|
| published | merged on its trunk, recorded in a batch; it waits for its plan to start | 30 minutes | `superseded` when a plan that does not answer its merge took over its plan; `go` when its plan started |
|
|
|
|
|
| held | it waits for a person: a failed backup (ADR 0404 §4), a policy of record; it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
|
|
|
|
|
| delivering | its plan runs: backups, then builds asked and sent tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the plan's key says |
|
|
|
|
|
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
|
|
|
|
|
|
|
|
|
|
**A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head
|
|
|
|
|
at the moment the build seat clones it, the **trunk commit**, and the verdict names both. When another merge
|
|
|
|
|
moves the trunk, a `ready` delivery goes back to `proposed` on the observed event `trunk moved`, with both
|
|
|
|
|
commits in the transition's reason; a `rejected` one stays `rejected`, its statuses saying the verdict is out
|
|
|
|
|
of date. The move itself asks the check again of every active open pull request into that trunk that merges
|
|
|
|
|
cleanly (ADR 0395, *A move of the trunk asks every check again* below); a recheck, a new head or the merge tool
|
|
|
|
|
asks it too. The row from
|
|
|
|
|
`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for
|
|
|
|
|
an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by
|
|
|
|
|
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout.
|
|
|
|
|
when the builder clones it, and the verdict names both. When another merge moves the trunk, a `ready`
|
|
|
|
|
delivery goes back to `proposed` on the event `trunk moved`, and mesh-delivery asks the check of every active,
|
|
|
|
|
cleanly merging open pull request into that trunk at once (ADR 0395, *A move of the trunk asks every check
|
|
|
|
|
again* below). The row from `proposed` to `checked` takes only a verdict that names the trunk's head.
|
|
|
|
|
|
|
|
|
|
What this leaves as it is, said so nobody reads more into it:
|
|
|
|
|
**What needs no person ends by itself** (ADR 0270): a merge that moves nothing is `delivered` at its merge; a
|
|
|
|
|
merge a later delivered merge contains is `covered`, by the forge's word that the later merge commit contains
|
|
|
|
|
it; a merge whose plan said its modules unchanged is `delivered` when that plan ends (issue 611); a stop whose
|
|
|
|
|
plan ended without building its modules is read as covered once a later merge covers it (issues 474, 480).
|
|
|
|
|
|
|
|
|
|
- **mesh-delivery's trunk head is the last merge it heard** into that trunk while the delivery was open. A push
|
|
|
|
|
to the trunk that is not a merge it hears does not move it; the forge's holder, which reads the branch, and
|
|
|
|
|
the merge tool, which compares commits when asked, still catch it.
|
|
|
|
|
- **A verdict that names no trunk commit**, from a build seat older than ADR 0323, counts in mesh-delivery as it
|
|
|
|
|
did: such a delivery stays `ready`. The forge reads the same verdict out of date once its trunk moves, and the
|
|
|
|
|
merge tool refuses it and asks the check. The two agree once the head is judged again.
|
|
|
|
|
- **A verdict no build judges** (a change that touches no module, in a repository with no `merge-check.sh` or
|
|
|
|
|
not the mesh's) names the base branch's head the forge announced with the head as its trunk commit.
|
|
|
|
|
- **In a busy repository the merge tool can refuse, ask, and refuse again**: another merge may move the trunk
|
|
|
|
|
while the asked check runs, and the new verdict is then out of date too. Each refusal asks once more; the
|
|
|
|
|
merge goes through once a check finishes before the next merge.
|
|
|
|
|
- **A check that was asked and never answers** is listed by `stalled` an hour after the ask, and is asked again
|
|
|
|
|
with `mesh-delivery.recheck`, which the merge tool's refusal names.
|
|
|
|
|
**History is retired, not held** (ADR 0250). A delivery made from the forge's word at its merge, heard fifteen
|
|
|
|
|
minutes or more after it, with no plan of it open, goes to `retired` at adoption. **Nothing of a published
|
|
|
|
|
delivery is registered before its plan starts.**
|
|
|
|
|
|
|
|
|
|
The transitions are those of ADR 0239 decision 2, ADR 0270, ADR 0323 and ADR 0395. A delivery that is
|
|
|
|
|
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
|
|
|
|
|
delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by
|
|
|
|
|
`stalled` at once. Nothing waits silently.
|
|
|
|
|
**The machine steps.** While a delivery is `delivering`, each machine its plan reaches has a row with the
|
|
|
|
|
module and the build: `backed-up`, `sent`, `applied` or `failed`, and `silent` when no report came within
|
|
|
|
|
ApplySilentAfter (fifteen minutes). They are read from the plan's key, written by the controller from the
|
|
|
|
|
nodes' reports, never inferred from time.
|
|
|
|
|
|
|
|
|
|
**What needs no person ends by itself** (ADR 0270):
|
|
|
|
|
## Order without a person
|
|
|
|
|
|
|
|
|
|
- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and
|
|
|
|
|
sends nowhere) is `delivered` as soon as it is merged, whatever its check said;
|
|
|
|
|
- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine
|
|
|
|
|
running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the
|
|
|
|
|
forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge
|
|
|
|
|
order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery`
|
|
|
|
|
says `covered: a later merge that contains it was delivered` as a success. A module that runs on one
|
|
|
|
|
machine only is covered too: its rollout records the empty rest as sent once its first machine passes
|
|
|
|
|
(ADR 0270, as corrected 2026-10-12);
|
|
|
|
|
- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends,
|
|
|
|
|
whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads
|
|
|
|
|
that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and
|
|
|
|
|
may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such
|
|
|
|
|
module with why, "its build source holds none of the changed files", and not in its modules, which keep
|
|
|
|
|
meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered,
|
|
|
|
|
saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below.
|
|
|
|
|
Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's
|
|
|
|
|
`TestAMergeWhoseModulesMatchWhatRunsIsDelivered`;
|
|
|
|
|
- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and
|
|
|
|
|
is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after
|
|
|
|
|
its stop` with the covering merge, once, as a success. No state or transition is added (issue 480);
|
|
|
|
|
- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes.
|
|
|
|
|
*ADR 0239 §11 and §12, ADR 0249, ADR 0402.*
|
|
|
|
|
|
|
|
|
|
**History is retired, not held** (ADR 0250).
|
|
|
|
|
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout
|
|
|
|
|
of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes
|
|
|
|
|
to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that
|
|
|
|
|
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout
|
|
|
|
|
starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a
|
|
|
|
|
build registered before its turn would be carried by somebody else's send.
|
|
|
|
|
A delivery group's order is a table of pairs, each with the rule that won (ADR 0249 §3). The controller's
|
|
|
|
|
`delivery-order` infers it from the manifests and the module graph; a person writes an `after: <repository>`
|
|
|
|
|
line only where the rules cannot see the order.
|
|
|
|
|
|
|
|
|
|
**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the
|
|
|
|
|
module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy
|
|
|
|
|
readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the
|
|
|
|
|
controller's `plan-moved` event, never inferred from time.
|
|
|
|
|
| Rank (highest first) | Rule | Read from |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| 1 | *declared*: an `after:` line | the member's description |
|
|
|
|
|
| 2 | *built by* and *version skew* | the module graph and the change |
|
|
|
|
|
| 2 | *verb before caller*: a member adding a verb to a seat's served verbs before a member adding that verb to an invoked verb | the manifests (ADR 0402) |
|
|
|
|
|
| 2 | *SDK before its users*: a member changing `mesh-sdk` before every member whose modules are built on it | the module graph (ADR 0402) |
|
|
|
|
|
| 3 | *engine before controller* | module names |
|
|
|
|
|
| 4 | the repositories' names | — |
|
|
|
|
|
|
|
|
|
|
Rules of one rank in opposite directions reject the group, naming both members, both rules and the line that
|
|
|
|
|
would settle it (ADR 0249 §4).
|
|
|
|
|
|
|
|
|
|
**The SDK by version.** A Go module names the SDK by its released version and fetches it from the mesh's forge,
|
|
|
|
|
checked against its recorded checksum; vendored copies go. Inside a group, a member may name the SDK member's
|
|
|
|
|
head: its merge check resolves that commit from the forge and builds against it, and the head stays fetchable
|
|
|
|
|
after the group merges because the forge keeps every pull request's head. The catalogue's check accepts an SDK
|
|
|
|
|
commit on the SDK's trunk or the head of an SDK pull request the forge holds, open or merged (ADR 0402 §2 to §4).
|
|
|
|
|
|
|
|
|
|
## Checks
|
|
|
|
|
|
|
|
|
|
### One check per pull request, kept by the bus
|
|
|
|
|
|
|
|
|
|
*ADR 0323, ADR 0350, ADR 0395, ADR 0406.*
|
|
|
|
|
|
|
|
|
|
- **Three work queues, taken in order.** Delivery builds go on a work-queue stream of their own; checks of pull
|
|
|
|
|
requests marked ready on a second; every other check on a third. A builder takes from the first, then the
|
|
|
|
|
second, then the third. Each check stream has one subject per pull request and keeps only the newest request:
|
|
|
|
|
a newer request replaces one still waiting, by the stream's rule. A ready mark publishes its pull request's
|
|
|
|
|
check request on the ready stream; a request left behind on the third for a head and trunk commit already
|
|
|
|
|
judged or running is acknowledged and dropped when taken, by reading the pull request's key. Nothing pauses the
|
|
|
|
|
worker or deletes and republishes a request (issue 558).
|
|
|
|
|
- **A running check stops itself.** It watches its pull request's key and stops when the head or the trunk
|
|
|
|
|
commit in it moves, without asking the forge (issue 726). An ended check delivers no verdict, so none of its
|
|
|
|
|
builds is reused.
|
|
|
|
|
- **A failed builder loses nothing.** A job a builder took and never acknowledged goes to another builder
|
|
|
|
|
after its wait.
|
|
|
|
|
- **A change is built once** (ADR 0350). A check that passes also builds what its merge would move into the
|
|
|
|
|
artifact store's scratch namespace under a reuse key, and the plan of the merge commit copies the build of the
|
|
|
|
|
same key, named by the check's verdict, instead of building.
|
|
|
|
|
|
|
|
|
|
### A move of the trunk asks every check again
|
|
|
|
|
|
|
|
|
|
*ADR 0395.*
|
|
|
|
|
|
|
|
|
|
- A trunk move asks the check of every active open pull request into that trunk that merges cleanly, against the
|
|
|
|
|
new head; a conflict is told once on the pull request, not checked.
|
|
|
|
|
- A pull request with no push and no ready mark for 7 days is left out; a push, a ready mark or a recheck asks
|
|
|
|
|
its check.
|
|
|
|
|
- For a group, a move asks each member's own check, and the composed check runs again through those rechecks; a
|
|
|
|
|
move made by a member merging in the group's order asks the other members nothing.
|
|
|
|
|
- Its cost is at most one check per active, cleanly merging pull request per trunk move; issue 944 measures it.
|
|
|
|
|
|
|
|
|
|
### A delivery group's checks
|
|
|
|
|
|
|
|
|
|
*ADR 0239 §14, ADR 0323 §6, ADR 0382, ADR 0403.*
|
|
|
|
|
|
|
|
|
|
- **The composed check is each member's gate.** The group's composed check composes every machine with all
|
|
|
|
|
members' heads, each merged into its own trunk's head first, judged by the controller built from the group's
|
|
|
|
|
controller member when it has one (ADR 0382 §3). Its verdict sets `mesh/delivery-group` and each member's
|
|
|
|
|
`mesh/merge-gate`, and names every member it read. No gate runs for a member alone.
|
|
|
|
|
- **Each member's own repository check stays required.** Each member's `mesh/repo-check` runs its repository's
|
|
|
|
|
suite with the siblings read beside it (ADR 0382 §2).
|
|
|
|
|
- **No composed verdict, no gate.** A member's gate is pending while no composed verdict at the members' current
|
|
|
|
|
heads exists. A member's new head, close or base move puts the composed verdict out of date, and with it every
|
|
|
|
|
member's gate. When mesh-delivery cannot say the groups, the pull request is checked alone and its verdict
|
|
|
|
|
says so (ADR 0382 §1).
|
|
|
|
|
- **A member merges only with its group passed and reviewed** (ADR 0382 §6), and in the group's order
|
|
|
|
|
(ADR 0364 §8).
|
|
|
|
|
|
|
|
|
|
## Ready marks and the merge
|
|
|
|
|
|
|
|
|
|
*ADR 0364, ADR 0406 §1.*
|
|
|
|
|
|
|
|
|
|
The independent reviewer marks a pull request ready at the reviewed head with the label `mesh/ready` and a
|
|
|
|
|
comment `ready at <full sha>`, through `mesh-delivery.mark-ready`. The forge's `labeled` and `commented` events
|
|
|
|
|
reach mesh-delivery in seconds; once both required checks passed against the trunk's head, it asks the operator's
|
|
|
|
|
Approve for a merge into main, with the pull request, its issues, the reviewer's verdict and the delivery plan, and
|
|
|
|
|
merges the marked head through the forge's merge tool on that Approve (ADR 0398 §3). When the trunk moved, the merge tool asks the fresh check, and the event
|
|
|
|
|
of its passing verdict asks the merge again; no timer asks it. A push clears the mark (the `synchronized`
|
|
|
|
|
event), and a failed check is said once and listed by `stalled`.
|
|
|
|
|
|
|
|
|
|
## The merge window and the batch
|
|
|
|
|
|
|
|
|
|
*2026-10-10, by ADR 0276.*
|
|
|
|
|
*ADR 0276, ADR 0282, ADR 0299, ADR 0304, ADR 0407.*
|
|
|
|
|
|
|
|
|
|
- **The window.** The controller puts every merge into a branch a module follows into the open **batch**.
|
|
|
|
|
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
|
|
|
|
|
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
|
|
|
|
|
merges and its times are kept in the controller's store and survive a restart.
|
|
|
|
|
- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for
|
|
|
|
|
every commit in it, tiered along the module graph across repositories, with at most one commit per
|
|
|
|
|
repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the
|
|
|
|
|
earlier merges of a repository that its later commit contains.
|
|
|
|
|
- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started
|
|
|
|
|
is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new
|
|
|
|
|
rollout, which answers every merge the folded one answered.
|
|
|
|
|
- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller,
|
|
|
|
|
the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for
|
|
|
|
|
mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its
|
|
|
|
|
rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again
|
|
|
|
|
behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10).
|
|
|
|
|
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is
|
|
|
|
|
answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends
|
|
|
|
|
`covered` at once.
|
|
|
|
|
- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for
|
|
|
|
|
an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own
|
|
|
|
|
commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own
|
|
|
|
|
commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose
|
|
|
|
|
merge it rollouts again nowhere (below).
|
|
|
|
|
- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's
|
|
|
|
|
tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is
|
|
|
|
|
at its turn.
|
|
|
|
|
- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a
|
|
|
|
|
folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting
|
|
|
|
|
rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
|
|
|
|
|
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
|
|
|
|
|
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
|
|
|
|
|
and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as
|
|
|
|
|
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
|
|
|
|
|
delivery it holds.
|
|
|
|
|
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch
|
|
|
|
|
waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout
|
|
|
|
|
answers within ten minutes is held with the true reason (the open rollouts of its repository and that none
|
|
|
|
|
names its merge, or that the controller holds none) and listed by `stalled` at once.
|
|
|
|
|
- **Every merge is recorded at once.** The forge's merged event records a merge into the open batch of what it
|
|
|
|
|
touches, or opens one, by whichever controller holds the plans. A merge is never refused, held or dropped,
|
|
|
|
|
and needs no person's release (issue 921). The bus's durable consumer keeps every merge event until it is
|
|
|
|
|
recorded; no catch-up timer looks for missed merges.
|
|
|
|
|
- **The window is two keys.** An open batch's window is a key whose time to live is `merge-window` (90 s),
|
|
|
|
|
refreshed by each merge into the batch, beside a key set once when the batch opens whose time to live is
|
|
|
|
|
`merge-window-at-most` (10 minutes). The first to expire closes the batch, heard by a watcher; no cutter
|
|
|
|
|
polls. A stop that records merges again into the open batch closes its window too (ADR 0299 §3).
|
|
|
|
|
- **A lone merge is cut at once**: no batch it shares a repository, module, graph edge or delivery group member
|
|
|
|
|
with is assembling (ADR 0282 §2, ADR 0304 §1, ADR 0306).
|
|
|
|
|
- **The plan.** A closed batch becomes one delivery plan: every commit in it, tiered along the module graph
|
|
|
|
|
across repositories, at most one commit per repository, the latest of its trunk. It answers every merge in the
|
|
|
|
|
batch, its own commits and the earlier merges a later commit contains (ADR 0276 §2, as ADR 0299 §7 changed it).
|
|
|
|
|
- **A plan fixes its commits when it starts.** A plan starts at its first send, after mesh-delivery's word, and
|
|
|
|
|
from then nothing that merges changes it. A plan still waiting for its word is folded into the next batch
|
|
|
|
|
(ADR 0276 §4).
|
|
|
|
|
- **Plans that share something run one after another.** A batch that shares a repository, a module or a graph
|
|
|
|
|
edge with an open plan waits for it, and starts with every merge recorded since when that plan ends. A batch
|
|
|
|
|
that shares none runs beside the open ones, so a hotfix is never held by an unrelated feature's delivery.
|
|
|
|
|
- **The ends.** A plan done ends every delivery it answers: `delivered` for its own commits, `covered` for an
|
|
|
|
|
earlier merge it carried by a later commit. A plan that failed fails its own commits' deliveries; each earlier
|
|
|
|
|
merge it answered is then delivered on its own commit, the newest first, until one is delivered (ADR 0276 §3).
|
|
|
|
|
- **Shown.** `plans` lists the open batch first, with its window's time left and its merges, and a batch waiting
|
|
|
|
|
behind an open plan names that plan; the controller says the batch as `plan-moved` in the state `assembling`.
|
|
|
|
|
|
|
|
|
|
## A backup before the send
|
|
|
|
|
|
|
|
|
|
## A stop on a shared rollout
|
|
|
|
|
*ADR 0214 §7, ADR 0233, ADR 0404.*
|
|
|
|
|
|
|
|
|
|
*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.*
|
|
|
|
|
When a delivery plan is ready to send, each node it reaches backs up, through its `node-backup` seat, every item
|
|
|
|
|
the plan's moved modules keep there whose protection is a backup (a copy or a dump). An item protected by
|
|
|
|
|
`redundancy`, one that says `backup: none`, and a cache are never asked, whatever their class. A node with no backup holder backs up
|
|
|
|
|
nothing, and the plan says so for that node.
|
|
|
|
|
|
|
|
|
|
One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other
|
|
|
|
|
delivery on it. A stop therefore ends only the change it is for:
|
|
|
|
|
- **The send waits for every backup.** The controller sends nothing of the plan to any node until every backup
|
|
|
|
|
it asked has answered done.
|
|
|
|
|
- **A failed backup holds the delivery.** A backup that fails, or does not answer within ten minutes, holds the
|
|
|
|
|
delivery with the node, the module and the backup's own words as the reason, and raises a condition for the
|
|
|
|
|
operator. A person's `release` asks the backups again, and the send goes on when they pass.
|
|
|
|
|
- **What it costs.** About 85% of data modules back up in under 30 s; the slowest measured, postgres on the home
|
|
|
|
|
server, about 165 s (research 055, finding 5). A plan moving no backed-up item costs nothing more.
|
|
|
|
|
|
|
|
|
|
## The send and the rollout
|
|
|
|
|
|
|
|
|
|
*ADR 0360, ADR 0362, ADR 0377, ADR 0406 §5.*
|
|
|
|
|
|
|
|
|
|
- **Every machine at once.** A plan's builds go to every machine that runs them in one step, tier by tier; a tier
|
|
|
|
|
is asked once the tier before it is built and sent (ADR 0360 §1, §6). No machine goes first and nothing waits
|
|
|
|
|
for a health reading. A module of policy `record` that would recreate a container or move data waits for a
|
|
|
|
|
person (ADR 0377).
|
|
|
|
|
- **Every move is said.** The controller says every move of a plan as `plan-moved`, a move a command makes among
|
|
|
|
|
them, so mesh-delivery needs no periodic reconcile; it reads `delivery-walks` once when it starts.
|
|
|
|
|
- **A rollout advances on events.** The `built` and sent events of a tier ask the next; the nodes' reports end
|
|
|
|
|
the rollout (ADR 0360 §7, with ADR 0362's provisioning report). Each send puts a key whose time to live is
|
|
|
|
|
ApplySilentAfter (fifteen minutes); its expiry names the machine silent. There is no rollout ticker.
|
|
|
|
|
- **A failed apply is loud and fixed forward** (ADR 0360 §8); the node-engine's witness still puts back a
|
|
|
|
|
controller or tool runner that does not come up (ADR 0360 §9).
|
|
|
|
|
|
|
|
|
|
## A stop on a shared plan
|
|
|
|
|
|
|
|
|
|
*ADR 0299.*
|
|
|
|
|
|
|
|
|
|
One plan carries every merge of its batch, so a stop ends only the change it is for:
|
|
|
|
|
|
|
|
|
|
- **A group's stop.** When a member after a failed one is stopped and its plan is open, mesh-delivery asks
|
|
|
|
|
`delivery-stop` for that plan with the stopped members' merges to leave out. The controller ends the plan and
|
|
|
|
|
records every other merge it answered again in the open batch of its kind, whose window the stop closes.
|
|
|
|
|
- **What is left out with it.** A later merge of the stopped merge's repository, and a merge that moves a
|
|
|
|
|
module a left-out merge moves, would deliver the stopped change; each is left out too, its delivery `stopped`,
|
|
|
|
|
naming why.
|
|
|
|
|
- **A person's stop.** `stop` of a delivery whose plan carries others first answers what it would take down, and
|
|
|
|
|
acts on `way`: `again` (the others delivered again without it) or `walk` (the whole plan stopped).
|
|
|
|
|
- **What it does not do.** What the stopped plan already sent stays on the machines it reached; putting it back
|
|
|
|
|
is issue 509's.
|
|
|
|
|
|
|
|
|
|
- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks
|
|
|
|
|
`delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass
|
|
|
|
|
stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge
|
|
|
|
|
it answered again.
|
|
|
|
|
- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A
|
|
|
|
|
module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too.
|
|
|
|
|
Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped
|
|
|
|
|
merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is
|
|
|
|
|
rolled out again.
|
|
|
|
|
- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran
|
|
|
|
|
already wait, so an older commit never rollouts after a newer one; only with none open do they form a new
|
|
|
|
|
batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop
|
|
|
|
|
before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules
|
|
|
|
|
where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes
|
|
|
|
|
to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every
|
|
|
|
|
merge moved are written in one transaction.
|
|
|
|
|
- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left
|
|
|
|
|
out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery
|
|
|
|
|
whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays
|
|
|
|
|
`published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every
|
|
|
|
|
delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop.
|
|
|
|
|
- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped
|
|
|
|
|
delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it
|
|
|
|
|
stops the whole rollout.
|
|
|
|
|
- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it
|
|
|
|
|
answers with each of them, which of them contain this merge and end either way, and two ways given with
|
|
|
|
|
`way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout
|
|
|
|
|
stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is
|
|
|
|
|
marked stopped only once its rollout is; a stop the controller refuses leaves it as it was.
|
|
|
|
|
- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its
|
|
|
|
|
asked builds still register, so a later send may carry them further; the stop names the machines, and
|
|
|
|
|
putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the
|
|
|
|
|
stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it
|
|
|
|
|
can see, and holding a merged change back for good is a revert's job.
|
|
|
|
|
## The delivery budget
|
|
|
|
|
|
|
|
|
|
*Decided 2026-10-10 by ADR 0282.
|
|
|
|
|
Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by
|
|
|
|
|
slice as each slice below is built; until a slice is built, the text above is what runs.*
|
|
|
|
|
*ADR 0282, ADR 0304 §4, ADR 0313, ADR 0360 §12, ADR 0362 §5.*
|
|
|
|
|
|
|
|
|
|
**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard
|
|
|
|
|
from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes;
|
|
|
|
|
a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is
|
|
|
|
|
shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes
|
|
|
|
|
(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers
|
|
|
|
|
14.2 (issue 338).
|
|
|
|
|
A delivery's **delivery time** runs from its merge until every machine its plan sent has reported the send
|
|
|
|
|
applied, and for a provisioned module also placed. A **leaf** delivery's budget is three minutes, a **core**
|
|
|
|
|
delivery's ten. A delivery held for a person, a failed backup among them, is shown and held to no budget. Each
|
|
|
|
|
phase is kept — the window, the wait behind a plan that shares something, the backups, the build, the send, the
|
|
|
|
|
reports — and a delivery over its budget raises `delivery.<class>.over-budget`, naming its longest phase. The
|
|
|
|
|
**fix time** (ADR 0313) runs from an issue's first move to `fixing` to its last fix delivered, and contains the
|
|
|
|
|
delivery time.
|
|
|
|
|
|
|
|
|
|
*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget
|
|
|
|
|
above runs from the merge; the operator's own target is wider, the **fix time**, which runs from an issue's
|
|
|
|
|
first move to `fixing` to the moment its last fix is delivered. Each issue's class carries a **fix budget** too
|
|
|
|
|
(five minutes leaf, ten core, the delivery budget's own figures), and the tracker splits a fix time into four
|
|
|
|
|
phases — implementing, review, checks, and merge to running (the delivery time above) — so a fix over its
|
|
|
|
|
budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget.
|
|
|
|
|
The delivery budget stays as written: a measure inside the fix budget, not replaced by it.
|
|
|
|
|
## Builds without a toolchain image
|
|
|
|
|
|
|
|
|
|
**What changes in the rollout.**
|
|
|
|
|
*ADR 0267, ADR 0350, ADR 0405.*
|
|
|
|
|
|
|
|
|
|
- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged.
|
|
|
|
|
- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One
|
|
|
|
|
node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf
|
|
|
|
|
build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged.
|
|
|
|
|
- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf
|
|
|
|
|
module with a long-running resource; one for a leaf module whose every resource is a tools bundle.
|
|
|
|
|
- **A tier's first nodes are judged at once.**
|
|
|
|
|
- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new
|
|
|
|
|
tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before
|
|
|
|
|
anything is judged; a step that would carry anything else is refused unsent and said on the gate line.
|
|
|
|
|
- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy
|
|
|
|
|
report; the plan shows an expected landing time from the last seven days' medians.
|
|
|
|
|
- **One build module per language** (Go; Node and TypeScript) installs its toolchain as a declared package on
|
|
|
|
|
every node holding the build seat, pinned to one version that is the same on every builder. The toolchain
|
|
|
|
|
image and its module retire.
|
|
|
|
|
- **Every builder runs the same toolchain, and says so.** Each builder names its toolchain versions with every
|
|
|
|
|
build outcome, and the controller's self-check raises a condition while two builders differ.
|
|
|
|
|
- **Builds and tests run directly**, as the build seat's own account on its node, in no container. A
|
|
|
|
|
repository's `# mesh-check-toolchain` line selects which installed toolchain runs its check.
|
|
|
|
|
- **A judge has an account of its own.** A judge built from a sibling's merge (ADR 0382 §3) is built and run as a
|
|
|
|
|
second account of the Go build module, whose home, configuration and cache the build seat's account cannot write,
|
|
|
|
|
so a check's script cannot change it.
|
|
|
|
|
- **Every dependency comes by the version its lock file names**, checked against its checksum: the mesh's own
|
|
|
|
|
code from the mesh's forge, its own packages from its package registry, any other from its public source.
|
|
|
|
|
- **A builder fetches from the forge over SSH with a key of its own**, as the build seat's account, under a forge
|
|
|
|
|
account for that builder that reads the mesh's repositories and pushes nothing.
|
|
|
|
|
- **A build's reuse key** holds the installed toolchain's version and the build module's recipe version in place
|
|
|
|
|
of the toolchain image's digest, so a toolchain update rebuilds only the modules of that language.
|
|
|
|
|
|
|
|
|
|
*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
|
|
|
|
|
a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it:
|
|
|
|
|
## One chain, on the bus alone
|
|
|
|
|
|
|
|
|
|
- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open
|
|
|
|
|
rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
|
|
|
|
|
judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open
|
|
|
|
|
rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
|
|
|
|
|
for builds an open rollout owns).
|
|
|
|
|
- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the
|
|
|
|
|
merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order.
|
|
|
|
|
- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or
|
|
|
|
|
send, the dependents of what its merges moved included; a change whose modules are not known yet is related to
|
|
|
|
|
every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node.
|
|
|
|
|
- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a
|
|
|
|
|
warning naming the rollout and the core rollouts that set it aside.
|
|
|
|
|
- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an
|
|
|
|
|
unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its
|
|
|
|
|
own builds.
|
|
|
|
|
- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by
|
|
|
|
|
issue 528.
|
|
|
|
|
*ADR 0408.*
|
|
|
|
|
|
|
|
|
|
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out
|
|
|
|
|
of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
|
|
|
|
|
overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every
|
|
|
|
|
open judgement on its node.
|
|
|
|
|
Module, commit, build, published package or artifact, rollout and assignment are one chain, each link
|
|
|
|
|
answerable from the next. A **rollout** here is the controller's record of one batch's delivery; a delivery's own
|
|
|
|
|
delivery plan (*A delivery* above) stays the delivery's, and names the rollout that delivers it.
|
|
|
|
|
|
|
|
|
|
*A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`,
|
|
|
|
|
a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge
|
|
|
|
|
into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace
|
|
|
|
|
such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once
|
|
|
|
|
it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy
|
|
|
|
|
container there, its first-node gate starts at the report that says the container replaced, and the wait is a
|
|
|
|
|
phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget
|
|
|
|
|
is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a
|
|
|
|
|
rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the
|
|
|
|
|
node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the
|
|
|
|
|
judgement.
|
|
|
|
|
| Link | Names | Written by | Bucket |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| build | its module, repository and commit, its reuse key, what it published: an artifact by digest or a package release by name and version | the controller | `mesh-controller_builds` |
|
|
|
|
|
| rollout | the commit it fixed per repository, its tiers, the builds it carried, the machine steps, and every assignment it changed with the build before and after; its states and transitions | the controller | `mesh-controller_rollouts` |
|
|
|
|
|
| assignment | the module and node, the build, artifact or package and commit it runs, and the rollout that put it there | the controller | `mesh-controller_assignments` |
|
|
|
|
|
|
|
|
|
|
**Who builds what.**
|
|
|
|
|
Beside the chain, on the bus too: deliveries and groups (mesh-delivery's `deliveries` and `groups`, ADR 0239 §37),
|
|
|
|
|
one key per open pull request (mesh-delivery), and each open batch with its merges and its window's two keys (the
|
|
|
|
|
controller, ADR 0407).
|
|
|
|
|
|
|
|
|
|
| Part | Owner | What it adds |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
|
|
|
|
|
| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` |
|
|
|
|
|
| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery.<class>.over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` |
|
|
|
|
|
| the lone merge | mesh-controller | the cut at once when nothing waits to join |
|
|
|
|
|
| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch |
|
|
|
|
|
| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once |
|
|
|
|
|
| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` |
|
|
|
|
|
|
|
|
|
|
**Slices**, each one pull request per repository, each with its acceptance sentence:
|
|
|
|
|
|
|
|
|
|
| Slice | Delivers | Done when |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
|
|
|
|
|
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases |
|
|
|
|
|
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout |
|
|
|
|
|
| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
|
|
|
|
|
| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median |
|
|
|
|
|
|
|
|
|
|
Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges
|
|
|
|
|
only once it is built.
|
|
|
|
|
|
|
|
|
|
## A delivery group
|
|
|
|
|
|
|
|
|
|
- **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull
|
|
|
|
|
requests. Two or more make a group. A group that has started delivering takes no new member.
|
|
|
|
|
- **Order**: `after: <repository>` lines in a member's description, and the controller's `delivery-order`
|
|
|
|
|
over the graph (ADR 0239 decision 4). The plan shows the order and why each pair is ordered: *declared*,
|
|
|
|
|
*built by*, *version skew*, *engine before controller*, *by name*.
|
|
|
|
|
- **Precedence** (ADR 0249): when rules order one pair both ways, the higher one decides. *Declared* comes
|
|
|
|
|
first, then *built by* and *version skew*, then *engine before controller*. The pair shows the rule that
|
|
|
|
|
won and the rules it won over. Rules of one rank both ways are a contradiction. The group is `rejected`,
|
|
|
|
|
and the reason names both members, both rules and the `after:` line that would settle it.
|
|
|
|
|
- **Check**: the members' heads composed together, one gate over every machine with all their definitions,
|
|
|
|
|
each head merged into its own trunk's head first and named with that trunk commit (ADR 0323). It is asked
|
|
|
|
|
by mesh-delivery through `delivery-check` when the group forms or a member's head moves, and posted on
|
|
|
|
|
every member's head as `mesh/delivery-group`. A member rebased or pushed is a new head with its own check:
|
|
|
|
|
it takes its old head's place in the group, even once the group is delivering, and the composed check runs
|
|
|
|
|
again for the new heads instead of holding the group. A merge into a member's trunk that is not one of the
|
|
|
|
|
group's members puts the composed check out of date: the group reads `checking` with the reason, and the
|
|
|
|
|
composed check runs again once a member is judged again. Checked by mesh-delivery's
|
|
|
|
|
`TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember` and `TestARebasedMemberIsComposedAgainNotHeld`.
|
|
|
|
|
- **Each member's own check reads its siblings** (ADR 0382): the group's other open members into a base of
|
|
|
|
|
the same name are read beside a member's own check at their head merged onto their base, and a mesh-controller
|
|
|
|
|
sibling's merge judges it, so changes that are only consistent together pass their own checks. The controller
|
|
|
|
|
learns the siblings from mesh-delivery's `groups`; the verdict names each one. A sibling's new head, close or
|
|
|
|
|
base move puts that verdict out of date, and the merge tool merges such a member only when
|
|
|
|
|
`mesh/delivery-group` passed at the heads read and every sibling head read is reviewed.
|
|
|
|
|
- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`.
|
|
|
|
|
A member published while its group's composed check is asked and not answered waits there for the answer;
|
|
|
|
|
one held because that check had not passed is *lifted* back to `published` once it passes or warns for the
|
|
|
|
|
heads that merged and its own rollout is still open (ADR 0270).
|
|
|
|
|
- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its
|
|
|
|
|
rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered.
|
|
|
|
|
Its own builds were put back at its gate.
|
|
|
|
|
|
|
|
|
|
A group's state is derived from its members and its composed check, and is shown, never stored apart from
|
|
|
|
|
them.
|
|
|
|
|
|
|
|
|
|
## A move of the trunk asks every check again
|
|
|
|
|
|
|
|
|
|
*ADR 0395, changing ADR 0323 §4.* When the forge's holder hears a repository's trunk move, mesh-delivery asks the
|
|
|
|
|
check of every active open pull request into that trunk against its new head at once, so a verdict is current by
|
|
|
|
|
the time anyone looks.
|
|
|
|
|
|
|
|
|
|
- **A conflict is told, not checked.** Before asking a check, mesh-delivery asks the forge whether the head merges
|
|
|
|
|
cleanly into the new head, which needs no builder. When it does not, no check is asked: its statuses fail naming the trunk commit and the files in
|
|
|
|
|
conflict where the forge gives them, and it is told once on the pull request, not again while the same conflict
|
|
|
|
|
stands. Every later trunk move and its next push ask the forge again; once it merges cleanly its check is asked.
|
|
|
|
|
- **One check per pull request.** When a check against a newer trunk commit is asked for a pull request, its check
|
|
|
|
|
against an older trunk commit is dropped if it waits and ended if it runs, said in the build log and on the
|
|
|
|
|
queue, as a new head already ends its old head's check (issue 726). An ended check delivers no verdict, so none of its builds is reused (ADR 0350 §7 keeps only builds a verdict
|
|
|
|
|
names); its scratch records wait for ADR 0350 §5's seven-day sweep. A try and a delivery's build are never touched.
|
|
|
|
|
- **The queue's order.** Delivery builds first; then checks of pull requests marked ready; then the rest, the most
|
|
|
|
|
recently pushed first.
|
|
|
|
|
- **Inactive pull requests are left out.** One with no push and no ready mark for 7 days is not asked again on a
|
|
|
|
|
trunk move; its statuses say out of date and that a push, a ready mark or a recheck asks its check. A recheck
|
|
|
|
|
takes a delivery that is still proposed and whose verdict is out of date, and asks its check (issue 888).
|
|
|
|
|
- **Groups.** For a group's members a move asks only each member's own check; the group's composed check runs again
|
|
|
|
|
through those rechecks (ADR 0323 §6) and is never asked by the move itself. A move made by a member merging in
|
|
|
|
|
the group's order asks the other members nothing, since they merge moments later; their verdicts are out of date
|
|
|
|
|
until then, and one that does not merge is asked again at the next move, its next push or a recheck.
|
|
|
|
|
|
|
|
|
|
**Its cost** is at most one check asked per active, cleanly merging pull request per trunk move: about 550 a day at
|
|
|
|
|
ADR 0323's measured load, fewer once conflicting and inactive ones are left out; ending an older check saves builder
|
|
|
|
|
time, not checks asked. Issue 944 measures it once built.
|
|
|
|
|
- **The bus is the only source.** Each bucket is written by the controller that holds the lease, by
|
|
|
|
|
compare-and-set, and the controller reads its own buckets to decide; a rollout resumes from its key after a
|
|
|
|
|
restart. No table of the controller's store holds builds, rollouts, batches or assignments, so no record mirror,
|
|
|
|
|
audit or healer exists for them; the nodes, settings, declared-config and predefined-keys mirrors stay as
|
|
|
|
|
ADR 0289 and ADR 0356 have them. A module reads a bucket by naming it in `reads`; mesh-config reads the
|
|
|
|
|
assignments bucket.
|
|
|
|
|
- **No key of the chain is removed.** Each bucket keeps up to 64 earlier revisions of each key as its history. A
|
|
|
|
|
value is never cut to a summary and stays under the bus's largest message. The bus's storage is declared as data
|
|
|
|
|
protected by a backup, so it is backed up nightly and before a delivery moves the bus.
|
|
|
|
|
- **One read answers each question.** For any module on any node, one read of its assignment names the commit,
|
|
|
|
|
the build, the artifact or package and the rollout that put it there; one read of a rollout lists every
|
|
|
|
|
assignment it changed.
|
|
|
|
|
- **The tables go by a numbered migration**: builds, rollouts and assignments are written once into the buckets,
|
|
|
|
|
waiting build requests onto the work queues and an open batch into its keys; then the tables are dropped from the
|
|
|
|
|
controller's store and its consolidated schema.
|
|
|
|
|
- **The artifact store and the package registry keep the bytes**; the chain names them (ADR 0251).
|
|
|
|
|
|
|
|
|
|
## What every transition does
|
|
|
|
|
|
|
|
|
|
In this order, each by its owner:
|
|
|
|
|
|
|
|
|
|
1. **kept**: mesh-delivery puts the delivery in its state (`deliveries`, one key per delivery; `groups`, one
|
|
|
|
|
key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried, later
|
|
|
|
|
each time, from ten seconds doubling to an hour. The forge's answer that what it is for does not exist ends
|
|
|
|
|
it at once, and sixty failed tries end it too, each said among the owner's refusals (ADR 0250);
|
|
|
|
|
1. **kept**: mesh-delivery puts the delivery in its state by compare-and-set before anything else; what it owes
|
|
|
|
|
outside is kept with it and retried from ten seconds doubling to an hour, at most sixty times (ADR 0250);
|
|
|
|
|
2. **said**: the event `mesh-delivery.transition` (and `mesh-delivery.group` when a group's derived state
|
|
|
|
|
changes), with the delivery's id, from, to, event, why and plan summary, and no secret or address;
|
|
|
|
|
3. **noted**: mesh-delivery asks the forge's holder (`gitea_note_append`) to append a line to the commit's
|
|
|
|
|
note: the head while the delivery is off the trunk, the commit it landed as once it is on it, and at its
|
|
|
|
|
end a line of what was executed there. The forge's holder writes it in its own repository as the forge's
|
|
|
|
|
own account, and a line already there is not appended again;
|
|
|
|
|
4. **reflected** (not for history, whose pull request the forge may no longer hold): the delivery's view, one comment on the pull request with the state, the plan, the machine
|
|
|
|
|
steps and the group, kept current (`gitea_delivery_view`); the status `mesh/delivery` on the head, and
|
|
|
|
|
`mesh/delivery-group` on every member's head (`gitea_commit_status`). Every status links to the pull
|
|
|
|
|
request, `mesh/merge-gate` among them.
|
|
|
|
|
changes), with no secret or address;
|
|
|
|
|
3. **noted**: a line on the commit's note, appended by the forge module as the forge's own account;
|
|
|
|
|
4. **reflected**: the delivery's view on the pull request, `mesh/delivery` on the head and `mesh/delivery-group`
|
|
|
|
|
on every member's head.
|
|
|
|
|
|
|
|
|
|
A doubled step is refused by compare-and-set and by the bus's message de-duplication, which also replaces the
|
|
|
|
|
forge module's files of what it announced (ADR 0406 §6).
|
|
|
|
|
|
|
|
|
|
## The verbs
|
|
|
|
|
|
|
|
|
|
**mesh-delivery's seat** (served by its holder; the first five read):
|
|
|
|
|
**mesh-delivery's seat**:
|
|
|
|
|
|
|
|
|
|
| Verb | Answers or does |
|
|
|
|
|
|---|---|
|
|
|
|
|
| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group |
|
|
|
|
|
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout |
|
|
|
|
|
| `deliveries` | every delivery not final, the final ones of the last day, and every one `stalled` still lists |
|
|
|
|
|
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the chain's links |
|
|
|
|
|
| `groups` | every group with its members in order and its derived state |
|
|
|
|
|
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
|
|
|
|
|
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner |
|
|
|
|
|
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
|
|
|
|
|
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds |
|
|
|
|
|
| `recheck` | a rejected or ready delivery back to proposed, with why; a delivery still proposed whose verdict is out of date has its check asked (ADR 0395) |
|
|
|
|
|
| `release` | a held delivery to delivering, with why: a person's word |
|
|
|
|
|
| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
|
|
|
|
|
| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise |
|
|
|
|
|
| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing |
|
|
|
|
|
| `stalled` | every delivery held past its bound, with the transition H2 may take |
|
|
|
|
|
| `mark-ready` | a reviewer's mark at a head: the label and the `ready at` comment, by the mesh's own forge account |
|
|
|
|
|
| `recheck` | a check asked again for a delivery whose verdict is out of date (ADR 0395 §6) |
|
|
|
|
|
| `release` | a held delivery goes on, with why; for a failed backup it asks the backups again |
|
|
|
|
|
| `stop` | a delivery to stopped, with why; on a shared plan it first answers what it would take down (ADR 0299) |
|
|
|
|
|
| `close` | H2's only verb: the transition the table names for a stalled delivery |
|
|
|
|
|
| `retire-history` | held history to retired (ADR 0250) |
|
|
|
|
|
| `times` | each class's budget, the week's median and worst delivery time, and the deliveries over budget |
|
|
|
|
|
|
|
|
|
|
**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`,
|
|
|
|
|
`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which
|
|
|
|
|
takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299),
|
|
|
|
|
`delivery-walks`. A person has `plans go <plan> --why`, which does what `deliver` does under their name and
|
|
|
|
|
is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command
|
|
|
|
|
too.
|
|
|
|
|
|
|
|
|
|
**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It
|
|
|
|
|
also announces `pull.closed`, and a merge's head and the statuses the forge holds on it.
|
|
|
|
|
|
|
|
|
|
## The store
|
|
|
|
|
|
|
|
|
|
Key-value state on the bus, declared by the module (ADR 0201): `deliveries` and `groups`. One writer, the
|
|
|
|
|
seat's holder, whose code changes state from a single goroutine. A final delivery is kept for thirty days and
|
|
|
|
|
then removed. Its note on the commit keeps its record after that. The bus's own snapshot (ADR 0235) backs it
|
|
|
|
|
up.
|
|
|
|
|
**The controller's verbs for it**: `delivery-plan`, `delivery-order`, `delivery-check`, `deliver`,
|
|
|
|
|
`delivery-stop`, `delivery-walks`, and `plans` (the open batch, the open plans, and a what-if for a repository
|
|
|
|
|
and its files). A person has `plans go <plan> --why`.
|
|
|
|
|
|
|
|
|
|
## The bootstrap
|
|
|
|
|
|
|
|
|
|
- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
|
|
|
|
|
controller's own, started by the merge and witnessed on the machine. mesh-delivery records it.
|
|
|
|
|
- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the
|
|
|
|
|
controller starts it as it always did.
|
|
|
|
|
- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises
|
|
|
|
|
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and
|
|
|
|
|
`plans go <walk> --why` as the way on.
|
|
|
|
|
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
|
|
|
|
|
- A plan that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
|
|
|
|
|
controller's own, started by the merge; mesh-delivery records it (ADR 0239 §31).
|
|
|
|
|
- Any other plan waits for `deliver` while the seat has a holder on record; with none, the controller starts it.
|
|
|
|
|
- A replaced controller or mesh-delivery carries on from the bus: the keys, the queues and the durable
|
|
|
|
|
consumers hold every step, so nothing is lost or repeated (ADR 0406 §6).
|
|
|
|
|
|
|
|
|
|
## The switch
|
|
|
|
|
## Acceptance tests
|
|
|
|
|
|
|
|
|
|
1. The controller's change (the seat, its verbs, `plan-moved`, the wait for a holder's word, `plans go`) is
|
|
|
|
|
merged and rolls out as the core does. With no holder on record it behaves exactly as before.
|
|
|
|
|
2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built
|
|
|
|
|
and published.
|
|
|
|
|
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start
|
|
|
|
|
from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in
|
|
|
|
|
`delivering`.
|
|
|
|
|
4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass.
|
|
|
|
|
What the build proves before this design is implemented (research 055's direction):
|
|
|
|
|
|
|
|
|
|
## Phases
|
|
|
|
|
1. A feature across the controller and the catalogue with a new seat verb is delivered as one group, with no
|
|
|
|
|
`after:` line, no two steps and one composed check.
|
|
|
|
|
2. An SDK change and a Go module using it are delivered as one group.
|
|
|
|
|
3. A data module is backed up before a delivery moves it, and a failed backup holds the delivery.
|
|
|
|
|
4. A build on any builder uses the installed toolchain with no container, and a toolchain update rebuilds only
|
|
|
|
|
modules of that language when its version changes.
|
|
|
|
|
5. Two merges into one repository within a minute leave each open pull request with one queued check, by the
|
|
|
|
|
stream's rule.
|
|
|
|
|
6. A ready mark is acted on within seconds of its event.
|
|
|
|
|
7. A controller replaced in the middle of a rollout resumes it with no step lost or repeated.
|
|
|
|
|
8. The merge window closes by its key's expiry.
|
|
|
|
|
9. A hotfix to a module the running feature's delivery does not touch reaches the machines in minutes, beside it.
|
|
|
|
|
10. For any module on any node, one read of its assignment names the commit, build, artifact or package and the
|
|
|
|
|
rollout that put it there, and one read of that rollout lists every assignment it changed.
|
|
|
|
|
11. Two merges into one repository while a delivery of that repository runs are both delivered by the next plan,
|
|
|
|
|
with no person's release and nothing held in mesh-delivery.
|
|
|
|
|
12. No table in the controller's store holds builds, rollouts or assignments.
|
|
|
|
|
|
|
|
|
|
| Phase | Delivers | Done when |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` |
|
|
|
|
|
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 |
|
|
|
|
|
| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
|
|
|
|
|
## What retires
|
|
|
|
|
|
|
|
|
|
- The "release branches" and the controller's judge of them (ADR 0358, superseded by ADR 0401).
|
|
|
|
|
- The toolchain image and its module (ADR 0405).
|
|
|
|
|
- The window cutter, the merge catch-up timer, the rollout ticker, mesh-delivery's tick, reconcile and
|
|
|
|
|
ready-mark poll, the forge's poll as a trigger, the queue reordering, the forge-asking cancel, and the forge
|
|
|
|
|
module's announcement files (ADR 0406, ADR 0407).
|
|
|
|
|
- The controller's tables of builds, build requests, rollouts and their tiers, batches and merges, and
|
|
|
|
|
assignments; the record mirrors, probe D17's audit and healer H6 for those kinds (ADR 0408).
|
|
|
|
|
|
|
|
|
|
## Milestones
|
|
|
|
|
|
|
|
|
|
Set by playbook 04 at the handoff, after the operator confirms them.
|
|
|
|
|
|
|
|
|
|
## What is not decided here
|
|
|
|
|
|
|
|
|
|
- A web view of deliveries beyond the pull request's comment.
|
|
|
|
|
- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge
|
|
|
|
|
module's protection tool.
|
|
|
|
|
- Moving the per-tier rollout out of the controller (Phase C).
|
|
|
|
|
- Whether `mesh/delivery-group` is required on the trunks: the operator's setting, through the forge module's
|
|
|
|
|
protection tool.
|
|
|
|
|
- How a builder's public SSH key reaches the forge: the forge's and the SSH client's mechanism.
|
|
|
|
|