Merge branch 'main' into feat/agents-write-git-ssh
mesh/merge-gate head 24701df23e8a merged into trunk dd5a06529556 beside mesh-catalog#340@ae9cda85 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 24701df23e8a merged into trunk dd5a06529556 beside mesh-catalog#340@ae9cda85 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/agents-write-git-ssh delivered: every member is delivered
mesh/merge-gate head 24701df23e8a merged into trunk dd5a06529556 beside mesh-catalog#340@ae9cda85 — pass: the change touches no module of the mesh's graph
mesh/repo-check head 24701df23e8a merged into trunk dd5a06529556 beside mesh-catalog#340@ae9cda85 — pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/agents-write-git-ssh delivered: every member is delivered
This commit is contained in:
+24
-11
@@ -219,10 +219,11 @@ every other domain has something to run on. **Recorded by** the controller and t
|
||||
(ADR 0229).
|
||||
|
||||
- **record mirror** — a key-value bucket on the bus that the controller writes from one kind of its
|
||||
records (nodes, assignments, rollouts, builds, settings), after each change commits in its store and only
|
||||
records (nodes, settings, declared config, predefined keys), after each change commits in its store and only
|
||||
when the value differs, with one event per write. The store stays the record; the self-check compares
|
||||
the two and raises `mirror-drifted` on any difference
|
||||
(ADR 0289). Shortened to *mirror* where the
|
||||
(ADR 0289, ADR 0356). Builds, rollouts and assignments are no mirrors: they are
|
||||
the chain, kept on the bus as their only source (ADR 0408). Shortened to *mirror* where the
|
||||
text is about the controller's records; the builder's *mirrored* `upstream` image (Change and delivery) is
|
||||
a different thing, a copy of an outside image in the artifact store.
|
||||
|
||||
@@ -393,7 +394,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
|
||||
**Purpose.** To take one commit from its pull request to every node that should run it, and to put it
|
||||
back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the
|
||||
planner, the rollout). **Downstream of** Module, Core, Health and repair and Data.
|
||||
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323.
|
||||
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276, 0323, 0401–0408.
|
||||
**Uses:** module, node, declaration and send (Placement), health (Health and repair), person.
|
||||
|
||||
- **merge check** — the two statuses a pull request carries before it may merge: the **merge gate**
|
||||
@@ -408,7 +409,10 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat
|
||||
**builder**. Any node may hold the seat, so there is no build node by nature
|
||||
(ADR 0237).
|
||||
*Not:* ~~build machine~~
|
||||
- **build queue** — the controller's queue of builds to run; one entry in it is a **build request**.
|
||||
- **build queue** — the builds waiting to run, kept by the bus as three work queues a builder takes in order:
|
||||
delivery builds, then checks of pull requests marked ready, then every other check; each check queue keeps one
|
||||
request per pull request, the newest (ADR 0406); one entry in it is a
|
||||
**build request**.
|
||||
The queue's verbs still say *ask* for an entry; the conversation's ask is another thing
|
||||
(see [Homonyms](#homonyms)).
|
||||
- **package** — what code resolves when it is **compiled**: an npm/cargo/pypi dependency, by **version**.
|
||||
@@ -450,9 +454,14 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat
|
||||
- **delivery group** — two or more deliveries sharing a pull request head branch name across the mesh's
|
||||
repositories, delivered as one unit in an order declared (`after:`) or inferred by the planner. One
|
||||
level: a group holds deliveries, never groups. Its state is derived from its members, never set.
|
||||
- **release branch** — a branch other than `main`, shared by name across repositories, that collects a feature's
|
||||
merged work and is brought into `main` as one delivery (design 57's `feat/57-config-and-provisioning`). While it
|
||||
is in flight, a change that touches its files waits for it (playbook 07, *Is the coast clear?*).
|
||||
- **hotfix** — one pull request in one repository, delivered at its merge (ADR 0401).
|
||||
- **feature** — one branch name across its repositories: in two or more, its pull requests are one delivery group,
|
||||
merged together and delivered member by member in the group's order; in one, a single pull request delivered as a
|
||||
lone delivery (ADR 0401). Work is never held back on a branch other than a trunk: the trunk is what runs.
|
||||
*Not:* ~~release branch~~ (hq)
|
||||
- **toolchain** — a language's compiler and tools, installed by that language's build module on every node holding
|
||||
the build seat, pinned to one version that is the same on every builder; its version is part of a build's reuse
|
||||
key (ADR 0405). No longer an image.
|
||||
- **delivery plan** — what a delivery does to the mesh: its **build plan** (the modules it moves and
|
||||
their dependents, in tiers), its **deploy plan** (per node, what it receives and what waits for a
|
||||
person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a
|
||||
@@ -460,15 +469,17 @@ planner, the rollout). **Downstream of** Module, Core, Health and repair and Dat
|
||||
*Not:* ~~change plan~~, ~~release plan~~
|
||||
- **merge window** — the rolling wait in which the controller collects merges before it plans them: each
|
||||
merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest
|
||||
10 minutes after its first merge (ADR 0276).
|
||||
10 minutes after its first merge (ADR 0276). Kept as two keys with a time to live on the bus; the first to
|
||||
expire closes the batch (ADR 0407).
|
||||
- **batch** — the merges one merge window collected, planned once as one rollout. For a repository merged more
|
||||
than once in it, the rollout takes the latest commit, and the earlier merges are answered by it. Not *group*,
|
||||
which is a delivery group.
|
||||
- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat
|
||||
of the same name. It records and decides; the controller sends, judges and rolls back when it is asked.
|
||||
- **rollout** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
|
||||
repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236,
|
||||
ADR 0276). One rollout is open at a time; it names every merge it answers. A primitive the delivering stage
|
||||
repository, tier by tier, to every node that runs each module at once (ADR 0276, ADR 0360). Rollouts that share a
|
||||
repository, a module or a graph edge run one after another; others run side by side (ADR 0407). It names every
|
||||
merge it answers, and its record is one key of the bus's rollouts bucket, a link of the chain (ADR 0408). A primitive the delivering stage
|
||||
asks for, not an object anyone manages. A **tier** is one step of a rollout: a module, then the modules
|
||||
built against it. The controller's verb `plans` lists the rollouts. A module is rolled out by its `roll`
|
||||
policy.
|
||||
@@ -732,6 +743,9 @@ Not a domain of the mesh but of the way it is built, listed because its words me
|
||||
([55 — The knowledge base](../03-DESIGN/01-to-be/55-the-knowledge-base.md)). Not **the record**, the literal reader
|
||||
of this repository.
|
||||
- **design** — a document in `03-DESIGN/`: **as-is** (what runs) or **to-be** (what is being built).
|
||||
- **milestone** — one step of a design's work breakdown, usable on its own once delivered and named by what the
|
||||
operator or the mesh can then do; delivered as one feature (playbook 04, ADR 0401).
|
||||
- **epic** — a run of milestones in one design, released milestone by milestone (ADR 0401).
|
||||
- **issue** — a question about the mesh that someone must answer, numbered and tracked by the issue tracker, which
|
||||
holds it whole: its report, diagnosis and resolution included (ADR 0291);
|
||||
the folders in `04-ISSUES/` are imported there and removed.
|
||||
@@ -784,7 +798,6 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|
||||
| release | a person letting a held delivery go on | **release** | Change and delivery |
|
||||
| | taking a consumer off a licence | **unbind** (verb `release`, to rename) | Identity and access |
|
||||
| | a copy of a work an engine can fetch | **media release** | Media |
|
||||
| | a branch brought into `main` as one delivery | **release branch** | Change and delivery |
|
||||
| gate | the judgement on a delivery's first node | **first-node gate** | Change and delivery |
|
||||
| | the pull request status | **merge gate** | Change and delivery |
|
||||
| | a failed step stopping its module's later steps (ADR 0136) | *a failed step holds its module* | Placement |
|
||||
|
||||
@@ -30,7 +30,8 @@ A design becomes **milestones**, and each milestone becomes **issues**:
|
||||
4. **Break it into milestones.** Write them into the design, in a section *Milestones*: for each, its number,
|
||||
its one-sentence outcome, the acceptance tests of the design it proves, the repositories it touches, and
|
||||
what it depends on. Order them so each is usable as soon as it is delivered, the smallest useful one
|
||||
first.
|
||||
first. A milestone is named by what it lets someone do; how it ships is shaped by the plan its changes would
|
||||
have, asked through `mesh-delivery.what-if`: the modules it moves and their fan-out (ADR 0401 §6).
|
||||
5. **Ask the operator to confirm the milestones**, in a short table: number, outcome, the acceptance tests it
|
||||
proves, repositories, the parts each repository builds. Change what they change.
|
||||
6. **Open the issues.** For each confirmed milestone, `issue-tracker.open` one issue per repository part, its
|
||||
|
||||
@@ -121,32 +121,10 @@ never into the running one, unless it blocks the running one.
|
||||
report, and the round does not wait for it. What a round asks is only what makes the change wrong against
|
||||
its issues and its decisions.
|
||||
|
||||
## Is the coast clear?
|
||||
|
||||
Before the push of phase 3 (and of a small fix, playbook [08](08-small-fixes.md) step 5), the lead checks, for every
|
||||
repository the change touches, whether a **release branch** is in flight there: a branch other than `main` that
|
||||
collects a feature's merged work to bring it into `main` as one delivery.
|
||||
|
||||
1. **Look for one.** A pull request into `main` whose head is a release branch (`gitea.gitea_open_pulls`), a
|
||||
release branch holding merged work that `main` does not have yet (`git log origin/main..origin/<branch>`), or
|
||||
a delivery plan of that release branch still running (`mesh-controller.plans`).
|
||||
2. **Compare the files.** `git diff --name-only origin/main...origin/<release branch>` against the files the
|
||||
change touches.
|
||||
3. **Overlap: hold.** If the release branch touches a file the change touches, nothing is pushed into that repository
|
||||
until the release branch's delivery plan is done. Then the feature branch is brought up to the new `main`, the unit
|
||||
tests run again, and the push follows. The hold is on the wait list (its bound: the release branch's delivery plan),
|
||||
and is said to the operator with what the change waits for.
|
||||
4. **No overlap: go on.** A release branch in another repository, or one that shares no file with the change, does not
|
||||
hold it; its merge joins the merge window as any other (ADR 0276).
|
||||
|
||||
Merging beside a release branch still in flight would make its lead resolve someone else's conflicts in the
|
||||
middle of it, and could send part of the change with its delivery (2026-10-16: the file module's
|
||||
controller part touched four files feature 57's release branch also changed).
|
||||
|
||||
## Phase 3 — Check (no review)
|
||||
|
||||
1. **Push every feature branch and open one pull request per repository, together, once the coast is clear**
|
||||
(above). The integration tests
|
||||
1. **Push every feature branch and open one pull request per repository, together.**
|
||||
The integration tests
|
||||
run only in the merge check (the operator, 2026-10-16), never before. Pull requests in two or
|
||||
more repositories that share the branch name are one delivery group (ADR 0239 §2).
|
||||
Where a session pushes through the forge tool `gitea_commit` (one commit on the forge's current base), the
|
||||
@@ -162,12 +140,16 @@ controller part touched four files feature 57's release branch also changed).
|
||||
5. **The reviewer marks each pull request ready as soon as the acceptance is shown** with
|
||||
`mesh-delivery.mark-ready` (the delivery of the head it approved, who reviewed it and why), which writes the
|
||||
label `mesh/ready` and then `ready at <full sha>`, the order the forge counts (issue 913); mesh-delivery merges it when its checks pass
|
||||
(ADR 0364). The operator's word is waited for only when the step changes what the operator sees, removes
|
||||
and the operator approved the merge into main on Telegram, the channel that proves who answered today (ADR 0364,
|
||||
ADR 0398 §3; an Approve at the terminal is issue 953). Beyond that
|
||||
Approve, the operator's word is waited for only when the step changes what the operator sees, removes
|
||||
data, or needs a step only the operator's terminal can take; the lead says which before the step starts.
|
||||
|
||||
**One exception: a catalogue change that uses a new manifest word.** The catalogue's merge check reads
|
||||
manifests with the controller the mesh runs, so the controller's pull request is merged and delivered first;
|
||||
then `mesh-delivery.recheck` the catalogue's and merge it on the new verdict. Plan such a feature as two steps.
|
||||
The exception ends when milestone 3 of design 47 is delivered (issue 977, ADR 0402, ADR 0403): the group's composed
|
||||
check, judged by the group's controller, is then each member's gate, and the feature is one group.
|
||||
|
||||
## Phase 4 — Deliver
|
||||
|
||||
@@ -202,7 +184,6 @@ on, with when it started and its bound.
|
||||
| the review | 15 minutes a round | ask the reviewer; stop at round 3 |
|
||||
| a merge check | 15 minutes | read the build queue (`mesh-controller.queue`) and the check's log |
|
||||
| the ready mark | 15 minutes after the acceptance is shown | the reviewer marks it, or the lead does |
|
||||
| a release branch in flight that shares files | its delivery plan done | report it every ten minutes; bring the branch up to `main` after |
|
||||
| a merge after ready | 5 minutes after the checks pass | read `mesh-delivery.show` for the delivery |
|
||||
| a delivery plan | its class's budget (`mesh-delivery.times`), at most 30 minutes | read `mesh-controller.plans`, then `mesh-delivery.stalled` |
|
||||
|
||||
@@ -270,8 +251,6 @@ issues in `fixing` that nobody works on reads to the operator as stuck work.
|
||||
review rules; `issue-tracker.list` with `flag: abandoned` lists issues left in `fixing` with nobody on them.
|
||||
- A pull request reviewed and green but not marked ready for more than 15 minutes is a lead that did not
|
||||
mark it.
|
||||
- A pull request opened into a repository whose release branch was in flight and shared a file with it is the failure of
|
||||
*Is the coast clear?*; the lead's report names the check it made before the push.
|
||||
- A step that adds or changes a module marked ready with no integration test that starts the module whole is
|
||||
the failure of phase 3's fourth step.
|
||||
- A bed run from `.work/<slug>/mesh-lab` that fails naming a `.work/<slug>/<repo>/…` path it cannot find is a
|
||||
|
||||
@@ -39,8 +39,8 @@ one delivery**. Everything before the push is local and fast.
|
||||
done-when is opened as an issue of its own, never asked as a change). The reviewer's findings go back to the same agent that wrote the fix, continued by
|
||||
its id, so the fix keeps its context. A playbook or design edit of a few lines may skip the reviewer: the
|
||||
operator's word on the pull request is its review, and the lead marks it ready on that word.
|
||||
5. **Push once and open the pull request, once the coast is clear** (playbook [07](07-feature-branches.md), *Is
|
||||
the coast clear?*: no release branch in flight in the repository that shares a file with the fix). The forge's merge
|
||||
5. **Push once and open the pull request.** The trunk is what runs and nothing holds merged work back
|
||||
(ADR 0401), so a fix waits for no other change's delivery. The forge's merge
|
||||
check runs the integration tests, the only
|
||||
place they run (the operator, 2026-10-16). A failed check
|
||||
goes back to the same agent with the failure, and the reviewer reads only the change it makes.
|
||||
@@ -62,6 +62,15 @@ done-when from holding, and that fault is in the same files, it is taken into th
|
||||
request with a review round of its own. A fault in other code is opened as an issue of its own, and the first
|
||||
issue waits for it (issue 937: mesh-catalog #335 was taken in; issue 943 was opened).
|
||||
|
||||
**A fix that moves a file to a new filler compares the values first.** When a fix makes a module's file filled a
|
||||
new way for the first time (the provisioner client in place of the controller, a provider's entry in place of a
|
||||
composed value), the fix lists every value the old way put into that file on each machine it reaches and shows,
|
||||
in its review, that the new way gives the same values, or says which differ and why. A difference nobody looked
|
||||
at is an outage: on 2026-10-16 the forge and the identity provider on the control node were down for 17 minutes
|
||||
because postgres's consumers were served the port inside its container instead of the port it is published on
|
||||
(issue 957). The comparison is read from the machines (the file as it is today) before the merge, not from the
|
||||
code.
|
||||
|
||||
## Bounds
|
||||
|
||||
| Waiting on | Bound | Past the bound |
|
||||
|
||||
@@ -1,100 +1,90 @@
|
||||
---
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [hal]
|
||||
updated: 2026-08-23
|
||||
code: [mesh-controller, mesh-catalog modules/mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/build-agent]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0010"
|
||||
- "ADR 0010"
|
||||
- "ADR 0010"
|
||||
- "ADR 0239"
|
||||
- "ADR 0267"
|
||||
- "ADR 0276"
|
||||
- "ADR 0282"
|
||||
- "ADR 0289"
|
||||
- "ADR 0304"
|
||||
- "ADR 0350"
|
||||
- "ADR 0358"
|
||||
- "ADR 0360"
|
||||
- "ADR 0364"
|
||||
- "ADR 0382"
|
||||
- "ADR 0395"
|
||||
- "ADR 0398"
|
||||
---
|
||||
|
||||
# Delivery — from a push to a running node
|
||||
# Delivery — from a merge to the nodes, as it runs
|
||||
|
||||
One trigger, three silos, and a fan-out point that is the most consequential boundary in the
|
||||
mesh.
|
||||
What runs on the mesh's main branches on 2026-10-16, read from the code and the live record by research 055
|
||||
(findings 1 to 12). It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) replaces, milestone
|
||||
by milestone. Until 2026-10-16 this document described the predecessor's pipeline, which no longer runs.
|
||||
|
||||
## The trigger
|
||||
## Who does what
|
||||
|
||||
A push to the forge. The forge calls a webhook; the receiving module verifies its signature
|
||||
and emits an event. The coordinator resolves which modules the pushed commits affect, orders
|
||||
them into dependency levels, and creates a pipeline per level.
|
||||
- **mesh-delivery** owns deliveries and delivery groups (ADR 0239): one delivery per commit, a group per branch
|
||||
name shared across repositories. It records each transition in its state on the bus (key-value buckets
|
||||
`deliveries` and `groups`), marks pull requests, asks the controller to deliver, and merges a pull request marked
|
||||
ready at its head once both required checks pass against the trunk's head (ADR 0364) and the operator approved
|
||||
the merge into main on a channel that proves who answered (ADR 0398 §3).
|
||||
- **The controller** plans, orders, sends and judges. It computes a delivery plan for a repository and the files a
|
||||
merge changes, moves only modules whose build source changed with their build dependents (ADR 0267), and
|
||||
reuses a merge check's build of the same reuse key (ADR 0350).
|
||||
- **The builders** (holders of the build seat) take build requests and checks from a work-queue stream on the bus
|
||||
and build inside a toolchain image another module provides, with the network off; on 2026-10-13 no node had Go
|
||||
or npm installed (issue 626).
|
||||
- **The forge module** publishes every forge event on the bus through the forge's system webhook since 2026-10-14
|
||||
(issue 729), and still polls the forge every 30 s, or every 300 s once it hears events. It keeps files of what it
|
||||
announced so it does not announce twice.
|
||||
|
||||
There is no second path. A manual trigger exists for a module the push detection missed, and
|
||||
using it routinely is a sign the detection is wrong rather than a workflow.
|
||||
## From a merge to a rollout
|
||||
|
||||
Detection is the pipeline's most fragile input. It has failed for reasons that have nothing to
|
||||
do with the change: a webhook truncating its commit list on a large merge, a forge address
|
||||
whose port broke the module-path matching. The characteristic outcome is the bad one — **a
|
||||
merge that created no pipeline, and nothing said so**.
|
||||
- **The merge window.** The controller collects merges into an open batch; each merge restarts the window, which
|
||||
closes after 90 seconds without a merge or 10 minutes after the batch's first merge (ADR 0276). A lone merge is
|
||||
cut at once (ADR 0282 §2, as ADR 0304 §1 changed it). The batch and its merges are rows in the controller's
|
||||
store, cut by a timer every 5 s, with a catch-up timer every 5 minutes for merges no batch holds.
|
||||
- **The rollout.** A closed batch becomes one rollout, at most one commit per repository, tiered along the module
|
||||
graph. Rollouts run one at a time: a batch cut while a rollout is open is queued behind it, whatever it shares
|
||||
with it. Running unrelated rollouts side by side (ADR 0282 §3, ADR 0304 §2) is decided and not built (issue 525). A rollout's builds go to every machine at once, with no first-machine gate, and the rollout ends on
|
||||
the machines' reports or after fifteen minutes of silence (ADR 0360). A controller timer advances rollouts every
|
||||
30 s.
|
||||
- **A merge during a rollout.** Its place in the next batch is a store row the open rollout's holder writes. On
|
||||
2026-10-16, when another controller worked the plans, a merge was not recorded and waited until a person released
|
||||
it (issue 921).
|
||||
- **"Release branches".** A pull request into a branch other than a trunk is judged by the controller built from that
|
||||
branch (ADR 0358); feature 57 shipped about 37 packages that way over about 2.5 days and held the controller for
|
||||
55.7 h.
|
||||
|
||||
## Three silos
|
||||
## Checks
|
||||
|
||||
Cardinality is the whole point, and the three differ
|
||||
(ADR 0010):
|
||||
- Every pull request's head gets `mesh/merge-gate` (the composed machines) and `mesh/repo-check` (the repository's
|
||||
`merge-check.sh`), each judged against one trunk commit (ADR 0323). A trunk move asks the check of every active,
|
||||
cleanly merging pull request again (ADR 0395).
|
||||
- A delivery group's members are each checked beside their siblings' heads, and the group also gets a composed
|
||||
check, `mesh/delivery-group`; each member keeps its own gate beside it (ADR 0382).
|
||||
- The queue's order (delivery builds first, then ready checks, then the rest) is kept by pausing the worker and
|
||||
deleting and republishing a request (issue 558); an old head's check is ended after asking the forge for the
|
||||
current head (issue 726).
|
||||
- mesh-delivery ticks every 10 s, reconciles with the controller every 30 s and polls the forge for ready marks
|
||||
every minute, although the forge module already publishes labeled, commented and synchronized events.
|
||||
|
||||
| Silo | Runs | Where | Does |
|
||||
|---|---|---|---|
|
||||
| **build** | once per module feature | the build node | compile and bundle into a self-contained output |
|
||||
| **publish** | once per module feature | the build node | package that output and upload it; a package-registry feature publishes here |
|
||||
| **deploy** | once per module feature **per node** | every assigned node | install, configure, start, verify |
|
||||
## Where the record of what runs lives
|
||||
|
||||
Commands and events are addressed per feature, not per module.
|
||||
Builds and build requests, rollouts with their tiers, batches and merges, and assignments are rows in the
|
||||
controller's store; artifacts are blobs in the artifact store by digest (ADR 0251); packages are versions in the
|
||||
package registry. The controller writes three record mirrors to the bus: assignments, declared config and
|
||||
predefined keys; of the five ADR 0289 §7 decided, the nodes, rollouts, builds and settings mirrors are not in the
|
||||
code. An assignment names its module and node but not the build, artifact or rollout that put it there: which commit
|
||||
runs where, and which delivery brought it, takes three verbs and a join by hand.
|
||||
|
||||
Build hands over a **staged tree**, not a package. Packaging belongs to publish, so a failed
|
||||
upload retries by re-packaging rather than by re-sending something stale, and build never needs
|
||||
to know how each module composes its artifact. The handover is a tree in a known location,
|
||||
because the build's own working directory is reference-counted and may be gone by the time a
|
||||
later stage runs.
|
||||
|
||||
## The artifact
|
||||
|
||||
The artifact is **build output** — compiled and bundled with its dependency graph inlined —
|
||||
never a filtered copy of source (ADR 0010).
|
||||
A deploy is extract-and-run and touches no network.
|
||||
|
||||
The consequence is the whole cost of the decision: **anything not in the build output does not
|
||||
ship.** Every file kind had to be brought into that rule separately, and each was discovered by
|
||||
something silently not happening after deploy — migrations reading a source layout, provisioning
|
||||
scripts reading a source layout, selection files never packaged at all.
|
||||
|
||||
## The fan-out, and the build node
|
||||
|
||||
After publish, work fans out to every assigned node. The build node is the only node that has
|
||||
already passed through two silos when this happens, and that asymmetry has its own defect
|
||||
class: anything advancing a node's stage must account for **both** pre-fan-out stages. Code
|
||||
that knew only about the first parked the build node forever while every other node deployed
|
||||
cleanly — and the recovery sweeper, which knew the same subset, reported nothing to recover.
|
||||
|
||||
A recovery mechanism that knows less than the thing it guards is worse than none, because it
|
||||
reports success over a stall it cannot see.
|
||||
|
||||
## Levels
|
||||
|
||||
A level completes before the next begins, so a module builds against its dependencies as they
|
||||
were just published. The shared library is at level zero, which means anything that breaks it
|
||||
breaks the first module of every cascade.
|
||||
|
||||
## What green proves
|
||||
|
||||
**A green pipeline proves transport, not effect.** The stages report that a message was
|
||||
dispatched and accepted, which is not the same as the thing being running, correct, or present.
|
||||
|
||||
This is the mesh's most consistent failure shape and it is not incidental to the design — it is
|
||||
what the stage reporting currently measures. Documented instances include a service reported
|
||||
started when the container command merely returned, an image pull failure that did not fail the
|
||||
deploy, a package install that 404ed from every mirror while the job went green (see
|
||||
[`04-ISSUES/001`](../../04-ISSUES/001-failed-package-install-reports-success/00-report.md)),
|
||||
and a node left on old code after a failed artifact download with a version marker that had
|
||||
already advanced.
|
||||
|
||||
A verify stage exists to close this gap. It has been built and, for a period, was never
|
||||
scheduled, because the coordinator's stage list did not include it.
|
||||
No backup is taken before a delivery moves a module that holds data; backups run nightly and on demand (ADR 0214).
|
||||
|
||||
## What is not covered
|
||||
|
||||
The end-to-end harness for this pipeline has not built since the workspace was removed, and
|
||||
nothing reports that nothing runs it
|
||||
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). The
|
||||
pipeline's coverage is currently assumed rather than checked — which is the same class of claim
|
||||
this repository exists to make people stop making.
|
||||
The integration tests run only in each repository's merge check; there are no end-to-end tests of delivery yet,
|
||||
and the lab is not used (the operator, 2026-10-16).
|
||||
|
||||
@@ -5,8 +5,9 @@ code:
|
||||
- mesh-controller cmd/mesh-builder
|
||||
- mesh-controller internal/builder
|
||||
- mesh-catalog modules/build-agent
|
||||
updated: 2026-10-08
|
||||
updated: 2026-10-16
|
||||
decisions:
|
||||
- "ADR 0405"
|
||||
- "ADR 0262"
|
||||
- "ADR 0233"
|
||||
- "ADR 0219"
|
||||
@@ -138,7 +139,7 @@ need to run. Third-party software still arrives as an image, because that is how
|
||||
## What this costs, stated before it is chosen
|
||||
|
||||
**Every language is permanent.** It needs an SDK — broker client, sealed-credential reading, the
|
||||
event envelope, tool serving — a toolchain image, and a bundler the builder understands. And the
|
||||
event envelope, tool serving — a build module that installs its toolchain on every builder (ADR 0405), and a bundler the builder understands. And the
|
||||
spine changes rarely but cascades when it does
|
||||
(ADR 0039): with one language a contract
|
||||
change is one edit; with four it is four that must land together, and a mesh whose SDKs disagree
|
||||
|
||||
@@ -4,69 +4,112 @@ status: in-progress
|
||||
code: [mesh-controller, mesh-catalog modules/mesh-delivery]
|
||||
updated: 2026-10-16
|
||||
decisions:
|
||||
- "ADR 0408"
|
||||
- "ADR 0407"
|
||||
- "ADR 0406"
|
||||
- "ADR 0405"
|
||||
- "ADR 0404"
|
||||
- "ADR 0403"
|
||||
- "ADR 0402"
|
||||
- "ADR 0401"
|
||||
- "ADR 0398"
|
||||
- "ADR 0395"
|
||||
- "ADR 0382"
|
||||
- "ADR 0377"
|
||||
- "ADR 0364"
|
||||
- "ADR 0362"
|
||||
- "ADR 0360"
|
||||
- "ADR 0350"
|
||||
- "ADR 0335"
|
||||
- "ADR 0323"
|
||||
- "ADR 0313"
|
||||
- "ADR 0306"
|
||||
- "ADR 0304"
|
||||
- "ADR 0299"
|
||||
- "ADR 0298"
|
||||
- "ADR 0282"
|
||||
- "ADR 0276"
|
||||
- "ADR 0270"
|
||||
- "ADR 0267"
|
||||
- "ADR 0250"
|
||||
- "ADR 0249"
|
||||
- "ADR 0239"
|
||||
- "ADR 0238"
|
||||
- "ADR 0236"
|
||||
- "ADR 0233"
|
||||
- "ADR 0214"
|
||||
- "ADR 0201"
|
||||
---
|
||||
|
||||
# 47 — Delivery: from a commit to delivered
|
||||
|
||||
**A delivery is one commit in one repository, from the moment its pull request's head is announced to the
|
||||
moment every machine of its deploy plan runs it, or to the moment it fails, is superseded or is stopped. A
|
||||
delivery group is two or more deliveries that share a head branch name, delivered as one unit in order. Both
|
||||
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
|
||||
(ADR 0239.)
|
||||
**A change reaches the machines in one of three ways, all owned by mesh-delivery: a hotfix (one pull request,
|
||||
delivered at its merge), a feature (one branch name across its repositories: in two or more, one delivery
|
||||
group merged together and delivered member by member in its order; in one, a lone delivery), or an epic (a run of milestones, each delivered as a feature). The trunk is what runs; there
|
||||
is no "release branch"** (ADR 0401).
|
||||
|
||||
**From its merge on, a delivery is delivered by the rollout of its batch.** The controller collects merges in a
|
||||
rolling merge window and plans each batch once, as one rollout with at most one commit per repository, one rollout
|
||||
at a time. Every delivery the rollout answers ends with it
|
||||
(ADR 0276).
|
||||
**Every step starts on an event, and the state lives on the bus.** The forge's events, the controller's
|
||||
events and the nodes' reports move a delivery; the bus's own rules keep the queues, the merge window and the
|
||||
order; and one chain, from module to commit to build to rollout to assignment, is kept in the bus's
|
||||
key-value store as its only source (ADR 0406, ADR 0407, ADR 0408). A running delivery never touches merging:
|
||||
every merge is recorded at once for the next delivery plan of what it touches.
|
||||
|
||||
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
|
||||
of source and artifacts decides, and nothing here changes that. This design says who answers for one
|
||||
commit's journey and in what order the steps are allowed.
|
||||
commit's journey, in what order the steps are allowed, and where each fact of it is kept. It was rewritten on
|
||||
2026-10-16 from the decisions of research 055, which replaced the "release branches", the toolchain image, the
|
||||
controller's delivery tables and the timers that polled them.
|
||||
|
||||
## The parts
|
||||
|
||||
```
|
||||
forge's holder ──pull.updated / .merged / .closed──► mesh-delivery ◄──checked / plan-moved── controller
|
||||
(statuses, the view, (deliveries, groups, (planner, gate,
|
||||
the notes) the table, its state) registration,
|
||||
▲ │ │ sending, walk)
|
||||
└── its tools: note, view, status ◄───────────────┘ └── its verbs: delivery-plan, -order,
|
||||
-check, deliver, delivery-stop,
|
||||
delivery-walks
|
||||
forge (system webhook) ──forge.pull-request.* / merged / status──► the bus ◄──built / checked / plan-moved── controller
|
||||
│ ▲ (planner, order,
|
||||
│ │ registration,
|
||||
▼ │ sending)
|
||||
mesh-delivery │
|
||||
(deliveries, groups, a key per │
|
||||
pull request, the state table) ▼
|
||||
node-engine on every node
|
||||
builders (build seat) ◄── work queues: delivery builds, ready checks, other checks ───────────────────────── (reports; backups first)
|
||||
```
|
||||
|
||||
- **The controller** stays the only writer of what it owns: machine declarations, bus objects, grants,
|
||||
registration. It keeps the primitives: the planner (one function, ADR 0238 decision 3), the gate, the trunk
|
||||
rule, a gated send, the rollback and the rollout of one trunk commit's tiers across machines. It still answers
|
||||
every pull request's head with its check itself, so a check never depends on mesh-delivery.
|
||||
- **mesh-delivery** is a Go bundle holding the mesh-scoped seat `mesh-delivery`. It owns the delivery, the
|
||||
group, the state table and their state, and nothing else.
|
||||
- **The forge's holder** sets a commit's statuses, keeps the delivery's view as one comment on the pull
|
||||
request, and appends to the commit's note under `refs/notes/mesh-plan`.
|
||||
- **The build seat** builds and checks, asked by the controller as before. **A change is built once** (ADR 0350):
|
||||
a check that passes also builds what its merge would move into the artifact store's scratch namespace, under a
|
||||
reuse key (the source fingerprint and the builder's version), and the rollout of the merge commit copies a build
|
||||
of the same key, which the check's verdict names, by digest instead of building. Otherwise the rollout builds and
|
||||
says why, per module, in its record. Check builds go when the rollout answering the merge ends, when the pull
|
||||
request closes unmerged, and after seven days.
|
||||
- **mesh-delivery** (the seat `mesh-delivery`) owns hotfixes, features and milestones as deliveries and delivery
|
||||
groups, decides each step on events, keeps one key per pull request, and asks the controller for each act
|
||||
and the nodes' backup seats for the backups a delivery needs. It never sends to a machine, writes a
|
||||
declaration or registers a build (ADR 0239 §25).
|
||||
- **The controller** keeps planning, ordering, registration, sending and judging. It writes the chain's
|
||||
links it owns — builds, rollouts with their tiers, assignments — and the merge window's
|
||||
keys, all on the bus, by compare-and-set, while it holds the lease.
|
||||
- **The builders**, the holders of the build seat, build and check with toolchains installed on their node, as
|
||||
the build seat's own account, and take their jobs from the bus's work queues, delivery builds first
|
||||
(ADR 0405, ADR 0406).
|
||||
- **The forge module** (`gitea`) publishes every forge event on the bus through the forge's system webhook
|
||||
(issue 729): pull requests updated, synchronized, labeled, unlabeled, commented, merged and closed, and
|
||||
commit statuses. It sets statuses, keeps the delivery's view as one comment on the pull request, and appends
|
||||
to the commit's note under `refs/notes/mesh-plan`.
|
||||
- **Each node's backup seat** (`node-backup`) backs up the data a delivery plan moves there before the plan
|
||||
sends anything (ADR 0404).
|
||||
|
||||
## Three ways a change reaches the machines
|
||||
|
||||
*ADR 0401.*
|
||||
|
||||
| Way | What it is | How it is delivered |
|
||||
|---|---|---|
|
||||
| **hotfix** | one pull request in one repository | at its merge, as a lone delivery; a lone merge is cut at once (ADR 0282 §2, as ADR 0304 §1 changed it) |
|
||||
| **feature** | one branch name across its repositories; in two or more, its pull requests are one delivery group; in one, a single pull request | a group is merged together once its composed check passes and every member is marked, reviewed and approved, then delivered member by member in its order (ADR 0239 §16); in one repository, as a lone delivery like a hotfix |
|
||||
| **epic** | a run of milestones in one design (playbook 04) | each milestone is a feature, delivered when its own group is ready; the next starts when the previous one's delivery plan is done |
|
||||
|
||||
- **The trunk is what runs.** Every delivery is cut from a repository's trunk. A merge into any other branch
|
||||
starts no delivery, and the controller says so for that merge (issue 889). Holding merged work back, on a
|
||||
branch beside the trunk or per module, was dropped: from 2026-10-13 23:52 to 2026-10-16 09:32 feature 57's
|
||||
branches held the controller for 55.7 h, and 28 of 68 other merges touched a held module (research 055,
|
||||
finding 3).
|
||||
- **A milestone is functional, its delivery shaped technically.** A milestone is named by what the operator or
|
||||
the mesh can do once it is delivered; how it ships is shaped by the delivery plan its changes would have, asked of
|
||||
the controller's planner through mesh-delivery's `what-if` (ADR 0239 §3): the modules it moves and their fan-out.
|
||||
- **What a merge waits for.** mesh-delivery merges a group's members once the group's composed check passed at
|
||||
their heads, the independent reviewer marked each member ready (ADR 0364 §1, §2), each member carries the
|
||||
review ADR 0382 §6 asks, and the operator approved each merge into main (ADR 0398 §3). The operator's Approve is
|
||||
asked for every merge into main, a hotfix's too (the operator, 2026-10-16 23:40).
|
||||
|
||||
## A delivery
|
||||
|
||||
@@ -75,12 +118,17 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
|
||||
- the pull request (number, base, head branch, description), or none for a commit that reached the trunk
|
||||
without one;
|
||||
- its **delivery plan**: build plan, deploy plan and verdict, as the controller's planner computed them for
|
||||
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
|
||||
its diffset (ADR 0238 §2, as ADR 0323 §1 changed it);
|
||||
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
|
||||
**machine steps** while delivering;
|
||||
- once merged, **the commit it landed on the trunk as**, and the rollout that delivers it: the rollout of the batch
|
||||
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
|
||||
window and the batch*).
|
||||
- once merged, **the commit it landed on the trunk as**, and the delivery plan that delivers it: the plan of the
|
||||
batch that answers its merge, which may carry a later commit of its repository that contains it.
|
||||
|
||||
**One key per pull request.** Beside the delivery, mesh-delivery keeps one key per open pull request on the
|
||||
bus: its head, the trunk commit its current check is asked against, and its verdict, written by
|
||||
compare-and-set (ADR 0406 §4). It writes a new head when the forge says one, and a new trunk commit only when it
|
||||
asks the pull request's check again (ADR 0395 §1). A running check watches that key and stops when the head or
|
||||
the trunk commit in it moves.
|
||||
|
||||
## The state table
|
||||
|
||||
@@ -95,418 +143,365 @@ allows.
|
||||
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
|
||||
| ready | it passed; it waits to be merged | none | — |
|
||||
| rejected | it failed its check | none | — |
|
||||
| published | merged on the trunk its modules follow, its rollout opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a rollout that does not answer its merge took over its rollout; `go` when its rollout started |
|
||||
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
|
||||
| delivering | its rollout runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the rollout's record says |
|
||||
| published | merged on its trunk, recorded in a batch; it waits for its plan to start | 30 minutes | `superseded` when a plan that does not answer its merge took over its plan; `go` when its plan started |
|
||||
| held | it waits for a person: a failed backup (ADR 0404 §4), a policy of record; it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
|
||||
| delivering | its plan runs: backups, then builds asked and sent tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the plan's key says |
|
||||
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
|
||||
|
||||
**A verdict holds for one trunk commit** (ADR 0323). The check judges the head merged into the trunk's head
|
||||
at the moment the build seat clones it, the **trunk commit**, and the verdict names both. When another merge
|
||||
moves the trunk, a `ready` delivery goes back to `proposed` on the observed event `trunk moved`, with both
|
||||
commits in the transition's reason; a `rejected` one stays `rejected`, its statuses saying the verdict is out
|
||||
of date. The move itself asks the check again of every active open pull request into that trunk that merges
|
||||
cleanly (ADR 0395, *A move of the trunk asks every check again* below); a recheck, a new head or the merge tool
|
||||
asks it too. The row from
|
||||
`proposed` to `checked` takes only a verdict that names the trunk's head as its trunk commit, so a verdict for
|
||||
an older trunk is kept and shown and does not count, and the rows reach a fixed point. Checked by
|
||||
mesh-delivery's `TestATrunkMoveSendsAReadyDeliveryBackToProposed` and the table's own rollout.
|
||||
when the builder clones it, and the verdict names both. When another merge moves the trunk, a `ready`
|
||||
delivery goes back to `proposed` on the event `trunk moved`, and mesh-delivery asks the check of every active,
|
||||
cleanly merging open pull request into that trunk at once (ADR 0395, *A move of the trunk asks every check
|
||||
again* below). The row from `proposed` to `checked` takes only a verdict that names the trunk's head.
|
||||
|
||||
What this leaves as it is, said so nobody reads more into it:
|
||||
**What needs no person ends by itself** (ADR 0270): a merge that moves nothing is `delivered` at its merge; a
|
||||
merge a later delivered merge contains is `covered`, by the forge's word that the later merge commit contains
|
||||
it; a merge whose plan said its modules unchanged is `delivered` when that plan ends (issue 611); a stop whose
|
||||
plan ended without building its modules is read as covered once a later merge covers it (issues 474, 480).
|
||||
|
||||
- **mesh-delivery's trunk head is the last merge it heard** into that trunk while the delivery was open. A push
|
||||
to the trunk that is not a merge it hears does not move it; the forge's holder, which reads the branch, and
|
||||
the merge tool, which compares commits when asked, still catch it.
|
||||
- **A verdict that names no trunk commit**, from a build seat older than ADR 0323, counts in mesh-delivery as it
|
||||
did: such a delivery stays `ready`. The forge reads the same verdict out of date once its trunk moves, and the
|
||||
merge tool refuses it and asks the check. The two agree once the head is judged again.
|
||||
- **A verdict no build judges** (a change that touches no module, in a repository with no `merge-check.sh` or
|
||||
not the mesh's) names the base branch's head the forge announced with the head as its trunk commit.
|
||||
- **In a busy repository the merge tool can refuse, ask, and refuse again**: another merge may move the trunk
|
||||
while the asked check runs, and the new verdict is then out of date too. Each refusal asks once more; the
|
||||
merge goes through once a check finishes before the next merge.
|
||||
- **A check that was asked and never answers** is listed by `stalled` an hour after the ask, and is asked again
|
||||
with `mesh-delivery.recheck`, which the merge tool's refusal names.
|
||||
**History is retired, not held** (ADR 0250). A delivery made from the forge's word at its merge, heard fifteen
|
||||
minutes or more after it, with no plan of it open, goes to `retired` at adoption. **Nothing of a published
|
||||
delivery is registered before its plan starts.**
|
||||
|
||||
The transitions are those of ADR 0239 decision 2, ADR 0270, ADR 0323 and ADR 0395. A delivery that is
|
||||
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
|
||||
delivery merged with no rollout answering it within ten minutes is held too, with the true reason, and listed by
|
||||
`stalled` at once. Nothing waits silently.
|
||||
**The machine steps.** While a delivery is `delivering`, each machine its plan reaches has a row with the
|
||||
module and the build: `backed-up`, `sent`, `applied` or `failed`, and `silent` when no report came within
|
||||
ApplySilentAfter (fifteen minutes). They are read from the plan's key, written by the controller from the
|
||||
nodes' reports, never inferred from time.
|
||||
|
||||
**What needs no person ends by itself** (ADR 0270):
|
||||
## Order without a person
|
||||
|
||||
- *a merge that moves nothing* (its plan builds nothing, or only adds a module the controller builds and
|
||||
sends nowhere) is `delivered` as soon as it is merged, whatever its check said;
|
||||
- *a merge a later delivered merge contains* is `covered`: every module it moves was sent to every machine
|
||||
running it by the rollout of a later merge into the same trunk, delivered on its own passing check, and the
|
||||
forge says that later merge commit contains it (`gitea_contains`, asked once per pair and kept). Merge
|
||||
order is only the first filter, and a forge that cannot answer means not yet covered. `mesh/delivery`
|
||||
says `covered: a later merge that contains it was delivered` as a success. A module that runs on one
|
||||
machine only is covered too: its rollout records the empty rest as sent once its first machine passes
|
||||
(ADR 0270, as corrected 2026-10-12);
|
||||
- *a merge whose rollout said its modules unchanged* moves nothing too, and is `delivered` when that rollout ends,
|
||||
whichever commit the rollout carried (issue 611). A delivery plan made while another rollout built a module reads
|
||||
that module whole and names it; the rollout, cut later, reads the build source that build recorded (ADR 0267) and
|
||||
may find none of the changed files in it. The rollout's record then says so in `delivery.unchanged`: each such
|
||||
module with why, "its build source holds none of the changed files", and not in its modules, which keep
|
||||
meaning what the rollout builds. A delivery whose every module the rollout built or said unchanged is delivered,
|
||||
saying which matched what runs. One with a module the rollout neither built nor said unchanged is read as below.
|
||||
Checked by the controller's `TestAWalkThatMovedNothingSaysWhichModulesMatchedWhatRuns` and mesh-delivery's
|
||||
`TestAMergeWhoseModulesMatchWhatRunsIsDelivered`;
|
||||
- *a stop whose rollout ended without building its modules* (issue 474) stays `stopped`, a final state, and
|
||||
is read as covered once a later merge covers it by the same test: `mesh/delivery` then says `covered after
|
||||
its stop` with the covering merge, once, as a success. No state or transition is added (issue 480);
|
||||
- the covered row is tried before `go`, from `held` and from `ready` past the ten minutes.
|
||||
*ADR 0239 §11 and §12, ADR 0249, ADR 0402.*
|
||||
|
||||
**History is retired, not held** (ADR 0250).
|
||||
A delivery made from the forge's word at its merge, heard fifteen minutes or more after that merge, with no rollout
|
||||
of it open, was replayed from the bus's history: it reached the machines before this owner was there. It goes
|
||||
to `retired` at adoption, before any check, hold or rollout, and its pull request is not told. A held delivery that
|
||||
fits the same rule is retired by a person's `retire-history`, never released. **Nothing of a published delivery is registered before its rollout
|
||||
starts**, because a send carries every registered build that waits on its machine (ADR 0236 §4a), and a
|
||||
build registered before its turn would be carried by somebody else's send.
|
||||
A delivery group's order is a table of pairs, each with the rule that won (ADR 0249 §3). The controller's
|
||||
`delivery-order` infers it from the manifests and the module graph; a person writes an `after: <repository>`
|
||||
line only where the rules cannot see the order.
|
||||
|
||||
**The machine steps.** While a delivery is `delivering`, each machine its rollout reaches has a row with the
|
||||
module and the build: `sent` (the first machine's send, or the rest's), `judging` (the gate's healthy
|
||||
readings so far), `passed`, `failed`, `rolled-back`. They are read from the rollout's own record in the
|
||||
controller's `plan-moved` event, never inferred from time.
|
||||
| Rank (highest first) | Rule | Read from |
|
||||
|---|---|---|
|
||||
| 1 | *declared*: an `after:` line | the member's description |
|
||||
| 2 | *built by* and *version skew* | the module graph and the change |
|
||||
| 2 | *verb before caller*: a member adding a verb to a seat's served verbs before a member adding that verb to an invoked verb | the manifests (ADR 0402) |
|
||||
| 2 | *SDK before its users*: a member changing `mesh-sdk` before every member whose modules are built on it | the module graph (ADR 0402) |
|
||||
| 3 | *engine before controller* | module names |
|
||||
| 4 | the repositories' names | — |
|
||||
|
||||
Rules of one rank in opposite directions reject the group, naming both members, both rules and the line that
|
||||
would settle it (ADR 0249 §4).
|
||||
|
||||
**The SDK by version.** A Go module names the SDK by its released version and fetches it from the mesh's forge,
|
||||
checked against its recorded checksum; vendored copies go. Inside a group, a member may name the SDK member's
|
||||
head: its merge check resolves that commit from the forge and builds against it, and the head stays fetchable
|
||||
after the group merges because the forge keeps every pull request's head. The catalogue's check accepts an SDK
|
||||
commit on the SDK's trunk or the head of an SDK pull request the forge holds, open or merged (ADR 0402 §2 to §4).
|
||||
|
||||
## Checks
|
||||
|
||||
### One check per pull request, kept by the bus
|
||||
|
||||
*ADR 0323, ADR 0350, ADR 0395, ADR 0406.*
|
||||
|
||||
- **Three work queues, taken in order.** Delivery builds go on a work-queue stream of their own; checks of pull
|
||||
requests marked ready on a second; every other check on a third. A builder takes from the first, then the
|
||||
second, then the third. Each check stream has one subject per pull request and keeps only the newest request:
|
||||
a newer request replaces one still waiting, by the stream's rule. A ready mark publishes its pull request's
|
||||
check request on the ready stream; a request left behind on the third for a head and trunk commit already
|
||||
judged or running is acknowledged and dropped when taken, by reading the pull request's key. Nothing pauses the
|
||||
worker or deletes and republishes a request (issue 558).
|
||||
- **A running check stops itself.** It watches its pull request's key and stops when the head or the trunk
|
||||
commit in it moves, without asking the forge (issue 726). An ended check delivers no verdict, so none of its
|
||||
builds is reused.
|
||||
- **A failed builder loses nothing.** A job a builder took and never acknowledged goes to another builder
|
||||
after its wait.
|
||||
- **A change is built once** (ADR 0350). A check that passes also builds what its merge would move into the
|
||||
artifact store's scratch namespace under a reuse key, and the plan of the merge commit copies the build of the
|
||||
same key, named by the check's verdict, instead of building.
|
||||
|
||||
### A move of the trunk asks every check again
|
||||
|
||||
*ADR 0395.*
|
||||
|
||||
- A trunk move asks the check of every active open pull request into that trunk that merges cleanly, against the
|
||||
new head; a conflict is told once on the pull request, not checked.
|
||||
- A pull request with no push and no ready mark for 7 days is left out; a push, a ready mark or a recheck asks
|
||||
its check.
|
||||
- For a group, a move asks each member's own check, and the composed check runs again through those rechecks; a
|
||||
move made by a member merging in the group's order asks the other members nothing.
|
||||
- Its cost is at most one check per active, cleanly merging pull request per trunk move; issue 944 measures it.
|
||||
|
||||
### A delivery group's checks
|
||||
|
||||
*ADR 0239 §14, ADR 0323 §6, ADR 0382, ADR 0403.*
|
||||
|
||||
- **The composed check is each member's gate.** The group's composed check composes every machine with all
|
||||
members' heads, each merged into its own trunk's head first, judged by the controller built from the group's
|
||||
controller member when it has one (ADR 0382 §3). Its verdict sets `mesh/delivery-group` and each member's
|
||||
`mesh/merge-gate`, and names every member it read. No gate runs for a member alone.
|
||||
- **Each member's own repository check stays required.** Each member's `mesh/repo-check` runs its repository's
|
||||
suite with the siblings read beside it (ADR 0382 §2).
|
||||
- **No composed verdict, no gate.** A member's gate is pending while no composed verdict at the members' current
|
||||
heads exists. A member's new head, close or base move puts the composed verdict out of date, and with it every
|
||||
member's gate. When mesh-delivery cannot say the groups, the pull request is checked alone and its verdict
|
||||
says so (ADR 0382 §1).
|
||||
- **A member merges only with its group passed and reviewed** (ADR 0382 §6), and in the group's order
|
||||
(ADR 0364 §8).
|
||||
|
||||
## Ready marks and the merge
|
||||
|
||||
*ADR 0364, ADR 0406 §1.*
|
||||
|
||||
The independent reviewer marks a pull request ready at the reviewed head with the label `mesh/ready` and a
|
||||
comment `ready at <full sha>`, through `mesh-delivery.mark-ready`. The forge's `labeled` and `commented` events
|
||||
reach mesh-delivery in seconds; once both required checks passed against the trunk's head, it asks the operator's
|
||||
Approve for a merge into main, with the pull request, its issues, the reviewer's verdict and the delivery plan, and
|
||||
merges the marked head through the forge's merge tool on that Approve (ADR 0398 §3). When the trunk moved, the merge tool asks the fresh check, and the event
|
||||
of its passing verdict asks the merge again; no timer asks it. A push clears the mark (the `synchronized`
|
||||
event), and a failed check is said once and listed by `stalled`.
|
||||
|
||||
## The merge window and the batch
|
||||
|
||||
*2026-10-10, by ADR 0276.*
|
||||
*ADR 0276, ADR 0282, ADR 0299, ADR 0304, ADR 0407.*
|
||||
|
||||
- **The window.** The controller puts every merge into a branch a module follows into the open **batch**.
|
||||
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
|
||||
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
|
||||
merges and its times are kept in the controller's store and survive a restart.
|
||||
- **The cut.** When the window closes and no rollout is open, the batch becomes one rollout: one delivery plan for
|
||||
every commit in it, tiered along the module graph across repositories, with at most one commit per
|
||||
repository, the latest of its branch. The rollout names every merge it answers: its own commits, and the
|
||||
earlier merges of a repository that its later commit contains.
|
||||
- **One rollout at a time.** Merges that arrive while a rollout is open join the next batch. A rollout that started
|
||||
is never superseded. A rollout still waiting for its word when the next batch is cut is folded into the new
|
||||
rollout, which answers every merge the folded one answered.
|
||||
- **The controller's own path.** A merge that moves a module whose rollout waits for nobody's word (the controller,
|
||||
the node-engine, the tool runner, the bus, mesh-delivery) never shares a batch with one that waits for
|
||||
mesh-delivery's: it has a batch of its own, with its own window, and the catalogue's batch queues behind its
|
||||
rollout like any other. An own-path rollout folds no waiting catalogue rollout; that rollout's merges are batched again
|
||||
behind it, so no catalogue delivery skips its turn (ADR 0276, decided during the build, 2026-10-10).
|
||||
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a rollout is
|
||||
answered by that rollout: the assembling batch, the open rollout, or, when that rollout was delivered, it ends
|
||||
`covered` at once.
|
||||
- **The ends.** A rollout done ends every delivery it answers: `delivered` for its own commits, `covered` for
|
||||
an earlier merge of a repository it carried later, naming that merge. A rollout that failed fails its own
|
||||
commits' deliveries and marks none delivered; each earlier merge it answered is then rolled out on its own
|
||||
commit, the newest first, one rollout each, until one is delivered. A rollout stopped stops every delivery whose
|
||||
merge it rollouts again nowhere (below).
|
||||
- **A group inside a rollout.** The pairs of a delivery group's order (ADR 0249) are edges of the rollout's
|
||||
tiers beside the module graph. mesh-delivery gives a rollout its word only when every delivery it carries is
|
||||
at its turn.
|
||||
- **The bookkeeping.** mesh-delivery ties a rollout to every live delivery whose merge it answers, follows a
|
||||
folded rollout to the rollout that took it over, and never ties a rollout to a delivery that has ended. A waiting
|
||||
rollout no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
|
||||
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
|
||||
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
|
||||
and a batch waiting behind an open rollout names that rollout. The controller says the batch on the bus as
|
||||
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
|
||||
delivery it holds.
|
||||
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no rollout open, and a batch
|
||||
waiting behind an open rollout past that rollout's bound, each raise a condition. A merged delivery that no rollout
|
||||
answers within ten minutes is held with the true reason (the open rollouts of its repository and that none
|
||||
names its merge, or that the controller holds none) and listed by `stalled` at once.
|
||||
- **Every merge is recorded at once.** The forge's merged event records a merge into the open batch of what it
|
||||
touches, or opens one, by whichever controller holds the plans. A merge is never refused, held or dropped,
|
||||
and needs no person's release (issue 921). The bus's durable consumer keeps every merge event until it is
|
||||
recorded; no catch-up timer looks for missed merges.
|
||||
- **The window is two keys.** An open batch's window is a key whose time to live is `merge-window` (90 s),
|
||||
refreshed by each merge into the batch, beside a key set once when the batch opens whose time to live is
|
||||
`merge-window-at-most` (10 minutes). The first to expire closes the batch, heard by a watcher; no cutter
|
||||
polls. A stop that records merges again into the open batch closes its window too (ADR 0299 §3).
|
||||
- **A lone merge is cut at once**: no batch it shares a repository, module, graph edge or delivery group member
|
||||
with is assembling (ADR 0282 §2, ADR 0304 §1, ADR 0306).
|
||||
- **The plan.** A closed batch becomes one delivery plan: every commit in it, tiered along the module graph
|
||||
across repositories, at most one commit per repository, the latest of its trunk. It answers every merge in the
|
||||
batch, its own commits and the earlier merges a later commit contains (ADR 0276 §2, as ADR 0299 §7 changed it).
|
||||
- **A plan fixes its commits when it starts.** A plan starts at its first send, after mesh-delivery's word, and
|
||||
from then nothing that merges changes it. A plan still waiting for its word is folded into the next batch
|
||||
(ADR 0276 §4).
|
||||
- **Plans that share something run one after another.** A batch that shares a repository, a module or a graph
|
||||
edge with an open plan waits for it, and starts with every merge recorded since when that plan ends. A batch
|
||||
that shares none runs beside the open ones, so a hotfix is never held by an unrelated feature's delivery.
|
||||
- **The ends.** A plan done ends every delivery it answers: `delivered` for its own commits, `covered` for an
|
||||
earlier merge it carried by a later commit. A plan that failed fails its own commits' deliveries; each earlier
|
||||
merge it answered is then delivered on its own commit, the newest first, until one is delivered (ADR 0276 §3).
|
||||
- **Shown.** `plans` lists the open batch first, with its window's time left and its merges, and a batch waiting
|
||||
behind an open plan names that plan; the controller says the batch as `plan-moved` in the state `assembling`.
|
||||
|
||||
## A backup before the send
|
||||
|
||||
## A stop on a shared rollout
|
||||
*ADR 0214 §7, ADR 0233, ADR 0404.*
|
||||
|
||||
*2026-10-11, by ADR 0299 (issue 459), with its progressive insight of the same day.*
|
||||
When a delivery plan is ready to send, each node it reaches backs up, through its `node-backup` seat, every item
|
||||
the plan's moved modules keep there whose protection is a backup (a copy or a dump). An item protected by
|
||||
`redundancy`, one that says `backup: none`, and a cache are never asked, whatever their class. A node with no backup holder backs up
|
||||
nothing, and the plan says so for that node.
|
||||
|
||||
One rollout carries every merge of its batch, so ending a rollout for one delivery ends it for every other
|
||||
delivery on it. A stop therefore ends only the change it is for:
|
||||
- **The send waits for every backup.** The controller sends nothing of the plan to any node until every backup
|
||||
it asked has answered done.
|
||||
- **A failed backup holds the delivery.** A backup that fails, or does not answer within ten minutes, holds the
|
||||
delivery with the node, the module and the backup's own words as the reason, and raises a condition for the
|
||||
operator. A person's `release` asks the backups again, and the send goes on when they pass.
|
||||
- **What it costs.** About 85% of data modules back up in under 30 s; the slowest measured, postgres on the home
|
||||
server, about 165 s (research 055, finding 5). A plan moving no backed-up item costs nothing more.
|
||||
|
||||
## The send and the rollout
|
||||
|
||||
*ADR 0360, ADR 0362, ADR 0377, ADR 0406 §5.*
|
||||
|
||||
- **Every machine at once.** A plan's builds go to every machine that runs them in one step, tier by tier; a tier
|
||||
is asked once the tier before it is built and sent (ADR 0360 §1, §6). No machine goes first and nothing waits
|
||||
for a health reading. A module of policy `record` that would recreate a container or move data waits for a
|
||||
person (ADR 0377).
|
||||
- **Every move is said.** The controller says every move of a plan as `plan-moved`, a move a command makes among
|
||||
them, so mesh-delivery needs no periodic reconcile; it reads `delivery-walks` once when it starts.
|
||||
- **A rollout advances on events.** The `built` and sent events of a tier ask the next; the nodes' reports end
|
||||
the rollout (ADR 0360 §7, with ADR 0362's provisioning report). Each send puts a key whose time to live is
|
||||
ApplySilentAfter (fifteen minutes); its expiry names the machine silent. There is no rollout ticker.
|
||||
- **A failed apply is loud and fixed forward** (ADR 0360 §8); the node-engine's witness still puts back a
|
||||
controller or tool runner that does not come up (ADR 0360 §9).
|
||||
|
||||
## A stop on a shared plan
|
||||
|
||||
*ADR 0299.*
|
||||
|
||||
One plan carries every merge of its batch, so a stop ends only the change it is for:
|
||||
|
||||
- **A group's stop.** When a member after a failed one is stopped and its plan is open, mesh-delivery asks
|
||||
`delivery-stop` for that plan with the stopped members' merges to leave out. The controller ends the plan and
|
||||
records every other merge it answered again in the open batch of its kind, whose window the stop closes.
|
||||
- **What is left out with it.** A later merge of the stopped merge's repository, and a merge that moves a
|
||||
module a left-out merge moves, would deliver the stopped change; each is left out too, its delivery `stopped`,
|
||||
naming why.
|
||||
- **A person's stop.** `stop` of a delivery whose plan carries others first answers what it would take down, and
|
||||
acts on `way`: `again` (the others delivered again without it) or `walk` (the whole plan stopped).
|
||||
- **What it does not do.** What the stopped plan already sent stays on the machines it reached; putting it back
|
||||
is issue 509's.
|
||||
|
||||
- **A group's stop.** When a member after a failed one is stopped and its rollout is open, mesh-delivery asks
|
||||
`delivery-stop` for that rollout with the stopped members' merges to leave out, every member the same pass
|
||||
stops on that rollout in one ask. The controller ends the rollout, said as stopped, and rolls out every other merge
|
||||
it answered again.
|
||||
- **What is left out with it.** A later merge of the stopped merge's repository and branch contains it. A
|
||||
module is built at its branch, so a merge that moves a module a left-out merge moves would deliver it too.
|
||||
Each such merge is left out as well, until no more is, and its delivery ends `stopped`, naming the stopped
|
||||
merge it contains or whose modules it shares. An earlier merge of that repository sharing no module is
|
||||
rolled out again.
|
||||
- **Where the others go.** They join the open batch of their kind, where merges heard while the rollout ran
|
||||
already wait, so an older commit never rollouts after a newer one; only with none open do they form a new
|
||||
batch. The stop closes that batch's window, so the unrelated merges in it, and any heard after the stop
|
||||
before it is cut, are cut without the rest of their window. Its rollout withholds the left-out merges' modules
|
||||
where it would build them only as dependents, and its record names them. A batch not yet cut can be stopped without a merge too: the merge goes
|
||||
to a stopped record of its own, and the batch goes on without it. The stopped record, the batch and every
|
||||
merge moved are written in one transaction.
|
||||
- **The rollout's record.** The stopped rollout names the batch that rolls out its merges again, every merge it left
|
||||
out, and each module it already sent that nothing rollouts again, with the machines it reached. A delivery
|
||||
whose merge is rolled out again goes from `delivering` back to `published` (the row `rewalk`), or stays
|
||||
`published`, and follows the fresh rollout once it is cut. A delivery whose merge was left out, and every
|
||||
delivery on a rollout stopped whole, ends `stopped`, naming the rollout and the stop.
|
||||
- **A stopped merge never ships by a rollout.** mesh-delivery asks every open rollout or batch that names a stopped
|
||||
delivery's merge to leave it out, and never gives such a rollout its word. When the controller refuses, it
|
||||
stops the whole rollout.
|
||||
- **A person's stop.** `stop` of a delivery whose open rollout carries other deliveries does nothing at first: it
|
||||
answers with each of them, which of them contain this merge and end either way, and two ways given with
|
||||
`way`: `again` (the rollout ended and the others rolled out again without this merge) or `walk` (the whole rollout
|
||||
stopped, every delivery on it with it). A delivery alone on its rollout is stopped at once. A delivery is
|
||||
marked stopped only once its rollout is; a stop the controller refuses leaves it as it was.
|
||||
- **What it does not do.** What the stopped rollout already sent stays on the machines it reached, and its
|
||||
asked builds still register, so a later send may carry them further; the stop names the machines, and
|
||||
putting them back is issue 509's. A merge of the same repository waiting elsewhere, or one that moves the
|
||||
stopped merge's modules, contains the stopped change and delivers it when rolled out; the stop names each it
|
||||
can see, and holding a merged change back for good is a revert's job.
|
||||
## The delivery budget
|
||||
|
||||
*Decided 2026-10-10 by ADR 0282.
|
||||
Where it differs, this section replaces the window for every merge and the one rollout at a time above, slice by
|
||||
slice as each slice below is built; until a slice is built, the text above is what runs.*
|
||||
*ADR 0282, ADR 0304 §4, ADR 0313, ADR 0360 §12, ADR 0362 §5.*
|
||||
|
||||
**The target.** A delivery's **delivery time** runs from its merge to every node of its deploy plan that is heard
|
||||
from running its modules' builds and reporting them healthy. A **leaf module's** delivery budget is five minutes;
|
||||
a **core module's** (the controller's own path and the mesh's resolver) is ten. A delivery held for a person is
|
||||
shown and held to no budget. Measured on 51 rollouts on 2026-10-09 and 10, a leaf rollout took a median of 6.0 minutes
|
||||
(7 of 20 within five), a core rollout of one tier 5.6 (worst 8.8), and a controller rollout dragging two catalogue tiers
|
||||
14.2 (issue 338).
|
||||
A delivery's **delivery time** runs from its merge until every machine its plan sent has reported the send
|
||||
applied, and for a provisioned module also placed. A **leaf** delivery's budget is three minutes, a **core**
|
||||
delivery's ten. A delivery held for a person, a failed backup among them, is shown and held to no budget. Each
|
||||
phase is kept — the window, the wait behind a plan that shares something, the backups, the build, the send, the
|
||||
reports — and a delivery over its budget raises `delivery.<class>.over-budget`, naming its longest phase. The
|
||||
**fix time** (ADR 0313) runs from an issue's first move to `fixing` to its last fix delivered, and contains the
|
||||
delivery time.
|
||||
|
||||
*A wider measure, around the delivery budget (2026-10-12, ADR 0313, extending ADR 0282).* The delivery budget
|
||||
above runs from the merge; the operator's own target is wider, the **fix time**, which runs from an issue's
|
||||
first move to `fixing` to the moment its last fix is delivered. Each issue's class carries a **fix budget** too
|
||||
(five minutes leaf, ten core, the delivery budget's own figures), and the tracker splits a fix time into four
|
||||
phases — implementing, review, checks, and merge to running (the delivery time above) — so a fix over its
|
||||
budget is said loud, naming its longest phase, and clears at the next fix of its class delivered within budget.
|
||||
The delivery budget stays as written: a measure inside the fix budget, not replaced by it.
|
||||
## Builds without a toolchain image
|
||||
|
||||
**What changes in the rollout.**
|
||||
*ADR 0267, ADR 0350, ADR 0405.*
|
||||
|
||||
- **A lone merge is cut at once**: no batch assembling, and no member of its delivery group still unmerged.
|
||||
- **Independent rollouts run at once**: no shared repository, module or module-graph edge with any open rollout. One
|
||||
node judges one rollout at a time; a send to a node under another rollout's judgement waits for its verdict; no leaf
|
||||
build goes to a node where a core build is judged, and no leaf reading counts while a new controller is judged.
|
||||
- **The first-node gate reads by class**: three readings for a core module, as ADR 0236 decided; two for a leaf
|
||||
module with a long-running resource; one for a leaf module whose every resource is a tools bundle.
|
||||
- **A tier's first nodes are judged at once.**
|
||||
- **The bus's grants arrive before the gate** (ADR 0298): when a rollout's builds change the bus's user list (a new
|
||||
tool, subject or user), the machine holding the bus is sent that list alone, every build there kept, before
|
||||
anything is judged; a step that would carry anything else is refused unsent and said on the gate line.
|
||||
- **Every phase is kept**: the window, the wait, the build, the send, each reading, the rest's first healthy
|
||||
report; the plan shows an expected landing time from the last seven days' medians.
|
||||
- **One build module per language** (Go; Node and TypeScript) installs its toolchain as a declared package on
|
||||
every node holding the build seat, pinned to one version that is the same on every builder. The toolchain
|
||||
image and its module retire.
|
||||
- **Every builder runs the same toolchain, and says so.** Each builder names its toolchain versions with every
|
||||
build outcome, and the controller's self-check raises a condition while two builders differ.
|
||||
- **Builds and tests run directly**, as the build seat's own account on its node, in no container. A
|
||||
repository's `# mesh-check-toolchain` line selects which installed toolchain runs its check.
|
||||
- **A judge has an account of its own.** A judge built from a sibling's merge (ADR 0382 §3) is built and run as a
|
||||
second account of the Go build module, whose home, configuration and cache the build seat's account cannot write,
|
||||
so a check's script cannot change it.
|
||||
- **Every dependency comes by the version its lock file names**, checked against its checksum: the mesh's own
|
||||
code from the mesh's forge, its own packages from its package registry, any other from its public source.
|
||||
- **A builder fetches from the forge over SSH with a key of its own**, as the build seat's account, under a forge
|
||||
account for that builder that reads the mesh's repositories and pushes nothing.
|
||||
- **A build's reuse key** holds the installed toolchain's version and the build module's recipe version in place
|
||||
of the toolchain image's digest, so a toolchain update rebuilds only the modules of that language.
|
||||
|
||||
*Rollouts beside each other (2026-10-12, ADR 0306, extending ADR 0304, which narrowed the bullets above to what
|
||||
a change shares).* Building the rollouts at once found what those bullets left open, and ADR 0306 decides it:
|
||||
## One chain, on the bus alone
|
||||
|
||||
- **A rollout's builds are its own, never another's backlog.** A send composes a module whose build another open
|
||||
rollout owns at the build its node was last sent, and neither carries nor judges it: that build is moved and
|
||||
judged only by its own rollout, so a failed rollout puts back only its own builds. What waits on a node with no open
|
||||
rollout to move it is still backlog, carried and judged by the next gated send there (this changes ADR 0236 §17
|
||||
for builds an open rollout owns).
|
||||
- **A delivery group still assembles.** A merge whose delivery group has a member not yet merged waits for the
|
||||
merge window whether or not a batch is assembling, so a group's members roll out in one batch, in their order.
|
||||
- **Related means everything the rollouts would move.** A module in common is any module a rollout plans to build or
|
||||
send, the dependents of what its merges moved included; a change whose modules are not known yet is related to
|
||||
every change. Two related rollouts are neither cut beside each other nor judged beside each other on one node.
|
||||
- **A starved judgement is said.** A leaf judgement set aside by core builds three times in a row raises a
|
||||
warning naming the rollout and the core rollouts that set it aside.
|
||||
- **A node's own fault fails every judgement on it.** A node found unwell as a whole (a failed apply, an
|
||||
unhealthy network or engine) fails every judgement open on it, leaf and core, each rollout putting back only its
|
||||
own builds.
|
||||
- **A rollback beside another rollout's judgement** behaves as it does today; whether it should wait is tracked by
|
||||
issue 528.
|
||||
*ADR 0408.*
|
||||
|
||||
*How it is checked:* the controller's tests ADR 0306 lists, one per clause: an unsent build of rollout A left out
|
||||
of rollout B's send and not taken up as backlog while A is open; a group's second member joining its first's batch;
|
||||
overlapping dependents keeping two rollouts apart; the warning on the third set-aside; a failed apply failing every
|
||||
open judgement on its node.
|
||||
Module, commit, build, published package or artifact, rollout and assignment are one chain, each link
|
||||
answerable from the next. A **rollout** here is the controller's record of one batch's delivery; a delivery's own
|
||||
delivery plan (*A delivery* above) stays the delivery's, and names the rollout that delivers it.
|
||||
|
||||
*A busy container on the first node (2026-10-13, ADR 0335, citing ADR 0282 §1).* A container may declare `busy`,
|
||||
a command that says whether it holds work a replacement would lose; the build agent declares it, so a merge
|
||||
into the controller's repository no longer throws away the checks in flight. When a rollout's send would replace
|
||||
such a container while it says busy, the node-engine leaves it running, applies the rest, and replaces it once
|
||||
it says idle, or after 15 minutes at the latest. Only that node's rollout waits: the rollout says it waits for a busy
|
||||
container there, its first-node gate starts at the report that says the container replaced, and the wait is a
|
||||
phase of the delivery time of its own, never exempted from the delivery budget. A delivery it puts over budget
|
||||
is said over budget, with that wait as its longest phase. *How it is checked:* the controller's test that a
|
||||
rollout whose module's container waits as busy on its first node says so and starts that node's gate only once the
|
||||
node reports it replaced, and the walk-phase test that the wait is a measured phase between the send and the
|
||||
judgement.
|
||||
| Link | Names | Written by | Bucket |
|
||||
|---|---|---|---|
|
||||
| build | its module, repository and commit, its reuse key, what it published: an artifact by digest or a package release by name and version | the controller | `mesh-controller_builds` |
|
||||
| rollout | the commit it fixed per repository, its tiers, the builds it carried, the machine steps, and every assignment it changed with the build before and after; its states and transitions | the controller | `mesh-controller_rollouts` |
|
||||
| assignment | the module and node, the build, artifact or package and commit it runs, and the rollout that put it there | the controller | `mesh-controller_assignments` |
|
||||
|
||||
**Who builds what.**
|
||||
Beside the chain, on the bus too: deliveries and groups (mesh-delivery's `deliveries` and `groups`, ADR 0239 §37),
|
||||
one key per open pull request (mesh-delivery), and each open batch with its merges and its window's two keys (the
|
||||
controller, ADR 0407).
|
||||
|
||||
| Part | Owner | What it adds |
|
||||
|---|---|---|
|
||||
| the rollout's phases and readings | mesh-controller | each phase's start and end and each reading in the rollout and in `plan-moved`, the rest's first healthy report per node, `durations` of kind `walk-phase`; a numbered migration |
|
||||
| the delivery time | mesh-delivery | each delivery's delivery time, class and phases; the verb `times` (budgets, the week's median and worst per class, the deliveries over budget and their longest phase); a line atop `deliveries`; phases in `show` |
|
||||
| the condition | mesh-controller | a self-check probe reading `times`, raising `delivery.<class>.over-budget` (a warning) and clearing it at the next delivery of that class within budget; the line in `status` |
|
||||
| the lone merge | mesh-controller | the cut at once when nothing waits to join |
|
||||
| rollouts at once | mesh-controller | the independence test, one judgement per node, a core rollout's ground, a repository's next batch |
|
||||
| judgement by class | mesh-controller | the readings per class in the first-node gate; a tier's first nodes judged at once |
|
||||
| the landing time | mesh-controller, mesh-delivery | an expected landing time in `plans`, in `plan-moved`, and in `show` |
|
||||
|
||||
**Slices**, each one pull request per repository, each with its acceptance sentence:
|
||||
|
||||
| Slice | Delivers | Done when |
|
||||
|---|---|---|
|
||||
| 1 — the measure | the rollout's phases and readings (controller); the delivery time, `times` and the line in `deliveries` (mesh-delivery); the probe and its condition (controller) | `mesh-delivery.times` shows the week's leaf and core median and worst delivery time, and a leaf delivery over five minutes raises `delivery.leaf.over-budget` naming its longest phase |
|
||||
| 2 — no window for a lone merge | the cut at once | a lone leaf merge's build is asked within fifteen seconds of its merge, read from its rollout's phases |
|
||||
| 3 — judgement by class | the readings by class, a tier's first nodes at once | a tools-only leaf module passes its first-node gate on one reading, read from its rollout |
|
||||
| 4 — rollouts at once | the independence test and the per-node rule | two unrelated catalogue merges a minute apart are both delivered within five minutes, neither waiting for the other |
|
||||
| 5 — the landing time | the expected landing time | `plans` shows it for an open rollout, and the week's deliveries land within a minute of it at the median |
|
||||
|
||||
Issue 338's path precision is the planner's, tracked by issue 338; the core budget is met by controller merges
|
||||
only once it is built.
|
||||
|
||||
## A delivery group
|
||||
|
||||
- **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull
|
||||
requests. Two or more make a group. A group that has started delivering takes no new member.
|
||||
- **Order**: `after: <repository>` lines in a member's description, and the controller's `delivery-order`
|
||||
over the graph (ADR 0239 decision 4). The plan shows the order and why each pair is ordered: *declared*,
|
||||
*built by*, *version skew*, *engine before controller*, *by name*.
|
||||
- **Precedence** (ADR 0249): when rules order one pair both ways, the higher one decides. *Declared* comes
|
||||
first, then *built by* and *version skew*, then *engine before controller*. The pair shows the rule that
|
||||
won and the rules it won over. Rules of one rank both ways are a contradiction. The group is `rejected`,
|
||||
and the reason names both members, both rules and the `after:` line that would settle it.
|
||||
- **Check**: the members' heads composed together, one gate over every machine with all their definitions,
|
||||
each head merged into its own trunk's head first and named with that trunk commit (ADR 0323). It is asked
|
||||
by mesh-delivery through `delivery-check` when the group forms or a member's head moves, and posted on
|
||||
every member's head as `mesh/delivery-group`. A member rebased or pushed is a new head with its own check:
|
||||
it takes its old head's place in the group, even once the group is delivering, and the composed check runs
|
||||
again for the new heads instead of holding the group. A merge into a member's trunk that is not one of the
|
||||
group's members puts the composed check out of date: the group reads `checking` with the reason, and the
|
||||
composed check runs again once a member is judged again. Checked by mesh-delivery's
|
||||
`TestAGroupsComposedCheckGoesOutOfDateOnAMergeThatIsNotItsMember` and `TestARebasedMemberIsComposedAgainNotHeld`.
|
||||
- **Each member's own check reads its siblings** (ADR 0382): the group's other open members into a base of
|
||||
the same name are read beside a member's own check at their head merged onto their base, and a mesh-controller
|
||||
sibling's merge judges it, so changes that are only consistent together pass their own checks. The controller
|
||||
learns the siblings from mesh-delivery's `groups`; the verdict names each one. A sibling's new head, close or
|
||||
base move puts that verdict out of date, and the merge tool merges such a member only when
|
||||
`mesh/delivery-group` passed at the heads read and every sibling head read is reviewed.
|
||||
- **Delivering**: in order. A member waits in `published` until the one before is `delivered` or `covered`.
|
||||
A member published while its group's composed check is asked and not answered waits there for the answer;
|
||||
one held because that check had not passed is *lifted* back to `published` once it passes or warns for the
|
||||
heads that merged and its own rollout is still open (ADR 0270).
|
||||
- **A failed member**: the members after it are `stopped`, naming it, and each one's merge is left out of its
|
||||
rollout while the rollout's other merges are rolled out again (ADR 0299). The members before it stay delivered.
|
||||
Its own builds were put back at its gate.
|
||||
|
||||
A group's state is derived from its members and its composed check, and is shown, never stored apart from
|
||||
them.
|
||||
|
||||
## A move of the trunk asks every check again
|
||||
|
||||
*ADR 0395, changing ADR 0323 §4.* When the forge's holder hears a repository's trunk move, mesh-delivery asks the
|
||||
check of every active open pull request into that trunk against its new head at once, so a verdict is current by
|
||||
the time anyone looks.
|
||||
|
||||
- **A conflict is told, not checked.** Before asking a check, mesh-delivery asks the forge whether the head merges
|
||||
cleanly into the new head, which needs no builder. When it does not, no check is asked: its statuses fail naming the trunk commit and the files in
|
||||
conflict where the forge gives them, and it is told once on the pull request, not again while the same conflict
|
||||
stands. Every later trunk move and its next push ask the forge again; once it merges cleanly its check is asked.
|
||||
- **One check per pull request.** When a check against a newer trunk commit is asked for a pull request, its check
|
||||
against an older trunk commit is dropped if it waits and ended if it runs, said in the build log and on the
|
||||
queue, as a new head already ends its old head's check (issue 726). An ended check delivers no verdict, so none of its builds is reused (ADR 0350 §7 keeps only builds a verdict
|
||||
names); its scratch records wait for ADR 0350 §5's seven-day sweep. A try and a delivery's build are never touched.
|
||||
- **The queue's order.** Delivery builds first; then checks of pull requests marked ready; then the rest, the most
|
||||
recently pushed first.
|
||||
- **Inactive pull requests are left out.** One with no push and no ready mark for 7 days is not asked again on a
|
||||
trunk move; its statuses say out of date and that a push, a ready mark or a recheck asks its check. A recheck
|
||||
takes a delivery that is still proposed and whose verdict is out of date, and asks its check (issue 888).
|
||||
- **Groups.** For a group's members a move asks only each member's own check; the group's composed check runs again
|
||||
through those rechecks (ADR 0323 §6) and is never asked by the move itself. A move made by a member merging in
|
||||
the group's order asks the other members nothing, since they merge moments later; their verdicts are out of date
|
||||
until then, and one that does not merge is asked again at the next move, its next push or a recheck.
|
||||
|
||||
**Its cost** is at most one check asked per active, cleanly merging pull request per trunk move: about 550 a day at
|
||||
ADR 0323's measured load, fewer once conflicting and inactive ones are left out; ending an older check saves builder
|
||||
time, not checks asked. Issue 944 measures it once built.
|
||||
- **The bus is the only source.** Each bucket is written by the controller that holds the lease, by
|
||||
compare-and-set, and the controller reads its own buckets to decide; a rollout resumes from its key after a
|
||||
restart. No table of the controller's store holds builds, rollouts, batches or assignments, so no record mirror,
|
||||
audit or healer exists for them; the nodes, settings, declared-config and predefined-keys mirrors stay as
|
||||
ADR 0289 and ADR 0356 have them. A module reads a bucket by naming it in `reads`; mesh-config reads the
|
||||
assignments bucket.
|
||||
- **No key of the chain is removed.** Each bucket keeps up to 64 earlier revisions of each key as its history. A
|
||||
value is never cut to a summary and stays under the bus's largest message. The bus's storage is declared as data
|
||||
protected by a backup, so it is backed up nightly and before a delivery moves the bus.
|
||||
- **One read answers each question.** For any module on any node, one read of its assignment names the commit,
|
||||
the build, the artifact or package and the rollout that put it there; one read of a rollout lists every
|
||||
assignment it changed.
|
||||
- **The tables go by a numbered migration**: builds, rollouts and assignments are written once into the buckets,
|
||||
waiting build requests onto the work queues and an open batch into its keys; then the tables are dropped from the
|
||||
controller's store and its consolidated schema.
|
||||
- **The artifact store and the package registry keep the bytes**; the chain names them (ADR 0251).
|
||||
|
||||
## What every transition does
|
||||
|
||||
In this order, each by its owner:
|
||||
|
||||
1. **kept**: mesh-delivery puts the delivery in its state (`deliveries`, one key per delivery; `groups`, one
|
||||
key per group). Nothing is said before it is kept. What it owes outside is kept with it and retried, later
|
||||
each time, from ten seconds doubling to an hour. The forge's answer that what it is for does not exist ends
|
||||
it at once, and sixty failed tries end it too, each said among the owner's refusals (ADR 0250);
|
||||
1. **kept**: mesh-delivery puts the delivery in its state by compare-and-set before anything else; what it owes
|
||||
outside is kept with it and retried from ten seconds doubling to an hour, at most sixty times (ADR 0250);
|
||||
2. **said**: the event `mesh-delivery.transition` (and `mesh-delivery.group` when a group's derived state
|
||||
changes), with the delivery's id, from, to, event, why and plan summary, and no secret or address;
|
||||
3. **noted**: mesh-delivery asks the forge's holder (`gitea_note_append`) to append a line to the commit's
|
||||
note: the head while the delivery is off the trunk, the commit it landed as once it is on it, and at its
|
||||
end a line of what was executed there. The forge's holder writes it in its own repository as the forge's
|
||||
own account, and a line already there is not appended again;
|
||||
4. **reflected** (not for history, whose pull request the forge may no longer hold): the delivery's view, one comment on the pull request with the state, the plan, the machine
|
||||
steps and the group, kept current (`gitea_delivery_view`); the status `mesh/delivery` on the head, and
|
||||
`mesh/delivery-group` on every member's head (`gitea_commit_status`). Every status links to the pull
|
||||
request, `mesh/merge-gate` among them.
|
||||
changes), with no secret or address;
|
||||
3. **noted**: a line on the commit's note, appended by the forge module as the forge's own account;
|
||||
4. **reflected**: the delivery's view on the pull request, `mesh/delivery` on the head and `mesh/delivery-group`
|
||||
on every member's head.
|
||||
|
||||
A doubled step is refused by compare-and-set and by the bus's message de-duplication, which also replaces the
|
||||
forge module's files of what it announced (ADR 0406 §6).
|
||||
|
||||
## The verbs
|
||||
|
||||
**mesh-delivery's seat** (served by its holder; the first five read):
|
||||
**mesh-delivery's seat**:
|
||||
|
||||
| Verb | Answers or does |
|
||||
|---|---|
|
||||
| `deliveries` | every delivery not final, the final ones of the last day, and every final one `stalled` still lists, one line each; filtered by state, repository or group |
|
||||
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the rollout |
|
||||
| `deliveries` | every delivery not final, the final ones of the last day, and every one `stalled` still lists |
|
||||
| `show` | one delivery or group whole: plan, transitions, machine steps, order, the chain's links |
|
||||
| `groups` | every group with its members in order and its derived state |
|
||||
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
|
||||
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner |
|
||||
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
|
||||
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting rollout no live delivery holds |
|
||||
| `recheck` | a rejected or ready delivery back to proposed, with why; a delivery still proposed whose verdict is out of date has its check asked (ADR 0395) |
|
||||
| `release` | a held delivery to delivering, with why: a person's word |
|
||||
| `stop` | any delivery not final to stopped, with why; its rollout is ended through the controller. On a rollout that carries other deliveries it first answers what ending the rollout would take down, and acts on `way`: `again` or `walk` (ADR 0299) |
|
||||
| `close` | H2's only verb: the transition the table names for a stalled delivery, refused otherwise |
|
||||
| `retire-history` | held history to retired, with why: one by id, refused when the rule does not take it, or every held one, answering how many and naming those it left; `dry` changes nothing |
|
||||
| `stalled` | every delivery held past its bound, with the transition H2 may take |
|
||||
| `mark-ready` | a reviewer's mark at a head: the label and the `ready at` comment, by the mesh's own forge account |
|
||||
| `recheck` | a check asked again for a delivery whose verdict is out of date (ADR 0395 §6) |
|
||||
| `release` | a held delivery goes on, with why; for a failed backup it asks the backups again |
|
||||
| `stop` | a delivery to stopped, with why; on a shared plan it first answers what it would take down (ADR 0299) |
|
||||
| `close` | H2's only verb: the transition the table names for a stalled delivery |
|
||||
| `retire-history` | held history to retired (ADR 0250) |
|
||||
| `times` | each class's budget, the week's median and worst delivery time, and the deliveries over budget |
|
||||
|
||||
**The controller's verbs for it** (on the mesh-controller seat): `delivery-plan`, `delivery-order`,
|
||||
`delivery-check` (a group's heads composed, or one head checked again), `deliver`, `delivery-stop` (which
|
||||
takes the merges to leave out and rolls out the rollout's other merges again without them, ADR 0299),
|
||||
`delivery-walks`. A person has `plans go <plan> --why`, which does what `deliver` does under their name and
|
||||
is recorded as a hand act. The controller says every rollout it keeps as `plan-moved`, from a verb's command
|
||||
too.
|
||||
|
||||
**The forge's holder's tools for it**: `gitea_note_append`, `gitea_delivery_view`, `gitea_commit_status`. It
|
||||
also announces `pull.closed`, and a merge's head and the statuses the forge holds on it.
|
||||
|
||||
## The store
|
||||
|
||||
Key-value state on the bus, declared by the module (ADR 0201): `deliveries` and `groups`. One writer, the
|
||||
seat's holder, whose code changes state from a single goroutine. A final delivery is kept for thirty days and
|
||||
then removed. Its note on the commit keeps its record after that. The bus's own snapshot (ADR 0235) backs it
|
||||
up.
|
||||
**The controller's verbs for it**: `delivery-plan`, `delivery-order`, `delivery-check`, `deliver`,
|
||||
`delivery-stop`, `delivery-walks`, and `plans` (the open batch, the open plans, and a what-if for a repository
|
||||
and its files). A person has `plans go <plan> --why`.
|
||||
|
||||
## The bootstrap
|
||||
|
||||
- A rollout that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
|
||||
controller's own, started by the merge and witnessed on the machine. mesh-delivery records it.
|
||||
- Any other rollout waits for `deliver` only while the seat has a holder on record. With none on record, the
|
||||
controller starts it as it always did.
|
||||
- A waiting rollout is said by the controller whatever mesh-delivery says of itself: S16 raises
|
||||
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the rollout answers and
|
||||
`plans go <walk> --why` as the way on.
|
||||
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
|
||||
- A plan that moves the controller, the node-engine, the tool runner, the bus or mesh-delivery is the
|
||||
controller's own, started by the merge; mesh-delivery records it (ADR 0239 §31).
|
||||
- Any other plan waits for `deliver` while the seat has a holder on record; with none, the controller starts it.
|
||||
- A replaced controller or mesh-delivery carries on from the bus: the keys, the queues and the durable
|
||||
consumers hold every step, so nothing is lost or repeated (ADR 0406 §6).
|
||||
|
||||
## The switch
|
||||
## Acceptance tests
|
||||
|
||||
1. The controller's change (the seat, its verbs, `plan-moved`, the wait for a holder's word, `plans go`) is
|
||||
merged and rolls out as the core does. With no holder on record it behaves exactly as before.
|
||||
2. The catalogue's change (mesh-delivery, the forge holder's notes and view) is merged. mesh-delivery is built
|
||||
and published.
|
||||
3. A person assigns mesh-delivery to the control node. The seat now has a holder on record. Rollouts that start
|
||||
from then on wait for its word. Rollouts already open finish on their own and are adopted as deliveries in
|
||||
`delivering`.
|
||||
4. To step back, unassign it. Waiting rollouts are then started by the controller at its next pass.
|
||||
What the build proves before this design is implemented (research 055's direction):
|
||||
|
||||
## Phases
|
||||
1. A feature across the controller and the catalogue with a new seat verb is delivered as one group, with no
|
||||
`after:` line, no two steps and one composed check.
|
||||
2. An SDK change and a Go module using it are delivered as one group.
|
||||
3. A data module is backed up before a delivery moves it, and a failed backup holds the delivery.
|
||||
4. A build on any builder uses the installed toolchain with no container, and a toolchain update rebuilds only
|
||||
modules of that language when its version changes.
|
||||
5. Two merges into one repository within a minute leave each open pull request with one queued check, by the
|
||||
stream's rule.
|
||||
6. A ready mark is acted on within seconds of its event.
|
||||
7. A controller replaced in the middle of a rollout resumes it with no step lost or repeated.
|
||||
8. The merge window closes by its key's expiry.
|
||||
9. A hotfix to a module the running feature's delivery does not touch reaches the machines in minutes, beside it.
|
||||
10. For any module on any node, one read of its assignment names the commit, build, artifact or package and the
|
||||
rollout that put it there, and one read of that rollout lists every assignment it changed.
|
||||
11. Two merges into one repository while a delivery of that repository runs are both delivered by the next plan,
|
||||
with no person's release and nothing held in mesh-delivery.
|
||||
12. No table in the controller's store holds builds, rollouts or assignments.
|
||||
|
||||
| Phase | Delivers | Done when |
|
||||
|---|---|---|
|
||||
| A — the owner | mesh-delivery with its table, state, verbs, events and adoption; the controller's seat, verbs, `plan-moved` and the wait; the forge holder's view, notes and statuses; the group's order and composed check | the tests of ADR 0239's *How it is checked* pass; a merge on the live mesh with mesh-delivery held is delivered by it and shown by `deliveries` |
|
||||
| B — the conditions | S16, the controller's own `plan.<walk>.waiting`; probe D14 reads `stalled` and raises `delivery.<id>.stalled`; H2 calling `close` — **built 2026-10-07**, on a branch | a delivery stalled on purpose is raised and closed by H2 through `close`; a rollout never let go is raised by S16 |
|
||||
| C — the rollout itself | the per-tier rollout moved into mesh-delivery behind single-send verbs, the controller's planner kept only for the built-in path | decided by its own record once Phase A has delivered for a while |
|
||||
## What retires
|
||||
|
||||
- The "release branches" and the controller's judge of them (ADR 0358, superseded by ADR 0401).
|
||||
- The toolchain image and its module (ADR 0405).
|
||||
- The window cutter, the merge catch-up timer, the rollout ticker, mesh-delivery's tick, reconcile and
|
||||
ready-mark poll, the forge's poll as a trigger, the queue reordering, the forge-asking cancel, and the forge
|
||||
module's announcement files (ADR 0406, ADR 0407).
|
||||
- The controller's tables of builds, build requests, rollouts and their tiers, batches and merges, and
|
||||
assignments; the record mirrors, probe D17's audit and healer H6 for those kinds (ADR 0408).
|
||||
|
||||
## Milestones
|
||||
|
||||
Confirmed by the operator on 2026-10-17 (playbook 04). Each is one feature step of playbook 07; the next starts
|
||||
when the previous one's delivery is done. Milestones 5 and 6 are split into steps when they come up.
|
||||
|
||||
| # | Outcome | Acceptance tests | Repositories | Issues |
|
||||
|---|---|---|---|---|
|
||||
| 1 | A merge during a running delivery is never dropped and needs no person's release; the merge window lives on the bus; a rollout that shares nothing with an open one runs beside it | 8, 9, 11 | mesh-controller | 973 (with the rest of 525) |
|
||||
| 2 | A ready mark is acted on within seconds, and each pull request has at most one waiting check | 5, 6, 7 | mesh-controller, mesh-catalog (mesh-delivery, gitea) | 974, 975, 976 |
|
||||
| 3 | A feature with a new seat verb across the controller and the catalogue is delivered as one group: no `after:` line, one composed check, no "release branches" | 1 | mesh-controller, mesh-catalog (mesh-delivery) | 977, 978 |
|
||||
| 4 | A delivery backs up the data it moves first, and a failed backup holds it and is said | 3 | mesh-controller, mesh-catalog (mesh-delivery, the backup seat's holder) | 980 |
|
||||
| 5 | One read of a module on a node names the commit, build and rollout that put it there | 10, 12 | mesh-controller, mesh-catalog (mesh-config, nats) | 981 |
|
||||
| 6 | Builds use installed toolchains with no container, builders fetch over SSH with their own key, and an SDK change ships in one group with its users | 2, 4 | mesh-catalog (build modules, build-agent, ssh-client, gitea), mesh-controller, mesh-sdk | 979, 971 |
|
||||
|
||||
The issues are linked `depends-on` in this order; 976 depends on 975, and 979 also on 971.
|
||||
|
||||
## What is not decided here
|
||||
|
||||
- A web view of deliveries beyond the pull request's comment.
|
||||
- Whether `mesh/delivery-group` is required on the trunks. That is the operator's setting, through the forge
|
||||
module's protection tool.
|
||||
- Moving the per-tier rollout out of the controller (Phase C).
|
||||
- Whether `mesh/delivery-group` is required on the trunks: the operator's setting, through the forge module's
|
||||
protection tool.
|
||||
- How a builder's public SSH key reaches the forge: the forge's and the SSH client's mechanism.
|
||||
|
||||
Reference in New Issue
Block a user