Describe delivery as it runs after R4g: the parts, the build modules, the one check (issue 1296, H9)
This commit is contained in:
@@ -1,8 +1,8 @@
|
||||
---
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [mesh-controller, mesh-catalog modules/mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/build-agent]
|
||||
updated: 2026-10-17
|
||||
code: [mesh-controller, mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/mesh-build-go]
|
||||
updated: 2026-10-24
|
||||
decisions:
|
||||
- "ADR 0239"
|
||||
- "ADR 0267"
|
||||
@@ -21,70 +21,62 @@ decisions:
|
||||
|
||||
# Delivery — from a merge to the nodes, as it runs
|
||||
|
||||
What runs on the mesh's main branches on 2026-10-16, read from the code and the live record by research 055
|
||||
(findings 1 to 12). It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) replaces, milestone
|
||||
by milestone. Until 2026-10-16 this document described the predecessor's pipeline, which no longer runs.
|
||||
What runs on 2026-10-24, after the delivery of release 4 and release 5's first half (milestone R4g, design 47),
|
||||
read from the live controller (`mesh-controller.modules`, `.seats`, `.plans`), the delivery checker's `checker_status`
|
||||
and the machines' module lists. It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) has not yet
|
||||
finished: the node agent applying wants and the deletion of plans are release 6, and the controller still plans,
|
||||
orders and sends a delivery's builds to the machines. Until 2026-10-23 this document described the pipeline with the
|
||||
build agent, the build queue and the two statuses `mesh/merge-gate` and `mesh/repo-check`, which no longer run.
|
||||
|
||||
## Who does what
|
||||
|
||||
- **mesh-delivery** owns deliveries and delivery groups (ADR 0239): one delivery per commit, a group per branch
|
||||
name shared across repositories. It records each transition in its state on the bus (key-value buckets
|
||||
`deliveries` and `groups`), marks pull requests, asks the controller to deliver, and merges a pull request marked
|
||||
ready at its head once both required checks pass against the trunk's head (ADR 0364) and the operator approved
|
||||
the merge into main on a channel that proves who answered (ADR 0398 §3).
|
||||
- **The controller** plans, orders, sends and judges. It computes a delivery plan for a repository and the files a
|
||||
merge changes, moves only modules whose build source changed with their build dependents (ADR 0267), and
|
||||
reuses a merge check's build of the same reuse key (ADR 0350).
|
||||
- **The builders** (holders of the build seat) take build requests and checks from a work-queue stream on the bus
|
||||
and build inside a toolchain image another module provides, with the network off; on 2026-10-13 no node had Go
|
||||
or npm installed (issue 626).
|
||||
- **The forge module** publishes every forge event on the bus through the forge's system webhook since 2026-10-14
|
||||
(issue 729), and still polls the forge every 30 s, or every 300 s once it hears events. It keeps files of what it
|
||||
announced so it does not announce twice.
|
||||
- **The parts of `mesh-delivery`** are modules of one repository, each a process with its own bus credential that
|
||||
watches keys and writes keys (design 47, ADR 0429). Running now: the forge bridge (`mesh-delivery-forge-bridge`,
|
||||
novox), the indexer (novox), the builder (`mesh-delivery-builder`, shanks), the checker (novox), the gate
|
||||
(`mesh-delivery-gate`, which holds the seat `mesh-delivery`, novox), the teller (novox), the watchman (novox), the
|
||||
status view (shanks) and the assigner (`mesh-delivery-assigner`, ace).
|
||||
- **The forge bridge** turns the forge's events into stream messages and is the only part that calls the forge. The
|
||||
forge module (`gitea`, novox) still announces them through the forge's system webhook.
|
||||
- **The indexer** computes each module's fingerprint at the merge preview; **the builder** claims a build once per
|
||||
fingerprint and orders it as an ask on the stream of the artifact's language.
|
||||
- **The build modules** (`mesh-build-go` and `mesh-build-node` on three machines, `mesh-build-image` on one) hold the
|
||||
node seats `node-build-go`, `node-build-node` and `node-build-image`. Each pulls asks from the bus's work-queue
|
||||
streams (deliveries first, then checks, then re-checks) while its node has a free slot of build capacity; no part
|
||||
chooses the node. The seat `node-build-agent` and the module `build-agent` have no holder any more, and the seat
|
||||
`mesh-build-machine` has none either.
|
||||
- **The checker** (`mesh-delivery-checker`) runs the merge check: a change's tests through the build module of each
|
||||
language, running only packages whose inputs changed, and writes the one status `mesh/check`. It also serves a
|
||||
**try** (`checker_try`, `checker_try_read`): the same check on a pushed branch, before a pull request. `checker_status`
|
||||
shows the steps in flight by node and the candidates still without a verdict.
|
||||
- **The gate** merges a delivery group's pull requests once the check, the review and the operator's Approve agree on
|
||||
the same heads. The forge requires no status on any repository (ADR 0534).
|
||||
- **The assigner** composes the machines a change touches (the composition check the checker reads) and writes which
|
||||
fingerprint each node should want, by its six rules; it only compares wants with what the controller sends.
|
||||
- **The controller** (`mesh-controller`, novox) plans, orders, sends and judges. It computes a delivery plan for the
|
||||
files a merge changes, moves only modules whose build source changed with their build dependents, and writes the
|
||||
chain's links it owns. On 2026-10-24 the seat still lists 64 verbs, `queue` and `replay` among them; whether
|
||||
those still do work was not read here, and the build agent they served has no holder.
|
||||
|
||||
## From a merge to a rollout
|
||||
## From a merge to a delivery plan
|
||||
|
||||
- **The merge window.** The controller collects merges into an open batch; each merge restarts the window, which
|
||||
closes after 90 seconds without a merge or 10 minutes after the batch's first merge (ADR 0276). A lone merge is
|
||||
cut at once (ADR 0282 §2, as ADR 0304 §1 changed it). The batch and its merges are rows in the controller's
|
||||
store, cut by a timer every 5 s, with a catch-up timer every 5 minutes for merges no batch holds.
|
||||
- **The rollout.** A closed batch becomes one rollout, at most one commit per repository, tiered along the module
|
||||
graph. Rollouts run one at a time: a batch cut while a rollout is open is queued behind it, whatever it shares
|
||||
with it. Running unrelated rollouts side by side (ADR 0282 §3, ADR 0304 §2) is decided and not built (issue 525). A rollout's builds go to every machine at once, with no first-machine gate, and the rollout ends on
|
||||
the machines' reports or after fifteen minutes of silence (ADR 0360). A controller timer advances rollouts every
|
||||
30 s.
|
||||
- **A merge during a rollout.** Its place in the next batch is a store row the open rollout's holder writes. On
|
||||
2026-10-16, when another controller worked the plans, a merge was not recorded and waited until a person released
|
||||
it (issue 921).
|
||||
- **"Release branches".** A pull request into a branch other than a trunk is judged by the controller built from that
|
||||
branch (ADR 0358); feature 57 shipped about 37 packages that way over about 2.5 days and held the controller for
|
||||
55.7 h.
|
||||
|
||||
## Checks
|
||||
|
||||
- Every pull request's head gets `mesh/merge-gate` (the composed machines) and `mesh/repo-check` (the repository's
|
||||
`merge-check.sh`), each judged against one trunk commit (ADR 0323). A trunk move asks the check of every active,
|
||||
cleanly merging pull request again (ADR 0395).
|
||||
- A delivery group's members are each checked beside their siblings' heads, and the group also gets a composed
|
||||
check, `mesh/delivery-group`; each member keeps its own gate beside it (ADR 0382).
|
||||
- The queue's order (delivery builds first, then ready checks, then the rest) is kept by pausing the worker and
|
||||
deleting and republishing a request (issue 558); an old head's check is ended after asking the forge for the
|
||||
current head (issue 726).
|
||||
- mesh-delivery ticks every 10 s, reconciles with the controller every 30 s and polls the forge for ready marks
|
||||
every minute, although the forge module already publishes labeled, commented and synchronized events.
|
||||
- A pull request's head is built and checked as it comes in. One status, `mesh/check`, carries the verdict; a delivery
|
||||
group has one verdict for every member. A move of the trunk asks the checks again, and a running check stops itself
|
||||
when its head or trunk commit moves.
|
||||
- A merge is made by the gate on the operator's Approve; the controller then makes a delivery plan, tiered along the
|
||||
module graph, and sends it. `mesh-controller.plans` lists the plans and the delivery plans' tiers; a plan that
|
||||
fails on a node says so and is retried by hand.
|
||||
- **hq** builds no module: the checker runs hq's Go tests (`go test ./...`) as its check.
|
||||
|
||||
## Where the record of what runs lives
|
||||
|
||||
Builds and build requests, rollouts with their tiers, batches and merges, and assignments are rows in the
|
||||
controller's store; artifacts are blobs in the artifact store by digest (ADR 0251); packages are versions in the
|
||||
package registry. The controller writes three record mirrors to the bus: assignments, declared config and
|
||||
predefined keys; of the five ADR 0289 §7 decided, the nodes, rollouts, builds and settings mirrors are not in the
|
||||
code. An assignment names its module and node but not the build, artifact or rollout that put it there: which commit
|
||||
runs where, and which delivery brought it, takes three verbs and a join by hand.
|
||||
|
||||
No backup is taken before a delivery moves a module that holds data; backups run nightly and on demand (ADR 0214).
|
||||
Fingerprints, builds, verdicts and wants are keys in the bus's key-value buckets of the parts; builds' files are in
|
||||
the artifact store by digest; packages are versions in the package registry. Plans, rollouts with their tiers,
|
||||
assignments and settings are still rows in the controller's store. An assignment names its module and node but not
|
||||
the build, artifact or plan that put it there.
|
||||
|
||||
## What is not covered
|
||||
|
||||
The integration tests run only in each repository's merge check; there are no end-to-end tests of delivery yet,
|
||||
and the lab is not used (the operator, 2026-10-16).
|
||||
The node agent does not yet apply wants: a node runs what the controller sent. The checker's left-running keys of
|
||||
build modules that stopped (`checker_status`'s `left_running`) are listed and hold no place. The integration tests run
|
||||
in each repository's check; there are no end-to-end tests of delivery yet, and the lab is not used (the operator,
|
||||
2026-10-16).
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [mesh-tools, mesh-host, mesh-controller]
|
||||
updated: 2026-10-17
|
||||
updated: 2026-10-24
|
||||
decisions:
|
||||
- "ADR 0152"
|
||||
- "ADR 0175"
|
||||
@@ -92,12 +92,12 @@ command, not inline, because one call blocks until it answers or times out.
|
||||
|
||||
**These already answer directly, not as a stated future.** Beside every assigned module's tools,
|
||||
the tool runner asks the `mesh-controller` seat for its own verbs the same way (ADR 0154): measured
|
||||
live, the mesh-controller runtime is one instance, mesh-wide, and it answers 63 verbs for that one
|
||||
live, the mesh-controller runtime is one instance, mesh-wide, and it answers 64 verbs for that one
|
||||
seat (`status`, `push`, `assign`, `build`, `artifacts` and the rest), and the `mesh-controller`
|
||||
*module* answers none of its own — it serves no module tools, only its seat's, exactly as design 33
|
||||
described while it was still only designed. The 56-to-93 seat-verb counts each node's tool runner
|
||||
answers (below) are a different thing: the tool runner serves the verbs of whichever node-scoped
|
||||
seats are held on that node (`node-backup`, `node-build-agent`, `node-service-manager` among them),
|
||||
seats are held on that node (`node-backup`, `node-build-go`, `node-service-manager` among them),
|
||||
on behalf of the modules holding them, not the mesh-controller seat's — that one is served by the
|
||||
mesh-controller runtime itself, a separate process, and the tool runner only asks it and relays what
|
||||
it answers.
|
||||
|
||||
Reference in New Issue
Block a user