diff --git a/03-DESIGN/00-as-is/04-delivery.md b/03-DESIGN/00-as-is/04-delivery.md index 2351daeb..828f7f00 100644 --- a/03-DESIGN/00-as-is/04-delivery.md +++ b/03-DESIGN/00-as-is/04-delivery.md @@ -1,100 +1,90 @@ --- layer: as-is status: implemented -code: [hal] -updated: 2026-08-23 +code: [mesh-controller, mesh-catalog modules/mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/build-agent] +updated: 2026-10-17 decisions: - - "ADR 0010" - - "ADR 0010" - - "ADR 0010" + - "ADR 0239" + - "ADR 0267" + - "ADR 0276" + - "ADR 0282" + - "ADR 0289" + - "ADR 0304" + - "ADR 0350" + - "ADR 0358" + - "ADR 0360" + - "ADR 0364" + - "ADR 0382" + - "ADR 0395" + - "ADR 0398" --- -# Delivery — from a push to a running node +# Delivery — from a merge to the nodes, as it runs -One trigger, three silos, and a fan-out point that is the most consequential boundary in the -mesh. +What runs on the mesh's main branches on 2026-10-16, read from the code and the live record by research 055 +(findings 1 to 12). It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) replaces, milestone +by milestone. Until 2026-10-16 this document described the predecessor's pipeline, which no longer runs. -## The trigger +## Who does what -A push to the forge. The forge calls a webhook; the receiving module verifies its signature -and emits an event. The coordinator resolves which modules the pushed commits affect, orders -them into dependency levels, and creates a pipeline per level. +- **mesh-delivery** owns deliveries and delivery groups (ADR 0239): one delivery per commit, a group per branch + name shared across repositories. It records each transition in its state on the bus (key-value buckets + `deliveries` and `groups`), marks pull requests, asks the controller to deliver, and merges a pull request marked + ready at its head once both required checks pass against the trunk's head (ADR 0364) and the operator approved + the merge into main on a channel that proves who answered (ADR 0398 §3). +- **The controller** plans, orders, sends and judges. It computes a delivery plan for a repository and the files a + merge changes, moves only modules whose build source changed with their build dependents (ADR 0267), and + reuses a merge check's build of the same reuse key (ADR 0350). +- **The builders** (holders of the build seat) take build requests and checks from a work-queue stream on the bus + and build inside a toolchain image another module provides, with the network off; on 2026-10-13 no node had Go + or npm installed (issue 626). +- **The forge module** publishes every forge event on the bus through the forge's system webhook since 2026-10-14 + (issue 729), and still polls the forge every 30 s, or every 300 s once it hears events. It keeps files of what it + announced so it does not announce twice. -There is no second path. A manual trigger exists for a module the push detection missed, and -using it routinely is a sign the detection is wrong rather than a workflow. +## From a merge to a rollout -Detection is the pipeline's most fragile input. It has failed for reasons that have nothing to -do with the change: a webhook truncating its commit list on a large merge, a forge address -whose port broke the module-path matching. The characteristic outcome is the bad one — **a -merge that created no pipeline, and nothing said so**. +- **The merge window.** The controller collects merges into an open batch; each merge restarts the window, which + closes after 90 seconds without a merge or 10 minutes after the batch's first merge (ADR 0276). A lone merge is + cut at once (ADR 0282 §2, as ADR 0304 §1 changed it). The batch and its merges are rows in the controller's + store, cut by a timer every 5 s, with a catch-up timer every 5 minutes for merges no batch holds. +- **The rollout.** A closed batch becomes one rollout, at most one commit per repository, tiered along the module + graph. Rollouts that share no repository, module or graph edge run side by side; overlapping ones wait (ADR 0282 + §3, ADR 0304). A rollout's builds go to every machine at once, with no first-machine gate, and the rollout ends on + the machines' reports or after fifteen minutes of silence (ADR 0360). A controller timer advances rollouts every + 30 s. +- **A merge during a rollout.** Its place in the next batch is a store row the open rollout's holder writes. On + 2026-10-16, when another controller worked the plans, a merge was not recorded and waited until a person released + it (issue 921). +- **"Release branches".** A pull request into a branch other than a trunk is judged by the controller built from that + branch (ADR 0358); feature 57 shipped about 37 packages that way over about 2.5 days and held the controller for + 55.7 h. -## Three silos +## Checks -Cardinality is the whole point, and the three differ -(ADR 0010): +- Every pull request's head gets `mesh/merge-gate` (the composed machines) and `mesh/repo-check` (the repository's + `merge-check.sh`), each judged against one trunk commit (ADR 0323). A trunk move asks the check of every active, + cleanly merging pull request again (ADR 0395). +- A delivery group's members are each checked beside their siblings' heads, and the group also gets a composed + check, `mesh/delivery-group`; each member keeps its own gate beside it (ADR 0382). +- The queue's order (delivery builds first, then ready checks, then the rest) is kept by pausing the worker and + deleting and republishing a request (issue 558); an old head's check is ended after asking the forge for the + current head (issue 726). +- mesh-delivery ticks every 10 s, reconciles with the controller every 30 s and polls the forge for ready marks + every minute, although the forge module already publishes labeled, commented and synchronized events. -| Silo | Runs | Where | Does | -|---|---|---|---| -| **build** | once per module feature | the build node | compile and bundle into a self-contained output | -| **publish** | once per module feature | the build node | package that output and upload it; a package-registry feature publishes here | -| **deploy** | once per module feature **per node** | every assigned node | install, configure, start, verify | +## Where the record of what runs lives -Commands and events are addressed per feature, not per module. +Builds and build requests, rollouts with their tiers, batches and merges, and assignments are rows in the +controller's store; artifacts are blobs in the artifact store by digest (ADR 0251); packages are versions in the +package registry. The controller writes three record mirrors to the bus: assignments, declared config and +predefined keys; of the five ADR 0289 §7 decided, the nodes, rollouts, builds and settings mirrors are not in the +code. An assignment names its module and node but not the build, artifact or rollout that put it there: which commit +runs where, and which delivery brought it, takes three verbs and a join by hand. -Build hands over a **staged tree**, not a package. Packaging belongs to publish, so a failed -upload retries by re-packaging rather than by re-sending something stale, and build never needs -to know how each module composes its artifact. The handover is a tree in a known location, -because the build's own working directory is reference-counted and may be gone by the time a -later stage runs. - -## The artifact - -The artifact is **build output** — compiled and bundled with its dependency graph inlined — -never a filtered copy of source (ADR 0010). -A deploy is extract-and-run and touches no network. - -The consequence is the whole cost of the decision: **anything not in the build output does not -ship.** Every file kind had to be brought into that rule separately, and each was discovered by -something silently not happening after deploy — migrations reading a source layout, provisioning -scripts reading a source layout, selection files never packaged at all. - -## The fan-out, and the build node - -After publish, work fans out to every assigned node. The build node is the only node that has -already passed through two silos when this happens, and that asymmetry has its own defect -class: anything advancing a node's stage must account for **both** pre-fan-out stages. Code -that knew only about the first parked the build node forever while every other node deployed -cleanly — and the recovery sweeper, which knew the same subset, reported nothing to recover. - -A recovery mechanism that knows less than the thing it guards is worse than none, because it -reports success over a stall it cannot see. - -## Levels - -A level completes before the next begins, so a module builds against its dependencies as they -were just published. The shared library is at level zero, which means anything that breaks it -breaks the first module of every cascade. - -## What green proves - -**A green pipeline proves transport, not effect.** The stages report that a message was -dispatched and accepted, which is not the same as the thing being running, correct, or present. - -This is the mesh's most consistent failure shape and it is not incidental to the design — it is -what the stage reporting currently measures. Documented instances include a service reported -started when the container command merely returned, an image pull failure that did not fail the -deploy, a package install that 404ed from every mirror while the job went green (see -[`04-ISSUES/001`](../../04-ISSUES/001-failed-package-install-reports-success/00-report.md)), -and a node left on old code after a failed artifact download with a version marker that had -already advanced. - -A verify stage exists to close this gap. It has been built and, for a period, was never -scheduled, because the coordinator's stage list did not include it. +No backup is taken before a delivery moves a module that holds data; backups run nightly and on demand (ADR 0214). ## What is not covered -The end-to-end harness for this pipeline has not built since the workspace was removed, and -nothing reports that nothing runs it -([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). The -pipeline's coverage is currently assumed rather than checked — which is the same class of claim -this repository exists to make people stop making. +The integration tests run only in each repository's merge check; there are no end-to-end tests of delivery yet, +and the lab is not used (the operator, 2026-10-16). diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index 7af5b4cd..d583467c 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -485,7 +485,19 @@ What the build proves before this design is implemented (research 055's directio ## Milestones -Set by playbook 04 at the handoff, after the operator confirms them. +Confirmed by the operator on 2026-10-17 (playbook 04). Each is one feature step of playbook 07; the next starts +when the previous one's delivery is done. Milestones 5 and 6 are split into steps when they come up. + +| # | Outcome | Acceptance tests | Repositories | Issues | +|---|---|---|---|---| +| 1 | A merge during a running delivery is never dropped and needs no person's release; the merge window lives on the bus | 8, 11 | mesh-controller | 973 | +| 2 | A ready mark is acted on within seconds, and each pull request has at most one waiting check | 5, 6, 7 | mesh-controller, mesh-catalog (mesh-delivery, gitea) | 974, 975, 976 | +| 3 | A feature with a new seat verb across the controller and the catalogue is delivered as one group: no `after:` line, one composed check, no "release branches" | 1, 9 | mesh-controller, mesh-catalog (mesh-delivery) | 977, 978 | +| 4 | A delivery backs up the data it moves first, and a failed backup holds it and is said | 3 | mesh-controller, mesh-catalog (mesh-delivery, the backup seat's holder) | 980 | +| 5 | One read of a module on a node names the commit, build and rollout that put it there | 10, 12 | mesh-controller, mesh-catalog (mesh-config, nats) | 981 | +| 6 | Builds use installed toolchains with no container, builders fetch over SSH with their own key, and an SDK change ships in one group with its users | 2, 4 | mesh-catalog (build modules, build-agent, ssh-client, gitea), mesh-controller, mesh-sdk | 979, 971 | + +The issues are linked `depends-on` in this order. ## What is not decided here