|
|
|
@@ -1,100 +1,90 @@
|
|
|
|
|
---
|
|
|
|
|
layer: as-is
|
|
|
|
|
status: implemented
|
|
|
|
|
code: [hal]
|
|
|
|
|
updated: 2026-08-23
|
|
|
|
|
code: [mesh-controller, mesh-catalog modules/mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/build-agent]
|
|
|
|
|
updated: 2026-10-17
|
|
|
|
|
decisions:
|
|
|
|
|
- "ADR 0010"
|
|
|
|
|
- "ADR 0010"
|
|
|
|
|
- "ADR 0010"
|
|
|
|
|
- "ADR 0239"
|
|
|
|
|
- "ADR 0267"
|
|
|
|
|
- "ADR 0276"
|
|
|
|
|
- "ADR 0282"
|
|
|
|
|
- "ADR 0289"
|
|
|
|
|
- "ADR 0304"
|
|
|
|
|
- "ADR 0350"
|
|
|
|
|
- "ADR 0358"
|
|
|
|
|
- "ADR 0360"
|
|
|
|
|
- "ADR 0364"
|
|
|
|
|
- "ADR 0382"
|
|
|
|
|
- "ADR 0395"
|
|
|
|
|
- "ADR 0398"
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# Delivery — from a push to a running node
|
|
|
|
|
# Delivery — from a merge to the nodes, as it runs
|
|
|
|
|
|
|
|
|
|
One trigger, three silos, and a fan-out point that is the most consequential boundary in the
|
|
|
|
|
mesh.
|
|
|
|
|
What runs on the mesh's main branches on 2026-10-16, read from the code and the live record by research 055
|
|
|
|
|
(findings 1 to 12). It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) replaces, milestone
|
|
|
|
|
by milestone. Until 2026-10-16 this document described the predecessor's pipeline, which no longer runs.
|
|
|
|
|
|
|
|
|
|
## The trigger
|
|
|
|
|
## Who does what
|
|
|
|
|
|
|
|
|
|
A push to the forge. The forge calls a webhook; the receiving module verifies its signature
|
|
|
|
|
and emits an event. The coordinator resolves which modules the pushed commits affect, orders
|
|
|
|
|
them into dependency levels, and creates a pipeline per level.
|
|
|
|
|
- **mesh-delivery** owns deliveries and delivery groups (ADR 0239): one delivery per commit, a group per branch
|
|
|
|
|
name shared across repositories. It records each transition in its state on the bus (key-value buckets
|
|
|
|
|
`deliveries` and `groups`), marks pull requests, asks the controller to deliver, and merges a pull request marked
|
|
|
|
|
ready at its head once both required checks pass against the trunk's head (ADR 0364) and the operator approved
|
|
|
|
|
the merge into main on a channel that proves who answered (ADR 0398 §3).
|
|
|
|
|
- **The controller** plans, orders, sends and judges. It computes a delivery plan for a repository and the files a
|
|
|
|
|
merge changes, moves only modules whose build source changed with their build dependents (ADR 0267), and
|
|
|
|
|
reuses a merge check's build of the same reuse key (ADR 0350).
|
|
|
|
|
- **The builders** (holders of the build seat) take build requests and checks from a work-queue stream on the bus
|
|
|
|
|
and build inside a toolchain image another module provides, with the network off; on 2026-10-13 no node had Go
|
|
|
|
|
or npm installed (issue 626).
|
|
|
|
|
- **The forge module** publishes every forge event on the bus through the forge's system webhook since 2026-10-14
|
|
|
|
|
(issue 729), and still polls the forge every 30 s, or every 300 s once it hears events. It keeps files of what it
|
|
|
|
|
announced so it does not announce twice.
|
|
|
|
|
|
|
|
|
|
There is no second path. A manual trigger exists for a module the push detection missed, and
|
|
|
|
|
using it routinely is a sign the detection is wrong rather than a workflow.
|
|
|
|
|
## From a merge to a rollout
|
|
|
|
|
|
|
|
|
|
Detection is the pipeline's most fragile input. It has failed for reasons that have nothing to
|
|
|
|
|
do with the change: a webhook truncating its commit list on a large merge, a forge address
|
|
|
|
|
whose port broke the module-path matching. The characteristic outcome is the bad one — **a
|
|
|
|
|
merge that created no pipeline, and nothing said so**.
|
|
|
|
|
- **The merge window.** The controller collects merges into an open batch; each merge restarts the window, which
|
|
|
|
|
closes after 90 seconds without a merge or 10 minutes after the batch's first merge (ADR 0276). A lone merge is
|
|
|
|
|
cut at once (ADR 0282 §2, as ADR 0304 §1 changed it). The batch and its merges are rows in the controller's
|
|
|
|
|
store, cut by a timer every 5 s, with a catch-up timer every 5 minutes for merges no batch holds.
|
|
|
|
|
- **The rollout.** A closed batch becomes one rollout, at most one commit per repository, tiered along the module
|
|
|
|
|
graph. Rollouts run one at a time: a batch cut while a rollout is open is queued behind it, whatever it shares
|
|
|
|
|
with it. Running unrelated rollouts side by side (ADR 0282 §3, ADR 0304 §2) is decided and not built (issue 525). A rollout's builds go to every machine at once, with no first-machine gate, and the rollout ends on
|
|
|
|
|
the machines' reports or after fifteen minutes of silence (ADR 0360). A controller timer advances rollouts every
|
|
|
|
|
30 s.
|
|
|
|
|
- **A merge during a rollout.** Its place in the next batch is a store row the open rollout's holder writes. On
|
|
|
|
|
2026-10-16, when another controller worked the plans, a merge was not recorded and waited until a person released
|
|
|
|
|
it (issue 921).
|
|
|
|
|
- **"Release branches".** A pull request into a branch other than a trunk is judged by the controller built from that
|
|
|
|
|
branch (ADR 0358); feature 57 shipped about 37 packages that way over about 2.5 days and held the controller for
|
|
|
|
|
55.7 h.
|
|
|
|
|
|
|
|
|
|
## Three silos
|
|
|
|
|
## Checks
|
|
|
|
|
|
|
|
|
|
Cardinality is the whole point, and the three differ
|
|
|
|
|
(ADR 0010):
|
|
|
|
|
- Every pull request's head gets `mesh/merge-gate` (the composed machines) and `mesh/repo-check` (the repository's
|
|
|
|
|
`merge-check.sh`), each judged against one trunk commit (ADR 0323). A trunk move asks the check of every active,
|
|
|
|
|
cleanly merging pull request again (ADR 0395).
|
|
|
|
|
- A delivery group's members are each checked beside their siblings' heads, and the group also gets a composed
|
|
|
|
|
check, `mesh/delivery-group`; each member keeps its own gate beside it (ADR 0382).
|
|
|
|
|
- The queue's order (delivery builds first, then ready checks, then the rest) is kept by pausing the worker and
|
|
|
|
|
deleting and republishing a request (issue 558); an old head's check is ended after asking the forge for the
|
|
|
|
|
current head (issue 726).
|
|
|
|
|
- mesh-delivery ticks every 10 s, reconciles with the controller every 30 s and polls the forge for ready marks
|
|
|
|
|
every minute, although the forge module already publishes labeled, commented and synchronized events.
|
|
|
|
|
|
|
|
|
|
| Silo | Runs | Where | Does |
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
| **build** | once per module feature | the build node | compile and bundle into a self-contained output |
|
|
|
|
|
| **publish** | once per module feature | the build node | package that output and upload it; a package-registry feature publishes here |
|
|
|
|
|
| **deploy** | once per module feature **per node** | every assigned node | install, configure, start, verify |
|
|
|
|
|
## Where the record of what runs lives
|
|
|
|
|
|
|
|
|
|
Commands and events are addressed per feature, not per module.
|
|
|
|
|
Builds and build requests, rollouts with their tiers, batches and merges, and assignments are rows in the
|
|
|
|
|
controller's store; artifacts are blobs in the artifact store by digest (ADR 0251); packages are versions in the
|
|
|
|
|
package registry. The controller writes three record mirrors to the bus: assignments, declared config and
|
|
|
|
|
predefined keys; of the five ADR 0289 §7 decided, the nodes, rollouts, builds and settings mirrors are not in the
|
|
|
|
|
code. An assignment names its module and node but not the build, artifact or rollout that put it there: which commit
|
|
|
|
|
runs where, and which delivery brought it, takes three verbs and a join by hand.
|
|
|
|
|
|
|
|
|
|
Build hands over a **staged tree**, not a package. Packaging belongs to publish, so a failed
|
|
|
|
|
upload retries by re-packaging rather than by re-sending something stale, and build never needs
|
|
|
|
|
to know how each module composes its artifact. The handover is a tree in a known location,
|
|
|
|
|
because the build's own working directory is reference-counted and may be gone by the time a
|
|
|
|
|
later stage runs.
|
|
|
|
|
|
|
|
|
|
## The artifact
|
|
|
|
|
|
|
|
|
|
The artifact is **build output** — compiled and bundled with its dependency graph inlined —
|
|
|
|
|
never a filtered copy of source (ADR 0010).
|
|
|
|
|
A deploy is extract-and-run and touches no network.
|
|
|
|
|
|
|
|
|
|
The consequence is the whole cost of the decision: **anything not in the build output does not
|
|
|
|
|
ship.** Every file kind had to be brought into that rule separately, and each was discovered by
|
|
|
|
|
something silently not happening after deploy — migrations reading a source layout, provisioning
|
|
|
|
|
scripts reading a source layout, selection files never packaged at all.
|
|
|
|
|
|
|
|
|
|
## The fan-out, and the build node
|
|
|
|
|
|
|
|
|
|
After publish, work fans out to every assigned node. The build node is the only node that has
|
|
|
|
|
already passed through two silos when this happens, and that asymmetry has its own defect
|
|
|
|
|
class: anything advancing a node's stage must account for **both** pre-fan-out stages. Code
|
|
|
|
|
that knew only about the first parked the build node forever while every other node deployed
|
|
|
|
|
cleanly — and the recovery sweeper, which knew the same subset, reported nothing to recover.
|
|
|
|
|
|
|
|
|
|
A recovery mechanism that knows less than the thing it guards is worse than none, because it
|
|
|
|
|
reports success over a stall it cannot see.
|
|
|
|
|
|
|
|
|
|
## Levels
|
|
|
|
|
|
|
|
|
|
A level completes before the next begins, so a module builds against its dependencies as they
|
|
|
|
|
were just published. The shared library is at level zero, which means anything that breaks it
|
|
|
|
|
breaks the first module of every cascade.
|
|
|
|
|
|
|
|
|
|
## What green proves
|
|
|
|
|
|
|
|
|
|
**A green pipeline proves transport, not effect.** The stages report that a message was
|
|
|
|
|
dispatched and accepted, which is not the same as the thing being running, correct, or present.
|
|
|
|
|
|
|
|
|
|
This is the mesh's most consistent failure shape and it is not incidental to the design — it is
|
|
|
|
|
what the stage reporting currently measures. Documented instances include a service reported
|
|
|
|
|
started when the container command merely returned, an image pull failure that did not fail the
|
|
|
|
|
deploy, a package install that 404ed from every mirror while the job went green (see
|
|
|
|
|
[`04-ISSUES/001`](../../04-ISSUES/001-failed-package-install-reports-success/00-report.md)),
|
|
|
|
|
and a node left on old code after a failed artifact download with a version marker that had
|
|
|
|
|
already advanced.
|
|
|
|
|
|
|
|
|
|
A verify stage exists to close this gap. It has been built and, for a period, was never
|
|
|
|
|
scheduled, because the coordinator's stage list did not include it.
|
|
|
|
|
No backup is taken before a delivery moves a module that holds data; backups run nightly and on demand (ADR 0214).
|
|
|
|
|
|
|
|
|
|
## What is not covered
|
|
|
|
|
|
|
|
|
|
The end-to-end harness for this pipeline has not built since the workspace was removed, and
|
|
|
|
|
nothing reports that nothing runs it
|
|
|
|
|
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). The
|
|
|
|
|
pipeline's coverage is currently assumed rather than checked — which is the same class of claim
|
|
|
|
|
this repository exists to make people stop making.
|
|
|
|
|
The integration tests run only in each repository's merge check; there are no end-to-end tests of delivery yet,
|
|
|
|
|
and the lab is not used (the operator, 2026-10-16).
|
|
|
|
|