Merge pull request 'Design 47's milestones and issues, and the as-is delivery as it runs today' (#393) from handoff/47-milestones into main
This commit was merged in pull request #393.
This commit is contained in:
@@ -30,7 +30,8 @@ A design becomes **milestones**, and each milestone becomes **issues**:
|
||||
4. **Break it into milestones.** Write them into the design, in a section *Milestones*: for each, its number,
|
||||
its one-sentence outcome, the acceptance tests of the design it proves, the repositories it touches, and
|
||||
what it depends on. Order them so each is usable as soon as it is delivered, the smallest useful one
|
||||
first.
|
||||
first. A milestone is named by what it lets someone do; how it ships is shaped by the plan its changes would
|
||||
have, asked through `mesh-delivery.what-if`: the modules it moves and their fan-out (ADR 0401 §6).
|
||||
5. **Ask the operator to confirm the milestones**, in a short table: number, outcome, the acceptance tests it
|
||||
proves, repositories, the parts each repository builds. Change what they change.
|
||||
6. **Open the issues.** For each confirmed milestone, `issue-tracker.open` one issue per repository part, its
|
||||
|
||||
@@ -140,12 +140,16 @@ never into the running one, unless it blocks the running one.
|
||||
5. **The reviewer marks each pull request ready as soon as the acceptance is shown** with
|
||||
`mesh-delivery.mark-ready` (the delivery of the head it approved, who reviewed it and why), which writes the
|
||||
label `mesh/ready` and then `ready at <full sha>`, the order the forge counts (issue 913); mesh-delivery merges it when its checks pass
|
||||
(ADR 0364). The operator's word is waited for only when the step changes what the operator sees, removes
|
||||
and the operator approved the merge into main on Telegram, the channel that proves who answered today (ADR 0364,
|
||||
ADR 0398 §3; an Approve at the terminal is issue 953). Beyond that
|
||||
Approve, the operator's word is waited for only when the step changes what the operator sees, removes
|
||||
data, or needs a step only the operator's terminal can take; the lead says which before the step starts.
|
||||
|
||||
**One exception: a catalogue change that uses a new manifest word.** The catalogue's merge check reads
|
||||
manifests with the controller the mesh runs, so the controller's pull request is merged and delivered first;
|
||||
then `mesh-delivery.recheck` the catalogue's and merge it on the new verdict. Plan such a feature as two steps.
|
||||
The exception ends when milestone 3 of design 47 is delivered (issue 977, ADR 0402, ADR 0403): the group's composed
|
||||
check, judged by the group's controller, is then each member's gate, and the feature is one group.
|
||||
|
||||
## Phase 4 — Deliver
|
||||
|
||||
|
||||
@@ -1,100 +1,90 @@
|
||||
---
|
||||
layer: as-is
|
||||
status: implemented
|
||||
code: [hal]
|
||||
updated: 2026-08-23
|
||||
code: [mesh-controller, mesh-catalog modules/mesh-delivery, mesh-catalog modules/gitea, mesh-catalog modules/build-agent]
|
||||
updated: 2026-10-17
|
||||
decisions:
|
||||
- "ADR 0010"
|
||||
- "ADR 0010"
|
||||
- "ADR 0010"
|
||||
- "ADR 0239"
|
||||
- "ADR 0267"
|
||||
- "ADR 0276"
|
||||
- "ADR 0282"
|
||||
- "ADR 0289"
|
||||
- "ADR 0304"
|
||||
- "ADR 0350"
|
||||
- "ADR 0358"
|
||||
- "ADR 0360"
|
||||
- "ADR 0364"
|
||||
- "ADR 0382"
|
||||
- "ADR 0395"
|
||||
- "ADR 0398"
|
||||
---
|
||||
|
||||
# Delivery — from a push to a running node
|
||||
# Delivery — from a merge to the nodes, as it runs
|
||||
|
||||
One trigger, three silos, and a fan-out point that is the most consequential boundary in the
|
||||
mesh.
|
||||
What runs on the mesh's main branches on 2026-10-16, read from the code and the live record by research 055
|
||||
(findings 1 to 12). It is what [to-be 47](../01-to-be/47-delivery-from-commit-to-delivered.md) replaces, milestone
|
||||
by milestone. Until 2026-10-16 this document described the predecessor's pipeline, which no longer runs.
|
||||
|
||||
## The trigger
|
||||
## Who does what
|
||||
|
||||
A push to the forge. The forge calls a webhook; the receiving module verifies its signature
|
||||
and emits an event. The coordinator resolves which modules the pushed commits affect, orders
|
||||
them into dependency levels, and creates a pipeline per level.
|
||||
- **mesh-delivery** owns deliveries and delivery groups (ADR 0239): one delivery per commit, a group per branch
|
||||
name shared across repositories. It records each transition in its state on the bus (key-value buckets
|
||||
`deliveries` and `groups`), marks pull requests, asks the controller to deliver, and merges a pull request marked
|
||||
ready at its head once both required checks pass against the trunk's head (ADR 0364) and the operator approved
|
||||
the merge into main on a channel that proves who answered (ADR 0398 §3).
|
||||
- **The controller** plans, orders, sends and judges. It computes a delivery plan for a repository and the files a
|
||||
merge changes, moves only modules whose build source changed with their build dependents (ADR 0267), and
|
||||
reuses a merge check's build of the same reuse key (ADR 0350).
|
||||
- **The builders** (holders of the build seat) take build requests and checks from a work-queue stream on the bus
|
||||
and build inside a toolchain image another module provides, with the network off; on 2026-10-13 no node had Go
|
||||
or npm installed (issue 626).
|
||||
- **The forge module** publishes every forge event on the bus through the forge's system webhook since 2026-10-14
|
||||
(issue 729), and still polls the forge every 30 s, or every 300 s once it hears events. It keeps files of what it
|
||||
announced so it does not announce twice.
|
||||
|
||||
There is no second path. A manual trigger exists for a module the push detection missed, and
|
||||
using it routinely is a sign the detection is wrong rather than a workflow.
|
||||
## From a merge to a rollout
|
||||
|
||||
Detection is the pipeline's most fragile input. It has failed for reasons that have nothing to
|
||||
do with the change: a webhook truncating its commit list on a large merge, a forge address
|
||||
whose port broke the module-path matching. The characteristic outcome is the bad one — **a
|
||||
merge that created no pipeline, and nothing said so**.
|
||||
- **The merge window.** The controller collects merges into an open batch; each merge restarts the window, which
|
||||
closes after 90 seconds without a merge or 10 minutes after the batch's first merge (ADR 0276). A lone merge is
|
||||
cut at once (ADR 0282 §2, as ADR 0304 §1 changed it). The batch and its merges are rows in the controller's
|
||||
store, cut by a timer every 5 s, with a catch-up timer every 5 minutes for merges no batch holds.
|
||||
- **The rollout.** A closed batch becomes one rollout, at most one commit per repository, tiered along the module
|
||||
graph. Rollouts run one at a time: a batch cut while a rollout is open is queued behind it, whatever it shares
|
||||
with it. Running unrelated rollouts side by side (ADR 0282 §3, ADR 0304 §2) is decided and not built (issue 525). A rollout's builds go to every machine at once, with no first-machine gate, and the rollout ends on
|
||||
the machines' reports or after fifteen minutes of silence (ADR 0360). A controller timer advances rollouts every
|
||||
30 s.
|
||||
- **A merge during a rollout.** Its place in the next batch is a store row the open rollout's holder writes. On
|
||||
2026-10-16, when another controller worked the plans, a merge was not recorded and waited until a person released
|
||||
it (issue 921).
|
||||
- **"Release branches".** A pull request into a branch other than a trunk is judged by the controller built from that
|
||||
branch (ADR 0358); feature 57 shipped about 37 packages that way over about 2.5 days and held the controller for
|
||||
55.7 h.
|
||||
|
||||
## Three silos
|
||||
## Checks
|
||||
|
||||
Cardinality is the whole point, and the three differ
|
||||
(ADR 0010):
|
||||
- Every pull request's head gets `mesh/merge-gate` (the composed machines) and `mesh/repo-check` (the repository's
|
||||
`merge-check.sh`), each judged against one trunk commit (ADR 0323). A trunk move asks the check of every active,
|
||||
cleanly merging pull request again (ADR 0395).
|
||||
- A delivery group's members are each checked beside their siblings' heads, and the group also gets a composed
|
||||
check, `mesh/delivery-group`; each member keeps its own gate beside it (ADR 0382).
|
||||
- The queue's order (delivery builds first, then ready checks, then the rest) is kept by pausing the worker and
|
||||
deleting and republishing a request (issue 558); an old head's check is ended after asking the forge for the
|
||||
current head (issue 726).
|
||||
- mesh-delivery ticks every 10 s, reconciles with the controller every 30 s and polls the forge for ready marks
|
||||
every minute, although the forge module already publishes labeled, commented and synchronized events.
|
||||
|
||||
| Silo | Runs | Where | Does |
|
||||
|---|---|---|---|
|
||||
| **build** | once per module feature | the build node | compile and bundle into a self-contained output |
|
||||
| **publish** | once per module feature | the build node | package that output and upload it; a package-registry feature publishes here |
|
||||
| **deploy** | once per module feature **per node** | every assigned node | install, configure, start, verify |
|
||||
## Where the record of what runs lives
|
||||
|
||||
Commands and events are addressed per feature, not per module.
|
||||
Builds and build requests, rollouts with their tiers, batches and merges, and assignments are rows in the
|
||||
controller's store; artifacts are blobs in the artifact store by digest (ADR 0251); packages are versions in the
|
||||
package registry. The controller writes three record mirrors to the bus: assignments, declared config and
|
||||
predefined keys; of the five ADR 0289 §7 decided, the nodes, rollouts, builds and settings mirrors are not in the
|
||||
code. An assignment names its module and node but not the build, artifact or rollout that put it there: which commit
|
||||
runs where, and which delivery brought it, takes three verbs and a join by hand.
|
||||
|
||||
Build hands over a **staged tree**, not a package. Packaging belongs to publish, so a failed
|
||||
upload retries by re-packaging rather than by re-sending something stale, and build never needs
|
||||
to know how each module composes its artifact. The handover is a tree in a known location,
|
||||
because the build's own working directory is reference-counted and may be gone by the time a
|
||||
later stage runs.
|
||||
|
||||
## The artifact
|
||||
|
||||
The artifact is **build output** — compiled and bundled with its dependency graph inlined —
|
||||
never a filtered copy of source (ADR 0010).
|
||||
A deploy is extract-and-run and touches no network.
|
||||
|
||||
The consequence is the whole cost of the decision: **anything not in the build output does not
|
||||
ship.** Every file kind had to be brought into that rule separately, and each was discovered by
|
||||
something silently not happening after deploy — migrations reading a source layout, provisioning
|
||||
scripts reading a source layout, selection files never packaged at all.
|
||||
|
||||
## The fan-out, and the build node
|
||||
|
||||
After publish, work fans out to every assigned node. The build node is the only node that has
|
||||
already passed through two silos when this happens, and that asymmetry has its own defect
|
||||
class: anything advancing a node's stage must account for **both** pre-fan-out stages. Code
|
||||
that knew only about the first parked the build node forever while every other node deployed
|
||||
cleanly — and the recovery sweeper, which knew the same subset, reported nothing to recover.
|
||||
|
||||
A recovery mechanism that knows less than the thing it guards is worse than none, because it
|
||||
reports success over a stall it cannot see.
|
||||
|
||||
## Levels
|
||||
|
||||
A level completes before the next begins, so a module builds against its dependencies as they
|
||||
were just published. The shared library is at level zero, which means anything that breaks it
|
||||
breaks the first module of every cascade.
|
||||
|
||||
## What green proves
|
||||
|
||||
**A green pipeline proves transport, not effect.** The stages report that a message was
|
||||
dispatched and accepted, which is not the same as the thing being running, correct, or present.
|
||||
|
||||
This is the mesh's most consistent failure shape and it is not incidental to the design — it is
|
||||
what the stage reporting currently measures. Documented instances include a service reported
|
||||
started when the container command merely returned, an image pull failure that did not fail the
|
||||
deploy, a package install that 404ed from every mirror while the job went green (see
|
||||
[`04-ISSUES/001`](../../04-ISSUES/001-failed-package-install-reports-success/00-report.md)),
|
||||
and a node left on old code after a failed artifact download with a version marker that had
|
||||
already advanced.
|
||||
|
||||
A verify stage exists to close this gap. It has been built and, for a period, was never
|
||||
scheduled, because the coordinator's stage list did not include it.
|
||||
No backup is taken before a delivery moves a module that holds data; backups run nightly and on demand (ADR 0214).
|
||||
|
||||
## What is not covered
|
||||
|
||||
The end-to-end harness for this pipeline has not built since the workspace was removed, and
|
||||
nothing reports that nothing runs it
|
||||
([`04-ISSUES/005`](../../04-ISSUES/005-pipeline-test-harness-unbuildable/00-report.md)). The
|
||||
pipeline's coverage is currently assumed rather than checked — which is the same class of claim
|
||||
this repository exists to make people stop making.
|
||||
The integration tests run only in each repository's merge check; there are no end-to-end tests of delivery yet,
|
||||
and the lab is not used (the operator, 2026-10-16).
|
||||
|
||||
@@ -485,7 +485,19 @@ What the build proves before this design is implemented (research 055's directio
|
||||
|
||||
## Milestones
|
||||
|
||||
Set by playbook 04 at the handoff, after the operator confirms them.
|
||||
Confirmed by the operator on 2026-10-17 (playbook 04). Each is one feature step of playbook 07; the next starts
|
||||
when the previous one's delivery is done. Milestones 5 and 6 are split into steps when they come up.
|
||||
|
||||
| # | Outcome | Acceptance tests | Repositories | Issues |
|
||||
|---|---|---|---|---|
|
||||
| 1 | A merge during a running delivery is never dropped and needs no person's release; the merge window lives on the bus; a rollout that shares nothing with an open one runs beside it | 8, 9, 11 | mesh-controller | 973 (with the rest of 525) |
|
||||
| 2 | A ready mark is acted on within seconds, and each pull request has at most one waiting check | 5, 6, 7 | mesh-controller, mesh-catalog (mesh-delivery, gitea) | 974, 975, 976 |
|
||||
| 3 | A feature with a new seat verb across the controller and the catalogue is delivered as one group: no `after:` line, one composed check, no "release branches" | 1 | mesh-controller, mesh-catalog (mesh-delivery) | 977, 978 |
|
||||
| 4 | A delivery backs up the data it moves first, and a failed backup holds it and is said | 3 | mesh-controller, mesh-catalog (mesh-delivery, the backup seat's holder) | 980 |
|
||||
| 5 | One read of a module on a node names the commit, build and rollout that put it there | 10, 12 | mesh-controller, mesh-catalog (mesh-config, nats) | 981 |
|
||||
| 6 | Builds use installed toolchains with no container, builders fetch over SSH with their own key, and an SDK change ships in one group with its users | 2, 4 | mesh-catalog (build modules, build-agent, ssh-client, gitea), mesh-controller, mesh-sdk | 979, 971 |
|
||||
|
||||
The issues are linked `depends-on` in this order; 976 depends on 975, and 979 also on 971.
|
||||
|
||||
## What is not decided here
|
||||
|
||||
|
||||
Reference in New Issue
Block a user