Design 47 G: write in the review of the composition runs and test runs

The four reviewers by architectural domain found results crossing candidates, a read of a
build seat's state, fixed durations and caps, unstated seat contract fields and a rerun rule
that differed between the two kinds; the lead's answers A-466 to A-466l settle each. The
lines move to a file of their own, G-24, so G-2 stays under its bound. The split of the main
file and G-22 waits until after G, as the operator decided at 10:31 CET.
This commit is contained in:
2026-11-17 10:31:28 +01:00
parent 77318ca60e
commit 6b7adc5ec2
7 changed files with 291 additions and 159 deletions
+3 -3
View File
@@ -625,9 +625,9 @@ controller (the planner, the rollout) until release 6 of ADR 0429; from each rel
every composition step of the candidate whose module is placed on the machine reads its verdict (design 47 G §718).
Not a **composition**, one module's resources composed for one node; nor the **composition step**, the pipeline
step that reads one composition run per machine.
- **test run** — the one run of a lint, unit test or integration test pipeline step for one level and one set of
tested packages at their digests, kept by the runner in its state `test-runs`; every artifact's pipeline step naming
the same packages reads its result (design 47 G §724). Not a **pipeline step**, each artifact's own, nor a **run
- **test run** — the one run of a lint, unit test or integration test pipeline step for one candidate, one level and
one set of tested packages at their digests, kept by the runner in its state `test-runs`; every artifact's pipeline step of that candidate
naming the same packages reads its result (design 47 G §724). Not a **pipeline step**, each artifact's own, nor a **run
on request**, a whole pipeline that delivers nothing.
- **merge check** — what a pull request carries before it may merge, judging the merge of its head into a
named **trunk commit**, made in a clone of its own on a build node and never pushed (ADR 0323). One status,
@@ -1972,16 +1972,23 @@ ADR 0667's checks, of milestone G itself:
`condition-raised` (whole, condition keeper and a fixture observer, mesh-catalog and mesh-sdk). Live: after G's
delivery, a node-engine restarted with the full seat list clears its `seat-list-unread`, read with
`mesh-conditions.conditions`.
108. **One composition run per candidate and machine** (design 47 G §718 to §723). Three composition steps of one
catalog revision on two machines start two composition runs and put two run asks, each step reading both
verdicts; a machine whose verdict blames one module fails that module's composition step alone and says it as a
warning in the others; steps asked again together start one run per machine that did not pass (whole, assigner
and the node-engine's composer, mesh-delivery and `mesh-host`). Live: in the first publish after G's delivery, the
composition run asks of a revision number at most its machines, and the state `inputs` stays under half its cap.
109. **One test run per level and tested packages** (design 47 G §724 to §726). A module whose program and tools
108. **One composition run per candidate and machine** (design 47 G §718 to §723, §727 to §731). Two composition
steps of one candidate started at once make one key, one request and one run ask; three steps of one revision on
two machines start two runs, each step reading both verdicts. A machine whose verdict blames one module fails that
module's composition step alone and says it as a warning in the others; a verdict with no `blamed` is judged as
today, naming no culprits. A run with no `step-ended` within its bound ends `could not run` with its readers; steps
asked again together start one run per machine that did not pass and move every former reader. Both drop tools end
the readers, and a reader-only account calling one is refused (whole, assigner and the node-engine's composer,
mesh-delivery and `mesh-host`). Live: in the first publish after G's delivery, the composition run asks of a
revision number at most its machines, read with `assigner_compositions` and `assigner_inputs`.
109. **One test run per level and tested packages** (design 47 G §724 to §731). A module whose program and tools
bundle test one directory runs its unit and integration tests once per level, both artifacts' pipeline steps
ending with that result and naming it in `shared`, a failing test failing once, and `pipeline-ended` counting the
module's tests once (whole, runner and Go build module, mesh-delivery and mesh-catalog).
module's tests once. A rerun that passes moves both artifacts' steps and the pipeline ends again; a runner
restarted during a run takes it up again; a replaced reader leaves the ask to the others; at the cap a step ends
`could not run` naming the fill; a second revision with the same digests runs a failed test again (whole, runner
and Go build module, mesh-delivery and mesh-catalog). Live: each module's test steps run once per level, read with
`runner_test_runs`.
## What retires
@@ -25,7 +25,6 @@ decisions:
- "ADR 0448"
- "ADR 0363"
- "ADR 0283"
- "ADR 0267"
- "ADR 0240"
---
@@ -113,143 +112,6 @@ One file of milestone G's part of [design 47](../47-delivery-from-commit-to-deli
pipelines up again. Live: once the checker is removed no module claims the seat `mesh-delivery-checker`. Tests 64,
84 and 88.
### One composition run per candidate and machine, one test run per tested packages
*§718 to §728 come from milestone G's live delivery of 2026-11-17. The 24 pipelines of one catalog revision asked 96
composition runs, one per pipeline and machine, and 80 more when they were asked again; each run kept its own copy of
a node's input of 1.1 to 1.5 MB in the assigner's state `inputs`, which reached its 64 MiB cap twice. The indexer's
and the teller's same failing tests failed once per artifact. The operator, 2026-11-17 about 10:20 CET: "In G, now".
Each line is a choice within ADR 0666 §7, §8 and §17, settled by the principle it names.*
- §718 **One composition run per candidate and machine.** (§354, A-226, A-243, A-252; ADR 0666 §17; principle 7) The
assigner, holding the seat `mesh-wants`, composes each machine once per candidate, whatever number of composition
steps name it. The candidate is named by its key (A-226): `<branch>.<revision>` for a branch pipeline, `run-<id>` for
a run on request. The first composition step of a candidate that needs a machine starts that machine's composition
run: one `compose-candidate` ask of the node, its request kept once in `inputs`, one `run` ask on `node-build-go`.
Every other composition step of that candidate whose module is placed on that machine (A-252) reads that run's
verdict. A run's request depends only on the candidate and the machine (the candidate's manifests, verdicts and
seat definitions, depth compose, the one machine, the case `change`), never on the module whose pipeline asked it,
so one verdict serves every module's pipeline. The composer program is chosen once, when the run starts, by A-252's
and A-451's rule, and recorded in the run. This replaces "one run ask per machine" for each composition step (A-243,
A-252) with one per candidate and machine.
- §719 **Kept in the assigner's state `composition-runs`.** (§354, §85; ADR 0666 §4, §7; principles 1, 2 and 7) Each
composition run is one key of the assigner's old `state` section `composition-runs`, whose `max-mib` is 16. The key
is `<candidate key>.<machine>`, each part escaped as a key's name (`keys.EscapeName`). The step that starts the run
writes the key create-only, so two steps of one candidate starting at once start one run and the second reads it.
The value holds:
- `candidate` and `machine`;
- `program`: repository, commit, directory, and the node-engine fingerprint it was chosen from;
- `input`: the digest of the kept request;
- `ask`: the id of the build seat's run ask;
- `started`;
- `readers`: the step keys of the composition steps reading the run, each added and removed by compare-and-set;
- at the run's end, `ended`, `outcome` (`passed`, `failed` or `could not run`, with its words) and `verdict` (that
machine's composer verdict, §720).
A step waiting on a running composition run hears its end through the assigner's own watch of that key (a module
watches its own state, principle 1). A restart of the assigner loses nothing: at its start the assigner takes up
again each key with no `ended`, by its `ask`'s `step-ended` (A-240), and ends `could not run` a key whose `ask` the
build seat no longer holds, naming the ask. The assigner deletes a key a day after its `ended`, and a key with no
`ended` a day after its `started`, ending each of its readers `could not run`; both are judged as the state `inputs`
is (A-465b). At the cap no run is started, and its steps end `could not run`, naming the fill (as A-465a).
- §720 **A machine's verdict names the modules it blames.** (ADR 0666 §17; A-452; principles 10 and 3) The composer's
verdict of a machine (mesh-delivery-sdk's `composer.Machine`, answered by the node-engine's `mesh-compose`) gains
`blamed`, a list of `{module, problem}`. It names each module that the candidate's composition of the machine holds
or refuses and the composition without the candidate does not (the modules in its input's `HeldBefore` are left
out). The culprits are found by the node-engine's own rule of holding one module at a time until the problem ends,
as the indexer's validator names a manifest whose removal ends a problem; two modules owning one path are each
blamed, with that path. `composes` stays false when the machine does not compose with every module of the
candidate. A machine that does not compose and blames no module fails on the node's own account (its own
composition, or no module composed at all, as A-452 has it).
- §721 **One module's fault fails only its own pipelines.** (§720; ADR 0666 §9, §17; A-452; principles 10 and 3) A
composition step judges each of its machines from that machine's verdict:
- `failed` when the verdict blames the step's module, or when the machine does not compose and blames no module;
- `could not run` when the composition run could not run;
- `passed` otherwise, with a warning for each module blamed there that is not the step's own (`<machine> holds
<module>: <problem>`). Another module's fault is so said in every pipeline it touches and fails none of them.
The step's outcome then follows from its machines, as today: failed before could not run before passed (§354). A
machine that is away passes with a warning and is not composed (§64).
- §722 **What asking a composition step again does.** (§83 `rerun`; ADR 0666 §11, §24; principle 7) The runner's
`rerun` of a composition step asks the assigner for the step again, marked as a rerun (§73). A machine that passed in
the step keeps its verdict. For each machine whose verdict in the step was not `passed`, the step:
- joins its candidate's and machine's composition run while that run is running;
- reads that run's verdict when the run started after the step's last end;
- otherwise starts a new run, replacing the ended key by compare-and-set.
So twenty composition steps of one revision asked again together start one run per machine. A new run's verdict
counts for no other step until that step is itself asked again or asked for the first time.
- §723 **How a composition run ends and is said.** (§73, §116, §496, §119, A-182; ADR 0666 §7, §21; principles 7 and 1)
A composition run says no event of its own. Its end is the build seat's `step-ended` of its `ask`, which the
assigner alone hears (A-240). Each composition step reading the run says its own `step-ended` on `mesh-wants`, as
today. That event's output names, per machine, the run it read (`shared`: the run's key and its `ask`) and the
verdict. So the runner, the forge module's `failed-steps` and the delivery record hear every pipeline's result from
the step's holder, as before (ADR 0666 §21). Each reading step says its `step-progress` once a minute while it waits
(A-262). A step that is superseded or cancelled leaves `readers`. The run's `ask` is dropped (§119, A-182) only when
no reader is left, so a superseded pipeline never takes away another pipeline's composition run.
- §724 **One test run per level and tested packages.** (§22, §25, §27, §416; ADR 0666 §8, §17; ADR 0267 §1;
principle 7) The runner asks a lint, unit test or integration test step of a build seat once per level and tested
packages, whatever number of artifacts' steps name them. A module's program and its tools bundle are built from
different import closures (ADR 0267 §1), but their test steps run the packages of one directory (the build seat's
`steps`, §22), so one set of tests is run. The share key is the level and the sha256 of the sorted lines
`<package>@<digest>`, one for each package the step would run after its carry (§25: those with no kept result for
their digest). The digests are those the runner already reads for its carry (§416). The first step of a key puts the
build seat's `step` ask; every other step with that key reads its result, and applies its own status from the policy
(§23). A later step of the same key reads the result of the key's ended run, a failed result too, so the same
failing tests fail once and not once per artifact.
- §725 **Kept in the runner's state `test-runs`.** (§27, §85, §330; ADR 0666 §4; principles 1, 2 and 7) Each test run is
one key of the runner's old `state` section `test-runs`, whose `max-mib` is 16. The key is `<level>.<digest>`, written
create-only by the step that asks the run. The value holds:
- `level`, and `packages` (each with its digest);
- `seat` and `ask`, and `node` once a holder has taken the ask;
- `started`;
- `readers`: each reading step's pipeline key and artifact, added and removed by compare-and-set;
- at the run's end, `ended`, `outcome`, its test counts, and the name of its output object in `step-outputs`.
The runner hears the run's end as the build seat's `step-ended` of its `ask`, as it hears every step (§24). At its
start the runner takes up again each key with no `ended`, by its `ask`. It deletes a key 7 days after its `ended`, in
the step log's order, as §330 ages its `pipelines`. A passed run's results go into `kept` per package, as today
(§27), so a step asked after the key is deleted carries them.
- §726 **How a test run ends, is said and asked again.** (§24, §73, §83 `rerun`, §119, §419, §496; ADR 0666 §8, §11,
§21; principles 3, 7 and 1)
- **Each reading step is said.** Each step reading a test run says its own `step-started` and `step-ended`. Its
`step-ended` carries the run's outcome and counts and `shared`: the key, its `ask`, and the pipeline and artifact of
the step that asked the run. Its `step-started` names the run's `ask`, so the watchman times it from that ask's
`step-progress` (§496).
- **Counted once.** `pipeline-ended` counts a catalog module's tests once per test run, never once per step reading
it (§24, §419).
- **Dropped only with its last reader.** A step that is superseded or cancelled leaves `readers`, and the run's ask
is dropped (§119) only when no reader is left.
- **Asked again, for every reader.** A `rerun` of a test step joins a run of its key while that run is running, or
reads one that started after the step's last end, and otherwise asks a new run, replacing the ended key by
compare-and-set. The new run's end moves every pipeline step that read the key's former run and did not pass: each
says `step-ended` again with the new outcome, marked as a rerun with the caller of the step asked again (§73). A
step left reading a failed result would say what is no longer true (principles 3 and 7; the coordinator,
2026-11-17).
- §727 **The tools.** (§83; ADR 0666 §25; principle 4) Two for each new state, in §83's table:
- `assigner_compositions` takes a candidate, a machine, or neither, and answers each composition run with its
program, input digest, ask, readers, outcome and verdict, and the state's fill against its cap.
- `assigner_compositions_drop` takes `{candidate, machine, why}`. It writes the hand-act log first, drops the run's
ask, ends each reader `could not run` naming the drop, and deletes the key.
- `runner_test_runs` takes a level, a digest, a module, or none of them, and answers each test run with its packages,
ask, node, readers, outcome and counts, and the state's fill against its cap.
- `runner_test_runs_drop` takes `{key, why}` and does the same as `assigner_compositions_drop`.
- §728 **How it is checked.** Tests 108 and 109.
- Whole (assigner, mesh-delivery): three composition steps of one revision, on two machines, start two runs, keep
two requests and put two run asks, and each step reads both verdicts. A fixture machine whose verdict blames module
A fails A's composition step and passes B's, with a warning naming A. A machine that does not compose and blames no
module fails both. Three steps asked again at once start one run for each machine that did not pass. A superseded
step leaves the run running for its other readers. An assigner restarted during a run takes it up again by its
ask. At the cap, a step ends `could not run` naming the fill.
- Whole (the node-engine's composer, `mesh-host`): a fixture module whose manifest is refused on the machine is
blamed by name, and two modules owning one path are both blamed.
- Whole (runner and Go build module): a module whose program and bundle test one directory asks one unit step and
one integration step. Both artifacts' steps end with that result and `shared`, a failing test fails once, and
`pipeline-ended` counts the module's tests once. A rerun of one artifact's failed step that passes moves the other
artifact's step to passed too.
- Live: in the first publish after G's delivery, the composition run asks of each revision number at most its
machines, and each module's test steps run once per level.
### Who may say what
- §68 **Callers per verb, in the seat's definition.** Each seat definition names, per verb, the callers allowed to call it,
@@ -287,7 +149,7 @@ Each line is a choice within ADR 0666 §7, §8 and §17, settled by the principl
| `pipeline-started`, `pipeline-ended` | a pipeline's start, and its end with every step's result, its test counts, the source revision and `widens` |
| `group-ended` | a group's one pipeline's end, on which alone its wants are written |
| `run-started` | a run on request's start, with its members at their commits |
| `step-started`, `step-ended` | each step's start and end: the step, its status in the policy, its outcome, its build-input fingerprint, its test counts, for a carried result what it was carried from, and for a rerun that it is one and the caller the runtime proved, for a step that read a test run or a composition run another step asked its `shared` (§723, §726); a bypassed step too |
| `step-started`, `step-ended` | each step's start and end: the step, its status in the policy, its outcome, its build-input fingerprint, its test counts, for a carried result what it was carried from, and for a rerun that it is one and the caller the runtime proved, for a step that read a test run or a composition run another step asked its `shared` (G-24 §729); a bypassed step too |
| `wait-held`, `wait-ended` | a step's wait: on a person (approval, code review), on a build node, on the revision view |
- §74 **What a session hears after its merge.** Between its `merge-pull` and the delivery, a session waits with `mesh_wait`
@@ -333,14 +195,14 @@ four lines a module's tools answer; every read tool of a store also answers its
| Part (seat) | Read everything it holds | Do by hand what it does itself | Recover without help |
|---|---|---|---|
| the runner (`mesh-pipeline-runner`) | `show` (one pipeline whole, its steps, step log and outputs), `list` (the pipelines of a module, a branch or a commit), `stands` (what stands still, since when), `why-carried` (where a step's result was carried from), `kept` (the kept results by digest), `runner_test_runs` (each test run, §727) | `start` (a delivering pipeline for a watched branch's head), `run` (the run on request), `rerun` (one step again) | `accept` (a failed `optional` step, with why), `cancel` (a pipeline, dropping its asks), `ask-again` (a declined `approval`), `runner_test_runs_drop` |
| the runner (`mesh-pipeline-runner`) | `show` (one pipeline whole, its steps, step log and outputs), `list` (the pipelines of a module, a branch or a commit), `stands` (what stands still, since when), `why-carried` (where a step's result was carried from), `kept` (the kept results by digest), `runner_test_runs` (each test run, G-24 §728) | `start` (a delivering pipeline for a watched branch's head), `run` (the run on request), `rerun` (one step again) | `accept` (a failed `optional` step, with why), `cancel` (a pipeline, dropping its asks), `ask-again` (a declined `approval`), `runner_test_runs_drop` |
| the indexer (`mesh-catalog`) | `registrations` (with each one's history); `indexer_edges` (an artifact's edges per watched branch), `indexer_order` (the order of a change, its cycles named), `indexer_catalog_difference` (two revisions); `indexer_pending` (a group with its missing members) | `register`, `policy`, `watch`, `unwatch` (each on the operator's Approve) | `indexer_seed_again` (the first registrations' ask raised again), `indexer_take_in_pending` (a group taken in as gathered), `indexer_drop_pending` (dropped) |
| the gate (`mesh-delivery`) | `source` with a branch (`gate_source`: every branch's source revision) | — | `gate_source_again` (say `source-moved` again for a branch) |
| the forge module (`git`) | `git_failed_steps` (the failed `required` steps it holds, by module and fingerprint, with the run that failed each) | — | `git_failed_steps_drop` (drop one, with why) |
| the grants part (`mesh-grants`) | `grants_callers` (the callers each verb admits, read from the seat definitions) | — | `grants_why_refused` (for an account and a verb, the grant it lacks and the definition's line that leaves it out) |
| the build seats | `steps` (the steps detected for an artifact) | `steps` with `again` (detect again) | `drop` (ADR 0664 §4) |
| the builder (`mesh-builds`) | `builder_reserved` (each module's reserved version, §369); the released versions are the catalog's `versions` | `builder_publish_again` (publish the missing artifacts under the reserved version) | — |
| the assigner (`mesh-wants`) | `assigner_why_not` (why a module is not on a machine yet: the pipeline it waits for, its hold, the placement, the node's last `running`), `assigner_compositions` (each composition run, §727) | — | `assigner_release`, `assigner_compositions_drop` |
| the assigner (`mesh-wants`) | `assigner_why_not` (why a module is not on a machine yet: the pipeline it waits for, its hold, the placement, the node's last `running`), `assigner_compositions` (each composition run, G-24 §728) | — | `assigner_release`, `assigner_compositions_drop` |
| the watchman | `watchman_views` (how far behind each part's view is, from each part's `view-waits` and first-view tools) | — | each part's `<module>_first_view_again` |
| the operator-channel | `asks` (with each asker's asks held back) | — | `ask-again` (offer a held-back ask now) |
| every module that imports mesh-sdk's small object store package | `<module>_state_objects`, `<module>_sent_objects` | — | `<module>_state_objects_drop`, `<module>_sent_objects_drop` |
@@ -368,8 +230,8 @@ and an estimated 2 mesh-sdk publishes a day, each rebuilding about 120 artifacts
| the builder | `reserved`: one key per module and `MAJOR.MINOR` (§369) | old `state` section | until its `version-released`, a release of another `MAJOR.MINOR` of the module, or `builder_reserved_drop` | a few KiB; at the cap the builder reserves nothing and says why |
| the assigner | `written`: the last source revision and end count per module and watched branch (§352) | old `state` section | until the branch is unwatched | a few KiB |
| the indexer and the assigner | `inputs`: a `run` ask's input by write time and sha256 (§354) | old `state` section | until the ask's `step-ended`, at most a day | 64 MiB; an estimated 300 asks a day of under 100 KiB each is under 30 MiB; at the cap the asker refuses the ask loudly, naming the store |
| the assigner | `composition-runs`: one key per candidate and machine, its program, input, ask, readers and verdict (§719) | old `state` section, `max-mib` 16 | a day after its end, or a day after its start with no end; deleted by the assigner (§719) | 16 MiB; an estimated 30 candidates a day on four machines, 120 keys of at most 8 KiB, is under 1 MiB a day |
| the runner | `test-runs`: one key per level and tested packages, its ask, readers, outcome and counts (§725) | old `state` section, `max-mib` 16 | 7 days after its end, deleted by the runner in the step log's order (§330) | 16 MiB; an estimated 1,000 test runs a day of about 1 KiB is about 7 MiB at 7 days |
| the assigner | `composition-runs`: one key per candidate and machine, its program, input, ask, readers and verdict (G-24 §719) | old `state` section, `max-mib` the setting `composition-runs-cap-mib` | the setting `composition-runs-kept` (default 24 hours) after its end; deleted by the assigner (§719) | default 16 MiB; an estimated 30 candidates a day on four machines, 120 keys of at most 8 KiB, is under 1 MiB a day |
| the runner | `test-runs`: one key per candidate, level and tested packages, its ask, readers, outcome and counts (G-24 §725) | old `state` section, `max-mib` the setting `test-runs-cap-mib` | the setting `test-runs-kept` (default 7 days) after its end, deleted by the runner in the step log's order with its output object (§330) | default 16 MiB; an estimated 1,000 test runs a day of about 1 KiB is about 7 MiB at 7 days |
| the runner and the assigner | `findings` (§355), read by their own read tools alone; the keeper hears the findings as events (§383) | old `state` section | an entry deleted when its finding clears | a few KiB |
| every observer (§387) | the stream `<module>-findings`: `finding-written`, `finding-cleared`, `findings-stated` (§383) | old `streams` section for a foundation part, the module's `streams` otherwise | 7 days, its oldest dropped when full; the next statement covers a lost one (§384) | 16 MiB; an estimate of a few hundred findings a day of about 1 KiB is under 2 MiB a week |
| the condition keeper | `findings-views`: one key per observer and node, its last stated set, pending keys and open asks (§388) | the form of its `conditions` | deleted when the module leaves the catalog or the node (§365) | under 1 MiB; about 30 observers on four machines |
@@ -1538,3 +1538,38 @@ the principle that settles it, cited as "design 47 G A-n".
time, and an input with its object once no ask's key names it, so no bookkeeping key of its own is kept. A key of the
old form `<time>.<digest>` is left to the day rule by the time it carries, never deleted earlier, so a composition
still running on one keeps its input. (From the develop of A-465a.)
- **A-466** (§720, §724; ADR 0666 §8, §17; principle 3) An ended run's result is read only by steps of the same
candidate; a step of another candidate asks a new run, a passed result crossing candidates only by §25's carry; a
`could not run` result is never read and always starts a new run, so a second revision with the same digests runs a
failed test again. (From the review by architectural domain of design 47 G §718 to §728, 2026-11-17.)
- **A-466a** (§727; principle 3) A rerun of a composition run or a test run moves every step that read the former run,
whose readers are carried into the new value; a moved step reopens its pipeline, which says `pipeline-ended` again,
and wants are written only from a pipeline's latest end; an `accepted` step does not move, a person's act standing.
- **A-466b** (§720, §725; ADR 0596; principle 1) A run whose ask has no `step-ended` within the step's own bound ends
`could not run`, heard on the durable `step-ended` consumer; no part reads a build seat's state to judge it.
- **A-466c** (§728, §391; principle 4) The two drop tools are of class `act`, with §391's default callers; a
reader-only account is refused (test 108).
- **A-466d** (§724, §25; principle 3) The test run's digest also covers the step's build-input fingerprint without the
artifact's own part, and its status in the policy, as §25's carry is keyed.
- **A-466e** (§719, §725; principle 11) Each run state's time kept and cap is a setting with a default: the assigner's
`composition-runs-kept` (24 hours) and `composition-runs-cap-mib` (16), the runner's `test-runs-kept` (7 days) and
`test-runs-cap-mib` (16).
- **A-466f** (§725; principle 7) `test-runs` keys a run by the package list's digest alone, the list kept in the run's
output object; at the cap a step ends `could not run` naming the fill with no condition of its own; the runner
deletes the output object with its key, after the last reader's end.
- **A-466g** (§721; principle 3) A composer verdict with no `blamed` (an older node-engine) is judged as today, saying
"the composer names no culprits"; the milestone plan orders mesh-delivery-sdk, then the node-engine's composer, then
the assigner.
- **A-466h** (§721; principles 7 and 11) The search for culprits holds only the candidate's changed modules placed on
the machine, one at a time, at most once per such module plus once and never past the composer's setting
`blame-tries` (20).
- **A-466i** (§726; principle 3) A module's tests are counted once per pipeline and test run.
- **A-466j** (§723, §726, §729; ADR 0666 §21; principles 1 and 8) `shared` is added to the seat definitions of
`mesh-wants` (`step-ended`, `step-progress`) and `mesh-pipeline-runner` (`step-started`, `step-ended`) in mesh-delivery,
generated by `seatgen`; a reading step names its own fingerprint and the run's ask, which meets ADR 0666 §21 as §25's
carry does.
- **A-466k** (§730; principle 3) Steps asked before the new assigner and runner run end by the old path; runs are made
only for steps asked after; a return to an earlier build leaves the new states unread, their readers ended `could not
run` by A-466b's bound.
- **A-466l** (§718; principle 3) A composition run's request carries the node's `compose-candidate` answer of the
moment it started, so a later step of the same candidate reuses that verdict knowingly, within the candidate.
@@ -177,6 +177,6 @@ deals with it, which is how the list is kept true.
node once, so this no longer multiplies anything and stays out of G (principle 7; the coordinator, 2026-11-17). After
G: find it by comparing two kept inputs of one node, read by digest through `mesh-wants`' `input`, and decide whether
the answer must be the same bytes while the node is unchanged.
- **Design 47's main file is split into milestone parts** (playbook 02, *The shape of a design*). It is about 250 KB
against its 80 KB bound. The split is a separate amendment after G's delivery, never folded into a line of G (the
coordinator, 2026-11-17).
- **Design 47's main file and G-22 are split** (playbook 02, *The shape of a design*). The main file is about 250 KB
against its 80 KB bound, and G-22 about 165 KB against a milestone file's 60 KB. The operator decided at 10:31 CET on
2026-11-17 that the split comes after G, as a separate amendment.
@@ -0,0 +1,226 @@
---
layer: to-be
status: in-progress
design: 47
milestone: G
updated: 2026-11-17
decisions:
- "ADR 0667"
- "ADR 0666"
- "ADR 0596"
- "ADR 0267"
---
# 47, milestone G: one composition run per candidate and machine, one test run per level and tested packages
One file of milestone G's part of [design 47](../47-delivery-from-commit-to-delivered.md); the part opens at
[G.md](G.md). Its design lines are numbered across every file of the part and cited as "design 47 G §n".
*§718 to §731 come from milestone G's live delivery of 2026-11-17. The 24 pipelines of one catalog revision asked 96
composition runs, one per pipeline and machine, and 80 more when they were asked again. Each run kept its own copy of
a node's input of 1.1 to 1.5 MB in the assigner's state `inputs`, which reached its 64 MiB cap twice. The indexer's
and the teller's same failing tests failed once per artifact. The operator, 2026-11-17 about 10:20 CET: "In G, now".
Each line is a choice within ADR 0666 §7, §8 and §17, settled by the principle it names. The review by architectural
domain of 2026-11-17 was answered by the lead in A-466 to A-466l (G-22), each cited at the line it settles.*
### Composition runs
- §718 **One composition run per candidate and machine.** (§354, A-226, A-243, A-252, A-466l; ADR 0666 §17; principle
7) The assigner, holding the seat `mesh-wants`, composes each machine once per candidate, whatever number of
composition steps name it. The candidate is named by its key (A-226): `<branch>.<revision>` for a branch pipeline,
`run-<id>` for a run on request. The first composition step of a candidate that needs a machine starts that machine's
composition run: one `compose-candidate` ask of the node, its request kept once in `inputs`, one `run` ask on
`node-build-go`. Every other composition step of that candidate whose module is placed on that machine (A-252) reads
that run's verdict, under §720's rules. The request carries the candidate's manifests, verdicts and seat
definitions, depth compose, the one machine and the case `change`, and the node's own `compose-candidate` answer at
the moment the run started; it carries nothing of the module whose pipeline asked it. A step of the same candidate
that comes later so reuses a verdict composed from the node's answer of that earlier moment, knowingly: within one
candidate, that answer is the machine as the candidate is judged on it (A-466l). The composer program is chosen once,
when the run starts, by A-252's and A-451's rule, and recorded in the run. This replaces "one run ask per machine" for
each composition step (A-243, A-252) with one per candidate and machine.
- §719 **Kept in the assigner's state `composition-runs`.** (§354, §85; ADR 0666 §4, §7; A-466e; principles 1, 2, 7 and
11) Each composition run is one key of the assigner's old `state` section `composition-runs`, whose `max-mib` is the
assigner's setting `composition-runs-cap-mib`, default 16. The key is `<candidate key>.<machine>`, each part escaped as
a key's name (`keys.EscapeName`). The step that starts the run writes the key create-only, so two steps of one
candidate starting at once start one run and the second reads it. The value holds:
- `candidate` and `machine`;
- `program`: repository, commit, directory, and the node-engine fingerprint it was chosen from;
- `input`: the digest of the kept request;
- `ask`: the id of the build seat's run ask;
- `started`;
- `readers`: the step keys of the composition steps reading the run, each added and removed by compare-and-set;
- at the run's end, `ended`, `outcome` (`passed`, `failed` or `could not run`, with its words) and `verdict` (that
machine's composer verdict, §721).
A step waiting on a running composition run hears its end through the assigner's own watch of that key (a module
watches its own state, principle 1). At its start the assigner takes up again each key with no `ended`, by its
`ask`'s `step-ended` (A-240) and the bound of §720. The assigner deletes a key once the time its setting
`composition-runs-kept`, default 24 hours, names has passed since its `ended`; both this and the bound are judged as
the state `inputs` is (A-465b). At the cap no run is started, and its steps end `could not run`, naming the fill (as
A-465a).
- §720 **What a run's result serves, and its bound.** (§718; ADR 0666 §8, §17; ADR 0596; A-466, A-466b; principles 1
and 3)
- **Within its candidate only.** An ended run's verdict is read only by composition steps of the same candidate. A
step of another candidate starts a run of its own; nothing composed for one candidate crosses to another.
- **`could not run` is never read.** A step that finds an ended run whose outcome is `could not run` starts a new
run, replacing the key by compare-and-set; its former readers have already ended `could not run` with it.
- **The bound.** A run whose `ask` has no `step-ended` within the bound of the step that started it (the step's own
bound key with its time to live, A-77, A-101) ends `could not run`, naming the ask and the bound, and every reader
with it. The end is heard on the assigner's durable consumer of the build seat's `step-ended` (A-240); the assigner
never reads the build seat's state (ADR 0596).
- §721 **A machine's verdict names the modules it blames.** (ADR 0666 §17; A-452, A-466g, A-466h; principles 10, 3, 7
and 11) The composer's verdict of a machine (mesh-delivery-sdk's `composer.Machine`, answered by the node-engine's
`mesh-compose`) gains `blamed`, a list of `{module, problem}`.
- **What it names.** Each module that the candidate's composition of the machine holds or refuses and the
composition without the candidate does not (the modules in its input's `HeldBefore` are left out). Two modules
owning one path are each blamed, with that path. `composes` stays false when the machine does not compose with
every module of the candidate. A machine that does not compose and blames no module fails on the node's own
account (its own composition, or no module composed at all, as A-452 has it).
- **How culprits are found, and its bound.** Only the candidate's changed modules placed on the machine (those whose
entry in the candidate differs from the composition without it) are held, one at a time, each try composing once
more; the search ends at the first module whose hold ends the problem, as the indexer's validator names a manifest
whose removal ends a problem. It composes at most once per such module plus once, never more than the composer's
setting `blame-tries`, default 20; past that bound, the modules not yet tried are named in `blamed` with the words
"not tried: the search reached its bound".
- **An old composer.** A verdict with no `blamed` field (a node-engine's composer from before this line) is judged as
today: a machine that does not compose fails every composition step placing a module on it, with the warning "the
composer names no culprits".
- **The order.** mesh-delivery-sdk's `composer` package is published before the node-engine and the assigner build (an
edge of kind 1); the node-engine's composer is delivered before the assigner, an order the milestone plan names.
No `needs` is declared for it, since an assigner meeting an old composer judges as today.
- §722 **One module's fault fails only its own pipelines.** (§721; ADR 0666 §9, §17; A-452; principles 10 and 3) A
composition step judges each of its machines from that machine's verdict:
- `failed` when the verdict blames the step's module, or when the machine does not compose and blames no module;
- `could not run` when the composition run could not run;
- `passed` otherwise, with a warning for each module blamed there that is not the step's own (`<machine> holds
<module>: <problem>`). Another module's fault is so said in every pipeline it touches and fails none of them.
The step's outcome then follows from its machines, as today: failed before could not run before passed (§354). A
machine that is away passes with a warning and is not composed (§64).
- §723 **How a composition run ends and is said.** (§73, §116, §119, §496, A-182; ADR 0666 §7, §21; A-466j;
principles 1, 7 and 8) A composition run says no event of its own. Its end is the build seat's `step-ended` of its
`ask`, which the assigner alone hears (A-240). Each composition step reading the run says its own `step-ended` on
`mesh-wants`, as today, with its own build-input fingerprint and `shared` (§729), naming per machine the run it read,
that run's `ask` and its verdict: the result was taken from the build seat's holder once, at that ask, and the step
names it, as §25's carry names where a carried result came from (ADR 0666 §21). Each reading step says its
`step-progress` once a minute while it waits (A-262), naming the runs' asks in `shared`. A step that is superseded or
cancelled leaves `readers`. The run's `ask` is dropped (§119, A-182) only when no reader is left, so a superseded
pipeline never takes away another pipeline's composition run.
### Test runs
- §724 **One test run per level and tested packages.** (§22, §25, §27, §416; ADR 0666 §8, §17; ADR 0267 §1; A-466,
A-466d; principles 3 and 7) The runner asks a lint, unit test or integration test step of a build seat once per
candidate, level and tested packages, whatever number of artifacts' steps name them. A test step's candidate is its
pipeline's in A-226's form: `<branch>.<revision>`, or `run-<id>` for a run on request. A module's program and its
tools bundle are built from different import closures (ADR 0267 §1), but their test steps run the packages of one
directory (the build seat's `steps`, §22), so one set of tests is run. The share digest is the sha256 of:
- the step's build-input fingerprint with the artifact's own part left out (its program's package and recipe), so two
artifacts of one directory meet;
- the step's status in the policy (§23), so the key is §25's carry key: two artifacts whose policies differ do not
share;
- the sorted lines `<package>@<digest>`, one for each package the step would run after its carry (§25: those with no
kept result for their digest), the digests being those the runner already reads for its carry (§416).
The first step of a key puts the build seat's `step` ask; every other step of the same candidate with that key reads
its result. An ended run's result is read only within its candidate; a step of another candidate asks a new run, and
a passed result crosses candidates only as §25's carry, through `kept`. A `could not run` result is never read: a
step finding one asks a new run, replacing the key by compare-and-set. So within a candidate the same failing tests
fail once, not once per artifact, and a second revision with the same digests runs a failed test again.
- §725 **Kept in the runner's state `test-runs`.** (§27, §85, §330; ADR 0666 §4; A-466e, A-466f; principles 1, 2, 7 and
11) Each test run is one key of the runner's old `state` section `test-runs`, whose `max-mib` is the runner's setting
`test-runs-cap-mib`, default 16. The key is `<candidate key>.<level>.<digest>`, each part escaped as a key's name,
written create-only by the step that asks the run. The value holds:
- `level` and `digest`; the list of packages with their digests is in the run's output object, never in the key or
the value;
- `seat` and `ask`, and `node` once a holder has taken the ask;
- `started`;
- `readers`: each reading step's pipeline key and artifact, added and removed by compare-and-set;
- at the run's end, `ended`, `outcome`, its test counts, and the name of its output object in `step-outputs`.
The runner hears the run's end as the build seat's `step-ended` of its `ask`, on its durable consumer, as it hears
every step (§24). A run whose `ask` has no `step-ended` within the bound of the step that asked it ends `could not
run`, as §720 has it, and every reader with it. At its start the runner takes up again each key with no `ended`, by its
`ask` and that bound. It deletes a key once the time its setting `test-runs-kept`, default 7 days, names has passed
since its `ended`, in the step log's order as §330 ages its `pipelines`, and deletes the run's output object with the
key, after the last reader's end. At the cap no run is asked, and the step ends `could not run`, naming the fill, as
§719 has it, with no condition of its own. A passed run's results go into `kept` per package, as today (§27), so a
step asked after the key is deleted carries them.
- §726 **How a test run ends and is said.** (§24, §73, §119, §419, §496; ADR 0666 §8, §21; A-466i, A-466j; principles
3, 7 and 1)
- **Each reading step is said.** Each step reading a test run says its own `step-started` and `step-ended`, with its
own build-input fingerprint and `shared` (§729). Its `step-ended` carries the run's outcome and counts. Its
`step-started` names the run's `ask`, so the watchman times it from that ask's `step-progress` (§496). The result
was taken from the build seat's holder once, at that ask, which the step names, as §25's carry does (ADR 0666 §21).
- **Counted once.** `pipeline-ended` counts a module's tests once per pipeline and test run, never once per step
reading it (§24, §419).
- **Dropped only with its last reader.** A step that is superseded or cancelled leaves `readers`, and the run's ask
is dropped (§119) only when no reader is left.
### For both kinds
- §727 **Asking a run again moves every reader.** (§30, §73, §83 `rerun`, §29; ADR 0666 §9, §11, §24; A-466a;
principle 3) One rule for composition runs and test runs. A `rerun` of a composition step (for each machine whose
verdict in it was not `passed`) or of a test step joins the run of its key while that run is running, reads one that
started after the step's last end, or otherwise starts a new run, replacing the ended key by compare-and-set and
carrying the former run's `readers` into the new value. The new run's end moves every step that read the former run
and did not pass: each says `step-ended` again with the new outcome, marked as a rerun with the caller of the step
asked again (§73). A moved step reopens its pipeline, which says `pipeline-ended` again with a higher end count; the
assigner writes wants only from a pipeline's latest end (§29). A step a person `accepted` (§30) does not move: a
person's act stands. A machine that passed in a composition step keeps its verdict. A step left reading a failed
result would say what is no longer true. So twenty composition steps of one revision asked again together start one
run per machine.
- §728 **The tools.** (§83, §391; ADR 0666 §25; A-466c; principle 4) Two for each new state, in §83's table:
- `assigner_compositions` (class `read`) takes a candidate, a machine, or neither, and answers each composition run
with its program, input digest, ask, readers, outcome and verdict, and the state's fill against its cap.
- `assigner_compositions_drop` (class `act`, §391's default callers) takes `{candidate, machine, why}`. It writes the
hand-act log first, drops the run's ask, ends each reader `could not run` naming the drop, and deletes the key.
- `runner_test_runs` (class `read`) takes a candidate, a level, a digest, a module, or none of them, and answers each
test run with its packages (read from its output object), ask, node, readers, outcome and counts, and the state's
fill against its cap.
- `runner_test_runs_drop` (class `act`, §391's default callers) takes `{key, why}` and does the same as
`assigner_compositions_drop`, deleting the run's output object with the key.
- §729 **`shared` in the seat contracts.** (§73, §116, §496; ADR 0666 §21, §24; A-466j; principles 1 and 8) The seat
definitions in mesh-delivery's `mesh-delivery-contracts` change, and the contracts' code is generated from them by
`seatgen`, never edited by hand:
- **`mesh-wants`' `step-ended`** gains `shared`: a list of `{machine, run, ask}`, one per machine composed, `run` the
`composition-runs` key and `ask` its run ask's id, all strings.
- **`mesh-wants`' `step-progress`** gains the same `shared` list, naming the runs the step waits on.
- **`mesh-pipeline-runner`'s `step-started`** gains `shared`: `{run, ask}` for a test step, `run` the `test-runs` key
and `ask` its build seat ask's id, strings.
- **`mesh-pipeline-runner`'s `step-ended`** gains `shared`: `{run, ask, asked-by}`, `asked-by` being `{pipeline,
artifact}` of the step that asked the run (strings), left out for that step itself.
A step that names `shared` names its own build-input fingerprint beside it, as every `step-ended` does (§73). A
composition step's `step-started` is said by the runner when it asks the assigner, before any run is chosen, so it
cannot name the runs' asks; the assigner's `step-progress` carries them instead, and the watchman times a composition
step from that (§496, A-262).
- §730 **What runs in flight, and going back.** (§65, §78; ADR 0666 §19, §24; ADR 0667 §9; A-466k; principle 3) A
composition step or test step asked before the new assigner or the new runner runs ends by the path it was asked
under, one run per step; composition runs and test runs are made only for steps asked after the new part runs. A
return to an earlier build (§78) leaves `composition-runs` and `test-runs` unread; their readers are ended `could not
run` by the bound of §720 and §725.
- §731 **How it is checked.** Tests 108 and 109.
- **Whole (assigner, mesh-delivery), test 108.** Two composition steps of one candidate started at once make one key,
one request and one run ask. Three composition steps of one revision on two machines start two runs, and each step
reads both verdicts. A machine whose verdict blames module A fails A's composition step and passes B's, with a
warning naming A; a machine that does not compose and blames no module fails both; a verdict with no `blamed` is
judged as today and says "the composer names no culprits". A step of another candidate starts its own run, and a
`could not run` result starts a new one. A run whose ask has no `step-ended` within the bound ends `could not run`,
with its readers. Steps asked again together start one run per machine that did not pass, and move every former
reader. A superseded step leaves the run running for its other readers. An assigner restarted during a run takes it
up again by its ask. At the cap a step ends `could not run`, naming the fill. `assigner_compositions_drop` ends the
readers and deletes the key, and a reader-only account calling it is refused.
- **Whole (the node-engine's composer, `mesh-host`).** A fixture module whose manifest is refused on the machine is
blamed by name, two modules owning one path are both blamed, and the search past `blame-tries` names the untried.
- **Whole (runner and Go build module, mesh-delivery and mesh-catalog), test 109.** A module whose program and bundle
test one directory asks one unit step and one integration step, and both artifacts' steps end with that result and
`shared`. A failing test fails once, and `pipeline-ended` counts the module's tests once. A rerun of one artifact's
failed step that passes moves the other artifact's step to passed too, and the pipeline says `pipeline-ended` again;
an `accepted` step does not move. A runner restarted during a run takes it up again by its ask. A replaced reader
leaves the ask to the other readers. At the cap a step ends `could not run`, naming the fill. A second revision with
the same digests runs a failed test again. `runner_test_runs_drop` deletes the key and its output object, and a
reader-only account calling it is refused.
- **Live.** In the first publish after G's delivery, the composition run asks of each revision number at most its
machines, read with `assigner_compositions` and `assigner_inputs`; each module's test steps run once per level, read
with `runner_test_runs`.
+4 -2
View File
@@ -3,7 +3,7 @@ layer: to-be
status: in-progress
design: 47
milestone: G
updated: 2026-11-12
updated: 2026-11-17
decisions:
- "ADR 0500"
- "ADR 0673"
@@ -88,7 +88,7 @@ file on 2026-11-10, and is cited as "design 47 G §n". A sentence that holds a d
stayed in the main file, and G builds what it says from G's delivery. A section named in italics is in this part, or,
when it is not here, in the main file.
The part is twenty-two files, each read whole, its design lines numbered across them:
The part is twenty-four files, each read whole, its design lines numbered across them:
- [G-1-pipelines-registration-and-steps](G-1-pipelines-registration-and-steps.md): pipelines per module: registration, watched branches, steps and policy, what holds a delivery, approval, groups, dependencies, a publish.
- [G-2-pipelines-sizes-checks-and-tools](G-2-pipelines-sizes-checks-and-tools.md): pipelines per module: any size, the pipeline makes the check, who may say what, health, recovery, tools, kept data, how G itself is delivered, the choices under ADR 0666 and ADR 0667.
@@ -112,5 +112,7 @@ The part is twenty-two files, each read whole, its design lines numbered across
- [G-20-lead-answers](G-20-lead-answers.md): the lead's one-line answers to develop's questions, each tested against the principles (A-1 on).
- [G-21-provider-and-revert-ask](G-21-provider-and-revert-ask.md): the provider a requirement points to, the named seat and the contribution edge (§708, §709, completing §357), and the assigner's ask of a held revert (§710, completing §300), the edge an acting rule gives (§711, completing §47's kind 5), the validation before take-in (§712, §713, completing §16), the take-in and groups of every watched branch (§714, §715), where the gate keeps a group's members (§716) and the seats of the status view's and the assigner's views, followed before their holder is known (§717).
- [G-22-lead-answers-2](G-22-lead-answers-2.md): the lead's answers continued, from A-182.
- [G-23-after-delivery](G-23-after-delivery.md): what is owed once G is delivered.
- [G-24-composition-runs-and-test-runs](G-24-composition-runs-and-test-runs.md): one composition run per candidate and machine, one test run per level and tested packages, from G's live delivery of 2026-11-17 (§718 to §731).
Design 34's milestone [every-event](../34-milestones/every-event.md) and design 46's milestone [7-approval-mark](../46-milestones/7-approval-mark.md) ship in milestone G's release, built in its repositories beside G's lines (lead-47, 2026-11-11 15:30 CET); where one of their lines and a line of G differ, G's later line holds and the other is corrected in place (§624 to §626).