From 31cd62aeecdbd6f2da736164a5c81aa49a8b842e Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 17 Nov 2026 10:24:28 +0100 Subject: [PATCH 1/9] Design 47 G: one composition run per candidate and machine, one test run per tested packages G's live delivery of 2026-11-17 composed each machine once per pipeline (96 runs for one revision, 80 more on rerun, the assigner's inputs state at its cap twice) and ran the same tests once per artifact. Pipelines of one candidate now read one composition run per machine, kept by the assigner, whose verdict names the modules it blames so one module's fault fails only its own pipelines; test steps naming the same packages read one test run, kept by the runner. The operator, 2026-11-17 about 10:20 CET: "In G, now". --- 00-META/glossary.md | 9 ++ .../47-delivery-from-commit-to-delivered.md | 12 +- .../G-2-pipelines-sizes-checks-and-tools.md | 144 +++++++++++++++++- .../47-milestones/G-22-lead-answers-2.md | 9 +- 4 files changed, 166 insertions(+), 8 deletions(-) diff --git a/00-META/glossary.md b/00-META/glossary.md index a40d471c..818c22a7 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -620,6 +620,15 @@ controller (the planner, the rollout) until release 6 of ADR 0429; from each rel - **composition step** — the pipeline step that composes each machine's declaration with a change, by the composer built from its heads in the sandbox, refusing a change in which two modules own one path; served by the seat `mesh-wants` (ADR 0666 §17). From milestone G's delivery it replaces the **composition check**. +- **composition run** — the one run of the composer for one candidate on one machine, started by the first + composition step of that candidate that needs the machine and kept by the assigner in its state `composition-runs`; + every composition step of the candidate whose module is placed on the machine reads its verdict (design 47 G §718). + Not a **composition**, one module's resources composed for one node; nor the **composition step**, the pipeline + step that reads one composition run per machine. +- **test run** — the one run of a lint, unit test or integration test pipeline step for one level and one set of + tested packages at their digests, kept by the runner in its state `test-runs`; every artifact's pipeline step naming + the same packages reads its result (design 47 G §724). Not a **pipeline step**, each artifact's own, nor a **run + on request**, a whole pipeline that delivers nothing. - **merge check** — what a pull request carries before it may merge, judging the merge of its head into a named **trunk commit**, made in a clone of its own on a build node and never pushed (ADR 0323). One status, `mesh/check`, written by the checker: the change's tests, run by the language's build module, and the diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index cb1f1fe5..721fc5ed 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -2,7 +2,7 @@ layer: to-be status: in-progress code: [mesh-controller, mesh-catalog modules/mesh-delivery-gate, mesh-catalog modules/gitea, mesh-delivery, mesh-host, mesh-sdk, mesh-vault, mesh-contracts, mesh-tools, mesh-networking, hq 00-META/checks] -updated: 2026-11-11 +updated: 2026-11-17 decisions: - "ADR 0673" - "ADR 0672" @@ -1972,6 +1972,16 @@ ADR 0667's checks, of milestone G itself: `condition-raised` (whole, condition keeper and a fixture observer, mesh-catalog and mesh-sdk). Live: after G's delivery, a node-engine restarted with the full seat list clears its `seat-list-unread`, read with `mesh-conditions.conditions`. +108. **One composition run per candidate and machine** (design 47 G §718 to §723). Three composition steps of one + catalog revision on two machines start two composition runs and put two run asks, each step reading both + verdicts; a machine whose verdict blames one module fails that module's composition step alone and says it as a + warning in the others; steps asked again together start one run per machine that did not pass (whole, assigner + and the node-engine's composer, mesh-delivery and `mesh-host`). Live: in the first publish after G's delivery, the + composition run asks of a revision number at most its machines, and the state `inputs` stays under half its cap. +109. **One test run per level and tested packages** (design 47 G §724 to §726). A module whose program and tools + bundle test one directory runs its unit and integration tests once per level, both artifacts' pipeline steps + ending with that result and naming it in `shared`, a failing test failing once, and `pipeline-ended` counting the + module's tests once (whole, runner and Go build module, mesh-delivery and mesh-catalog). ## What retires diff --git a/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md b/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md index 59f768e6..6c495532 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md +++ b/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md @@ -3,7 +3,7 @@ layer: to-be status: in-progress design: 47 milestone: G -updated: 2026-11-11 +updated: 2026-11-17 decisions: - "ADR 0667" - "ADR 0666" @@ -25,6 +25,7 @@ decisions: - "ADR 0448" - "ADR 0363" - "ADR 0283" + - "ADR 0267" - "ADR 0240" --- @@ -112,6 +113,139 @@ One file of milestone G's part of [design 47](../47-delivery-from-commit-to-deli pipelines up again. Live: once the checker is removed no module claims the seat `mesh-delivery-checker`. Tests 64, 84 and 88. +### One composition run per candidate and machine, one test run per tested packages + +*§718 to §728 come from milestone G's live delivery of 2026-11-17. The 24 pipelines of one catalog revision asked 96 +composition runs, one per pipeline and machine, and 80 more when they were asked again; each run kept its own copy of +a node's input of 1.1 to 1.5 MB in the assigner's state `inputs`, which reached its 64 MiB cap twice. The indexer's +and the teller's same failing tests failed once per artifact. The operator, 2026-11-17 about 10:20 CET: "In G, now". +Each line is a choice within ADR 0666 §7, §8 and §17, settled by the principle it names.* + +- §718 **One composition run per candidate and machine.** (§354, A-226, A-243, A-252; ADR 0666 §17; principle 7) The + assigner, holding the seat `mesh-wants`, composes each machine once per candidate, whatever number of composition + steps name it. The candidate is named by its key (A-226): `.` for a branch pipeline, `run-` for + a run on request. The first composition step of a candidate that needs a machine starts that machine's composition + run: one `compose-candidate` ask of the node, its request kept once in `inputs`, one `run` ask on `node-build-go`. + Every other composition step of that candidate whose module is placed on that machine (A-252) reads that run's + verdict. A run's request depends only on the candidate and the machine (the candidate's manifests, verdicts and + seat definitions, depth compose, the one machine, the case `change`), never on the module whose pipeline asked it, + so one verdict serves every module's pipeline. The composer program is chosen once, when the run starts, by A-252's + and A-451's rule, and recorded in the run. This replaces "one run ask per machine" for each composition step (A-243, + A-252) with one per candidate and machine. +- §719 **Kept in the assigner's state `composition-runs`.** (§354, §85; ADR 0666 §4, §7; principles 1, 2 and 7) Each + composition run is one key of the assigner's old `state` section `composition-runs`, whose `max-mib` is 16. The key + is `.`, each part escaped as a key's name (`keys.EscapeName`). The step that starts the run + writes the key create-only, so two steps of one candidate starting at once start one run and the second reads it. + The value holds: + - `candidate` and `machine`; + - `program`: repository, commit, directory, and the node-engine fingerprint it was chosen from; + - `input`: the digest of the kept request; + - `ask`: the id of the build seat's run ask; + - `started`; + - `readers`: the step keys of the composition steps reading the run, each added and removed by compare-and-set; + - at the run's end, `ended`, `outcome` (`passed`, `failed` or `could not run`, with its words) and `verdict` (that + machine's composer verdict, §720). + + A step waiting on a running composition run hears its end through the assigner's own watch of that key (a module + watches its own state, principle 1). A restart of the assigner loses nothing: at its start the assigner takes up + again each key with no `ended`, by its `ask`'s `step-ended` (A-240), and ends `could not run` a key whose `ask` the + build seat no longer holds, naming the ask. The assigner deletes a key a day after its `ended`, and a key with no + `ended` a day after its `started`, ending each of its readers `could not run`; both are judged as the state `inputs` + is (A-465b). At the cap no run is started, and its steps end `could not run`, naming the fill (as A-465a). +- §720 **A machine's verdict names the modules it blames.** (ADR 0666 §17; A-452; principles 10 and 3) The composer's + verdict of a machine (mesh-delivery-sdk's `composer.Machine`, answered by the node-engine's `mesh-compose`) gains + `blamed`, a list of `{module, problem}`. It names each module that the candidate's composition of the machine holds + or refuses and the composition without the candidate does not (the modules in its input's `HeldBefore` are left + out). The culprits are found by the node-engine's own rule of holding one module at a time until the problem ends, + as the indexer's validator names a manifest whose removal ends a problem; two modules owning one path are each + blamed, with that path. `composes` stays false when the machine does not compose with every module of the + candidate. A machine that does not compose and blames no module fails on the node's own account (its own + composition, or no module composed at all, as A-452 has it). +- §721 **One module's fault fails only its own pipelines.** (§720; ADR 0666 §9, §17; A-452; principles 10 and 3) A + composition step judges each of its machines from that machine's verdict: + - `failed` when the verdict blames the step's module, or when the machine does not compose and blames no module; + - `could not run` when the composition run could not run; + - `passed` otherwise, with a warning for each module blamed there that is not the step's own (` holds + : `). Another module's fault is so said in every pipeline it touches and fails none of them. + + The step's outcome then follows from its machines, as today: failed before could not run before passed (§354). A + machine that is away passes with a warning and is not composed (§64). +- §722 **What asking a composition step again does.** (§83 `rerun`; ADR 0666 §11, §24; principle 7) The runner's + `rerun` of a composition step asks the assigner for the step again, marked as a rerun (§73). A machine that passed in + the step keeps its verdict. For each machine whose verdict in the step was not `passed`, the step: + - joins its candidate's and machine's composition run while that run is running; + - reads that run's verdict when the run started after the step's last end; + - otherwise starts a new run, replacing the ended key by compare-and-set. + + So twenty composition steps of one revision asked again together start one run per machine. A new run's verdict + counts for no other step until that step is itself asked again or asked for the first time. +- §723 **How a composition run ends and is said.** (§73, §116, §496, §119, A-182; ADR 0666 §7, §21; principles 7 and 1) + A composition run says no event of its own. Its end is the build seat's `step-ended` of its `ask`, which the + assigner alone hears (A-240). Each composition step reading the run says its own `step-ended` on `mesh-wants`, as + today. That event's output names, per machine, the run it read (`shared`: the run's key and its `ask`) and the + verdict. So the runner, the forge module's `failed-steps` and the delivery record hear every pipeline's result from + the step's holder, as before (ADR 0666 §21). Each reading step says its `step-progress` once a minute while it waits + (A-262). A step that is superseded or cancelled leaves `readers`. The run's `ask` is dropped (§119, A-182) only when + no reader is left, so a superseded pipeline never takes away another pipeline's composition run. +- §724 **One test run per level and tested packages.** (§22, §25, §27, §416; ADR 0666 §8, §17; ADR 0267 §1; + principle 7) The runner asks a lint, unit test or integration test step of a build seat once per level and tested + packages, whatever number of artifacts' steps name them. A module's program and its tools bundle are built from + different import closures (ADR 0267 §1), but their test steps run the packages of one directory (the build seat's + `steps`, §22), so one set of tests is run. The share key is the level and the sha256 of the sorted lines + `@`, one for each package the step would run after its carry (§25: those with no kept result for + their digest). The digests are those the runner already reads for its carry (§416). The first step of a key puts the + build seat's `step` ask; every other step with that key reads its result, and applies its own status from the policy + (§23). A later step of the same key reads the result of the key's ended run, a failed result too, so the same + failing tests fail once and not once per artifact. +- §725 **Kept in the runner's state `test-runs`.** (§27, §85, §330; ADR 0666 §4; principles 1, 2 and 7) Each test run is + one key of the runner's old `state` section `test-runs`, whose `max-mib` is 16. The key is `.`, written + create-only by the step that asks the run. The value holds: + - `level`, and `packages` (each with its digest); + - `seat` and `ask`, and `node` once a holder has taken the ask; + - `started`; + - `readers`: each reading step's pipeline key and artifact, added and removed by compare-and-set; + - at the run's end, `ended`, `outcome`, its test counts, and the name of its output object in `step-outputs`. + + The runner hears the run's end as the build seat's `step-ended` of its `ask`, as it hears every step (§24). At its + start the runner takes up again each key with no `ended`, by its `ask`. It deletes a key 7 days after its `ended`, in + the step log's order, as §330 ages its `pipelines`. A passed run's results go into `kept` per package, as today + (§27), so a step asked after the key is deleted carries them. +- §726 **How a test run ends, is said and asked again.** (§24, §73, §83 `rerun`, §119, §419, §496; ADR 0666 §8, §11, + §21; principles 3, 7 and 1) + - **Each reading step is said.** Each step reading a test run says its own `step-started` and `step-ended`. Its + `step-ended` carries the run's outcome and counts and `shared`: the key, its `ask`, and the pipeline and artifact of + the step that asked the run. Its `step-started` names the run's `ask`, so the watchman times it from that ask's + `step-progress` (§496). + - **Counted once.** `pipeline-ended` counts a catalog module's tests once per test run, never once per step reading + it (§24, §419). + - **Dropped only with its last reader.** A step that is superseded or cancelled leaves `readers`, and the run's ask + is dropped (§119) only when no reader is left. + - **Asked again.** A `rerun` of a test step joins a run of its key while that run is running, or reads one that + started after the step's last end, and otherwise asks a new run, replacing the ended key by compare-and-set. A + sibling step with the same key moves only by its own rerun, which then reads that run. +- §727 **The tools.** (§83; ADR 0666 §25; principle 4) Two for each new state, in §83's table: + - `assigner_compositions` takes a candidate, a machine, or neither, and answers each composition run with its + program, input digest, ask, readers, outcome and verdict, and the state's fill against its cap. + - `assigner_compositions_drop` takes `{candidate, machine, why}`. It writes the hand-act log first, drops the run's + ask, ends each reader `could not run` naming the drop, and deletes the key. + - `runner_test_runs` takes a level, a digest, a module, or none of them, and answers each test run with its packages, + ask, node, readers, outcome and counts, and the state's fill against its cap. + - `runner_test_runs_drop` takes `{key, why}` and does the same as `assigner_compositions_drop`. +- §728 **How it is checked.** Tests 108 and 109. + - Whole (assigner, mesh-delivery): three composition steps of one revision, on two machines, start two runs, keep + two requests and put two run asks, and each step reads both verdicts. A fixture machine whose verdict blames module + A fails A's composition step and passes B's, with a warning naming A. A machine that does not compose and blames no + module fails both. Three steps asked again at once start one run for each machine that did not pass. A superseded + step leaves the run running for its other readers. An assigner restarted during a run takes it up again by its + ask. At the cap, a step ends `could not run` naming the fill. + - Whole (the node-engine's composer, `mesh-host`): a fixture module whose manifest is refused on the machine is + blamed by name, and two modules owning one path are both blamed. + - Whole (runner and Go build module): a module whose program and bundle test one directory asks one unit step and + one integration step. Both artifacts' steps end with that result and `shared`, a failing test fails once, and + `pipeline-ended` counts the module's tests once. + - Live: in the first publish after G's delivery, the composition run asks of each revision number at most its + machines, and each module's test steps run once per level. + ### Who may say what - §68 **Callers per verb, in the seat's definition.** Each seat definition names, per verb, the callers allowed to call it, @@ -149,7 +283,7 @@ One file of milestone G's part of [design 47](../47-delivery-from-commit-to-deli | `pipeline-started`, `pipeline-ended` | a pipeline's start, and its end with every step's result, its test counts, the source revision and `widens` | | `group-ended` | a group's one pipeline's end, on which alone its wants are written | | `run-started` | a run on request's start, with its members at their commits | - | `step-started`, `step-ended` | each step's start and end: the step, its status in the policy, its outcome, its build-input fingerprint, its test counts, for a carried result what it was carried from, and for a rerun that it is one and the caller the runtime proved; a bypassed step too | + | `step-started`, `step-ended` | each step's start and end: the step, its status in the policy, its outcome, its build-input fingerprint, its test counts, for a carried result what it was carried from, and for a rerun that it is one and the caller the runtime proved, for a step that read a test run or a composition run another step asked its `shared` (§723, §726); a bypassed step too | | `wait-held`, `wait-ended` | a step's wait: on a person (approval, code review), on a build node, on the revision view | - §74 **What a session hears after its merge.** Between its `merge-pull` and the delivery, a session waits with `mesh_wait` @@ -195,14 +329,14 @@ four lines a module's tools answer; every read tool of a store also answers its | Part (seat) | Read everything it holds | Do by hand what it does itself | Recover without help | |---|---|---|---| -| the runner (`mesh-pipeline-runner`) | `show` (one pipeline whole, its steps, step log and outputs), `list` (the pipelines of a module, a branch or a commit), `stands` (what stands still, since when), `why-carried` (where a step's result was carried from), `kept` (the kept results by digest) | `start` (a delivering pipeline for a watched branch's head), `run` (the run on request), `rerun` (one step again) | `accept` (a failed `optional` step, with why), `cancel` (a pipeline, dropping its asks), `ask-again` (a declined `approval`) | +| the runner (`mesh-pipeline-runner`) | `show` (one pipeline whole, its steps, step log and outputs), `list` (the pipelines of a module, a branch or a commit), `stands` (what stands still, since when), `why-carried` (where a step's result was carried from), `kept` (the kept results by digest), `runner_test_runs` (each test run, §727) | `start` (a delivering pipeline for a watched branch's head), `run` (the run on request), `rerun` (one step again) | `accept` (a failed `optional` step, with why), `cancel` (a pipeline, dropping its asks), `ask-again` (a declined `approval`), `runner_test_runs_drop` | | the indexer (`mesh-catalog`) | `registrations` (with each one's history); `indexer_edges` (an artifact's edges per watched branch), `indexer_order` (the order of a change, its cycles named), `indexer_catalog_difference` (two revisions); `indexer_pending` (a group with its missing members) | `register`, `policy`, `watch`, `unwatch` (each on the operator's Approve) | `indexer_seed_again` (the first registrations' ask raised again), `indexer_take_in_pending` (a group taken in as gathered), `indexer_drop_pending` (dropped) | | the gate (`mesh-delivery`) | `source` with a branch (`gate_source`: every branch's source revision) | — | `gate_source_again` (say `source-moved` again for a branch) | | the forge module (`git`) | `git_failed_steps` (the failed `required` steps it holds, by module and fingerprint, with the run that failed each) | — | `git_failed_steps_drop` (drop one, with why) | | the grants part (`mesh-grants`) | `grants_callers` (the callers each verb admits, read from the seat definitions) | — | `grants_why_refused` (for an account and a verb, the grant it lacks and the definition's line that leaves it out) | | the build seats | `steps` (the steps detected for an artifact) | `steps` with `again` (detect again) | `drop` (ADR 0664 §4) | | the builder (`mesh-builds`) | `builder_reserved` (each module's reserved version, §369); the released versions are the catalog's `versions` | `builder_publish_again` (publish the missing artifacts under the reserved version) | — | -| the assigner (`mesh-wants`) | `assigner_why_not` (why a module is not on a machine yet: the pipeline it waits for, its hold, the placement, the node's last `running`) | — | `assigner_release` | +| the assigner (`mesh-wants`) | `assigner_why_not` (why a module is not on a machine yet: the pipeline it waits for, its hold, the placement, the node's last `running`), `assigner_compositions` (each composition run, §727) | — | `assigner_release`, `assigner_compositions_drop` | | the watchman | `watchman_views` (how far behind each part's view is, from each part's `view-waits` and first-view tools) | — | each part's `_first_view_again` | | the operator-channel | `asks` (with each asker's asks held back) | — | `ask-again` (offer a held-back ask now) | | every module that imports mesh-sdk's small object store package | `_state_objects`, `_sent_objects` | — | `_state_objects_drop`, `_sent_objects_drop` | @@ -230,6 +364,8 @@ and an estimated 2 mesh-sdk publishes a day, each rebuilding about 120 artifacts | the builder | `reserved`: one key per module and `MAJOR.MINOR` (§369) | old `state` section | until its `version-released`, a release of another `MAJOR.MINOR` of the module, or `builder_reserved_drop` | a few KiB; at the cap the builder reserves nothing and says why | | the assigner | `written`: the last source revision and end count per module and watched branch (§352) | old `state` section | until the branch is unwatched | a few KiB | | the indexer and the assigner | `inputs`: a `run` ask's input by write time and sha256 (§354) | old `state` section | until the ask's `step-ended`, at most a day | 64 MiB; an estimated 300 asks a day of under 100 KiB each is under 30 MiB; at the cap the asker refuses the ask loudly, naming the store | +| the assigner | `composition-runs`: one key per candidate and machine, its program, input, ask, readers and verdict (§719) | old `state` section, `max-mib` 16 | a day after its end, or a day after its start with no end; deleted by the assigner (§719) | 16 MiB; an estimated 30 candidates a day on four machines, 120 keys of at most 8 KiB, is under 1 MiB a day | +| the runner | `test-runs`: one key per level and tested packages, its ask, readers, outcome and counts (§725) | old `state` section, `max-mib` 16 | 7 days after its end, deleted by the runner in the step log's order (§330) | 16 MiB; an estimated 1,000 test runs a day of about 1 KiB is about 7 MiB at 7 days | | the runner and the assigner | `findings` (§355), read by their own read tools alone; the keeper hears the findings as events (§383) | old `state` section | an entry deleted when its finding clears | a few KiB | | every observer (§387) | the stream `-findings`: `finding-written`, `finding-cleared`, `findings-stated` (§383) | old `streams` section for a foundation part, the module's `streams` otherwise | 7 days, its oldest dropped when full; the next statement covers a lost one (§384) | 16 MiB; an estimate of a few hundred findings a day of about 1 KiB is under 2 MiB a week | | the condition keeper | `findings-views`: one key per observer and node, its last stated set, pending keys and open asks (§388) | the form of its `conditions` | deleted when the module leaves the catalog or the node (§365) | under 1 MiB; about 30 observers on four machines | diff --git a/03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md b/03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md index bd634a92..e090b3b0 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md +++ b/03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md @@ -245,8 +245,9 @@ the principle that settles it, cited as "design 47 G A-n". `view_tiers` call for a source revision compares the kept assignment revision whose catalog for that branch is that source revision with the revision kept before it. The view keeps a state `waits` (key owner, step, node, module) written by its process from every part's `wait-held` and `wait-ended` it consumes, read by its own read tool. -- **A-243** (corrects A-226's joining; A-237, A-238; principle 7) The composition step runs one request per machine, - each carrying that machine's input; no inputs are joined. A-237 holds for every run's input (the assigner's and +- **A-243** (corrects A-226's joining; A-237, A-238; principle 7; amended by design 47 G §718 on 2026-11-17) The + composition step's machines are composed by one request per machine and candidate, shared by every composition step + of that candidate (§718), each request carrying that machine's input; no inputs are joined. A-237 holds for every run's input (the assigner's and `mesh-wants`' `input` too): any size, kept with an `-objects` store, answered in chunks. The composition run names no head (A-238); its program is the node-engine fingerprint's entry (repository, commit, directory) with the composer's package. @@ -284,7 +285,9 @@ the principle that settles it, cited as "design 47 G A-n". come from the step's catalog revision through mesh-catalog's `revision` (each entry's manifest digest and verdict, its `definitions`), depth compose, the one machine, one case `change`. A machine's node-engine build is what the assigner keeps from the node-engine seat's own events, never read from the node-engine's bucket; where the step's - revision's node-engine entry differs from the trunk's, that build serves every machine. One run ask per machine. + revision's node-engine entry differs from the trunk's, that build serves every machine. One run ask per machine and + candidate, which every composition step of that candidate placing a module on that machine reads (amended by design + 47 G §718 on 2026-11-17). - **A-253** (A-249; principle 8) The tool runner hands a bundle a chunked answer as the verb's own answer with the whole file added as the member `file` (base64). - **A-254** (§712, §354, A-240, A-247, ADR 0596; principles 1, 2 and 7) A build seat's `step-ended` of a run carries From 77318ca60e79344e1903ca34afc07ed44187849e Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 17 Nov 2026 10:25:15 +0100 Subject: [PATCH 2/9] Design 47 G: a test run asked again moves every step that reads it; two items after G The coordinator's answers of 2026-11-17: a step left reading a failed result would say what is no longer true, so a rerun of a test run moves all its readers; the node-engine's differing input bytes and the split of design 47's main file wait until after G. --- .../G-2-pipelines-sizes-checks-and-tools.md | 12 ++++++++---- .../01-to-be/47-milestones/G-23-after-delivery.md | 13 +++++++++++++ 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md b/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md index 6c495532..71c25412 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md +++ b/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md @@ -220,9 +220,12 @@ Each line is a choice within ADR 0666 §7, §8 and §17, settled by the principl it (§24, §419). - **Dropped only with its last reader.** A step that is superseded or cancelled leaves `readers`, and the run's ask is dropped (§119) only when no reader is left. - - **Asked again.** A `rerun` of a test step joins a run of its key while that run is running, or reads one that - started after the step's last end, and otherwise asks a new run, replacing the ended key by compare-and-set. A - sibling step with the same key moves only by its own rerun, which then reads that run. + - **Asked again, for every reader.** A `rerun` of a test step joins a run of its key while that run is running, or + reads one that started after the step's last end, and otherwise asks a new run, replacing the ended key by + compare-and-set. The new run's end moves every pipeline step that read the key's former run and did not pass: each + says `step-ended` again with the new outcome, marked as a rerun with the caller of the step asked again (§73). A + step left reading a failed result would say what is no longer true (principles 3 and 7; the coordinator, + 2026-11-17). - §727 **The tools.** (§83; ADR 0666 §25; principle 4) Two for each new state, in §83's table: - `assigner_compositions` takes a candidate, a machine, or neither, and answers each composition run with its program, input digest, ask, readers, outcome and verdict, and the state's fill against its cap. @@ -242,7 +245,8 @@ Each line is a choice within ADR 0666 §7, §8 and §17, settled by the principl blamed by name, and two modules owning one path are both blamed. - Whole (runner and Go build module): a module whose program and bundle test one directory asks one unit step and one integration step. Both artifacts' steps end with that result and `shared`, a failing test fails once, and - `pipeline-ended` counts the module's tests once. + `pipeline-ended` counts the module's tests once. A rerun of one artifact's failed step that passes moves the other + artifact's step to passed too. - Live: in the first publish after G's delivery, the composition run asks of each revision number at most its machines, and each module's test steps run once per level. diff --git a/03-DESIGN/01-to-be/47-milestones/G-23-after-delivery.md b/03-DESIGN/01-to-be/47-milestones/G-23-after-delivery.md index 738ec504..63b68f5c 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-23-after-delivery.md +++ b/03-DESIGN/01-to-be/47-milestones/G-23-after-delivery.md @@ -167,3 +167,16 @@ deals with it, which is how the list is kept true. glossary word in their place. - A composition that passes on every machine as "not judged: the composer was handed no seat list" is reported as passed: a composition step that judged nothing must fail or say so (principle 3). + +## From the composition runs of 2026-11-17 + +- **The node-engine answers `compose-candidate` with different bytes for one candidate.** Within 30 seconds the + assigner kept twelve inputs of exactly 1,491,623 bytes with different digests for the same revision. The bytes differ + in the node-engine's answer, not in the assigner's request; a guess, not checked: a slice whose order changes between + asks, perhaps `ComposedHow.Running`. Under design 47 G §718 one composition run per candidate and machine asks each + node once, so this no longer multiplies anything and stays out of G (principle 7; the coordinator, 2026-11-17). After + G: find it by comparing two kept inputs of one node, read by digest through `mesh-wants`' `input`, and decide whether + the answer must be the same bytes while the node is unchanged. +- **Design 47's main file is split into milestone parts** (playbook 02, *The shape of a design*). It is about 250 KB + against its 80 KB bound. The split is a separate amendment after G's delivery, never folded into a line of G (the + coordinator, 2026-11-17). From 6b7adc5ec2fe79f6890313f090a12b9bd49f8f8a Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 17 Nov 2026 10:31:28 +0100 Subject: [PATCH 3/9] Design 47 G: write in the review of the composition runs and test runs The four reviewers by architectural domain found results crossing candidates, a read of a build seat's state, fixed durations and caps, unstated seat contract fields and a rerun rule that differed between the two kinds; the lead's answers A-466 to A-466l settle each. The lines move to a file of their own, G-24, so G-2 stays under its bound. The split of the main file and G-22 waits until after G, as the operator decided at 10:31 CET. --- 00-META/glossary.md | 6 +- .../47-delivery-from-commit-to-delivered.md | 23 +- .../G-2-pipelines-sizes-checks-and-tools.md | 148 +----------- .../47-milestones/G-22-lead-answers-2.md | 35 +++ .../47-milestones/G-23-after-delivery.md | 6 +- .../G-24-composition-runs-and-test-runs.md | 226 ++++++++++++++++++ 03-DESIGN/01-to-be/47-milestones/G.md | 6 +- 7 files changed, 291 insertions(+), 159 deletions(-) create mode 100644 03-DESIGN/01-to-be/47-milestones/G-24-composition-runs-and-test-runs.md diff --git a/00-META/glossary.md b/00-META/glossary.md index 818c22a7..b99573dc 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -625,9 +625,9 @@ controller (the planner, the rollout) until release 6 of ADR 0429; from each rel every composition step of the candidate whose module is placed on the machine reads its verdict (design 47 G §718). Not a **composition**, one module's resources composed for one node; nor the **composition step**, the pipeline step that reads one composition run per machine. -- **test run** — the one run of a lint, unit test or integration test pipeline step for one level and one set of - tested packages at their digests, kept by the runner in its state `test-runs`; every artifact's pipeline step naming - the same packages reads its result (design 47 G §724). Not a **pipeline step**, each artifact's own, nor a **run +- **test run** — the one run of a lint, unit test or integration test pipeline step for one candidate, one level and + one set of tested packages at their digests, kept by the runner in its state `test-runs`; every artifact's pipeline step of that candidate + naming the same packages reads its result (design 47 G §724). Not a **pipeline step**, each artifact's own, nor a **run on request**, a whole pipeline that delivers nothing. - **merge check** — what a pull request carries before it may merge, judging the merge of its head into a named **trunk commit**, made in a clone of its own on a build node and never pushed (ADR 0323). One status, diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index 721fc5ed..d6a552c7 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -1972,16 +1972,23 @@ ADR 0667's checks, of milestone G itself: `condition-raised` (whole, condition keeper and a fixture observer, mesh-catalog and mesh-sdk). Live: after G's delivery, a node-engine restarted with the full seat list clears its `seat-list-unread`, read with `mesh-conditions.conditions`. -108. **One composition run per candidate and machine** (design 47 G §718 to §723). Three composition steps of one - catalog revision on two machines start two composition runs and put two run asks, each step reading both - verdicts; a machine whose verdict blames one module fails that module's composition step alone and says it as a - warning in the others; steps asked again together start one run per machine that did not pass (whole, assigner - and the node-engine's composer, mesh-delivery and `mesh-host`). Live: in the first publish after G's delivery, the - composition run asks of a revision number at most its machines, and the state `inputs` stays under half its cap. -109. **One test run per level and tested packages** (design 47 G §724 to §726). A module whose program and tools +108. **One composition run per candidate and machine** (design 47 G §718 to §723, §727 to §731). Two composition + steps of one candidate started at once make one key, one request and one run ask; three steps of one revision on + two machines start two runs, each step reading both verdicts. A machine whose verdict blames one module fails that + module's composition step alone and says it as a warning in the others; a verdict with no `blamed` is judged as + today, naming no culprits. A run with no `step-ended` within its bound ends `could not run` with its readers; steps + asked again together start one run per machine that did not pass and move every former reader. Both drop tools end + the readers, and a reader-only account calling one is refused (whole, assigner and the node-engine's composer, + mesh-delivery and `mesh-host`). Live: in the first publish after G's delivery, the composition run asks of a + revision number at most its machines, read with `assigner_compositions` and `assigner_inputs`. +109. **One test run per level and tested packages** (design 47 G §724 to §731). A module whose program and tools bundle test one directory runs its unit and integration tests once per level, both artifacts' pipeline steps ending with that result and naming it in `shared`, a failing test failing once, and `pipeline-ended` counting the - module's tests once (whole, runner and Go build module, mesh-delivery and mesh-catalog). + module's tests once. A rerun that passes moves both artifacts' steps and the pipeline ends again; a runner + restarted during a run takes it up again; a replaced reader leaves the ask to the others; at the cap a step ends + `could not run` naming the fill; a second revision with the same digests runs a failed test again (whole, runner + and Go build module, mesh-delivery and mesh-catalog). Live: each module's test steps run once per level, read with + `runner_test_runs`. ## What retires diff --git a/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md b/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md index 71c25412..0db8a6be 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md +++ b/03-DESIGN/01-to-be/47-milestones/G-2-pipelines-sizes-checks-and-tools.md @@ -25,7 +25,6 @@ decisions: - "ADR 0448" - "ADR 0363" - "ADR 0283" - - "ADR 0267" - "ADR 0240" --- @@ -113,143 +112,6 @@ One file of milestone G's part of [design 47](../47-delivery-from-commit-to-deli pipelines up again. Live: once the checker is removed no module claims the seat `mesh-delivery-checker`. Tests 64, 84 and 88. -### One composition run per candidate and machine, one test run per tested packages - -*§718 to §728 come from milestone G's live delivery of 2026-11-17. The 24 pipelines of one catalog revision asked 96 -composition runs, one per pipeline and machine, and 80 more when they were asked again; each run kept its own copy of -a node's input of 1.1 to 1.5 MB in the assigner's state `inputs`, which reached its 64 MiB cap twice. The indexer's -and the teller's same failing tests failed once per artifact. The operator, 2026-11-17 about 10:20 CET: "In G, now". -Each line is a choice within ADR 0666 §7, §8 and §17, settled by the principle it names.* - -- §718 **One composition run per candidate and machine.** (§354, A-226, A-243, A-252; ADR 0666 §17; principle 7) The - assigner, holding the seat `mesh-wants`, composes each machine once per candidate, whatever number of composition - steps name it. The candidate is named by its key (A-226): `.` for a branch pipeline, `run-` for - a run on request. The first composition step of a candidate that needs a machine starts that machine's composition - run: one `compose-candidate` ask of the node, its request kept once in `inputs`, one `run` ask on `node-build-go`. - Every other composition step of that candidate whose module is placed on that machine (A-252) reads that run's - verdict. A run's request depends only on the candidate and the machine (the candidate's manifests, verdicts and - seat definitions, depth compose, the one machine, the case `change`), never on the module whose pipeline asked it, - so one verdict serves every module's pipeline. The composer program is chosen once, when the run starts, by A-252's - and A-451's rule, and recorded in the run. This replaces "one run ask per machine" for each composition step (A-243, - A-252) with one per candidate and machine. -- §719 **Kept in the assigner's state `composition-runs`.** (§354, §85; ADR 0666 §4, §7; principles 1, 2 and 7) Each - composition run is one key of the assigner's old `state` section `composition-runs`, whose `max-mib` is 16. The key - is `.`, each part escaped as a key's name (`keys.EscapeName`). The step that starts the run - writes the key create-only, so two steps of one candidate starting at once start one run and the second reads it. - The value holds: - - `candidate` and `machine`; - - `program`: repository, commit, directory, and the node-engine fingerprint it was chosen from; - - `input`: the digest of the kept request; - - `ask`: the id of the build seat's run ask; - - `started`; - - `readers`: the step keys of the composition steps reading the run, each added and removed by compare-and-set; - - at the run's end, `ended`, `outcome` (`passed`, `failed` or `could not run`, with its words) and `verdict` (that - machine's composer verdict, §720). - - A step waiting on a running composition run hears its end through the assigner's own watch of that key (a module - watches its own state, principle 1). A restart of the assigner loses nothing: at its start the assigner takes up - again each key with no `ended`, by its `ask`'s `step-ended` (A-240), and ends `could not run` a key whose `ask` the - build seat no longer holds, naming the ask. The assigner deletes a key a day after its `ended`, and a key with no - `ended` a day after its `started`, ending each of its readers `could not run`; both are judged as the state `inputs` - is (A-465b). At the cap no run is started, and its steps end `could not run`, naming the fill (as A-465a). -- §720 **A machine's verdict names the modules it blames.** (ADR 0666 §17; A-452; principles 10 and 3) The composer's - verdict of a machine (mesh-delivery-sdk's `composer.Machine`, answered by the node-engine's `mesh-compose`) gains - `blamed`, a list of `{module, problem}`. It names each module that the candidate's composition of the machine holds - or refuses and the composition without the candidate does not (the modules in its input's `HeldBefore` are left - out). The culprits are found by the node-engine's own rule of holding one module at a time until the problem ends, - as the indexer's validator names a manifest whose removal ends a problem; two modules owning one path are each - blamed, with that path. `composes` stays false when the machine does not compose with every module of the - candidate. A machine that does not compose and blames no module fails on the node's own account (its own - composition, or no module composed at all, as A-452 has it). -- §721 **One module's fault fails only its own pipelines.** (§720; ADR 0666 §9, §17; A-452; principles 10 and 3) A - composition step judges each of its machines from that machine's verdict: - - `failed` when the verdict blames the step's module, or when the machine does not compose and blames no module; - - `could not run` when the composition run could not run; - - `passed` otherwise, with a warning for each module blamed there that is not the step's own (` holds - : `). Another module's fault is so said in every pipeline it touches and fails none of them. - - The step's outcome then follows from its machines, as today: failed before could not run before passed (§354). A - machine that is away passes with a warning and is not composed (§64). -- §722 **What asking a composition step again does.** (§83 `rerun`; ADR 0666 §11, §24; principle 7) The runner's - `rerun` of a composition step asks the assigner for the step again, marked as a rerun (§73). A machine that passed in - the step keeps its verdict. For each machine whose verdict in the step was not `passed`, the step: - - joins its candidate's and machine's composition run while that run is running; - - reads that run's verdict when the run started after the step's last end; - - otherwise starts a new run, replacing the ended key by compare-and-set. - - So twenty composition steps of one revision asked again together start one run per machine. A new run's verdict - counts for no other step until that step is itself asked again or asked for the first time. -- §723 **How a composition run ends and is said.** (§73, §116, §496, §119, A-182; ADR 0666 §7, §21; principles 7 and 1) - A composition run says no event of its own. Its end is the build seat's `step-ended` of its `ask`, which the - assigner alone hears (A-240). Each composition step reading the run says its own `step-ended` on `mesh-wants`, as - today. That event's output names, per machine, the run it read (`shared`: the run's key and its `ask`) and the - verdict. So the runner, the forge module's `failed-steps` and the delivery record hear every pipeline's result from - the step's holder, as before (ADR 0666 §21). Each reading step says its `step-progress` once a minute while it waits - (A-262). A step that is superseded or cancelled leaves `readers`. The run's `ask` is dropped (§119, A-182) only when - no reader is left, so a superseded pipeline never takes away another pipeline's composition run. -- §724 **One test run per level and tested packages.** (§22, §25, §27, §416; ADR 0666 §8, §17; ADR 0267 §1; - principle 7) The runner asks a lint, unit test or integration test step of a build seat once per level and tested - packages, whatever number of artifacts' steps name them. A module's program and its tools bundle are built from - different import closures (ADR 0267 §1), but their test steps run the packages of one directory (the build seat's - `steps`, §22), so one set of tests is run. The share key is the level and the sha256 of the sorted lines - `@`, one for each package the step would run after its carry (§25: those with no kept result for - their digest). The digests are those the runner already reads for its carry (§416). The first step of a key puts the - build seat's `step` ask; every other step with that key reads its result, and applies its own status from the policy - (§23). A later step of the same key reads the result of the key's ended run, a failed result too, so the same - failing tests fail once and not once per artifact. -- §725 **Kept in the runner's state `test-runs`.** (§27, §85, §330; ADR 0666 §4; principles 1, 2 and 7) Each test run is - one key of the runner's old `state` section `test-runs`, whose `max-mib` is 16. The key is `.`, written - create-only by the step that asks the run. The value holds: - - `level`, and `packages` (each with its digest); - - `seat` and `ask`, and `node` once a holder has taken the ask; - - `started`; - - `readers`: each reading step's pipeline key and artifact, added and removed by compare-and-set; - - at the run's end, `ended`, `outcome`, its test counts, and the name of its output object in `step-outputs`. - - The runner hears the run's end as the build seat's `step-ended` of its `ask`, as it hears every step (§24). At its - start the runner takes up again each key with no `ended`, by its `ask`. It deletes a key 7 days after its `ended`, in - the step log's order, as §330 ages its `pipelines`. A passed run's results go into `kept` per package, as today - (§27), so a step asked after the key is deleted carries them. -- §726 **How a test run ends, is said and asked again.** (§24, §73, §83 `rerun`, §119, §419, §496; ADR 0666 §8, §11, - §21; principles 3, 7 and 1) - - **Each reading step is said.** Each step reading a test run says its own `step-started` and `step-ended`. Its - `step-ended` carries the run's outcome and counts and `shared`: the key, its `ask`, and the pipeline and artifact of - the step that asked the run. Its `step-started` names the run's `ask`, so the watchman times it from that ask's - `step-progress` (§496). - - **Counted once.** `pipeline-ended` counts a catalog module's tests once per test run, never once per step reading - it (§24, §419). - - **Dropped only with its last reader.** A step that is superseded or cancelled leaves `readers`, and the run's ask - is dropped (§119) only when no reader is left. - - **Asked again, for every reader.** A `rerun` of a test step joins a run of its key while that run is running, or - reads one that started after the step's last end, and otherwise asks a new run, replacing the ended key by - compare-and-set. The new run's end moves every pipeline step that read the key's former run and did not pass: each - says `step-ended` again with the new outcome, marked as a rerun with the caller of the step asked again (§73). A - step left reading a failed result would say what is no longer true (principles 3 and 7; the coordinator, - 2026-11-17). -- §727 **The tools.** (§83; ADR 0666 §25; principle 4) Two for each new state, in §83's table: - - `assigner_compositions` takes a candidate, a machine, or neither, and answers each composition run with its - program, input digest, ask, readers, outcome and verdict, and the state's fill against its cap. - - `assigner_compositions_drop` takes `{candidate, machine, why}`. It writes the hand-act log first, drops the run's - ask, ends each reader `could not run` naming the drop, and deletes the key. - - `runner_test_runs` takes a level, a digest, a module, or none of them, and answers each test run with its packages, - ask, node, readers, outcome and counts, and the state's fill against its cap. - - `runner_test_runs_drop` takes `{key, why}` and does the same as `assigner_compositions_drop`. -- §728 **How it is checked.** Tests 108 and 109. - - Whole (assigner, mesh-delivery): three composition steps of one revision, on two machines, start two runs, keep - two requests and put two run asks, and each step reads both verdicts. A fixture machine whose verdict blames module - A fails A's composition step and passes B's, with a warning naming A. A machine that does not compose and blames no - module fails both. Three steps asked again at once start one run for each machine that did not pass. A superseded - step leaves the run running for its other readers. An assigner restarted during a run takes it up again by its - ask. At the cap, a step ends `could not run` naming the fill. - - Whole (the node-engine's composer, `mesh-host`): a fixture module whose manifest is refused on the machine is - blamed by name, and two modules owning one path are both blamed. - - Whole (runner and Go build module): a module whose program and bundle test one directory asks one unit step and - one integration step. Both artifacts' steps end with that result and `shared`, a failing test fails once, and - `pipeline-ended` counts the module's tests once. A rerun of one artifact's failed step that passes moves the other - artifact's step to passed too. - - Live: in the first publish after G's delivery, the composition run asks of each revision number at most its - machines, and each module's test steps run once per level. - ### Who may say what - §68 **Callers per verb, in the seat's definition.** Each seat definition names, per verb, the callers allowed to call it, @@ -287,7 +149,7 @@ Each line is a choice within ADR 0666 §7, §8 and §17, settled by the principl | `pipeline-started`, `pipeline-ended` | a pipeline's start, and its end with every step's result, its test counts, the source revision and `widens` | | `group-ended` | a group's one pipeline's end, on which alone its wants are written | | `run-started` | a run on request's start, with its members at their commits | - | `step-started`, `step-ended` | each step's start and end: the step, its status in the policy, its outcome, its build-input fingerprint, its test counts, for a carried result what it was carried from, and for a rerun that it is one and the caller the runtime proved, for a step that read a test run or a composition run another step asked its `shared` (§723, §726); a bypassed step too | + | `step-started`, `step-ended` | each step's start and end: the step, its status in the policy, its outcome, its build-input fingerprint, its test counts, for a carried result what it was carried from, and for a rerun that it is one and the caller the runtime proved, for a step that read a test run or a composition run another step asked its `shared` (G-24 §729); a bypassed step too | | `wait-held`, `wait-ended` | a step's wait: on a person (approval, code review), on a build node, on the revision view | - §74 **What a session hears after its merge.** Between its `merge-pull` and the delivery, a session waits with `mesh_wait` @@ -333,14 +195,14 @@ four lines a module's tools answer; every read tool of a store also answers its | Part (seat) | Read everything it holds | Do by hand what it does itself | Recover without help | |---|---|---|---| -| the runner (`mesh-pipeline-runner`) | `show` (one pipeline whole, its steps, step log and outputs), `list` (the pipelines of a module, a branch or a commit), `stands` (what stands still, since when), `why-carried` (where a step's result was carried from), `kept` (the kept results by digest), `runner_test_runs` (each test run, §727) | `start` (a delivering pipeline for a watched branch's head), `run` (the run on request), `rerun` (one step again) | `accept` (a failed `optional` step, with why), `cancel` (a pipeline, dropping its asks), `ask-again` (a declined `approval`), `runner_test_runs_drop` | +| the runner (`mesh-pipeline-runner`) | `show` (one pipeline whole, its steps, step log and outputs), `list` (the pipelines of a module, a branch or a commit), `stands` (what stands still, since when), `why-carried` (where a step's result was carried from), `kept` (the kept results by digest), `runner_test_runs` (each test run, G-24 §728) | `start` (a delivering pipeline for a watched branch's head), `run` (the run on request), `rerun` (one step again) | `accept` (a failed `optional` step, with why), `cancel` (a pipeline, dropping its asks), `ask-again` (a declined `approval`), `runner_test_runs_drop` | | the indexer (`mesh-catalog`) | `registrations` (with each one's history); `indexer_edges` (an artifact's edges per watched branch), `indexer_order` (the order of a change, its cycles named), `indexer_catalog_difference` (two revisions); `indexer_pending` (a group with its missing members) | `register`, `policy`, `watch`, `unwatch` (each on the operator's Approve) | `indexer_seed_again` (the first registrations' ask raised again), `indexer_take_in_pending` (a group taken in as gathered), `indexer_drop_pending` (dropped) | | the gate (`mesh-delivery`) | `source` with a branch (`gate_source`: every branch's source revision) | — | `gate_source_again` (say `source-moved` again for a branch) | | the forge module (`git`) | `git_failed_steps` (the failed `required` steps it holds, by module and fingerprint, with the run that failed each) | — | `git_failed_steps_drop` (drop one, with why) | | the grants part (`mesh-grants`) | `grants_callers` (the callers each verb admits, read from the seat definitions) | — | `grants_why_refused` (for an account and a verb, the grant it lacks and the definition's line that leaves it out) | | the build seats | `steps` (the steps detected for an artifact) | `steps` with `again` (detect again) | `drop` (ADR 0664 §4) | | the builder (`mesh-builds`) | `builder_reserved` (each module's reserved version, §369); the released versions are the catalog's `versions` | `builder_publish_again` (publish the missing artifacts under the reserved version) | — | -| the assigner (`mesh-wants`) | `assigner_why_not` (why a module is not on a machine yet: the pipeline it waits for, its hold, the placement, the node's last `running`), `assigner_compositions` (each composition run, §727) | — | `assigner_release`, `assigner_compositions_drop` | +| the assigner (`mesh-wants`) | `assigner_why_not` (why a module is not on a machine yet: the pipeline it waits for, its hold, the placement, the node's last `running`), `assigner_compositions` (each composition run, G-24 §728) | — | `assigner_release`, `assigner_compositions_drop` | | the watchman | `watchman_views` (how far behind each part's view is, from each part's `view-waits` and first-view tools) | — | each part's `_first_view_again` | | the operator-channel | `asks` (with each asker's asks held back) | — | `ask-again` (offer a held-back ask now) | | every module that imports mesh-sdk's small object store package | `_state_objects`, `_sent_objects` | — | `_state_objects_drop`, `_sent_objects_drop` | @@ -368,8 +230,8 @@ and an estimated 2 mesh-sdk publishes a day, each rebuilding about 120 artifacts | the builder | `reserved`: one key per module and `MAJOR.MINOR` (§369) | old `state` section | until its `version-released`, a release of another `MAJOR.MINOR` of the module, or `builder_reserved_drop` | a few KiB; at the cap the builder reserves nothing and says why | | the assigner | `written`: the last source revision and end count per module and watched branch (§352) | old `state` section | until the branch is unwatched | a few KiB | | the indexer and the assigner | `inputs`: a `run` ask's input by write time and sha256 (§354) | old `state` section | until the ask's `step-ended`, at most a day | 64 MiB; an estimated 300 asks a day of under 100 KiB each is under 30 MiB; at the cap the asker refuses the ask loudly, naming the store | -| the assigner | `composition-runs`: one key per candidate and machine, its program, input, ask, readers and verdict (§719) | old `state` section, `max-mib` 16 | a day after its end, or a day after its start with no end; deleted by the assigner (§719) | 16 MiB; an estimated 30 candidates a day on four machines, 120 keys of at most 8 KiB, is under 1 MiB a day | -| the runner | `test-runs`: one key per level and tested packages, its ask, readers, outcome and counts (§725) | old `state` section, `max-mib` 16 | 7 days after its end, deleted by the runner in the step log's order (§330) | 16 MiB; an estimated 1,000 test runs a day of about 1 KiB is about 7 MiB at 7 days | +| the assigner | `composition-runs`: one key per candidate and machine, its program, input, ask, readers and verdict (G-24 §719) | old `state` section, `max-mib` the setting `composition-runs-cap-mib` | the setting `composition-runs-kept` (default 24 hours) after its end; deleted by the assigner (§719) | default 16 MiB; an estimated 30 candidates a day on four machines, 120 keys of at most 8 KiB, is under 1 MiB a day | +| the runner | `test-runs`: one key per candidate, level and tested packages, its ask, readers, outcome and counts (G-24 §725) | old `state` section, `max-mib` the setting `test-runs-cap-mib` | the setting `test-runs-kept` (default 7 days) after its end, deleted by the runner in the step log's order with its output object (§330) | default 16 MiB; an estimated 1,000 test runs a day of about 1 KiB is about 7 MiB at 7 days | | the runner and the assigner | `findings` (§355), read by their own read tools alone; the keeper hears the findings as events (§383) | old `state` section | an entry deleted when its finding clears | a few KiB | | every observer (§387) | the stream `-findings`: `finding-written`, `finding-cleared`, `findings-stated` (§383) | old `streams` section for a foundation part, the module's `streams` otherwise | 7 days, its oldest dropped when full; the next statement covers a lost one (§384) | 16 MiB; an estimate of a few hundred findings a day of about 1 KiB is under 2 MiB a week | | the condition keeper | `findings-views`: one key per observer and node, its last stated set, pending keys and open asks (§388) | the form of its `conditions` | deleted when the module leaves the catalog or the node (§365) | under 1 MiB; about 30 observers on four machines | diff --git a/03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md b/03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md index e090b3b0..492e371b 100644 --- a/03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md +++ b/03-DESIGN/01-to-be/47-milestones/G-22-lead-answers-2.md @@ -1538,3 +1538,38 @@ the principle that settles it, cited as "design 47 G A-n". time, and an input with its object once no ask's key names it, so no bookkeeping key of its own is kept. A key of the old form `