diff --git a/00-META/glossary.md b/00-META/glossary.md index 55fdab79..2304a168 100644 --- a/00-META/glossary.md +++ b/00-META/glossary.md @@ -289,7 +289,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences` **Purpose.** To take one commit from its pull request to every node that should run it, and to put it back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the planner, the walk). **Downstream of** Module, Core, Health and repair and Data. -**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270. +**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276. **Uses:** module, node, declaration and send (Placement), health (Health and repair), person. - **merge check** — the two statuses a pull request carries before it may merge: the **merge gate** @@ -326,6 +326,8 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data. - **delivery** — one commit in one repository on its way to the nodes, from its pull request's head being announced to one of its final states: one pull request, one status, one note ([ADR 0239](../02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md)). + From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk + ([ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`, `held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`; [ADR 0250](../02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md), @@ -345,10 +347,17 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data. person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a diffset; the controller's verb `delivery-plan` shows it. *Deploy* is used in this one place. *Not:* ~~change plan~~, ~~release plan~~ +- **merge window** — the rolling wait in which the controller collects merges before it plans them: each + merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest + 10 minutes after its first merge ([ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). +- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more + than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*, + which is a delivery group. - **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat of the same name. It records and decides; the controller sends, judges and rolls back when it is asked. -- **walk** — the controller's sending of one trunk commit's builds across nodes, tier by tier, one node - first and judged at the first-node gate, then the rest (ADR 0236). A primitive the delivering stage +- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per + repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236, + [ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). One walk is open at a time; it names every merge it answers. A primitive the delivering stage asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own: a module *rolls out* by its `roll` policy. @@ -548,7 +557,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical. |---|---|---|---| | plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement | | | what a delivery would do to the mesh | **delivery plan** | Change and delivery | -| | the sending of one commit across nodes | **walk** (verb `plans`) | Change and delivery | +| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery | | | a step a person starts, like the bus's | **planned step** | Core | | membership | the subjects a bus account is issued | **membership** | Provisioning | | | an account in a group of the machine | **account group** | Placement | diff --git a/02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md b/02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md index 3c3fd577..4e8f6cfc 100644 --- a/02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md +++ b/02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md @@ -9,6 +9,11 @@ extends: 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md # 162. A merge produces a tiered plan the mesh keeps, and a module's dependencies are one relation in the catalogue +> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).** +> "A merge produces a tiered plan" no longer holds: merges are collected in a rolling merge window, and each +> batch is planned once, as one walk with at most one commit per repository (issue 362). The plan the mesh +> keeps, its tiers and the one dependency relation stand as decided here. + > **Progressive insight — 2026-10-02.** The context below says a dependent is *built by whichever build machine is running — the only one there could be*. That was a fact of the day, not of the decision: since [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md) a tier's asks are taken by every machine holding the build seat. The plan and its tiers are unchanged. > **Superseded in part — 2026-10-08, by [ADR 0267](0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md).** diff --git a/02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md b/02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md index 3623c6bc..b1389605 100644 --- a/02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md +++ b/02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md @@ -9,6 +9,12 @@ extends: 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md # 218. A plan sends grants before code, rolls a module out one machine first, and a newer merge takes over an older plan +> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).** +> Decision 3 no longer holds: a newer merge does not take over an older plan. Merges are collected in a +> rolling merge window and walked one batch at a time; a walk that started is never superseded, and only a +> walk still waiting for its word is folded into the next batch's walk (issue 362). Grants before code and +> one machine first stand as decided here. + ## Context On 2026-10-05 the delivery path was watched through a day of merges, by several sessions at once. Three diff --git a/02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md b/02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md index 6cddc47a..40fe4c7e 100644 --- a/02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md +++ b/02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md @@ -11,6 +11,11 @@ extends: 02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before- # 238. A commit is the build at hand: one commit, one change plan, checked off the trunk and published only on it +> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).** +> Decision 2's "one commit, one change plan" no longer holds for the release after the merge: a batch of +> merges collected in a rolling merge window is planned once, as one walk with at most one commit per +> repository (issue 362). A pull request's head is still checked on its own, with its own delivery plan. + > **Superseded in part — 2026-10-06, by [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md).** Decision 6's state machine is > built as a **delivery**'s, owned by the module `mesh-delivery` rather than by the controller. Its states are > renamed (`releasing` → `delivering`, `done` → `delivered`) and a delivery group sits above it. Decision 7's diff --git a/02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md b/02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md index 6cb798c5..a8d9084e 100644 --- a/02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md +++ b/02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md @@ -11,6 +11,14 @@ extends: 02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-p # 239. A delivery is owned by the mesh-delivery module and runs from commit to delivered +> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).** +> A delivery is no longer walked by a walk of its own trunk commit. From its merge on it is delivered by the +> walk of the batch that answers it, one walk per rolling merge window with at most one commit per +> repository, and it ends with that walk (`covered` for an earlier commit of a repository the walk carried +> later). The superseded row (published, delivering → superseded) is taken only for a walk folded before it +> started whose successor does not answer the delivery's merge; a delivery follows the walk that took its +> own over. The owner, the table, the group and its order stand as decided here (issue 362). + > **Superseded in part — 2026-10-08, by [ADR 0249](0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md).** > Decision 4's sentence "a declared order that contradicts an inferred one is a cycle" no longer holds. > Two rules ordering one pair both ways were always a cycle, and a group of the controller, the node-engine diff --git a/02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md b/02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md new file mode 100644 index 00000000..dabf0b04 --- /dev/null +++ b/02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md @@ -0,0 +1,218 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-10 +deciders: jochen +reconstructed: false +supersedes-in-part: + - 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md + - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md + - 02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md + - 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md +extends: 02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md +--- + +# 276. Merges are assembled in a rolling window, and each batch is delivered by one walk with one commit per repository + +## Context + +Shortly after midnight on 2026-10-10 two pull requests of the catalogue were merged 18 seconds apart, the +claude-code change at 00:20:52 and the dunst change at 00:21:09 +([issue 362](../04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md)). +The controller opened one walk for each merge, as [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) +decided ("a merge produces a tiered plan"). The second walk superseded the first, as +[ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md) +decision 3 decided, and both were planned at the dunst commit, as the fix for +[issue 349](../04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md) +decided. mesh-delivery then tied the surviving walk to the dunst delivery, which the supersession had already +ended, and tied no walk to the claude-code delivery. Nothing gave the walk its word. The change waited 58 +minutes, until the operator started the walk by hand with `plans go`. + +The diagnosis found two parts that each behave as written and disagree on what a walk's commit means. That +is the narrow cause. The wide one is that the mesh plans one walk per merge, while merges arrive in bursts: + +- **A burst is the normal case.** The operator merges a batch of reviewed pull requests at once, often + across repositories (a catalogue change with the controller change it needs). Every merge opened a walk, + and every walk after the first superseded the one before. Three supersession paths now exist (ADR 0218, + issue 254, issue 349), each with its own bookkeeping, and issue 362 is a fourth way for them to disagree. +- **A walk superseded mid-gate wastes the gate and confuses its record.** A first machine judged healthy + for a build that the next walk replaces a minute later proves nothing about the build that stays. On + 2026-10-09 a controller merge landing while another walk was at its gate rolled everything back; the rule + "one controller rollout at a time" was kept since then by the person merging, not by the mesh. +- **ADR 0239 made a delivery "one commit … from commit to delivered" and walked it alone**, while its own + decision 4 delivers a delivery group across repositories as one unit. The walk the commit gets is + decided by when the controller heard its merge, which is not something the person merging controls. + +On 2026-10-10 the operator decided the model, in their words: + +> "a delivery doesn't really 'run from commit to delivered', we've already discussed that a delivery plan +> might start with a set of commits spanning multiple repo's, however, only 1 commit per repo is allowed +> … we assemble all commits that are coming in with a 1 minute rolling window, so if a new commit comes +> in, less then 1 or 2 minutes after the last one, the builder again waits 1 or 2 minutes, until no new +> commit came in, then it looks at the 'group of commits' and decides if they all fit into 1 delivery +> plan … This is a hard problem to tackle, but it will improve our build setup I believe." + +> "the 'actively assembling plan' is very interesting, the controller can say 'Still waiting for 13 +> seconds for new commits, these are the currently grouped commits, plan is yet to be calculated'." + +On two commits of one repository in one window, the operator chose the later commit over splitting: +"i like your idea better, I just thought it would be more complex, but if it's reasonably doable, go for +it". + +**Checked against GENESIS.** *Failure must be loud*: an assembling batch and a waiting walk each have a +bound and a condition. *The mesh owns the truth*: the window, the batch and the walk are kept by the +controller and shown by its `plans`, not inferred from timing by a person. Nothing here conflicts with +GENESIS. + +**Words.** The glossary had no word for the merges one window collects, nor for the window. This record +adds **merge window** and **batch** to it (Change and delivery), with this record. *Group* is not used for +either, because **delivery group** already names deliveries that share a head branch name. + +## Considered Options + +**What a walk answers.** + +1. **One walk per merge, as today, with the bookkeeping repaired.** The walk names the merges it answers, + and mesh-delivery ties a superseding walk to every live delivery it contains. Rejected as the model, + kept as a repair (decision 8): it makes the 362 shape deliver, but every burst still opens and + supersedes walks, a walk can still be superseded at its gate, and a cross-repository change is still + walked as separate walks in whatever order the merges were heard. +2. **A rolling merge window, and one walk per batch** (chosen). Merges are collected while they keep + arriving; the batch they form is planned once, as one walk with at most one commit per repository. +3. **Every walk started by hand.** The operator named it as the alternative ("this does mean we need to + manually start all delivery plans, or, even better, …"). Rejected: a person in every delivery is the + wait ADR 0239 exists to remove, and a merge that passed its checks must reach its machines without a + person. + +**Two commits of one repository in one batch.** + +1. **Split into two walks in merge order** (the operator's first proposal). The second walk waits for the + first. Rejected: every machine running the module builds, is sent and is judged on the earlier commit, + and then again on the later one minutes after; the gate judges a build that is about to be replaced; + and the batch's walk stops being one unit. +2. **One walk with the later commit** (chosen). On a linear trunk the later merge commit contains the + earlier one, so building it delivers both. One build, one gate, one rollout. The earlier merge is + answered by its successor and ends `covered` when the walk is delivered. Only when that walk fails are + the earlier merges walked on their own, to find which one brought the failure. + +**Who keeps the window.** + +1. **mesh-delivery**, which owns deliveries (ADR 0239). Rejected: a walk on the controller's own path (the + controller, the node-engine, the node tools, the bus, mesh-delivery itself) must not wait for + mesh-delivery, and with no holder of the seat on record the controller plans alone. The window would + then exist twice, once in each. +2. **The controller** (chosen). It hears every merge (the bus, and the catch-up that reads back what the + bus missed), it owns the planner and the walk, and it already plans alone for its own path. mesh-delivery + keeps what it owns: each commit's delivery record, the order of a delivery group, and the word that lets + a waiting walk start. + +## Decision + +1. **A rolling merge window.** The controller collects every merge into a branch a module follows into the + open **batch**. The window is rolling: each merge restarts it. It closes when no merge arrived for the + window's length, 90 seconds by default (the controller's setting `merge-window`), and at the latest 10 + minutes after the batch's first merge (`merge-window-at-most`), so a busy period never starves delivery. + The batch, its merges and its times are kept in the controller's store, so a restarted controller + resumes the window where it stood rather than losing or doubling a merge. +2. **A batch is planned once, as one walk with at most one commit per repository.** When the window closes, + the planner computes one delivery plan for the batch: every module its commits move, their dependents, + in tiers along the module graph, across repositories. For a repository with two or more merges in the + batch the walk takes the latest commit of its branch, which contains the earlier ones. **Every merge + belongs to exactly one walk**: the walk names every merge it answers, its own commits and the earlier + ones they contain. A merge is never dropped and never walked twice. +3. **The earlier merge of a repository is answered by its successor.** Its delivery follows the walk and + ends `covered` when the walk is delivered, naming the merge that carried it (the glossary's word, + [ADR 0270](0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md); + here the containment is the trunk's order inside one batch, so the forge is not asked). **When that walk + fails**, the walk's own commits fail, and each earlier merge it answered is walked on its own commit, the + newest first, one walk each, in the next batches: the first that is delivered ends `delivered`, and the + search stops there. A walk that failed never marks a merge delivered. +4. **One walk at a time, and nothing supersedes a walk that started.** A batch becomes a walk only when no + walk is open. Merges that arrive while a walk runs join the next batch, whatever its window says. A walk + that has not started, because it waits for mesh-delivery's word, is folded into the next batch when that + batch is planned, and the new walk answers every merge the folded one answered. This is the only + supersession left; ADR 0218 decision 3 and the issue 349 rule of planning a late merge at a later commit + are replaced by it. A merge heard after a later merge of its repository and branch was put in a walk is + answered by that walk: by the batch still assembling, by the open walk, or, when that walk was delivered, + it ends `covered` at once. +5. **The order of a delivery group holds inside a walk** (ADR 0249 unchanged). The pairs its order rules + give (declared, built by, version skew, engine before controller, each with the rule that won) become + edges of the walk's tiers beside the module graph: the modules of a member ordered after another are in + a later tier than that member's. A contradiction still rejects the group before anything merges. + mesh-delivery gives a walk its word only when every delivery it carries is at its turn. +6. **A delivery is one commit's record, and its walk is the batch's.** A delivery keeps its pull request, + its status, its note and its check, as ADR 0239 decided. From its merge on it is delivered by the walk + that answers it, and every delivery a walk carries ends with that walk's result: `delivered` (its own + commit) or `covered` (an earlier commit of the same repository) when the walk is done, `failed` when it + fails, `stopped` when it is stopped. The walk's machine steps are recorded on each delivery it carries. +7. **The assembling state is shown.** The controller's `plans` lists the open batch first, as the operator + asked: "assembling: 13 s left (at the latest 00:31:09); grouped: novox/mesh-catalog@48bda475 (answers + 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated". It says the batch on the bus as + `plan-moved` with the state `assembling`, so mesh-delivery's `deliveries` and `show` say of a merged + delivery that its batch is assembling and how long is left, and of one whose batch waits that it waits + behind which walk. +8. **The bookkeeping is repaired as well** (the narrow cause of issue 362), because decision 4 keeps one + supersession path. A walk names every merge it answers, and a superseded walk names the walk that took it + over. mesh-delivery ties a walk to every live delivery whose merge it answers, follows a walk to the one + that took it over, and **never ties a walk to a delivery that has ended**. A walk waiting for the word + that no live delivery holds is either adopted as a delivery of its own or listed at once by `stalled`. +9. **Every wait has a bound and a condition.** A batch still assembling a minute past + `merge-window-at-most` while no walk is open raises a condition: the controller failed to cut it. A + batch waiting behind an open walk longer than that walk's bound raises one naming the walk. A walk + waiting for its word past 30 minutes is S16's, as before, and its words name the merges it answers. A + merged delivery no walk answers within ten minutes is held with the true reason (which open walks of its + repository exist and that none names its merge, or that the controller holds none) and listed by + `stalled` at once, so probe D14 raises it for the operator. + +## Consequences + +- **A merge waits up to 90 seconds before it is planned**, and up to 10 minutes in a busy period. A walk + takes minutes to build and longer to judge, so the wait is small beside it, and it removes the builds, + sends and gates that a superseded walk wasted. +- **A cross-repository change merged within a window is walked as one unit**, tiered by the graph and its + group's order. A person no longer has to merge in order and wait in between. +- **One walk at a time is mechanical.** A controller merge can no longer land in the middle of another + walk's gate; it waits for the next batch. +- **A failing walk fails every delivery it carried**, including one whose own modules passed before the + failure. Each delivery's machine steps say how far it got, and decision 3's search finds the culprit only + among the merges of one repository. A failure in a walk that carried several repositories is the + operator's to read. This is the price of one unit, and the operator chose it. +- **What got harder**: the controller's merge handling holds state across merges (the open batch) and is + restart-safe; the walk's record names several repositories and commits instead of one, and every reader of + it (`plans`, S3, S16, `plan-moved`, mesh-delivery) changes with it. The store gains the batch and the + merges a walk answers, as a numbered migration and in the consolidated schema. +- **The replay of issue 362** in the lab raises a mesh, merges two commits of one repository 18 seconds + apart, and passes when one walk delivers both without a person; it fails on the commit before the change. + +## How it is checked + +- The controller's tests: two merges of one repository 18 seconds apart make one batch and one walk at the + later commit, answering both; merges in three repositories within the window make one walk; a merge at + second 80 restarts the window; the window closes at `merge-window-at-most` however busy; a merge arriving + while a walk runs joins the next batch; a walk that has not started is folded into the next one; `plans` + shows the assembling state; the batch survives a restart; the two conditions of decision 9 are raised past + their bounds and not before. +- mesh-delivery's tests: both deliveries of one walk end `delivered` and `covered` with its done record; a + failed walk marks none delivered and walks the earlier merge alone; a walk is never tied to an ended + delivery; a walk is given its word only when every delivery it carries is at its turn; a group's order is + unchanged; a held delivery with no walk names the true reason and is listed by `stalled`. +- Each of those tests fails with its part of the change removed. +- mesh-lab's replay R362 (decision's consequence above), proved to fail before the change and pass after it. + +## References + +- [Issue 362](../04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md), + its diagnosis, and the replay R362 in mesh-lab. +- [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) (a merge produces a plan), + [ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md) + decision 3 (a newer merge takes over an older plan), [ADR 0238](0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md) + decision 2 (one commit, one change plan), [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md) + (a delivery walked by its own commit's walk), each superseded in the part this record names. +- [ADR 0249](0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md), + whose order rules this record keeps and uses as tier edges. +- [ADR 0270](0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md), + whose `covered` ends an earlier merge of a repository. +- [Issue 349](../04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md), + whose rule decision 4 replaces. +- To-be [47](../03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md) and + [10](../03-DESIGN/01-to-be/10-delivery.md), amended with this record. diff --git a/03-DESIGN/01-to-be/10-delivery.md b/03-DESIGN/01-to-be/10-delivery.md index 20c60006..19d74fe4 100644 --- a/03-DESIGN/01-to-be/10-delivery.md +++ b/03-DESIGN/01-to-be/10-delivery.md @@ -5,8 +5,9 @@ code: - mesh-controller internal/builder - mesh-controller cmd/mesh-controller (build, build --behind, push, status) - mesh-controller internal/inventory/builds.go -updated: 2026-10-06 +updated: 2026-10-10 decisions: + - 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md - 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md - 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md - 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md @@ -282,3 +283,9 @@ everything else while that module is held. A person can always deliver by hand. The controller keeps the comparison, the planner, the gate, sending and rollback. `mesh-delivery` keeps the record and the order, and asks. + +**Merges are walked in batches** (2026-10-10, by [ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). The controller collects +merges in a rolling merge window, 90 seconds by default and at most 10 minutes, and plans each batch once, as +one walk with at most one commit per repository: the latest of its branch, which contains the earlier ones. +One walk is open at a time, and a walk that started is never superseded. `plans` shows a batch while it +assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's walk. diff --git a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md index b69000e6..2949244b 100644 --- a/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md +++ b/03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md @@ -2,8 +2,9 @@ layer: to-be status: designed code: [] -updated: 2026-10-09 +updated: 2026-10-10 decisions: + - 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md - 02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md - 02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md - 02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md @@ -21,6 +22,11 @@ delivery group is two or more deliveries that share a head branch name, delivere are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.** ([ADR 0239](../../02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md).) +**From its merge on, a delivery is delivered by the walk of its batch.** The controller collects merges in a +rolling merge window and plans each batch once, as one walk with at most one commit per repository, one walk +at a time. Every delivery the walk answers ends with it +([ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). + This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison of source and artifacts decides, and nothing here changes that. This design says who answers for one commit's journey and in what order the steps are allowed. @@ -57,7 +63,9 @@ Its id is `/@`. It holds: its diffset (what ADR 0238 called the "change plan", under the glossary's name); - its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its **machine steps** while delivering; -- once merged, **the commit it landed on the trunk as**, and the walk that delivers it. +- once merged, **the commit it landed on the trunk as**, and the walk that delivers it: the walk of the batch + that answers its merge, which may carry a later commit of its repository that contains it (see *The merge + window and the batch*). ## The state table @@ -72,15 +80,15 @@ allows. | checked | a verdict arrived | 1 minute | nothing: it is decided at once | | ready | it passed; it waits to be merged | none | — | | rejected | it failed its check | none | — | -| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a newer delivery took over its walk; `go` when its walk started | +| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a walk that does not answer its merge took over its walk; `go` when its walk started | | held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's | | delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says | | delivered, failed, superseded, stopped, retired, covered | final | — | — | The transitions are those of ADR 0239 decision 2 and [ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md). A delivery that is merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A -delivery merged with no walk opened for it within ten minutes is held too, saying so: nothing it moves follows -that branch, or the merge was not heard. Nothing waits silently. +delivery merged with no walk answering it within ten minutes is held too, with the true reason, and listed by +`stalled` at once. Nothing waits silently. **What needs no person ends by itself** (ADR 0270): @@ -107,6 +115,44 @@ module and the build: `sent` (the first machine's send, or the rest's), `judging readings so far), `passed`, `failed`, `rolled-back`. They are read from the walk's own record in the controller's `plan-moved` event, never inferred from time. +## The merge window and the batch + +*2026-10-10, by [ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).* + +- **The window.** The controller puts every merge into a branch a module follows into the open **batch**. + Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by + default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its + merges and its times are kept in the controller's store and survive a restart. +- **The cut.** When the window closes and no walk is open, the batch becomes one walk: one delivery plan for + every commit in it, tiered along the module graph across repositories, with at most one commit per + repository, the latest of its branch. The walk names every merge it answers: its own commits, and the + earlier merges of a repository that its later commit contains. +- **One walk at a time.** Merges that arrive while a walk is open join the next batch. A walk that started + is never superseded. A walk still waiting for its word when the next batch is cut is folded into the new + walk, which answers every merge the folded one answered. +- **A late merge.** A merge heard after a later merge of its repository and branch was put in a walk is + answered by that walk: the assembling batch, the open walk, or, when that walk was delivered, it ends + `covered` at once. +- **The ends.** A walk done ends every delivery it answers: `delivered` for its own commits, `covered` for + an earlier merge of a repository it carried later, naming that merge. A walk that failed fails its own + commits' deliveries and marks none delivered; each earlier merge it answered is then walked on its own + commit, the newest first, one walk each, until one is delivered. A walk stopped stops them all. +- **A group inside a walk.** The pairs of a delivery group's order (ADR 0249) are edges of the walk's + tiers beside the module graph. mesh-delivery gives a walk its word only when every delivery it carries is + at its turn. +- **The bookkeeping.** mesh-delivery ties a walk to every live delivery whose merge it answers, follows a + folded walk to the walk that took it over, and never ties a walk to a delivery that has ended. A waiting + walk no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once. +- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped: + novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated", + and a batch waiting behind an open walk names that walk. The controller says the batch on the bus as + `plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged + delivery it holds. +- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no walk open, and a batch + waiting behind an open walk past that walk's bound, each raise a condition. A merged delivery that no walk + answers within ten minutes is held with the true reason (the open walks of its repository and that none + names its merge, or that the controller holds none) and listed by `stalled` at once. + ## A delivery group - **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull @@ -161,7 +207,7 @@ In this order, each by its owner: | `groups` | every group with its members in order and its derived state | | `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere | | `table` | the state table and the machine-step table, with bounds and H2's repairs | -| `stalled` | every delivery held past its bound, with the transition H2 may take | +| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting walk no live delivery holds | | `recheck` | a rejected or ready delivery back to proposed, with why | | `release` | a held delivery to delivering, with why: a person's word | | `stop` | any delivery not final to stopped, with why; its walk is ended through the controller | @@ -191,7 +237,8 @@ up. - Any other walk waits for `deliver` only while the seat has a holder on record. With none on record, the controller starts it as it always did. - A waiting walk is said by the controller whatever mesh-delivery says of itself: S16 raises - `plan..waiting` after 30 minutes, urgent after 4 hours, naming `plans go --why` as the way on. + `plan..waiting` after 30 minutes, urgent after 4 hours, naming the merges the walk answers and + `plans go --why` as the way on. A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone. ## The switch diff --git a/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md b/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md index 584b0a73..c1752d85 100644 --- a/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md +++ b/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md @@ -3,7 +3,7 @@ status: located opened: 2026-10-10 located-in: [mesh-catalog modules/mesh-delivery (holder.go deliveryWalking), mesh-controller cmd/mesh-controller (upgrades.go SourceMoved, release_plan.go laterOnTheBranch)] fixed-by: -amended-design: +amended-design: 03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md --- # 362. Two catalogue merges seconds apart left their delivery waiting for a person @@ -22,8 +22,10 @@ walk waits for mesh-delivery's word — `plans go … --why …` starts it by ha Both reasons were wrong. A walk had been opened, and it contained the held merge. Nothing gave it the word. The operator's change sat undelivered for 58 minutes, until the operator's hand-act `mesh-controller.plans -go` started the walk. **No condition was raised to anyone in that time**: the hold is bounded at 24 hours -before `mesh-delivery.stalled` lists it, and the walk's own wait raised nothing either. +go` started the walk. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours +before `mesh-delivery.stalled` lists it, and the walk's own wait raised only a warning, at 00:51, which said +there was nothing to do and pointed at the ended delivery (the diagnosis's second entry; this sentence said +before that the walk's wait raised nothing). ## Why it is an issue @@ -55,3 +57,11 @@ disagree on what a walk's commit means: as a delivery of its own, so that something always holds a waiting walk. - A walk waiting for mesh-delivery's word past a bound, and a delivery held because no walk was opened, raise a condition to the operator instead of waiting silently (failure must be loud). + +## Decided + +The operator decided the model that removes this shape, not only its bookkeeping: +[ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md). Merges are collected in a rolling merge window and each batch is walked once, with at most one +commit per repository, one walk at a time. The bookkeeping that tied a walk to an ended delivery is repaired as +well, and every wait of this shape has a bound and a condition. It is designed in to-be 47, *The merge window +and the batch*. diff --git a/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md b/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md index 52ace325..516e6ef1 100644 --- a/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md +++ b/04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/01-diagnosis.md @@ -57,3 +57,19 @@ Two parts, each correct by its own rule, disagree on what a walk's commit means. 349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a walk's commit is the merge commit of exactly one delivery. `located-in` names both. The fix is to make the walk say which merges it answers, or to make mesh-delivery match by containment, and to let no waiting walk be held by nothing. + +## 2026-10-10 — what the controller's side said + +The first entry says no condition reached the operator in the 58 minutes. The controller's condition history +says more precisely what happened. S16 (a walk waiting for its delivery's word) raised +`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the walk opened, as its bound says. +It was a warning, its explanation was "Nothing for you to do … It becomes a question for you if it still waits +after 4 hours", and its summary sent the reader to "`mesh-delivery.show` for the delivery that landed as +48bda475", which is the dunst delivery the supersession had already ended. It cleared at 01:20:32, when the +operator's `plans go` started the walk. + +So the controller's side was not silent; it was mild and pointed at the wrong delivery. S16 cannot tell a +group member waiting its turn from a walk nobody holds; only mesh-delivery knows that. That is why ADR 0276 +decision 9 has mesh-delivery list a waiting walk no live delivery holds through `stalled` at once, and has +S16 name the merges a walk answers. +