Merge pull request 'ADR 0276: merges are assembled in a rolling window, and each batch is delivered by one walk (issue 362)' (#271) from decisions/0276-merges-are-assembled-in-a-rolling-window into main
This commit was merged in pull request #271.
This commit is contained in:
+13
-4
@@ -289,7 +289,7 @@ and the operator act. **Recorded by** the controller's `identity` and `licences`
|
||||
**Purpose.** To take one commit from its pull request to every node that should run it, and to put it
|
||||
back when it fails. **Recorded by** the `mesh-delivery` module (deliveries) and the controller (the
|
||||
planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270.
|
||||
**Decided by** ADR 0162, 0218, 0236, 0237, 0238, 0239, 0250, 0267, 0270, 0276.
|
||||
**Uses:** module, node, declaration and send (Placement), health (Health and repair), person.
|
||||
|
||||
- **merge check** — the two statuses a pull request carries before it may merge: the **merge gate**
|
||||
@@ -326,6 +326,8 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
- **delivery** — one commit in one repository on its way to the nodes, from its pull request's head
|
||||
being announced to one of its final states: one pull request, one status, one note
|
||||
([ADR 0239](../02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md)).
|
||||
From its merge on it is delivered by the walk of the batch that answers it, and ends with that walk
|
||||
([ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)).
|
||||
Its states are one table (`proposed`, `checked`, `ready` or `rejected`, `published`, `delivering`,
|
||||
`held`, and the final six: `delivered`, `failed`, `superseded`, `stopped`, `retired` and `covered`;
|
||||
[ADR 0250](../02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md),
|
||||
@@ -345,10 +347,17 @@ planner, the walk). **Downstream of** Module, Core, Health and repair and Data.
|
||||
person) and its verdict (the composed nodes, the replays). Computed by the controller's planner from a
|
||||
diffset; the controller's verb `delivery-plan` shows it. *Deploy* is used in this one place.
|
||||
*Not:* ~~change plan~~, ~~release plan~~
|
||||
- **merge window** — the rolling wait in which the controller collects merges before it plans them: each
|
||||
merge restarts it, and it closes when none arrived for its length (90 seconds by default) or at the latest
|
||||
10 minutes after its first merge ([ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)).
|
||||
- **batch** — the merges one merge window collected, planned once as one walk. For a repository merged more
|
||||
than once in it, the walk takes the latest commit, and the earlier merges are answered by it. Not *group*,
|
||||
which is a delivery group.
|
||||
- **mesh-delivery** — the module that owns deliveries and delivery groups, holding the mesh-scoped seat
|
||||
of the same name. It records and decides; the controller sends, judges and rolls back when it is asked.
|
||||
- **walk** — the controller's sending of one trunk commit's builds across nodes, tier by tier, one node
|
||||
first and judged at the first-node gate, then the rest (ADR 0236). A primitive the delivering stage
|
||||
- **walk** — the controller's sending of one batch's builds across nodes, at most one trunk commit per
|
||||
repository, tier by tier, one node first and judged at the first-node gate, then the rest (ADR 0236,
|
||||
[ADR 0276](../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). One walk is open at a time; it names every merge it answers. A primitive the delivering stage
|
||||
asks for, not an object anyone manages. A **tier** is one step of a walk: a module, then the modules
|
||||
built against it. The controller's verb `plans` lists the walks. Walking is not a noun of its own:
|
||||
a module *rolls out* by its `roll` policy.
|
||||
@@ -548,7 +557,7 @@ renaming one sense, the old sense moves to a *Not:* line and becomes mechanical.
|
||||
|---|---|---|---|
|
||||
| plan | what the controller would send one node | that node's **declaration** (verb `plan`) | Placement |
|
||||
| | what a delivery would do to the mesh | **delivery plan** | Change and delivery |
|
||||
| | the sending of one commit across nodes | **walk** (verb `plans`) | Change and delivery |
|
||||
| | the sending of one batch of commits across nodes | **walk** (verb `plans`) | Change and delivery |
|
||||
| | a step a person starts, like the bus's | **planned step** | Core |
|
||||
| membership | the subjects a bus account is issued | **membership** | Provisioning |
|
||||
| | an account in a group of the machine | **account group** | Placement |
|
||||
|
||||
@@ -9,6 +9,11 @@ extends: 02-DECISIONS/0157-a-build-says-what-it-does-on-the-bus-as-it-happens.md
|
||||
|
||||
# 162. A merge produces a tiered plan the mesh keeps, and a module's dependencies are one relation in the catalogue
|
||||
|
||||
> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).**
|
||||
> "A merge produces a tiered plan" no longer holds: merges are collected in a rolling merge window, and each
|
||||
> batch is planned once, as one walk with at most one commit per repository (issue 362). The plan the mesh
|
||||
> keeps, its tiers and the one dependency relation stand as decided here.
|
||||
|
||||
> **Progressive insight — 2026-10-02.** The context below says a dependent is *built by whichever build machine is running — the only one there could be*. That was a fact of the day, not of the decision: since [ADR 0190](0190-a-seats-work-is-shared-by-its-holders-and-building-is-the-first-such-role.md) a tier's asks are taken by every machine holding the build seat. The plan and its tiers are unchanged.
|
||||
|
||||
> **Superseded in part — 2026-10-08, by [ADR 0267](0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md).**
|
||||
|
||||
+6
@@ -9,6 +9,12 @@ extends: 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md
|
||||
|
||||
# 218. A plan sends grants before code, rolls a module out one machine first, and a newer merge takes over an older plan
|
||||
|
||||
> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).**
|
||||
> Decision 3 no longer holds: a newer merge does not take over an older plan. Merges are collected in a
|
||||
> rolling merge window and walked one batch at a time; a walk that started is never superseded, and only a
|
||||
> walk still waiting for its word is folded into the next batch's walk (issue 362). Grants before code and
|
||||
> one machine first stand as decided here.
|
||||
|
||||
## Context
|
||||
|
||||
On 2026-10-05 the delivery path was watched through a day of merges, by several sessions at once. Three
|
||||
|
||||
+5
@@ -11,6 +11,11 @@ extends: 02-DECISIONS/0237-a-change-is-judged-against-the-mesh-that-runs-before-
|
||||
|
||||
# 238. A commit is the build at hand: one commit, one change plan, checked off the trunk and published only on it
|
||||
|
||||
> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).**
|
||||
> Decision 2's "one commit, one change plan" no longer holds for the release after the merge: a batch of
|
||||
> merges collected in a rolling merge window is planned once, as one walk with at most one commit per
|
||||
> repository (issue 362). A pull request's head is still checked on its own, with its own delivery plan.
|
||||
|
||||
> **Superseded in part — 2026-10-06, by [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md).** Decision 6's state machine is
|
||||
> built as a **delivery**'s, owned by the module `mesh-delivery` rather than by the controller. Its states are
|
||||
> renamed (`releasing` → `delivering`, `done` → `delivered`) and a delivery group sits above it. Decision 7's
|
||||
|
||||
+8
@@ -11,6 +11,14 @@ extends: 02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-p
|
||||
|
||||
# 239. A delivery is owned by the mesh-delivery module and runs from commit to delivered
|
||||
|
||||
> **Superseded in part — 2026-10-10, by [ADR 0276](0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).**
|
||||
> A delivery is no longer walked by a walk of its own trunk commit. From its merge on it is delivered by the
|
||||
> walk of the batch that answers it, one walk per rolling merge window with at most one commit per
|
||||
> repository, and it ends with that walk (`covered` for an earlier commit of a repository the walk carried
|
||||
> later). The superseded row (published, delivering → superseded) is taken only for a walk folded before it
|
||||
> started whose successor does not answer the delivery's merge; a delivery follows the walk that took its
|
||||
> own over. The owner, the table, the group and its order stand as decided here (issue 362).
|
||||
|
||||
> **Superseded in part — 2026-10-08, by [ADR 0249](0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md).**
|
||||
> Decision 4's sentence "a declared order that contradicts an inferred one is a cycle" no longer holds.
|
||||
> Two rules ordering one pair both ways were always a cycle, and a group of the controller, the node-engine
|
||||
|
||||
+218
@@ -0,0 +1,218 @@
|
||||
---
|
||||
topic: the mesh
|
||||
status: accepted
|
||||
date: 2026-10-10
|
||||
deciders: jochen
|
||||
reconstructed: false
|
||||
supersedes-in-part:
|
||||
- 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md
|
||||
- 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md
|
||||
- 02-DECISIONS/0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md
|
||||
- 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md
|
||||
extends: 02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md
|
||||
---
|
||||
|
||||
# 276. Merges are assembled in a rolling window, and each batch is delivered by one walk with one commit per repository
|
||||
|
||||
## Context
|
||||
|
||||
Shortly after midnight on 2026-10-10 two pull requests of the catalogue were merged 18 seconds apart, the
|
||||
claude-code change at 00:20:52 and the dunst change at 00:21:09
|
||||
([issue 362](../04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md)).
|
||||
The controller opened one walk for each merge, as [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
|
||||
decided ("a merge produces a tiered plan"). The second walk superseded the first, as
|
||||
[ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)
|
||||
decision 3 decided, and both were planned at the dunst commit, as the fix for
|
||||
[issue 349](../04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md)
|
||||
decided. mesh-delivery then tied the surviving walk to the dunst delivery, which the supersession had already
|
||||
ended, and tied no walk to the claude-code delivery. Nothing gave the walk its word. The change waited 58
|
||||
minutes, until the operator started the walk by hand with `plans go`.
|
||||
|
||||
The diagnosis found two parts that each behave as written and disagree on what a walk's commit means. That
|
||||
is the narrow cause. The wide one is that the mesh plans one walk per merge, while merges arrive in bursts:
|
||||
|
||||
- **A burst is the normal case.** The operator merges a batch of reviewed pull requests at once, often
|
||||
across repositories (a catalogue change with the controller change it needs). Every merge opened a walk,
|
||||
and every walk after the first superseded the one before. Three supersession paths now exist (ADR 0218,
|
||||
issue 254, issue 349), each with its own bookkeeping, and issue 362 is a fourth way for them to disagree.
|
||||
- **A walk superseded mid-gate wastes the gate and confuses its record.** A first machine judged healthy
|
||||
for a build that the next walk replaces a minute later proves nothing about the build that stays. On
|
||||
2026-10-09 a controller merge landing while another walk was at its gate rolled everything back; the rule
|
||||
"one controller rollout at a time" was kept since then by the person merging, not by the mesh.
|
||||
- **ADR 0239 made a delivery "one commit … from commit to delivered" and walked it alone**, while its own
|
||||
decision 4 delivers a delivery group across repositories as one unit. The walk the commit gets is
|
||||
decided by when the controller heard its merge, which is not something the person merging controls.
|
||||
|
||||
On 2026-10-10 the operator decided the model, in their words:
|
||||
|
||||
> "a delivery doesn't really 'run from commit to delivered', we've already discussed that a delivery plan
|
||||
> might start with a set of commits spanning multiple repo's, however, only 1 commit per repo is allowed
|
||||
> … we assemble all commits that are coming in with a 1 minute rolling window, so if a new commit comes
|
||||
> in, less then 1 or 2 minutes after the last one, the builder again waits 1 or 2 minutes, until no new
|
||||
> commit came in, then it looks at the 'group of commits' and decides if they all fit into 1 delivery
|
||||
> plan … This is a hard problem to tackle, but it will improve our build setup I believe."
|
||||
|
||||
> "the 'actively assembling plan' is very interesting, the controller can say 'Still waiting for 13
|
||||
> seconds for new commits, these are the currently grouped commits, plan is yet to be calculated'."
|
||||
|
||||
On two commits of one repository in one window, the operator chose the later commit over splitting:
|
||||
"i like your idea better, I just thought it would be more complex, but if it's reasonably doable, go for
|
||||
it".
|
||||
|
||||
**Checked against GENESIS.** *Failure must be loud*: an assembling batch and a waiting walk each have a
|
||||
bound and a condition. *The mesh owns the truth*: the window, the batch and the walk are kept by the
|
||||
controller and shown by its `plans`, not inferred from timing by a person. Nothing here conflicts with
|
||||
GENESIS.
|
||||
|
||||
**Words.** The glossary had no word for the merges one window collects, nor for the window. This record
|
||||
adds **merge window** and **batch** to it (Change and delivery), with this record. *Group* is not used for
|
||||
either, because **delivery group** already names deliveries that share a head branch name.
|
||||
|
||||
## Considered Options
|
||||
|
||||
**What a walk answers.**
|
||||
|
||||
1. **One walk per merge, as today, with the bookkeeping repaired.** The walk names the merges it answers,
|
||||
and mesh-delivery ties a superseding walk to every live delivery it contains. Rejected as the model,
|
||||
kept as a repair (decision 8): it makes the 362 shape deliver, but every burst still opens and
|
||||
supersedes walks, a walk can still be superseded at its gate, and a cross-repository change is still
|
||||
walked as separate walks in whatever order the merges were heard.
|
||||
2. **A rolling merge window, and one walk per batch** (chosen). Merges are collected while they keep
|
||||
arriving; the batch they form is planned once, as one walk with at most one commit per repository.
|
||||
3. **Every walk started by hand.** The operator named it as the alternative ("this does mean we need to
|
||||
manually start all delivery plans, or, even better, …"). Rejected: a person in every delivery is the
|
||||
wait ADR 0239 exists to remove, and a merge that passed its checks must reach its machines without a
|
||||
person.
|
||||
|
||||
**Two commits of one repository in one batch.**
|
||||
|
||||
1. **Split into two walks in merge order** (the operator's first proposal). The second walk waits for the
|
||||
first. Rejected: every machine running the module builds, is sent and is judged on the earlier commit,
|
||||
and then again on the later one minutes after; the gate judges a build that is about to be replaced;
|
||||
and the batch's walk stops being one unit.
|
||||
2. **One walk with the later commit** (chosen). On a linear trunk the later merge commit contains the
|
||||
earlier one, so building it delivers both. One build, one gate, one rollout. The earlier merge is
|
||||
answered by its successor and ends `covered` when the walk is delivered. Only when that walk fails are
|
||||
the earlier merges walked on their own, to find which one brought the failure.
|
||||
|
||||
**Who keeps the window.**
|
||||
|
||||
1. **mesh-delivery**, which owns deliveries (ADR 0239). Rejected: a walk on the controller's own path (the
|
||||
controller, the node-engine, the node tools, the bus, mesh-delivery itself) must not wait for
|
||||
mesh-delivery, and with no holder of the seat on record the controller plans alone. The window would
|
||||
then exist twice, once in each.
|
||||
2. **The controller** (chosen). It hears every merge (the bus, and the catch-up that reads back what the
|
||||
bus missed), it owns the planner and the walk, and it already plans alone for its own path. mesh-delivery
|
||||
keeps what it owns: each commit's delivery record, the order of a delivery group, and the word that lets
|
||||
a waiting walk start.
|
||||
|
||||
## Decision
|
||||
|
||||
1. **A rolling merge window.** The controller collects every merge into a branch a module follows into the
|
||||
open **batch**. The window is rolling: each merge restarts it. It closes when no merge arrived for the
|
||||
window's length, 90 seconds by default (the controller's setting `merge-window`), and at the latest 10
|
||||
minutes after the batch's first merge (`merge-window-at-most`), so a busy period never starves delivery.
|
||||
The batch, its merges and its times are kept in the controller's store, so a restarted controller
|
||||
resumes the window where it stood rather than losing or doubling a merge.
|
||||
2. **A batch is planned once, as one walk with at most one commit per repository.** When the window closes,
|
||||
the planner computes one delivery plan for the batch: every module its commits move, their dependents,
|
||||
in tiers along the module graph, across repositories. For a repository with two or more merges in the
|
||||
batch the walk takes the latest commit of its branch, which contains the earlier ones. **Every merge
|
||||
belongs to exactly one walk**: the walk names every merge it answers, its own commits and the earlier
|
||||
ones they contain. A merge is never dropped and never walked twice.
|
||||
3. **The earlier merge of a repository is answered by its successor.** Its delivery follows the walk and
|
||||
ends `covered` when the walk is delivered, naming the merge that carried it (the glossary's word,
|
||||
[ADR 0270](0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md);
|
||||
here the containment is the trunk's order inside one batch, so the forge is not asked). **When that walk
|
||||
fails**, the walk's own commits fail, and each earlier merge it answered is walked on its own commit, the
|
||||
newest first, one walk each, in the next batches: the first that is delivered ends `delivered`, and the
|
||||
search stops there. A walk that failed never marks a merge delivered.
|
||||
4. **One walk at a time, and nothing supersedes a walk that started.** A batch becomes a walk only when no
|
||||
walk is open. Merges that arrive while a walk runs join the next batch, whatever its window says. A walk
|
||||
that has not started, because it waits for mesh-delivery's word, is folded into the next batch when that
|
||||
batch is planned, and the new walk answers every merge the folded one answered. This is the only
|
||||
supersession left; ADR 0218 decision 3 and the issue 349 rule of planning a late merge at a later commit
|
||||
are replaced by it. A merge heard after a later merge of its repository and branch was put in a walk is
|
||||
answered by that walk: by the batch still assembling, by the open walk, or, when that walk was delivered,
|
||||
it ends `covered` at once.
|
||||
5. **The order of a delivery group holds inside a walk** (ADR 0249 unchanged). The pairs its order rules
|
||||
give (declared, built by, version skew, engine before controller, each with the rule that won) become
|
||||
edges of the walk's tiers beside the module graph: the modules of a member ordered after another are in
|
||||
a later tier than that member's. A contradiction still rejects the group before anything merges.
|
||||
mesh-delivery gives a walk its word only when every delivery it carries is at its turn.
|
||||
6. **A delivery is one commit's record, and its walk is the batch's.** A delivery keeps its pull request,
|
||||
its status, its note and its check, as ADR 0239 decided. From its merge on it is delivered by the walk
|
||||
that answers it, and every delivery a walk carries ends with that walk's result: `delivered` (its own
|
||||
commit) or `covered` (an earlier commit of the same repository) when the walk is done, `failed` when it
|
||||
fails, `stopped` when it is stopped. The walk's machine steps are recorded on each delivery it carries.
|
||||
7. **The assembling state is shown.** The controller's `plans` lists the open batch first, as the operator
|
||||
asked: "assembling: 13 s left (at the latest 00:31:09); grouped: novox/mesh-catalog@48bda475 (answers
|
||||
553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated". It says the batch on the bus as
|
||||
`plan-moved` with the state `assembling`, so mesh-delivery's `deliveries` and `show` say of a merged
|
||||
delivery that its batch is assembling and how long is left, and of one whose batch waits that it waits
|
||||
behind which walk.
|
||||
8. **The bookkeeping is repaired as well** (the narrow cause of issue 362), because decision 4 keeps one
|
||||
supersession path. A walk names every merge it answers, and a superseded walk names the walk that took it
|
||||
over. mesh-delivery ties a walk to every live delivery whose merge it answers, follows a walk to the one
|
||||
that took it over, and **never ties a walk to a delivery that has ended**. A walk waiting for the word
|
||||
that no live delivery holds is either adopted as a delivery of its own or listed at once by `stalled`.
|
||||
9. **Every wait has a bound and a condition.** A batch still assembling a minute past
|
||||
`merge-window-at-most` while no walk is open raises a condition: the controller failed to cut it. A
|
||||
batch waiting behind an open walk longer than that walk's bound raises one naming the walk. A walk
|
||||
waiting for its word past 30 minutes is S16's, as before, and its words name the merges it answers. A
|
||||
merged delivery no walk answers within ten minutes is held with the true reason (which open walks of its
|
||||
repository exist and that none names its merge, or that the controller holds none) and listed by
|
||||
`stalled` at once, so probe D14 raises it for the operator.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **A merge waits up to 90 seconds before it is planned**, and up to 10 minutes in a busy period. A walk
|
||||
takes minutes to build and longer to judge, so the wait is small beside it, and it removes the builds,
|
||||
sends and gates that a superseded walk wasted.
|
||||
- **A cross-repository change merged within a window is walked as one unit**, tiered by the graph and its
|
||||
group's order. A person no longer has to merge in order and wait in between.
|
||||
- **One walk at a time is mechanical.** A controller merge can no longer land in the middle of another
|
||||
walk's gate; it waits for the next batch.
|
||||
- **A failing walk fails every delivery it carried**, including one whose own modules passed before the
|
||||
failure. Each delivery's machine steps say how far it got, and decision 3's search finds the culprit only
|
||||
among the merges of one repository. A failure in a walk that carried several repositories is the
|
||||
operator's to read. This is the price of one unit, and the operator chose it.
|
||||
- **What got harder**: the controller's merge handling holds state across merges (the open batch) and is
|
||||
restart-safe; the walk's record names several repositories and commits instead of one, and every reader of
|
||||
it (`plans`, S3, S16, `plan-moved`, mesh-delivery) changes with it. The store gains the batch and the
|
||||
merges a walk answers, as a numbered migration and in the consolidated schema.
|
||||
- **The replay of issue 362** in the lab raises a mesh, merges two commits of one repository 18 seconds
|
||||
apart, and passes when one walk delivers both without a person; it fails on the commit before the change.
|
||||
|
||||
## How it is checked
|
||||
|
||||
- The controller's tests: two merges of one repository 18 seconds apart make one batch and one walk at the
|
||||
later commit, answering both; merges in three repositories within the window make one walk; a merge at
|
||||
second 80 restarts the window; the window closes at `merge-window-at-most` however busy; a merge arriving
|
||||
while a walk runs joins the next batch; a walk that has not started is folded into the next one; `plans`
|
||||
shows the assembling state; the batch survives a restart; the two conditions of decision 9 are raised past
|
||||
their bounds and not before.
|
||||
- mesh-delivery's tests: both deliveries of one walk end `delivered` and `covered` with its done record; a
|
||||
failed walk marks none delivered and walks the earlier merge alone; a walk is never tied to an ended
|
||||
delivery; a walk is given its word only when every delivery it carries is at its turn; a group's order is
|
||||
unchanged; a held delivery with no walk names the true reason and is listed by `stalled`.
|
||||
- Each of those tests fails with its part of the change removed.
|
||||
- mesh-lab's replay R362 (decision's consequence above), proved to fail before the change and pass after it.
|
||||
|
||||
## References
|
||||
|
||||
- [Issue 362](../04-ISSUES/362-two-merges-seconds-apart-left-their-delivery-waiting-for-a-person/00-report.md),
|
||||
its diagnosis, and the replay R362 in mesh-lab.
|
||||
- [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md) (a merge produces a plan),
|
||||
[ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md)
|
||||
decision 3 (a newer merge takes over an older plan), [ADR 0238](0238-a-commit-is-the-build-at-hand-one-commit-one-change-plan-checked-off-the-trunk-and-published-only-on-it.md)
|
||||
decision 2 (one commit, one change plan), [ADR 0239](0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md)
|
||||
(a delivery walked by its own commit's walk), each superseded in the part this record names.
|
||||
- [ADR 0249](0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md),
|
||||
whose order rules this record keeps and uses as tier edges.
|
||||
- [ADR 0270](0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md),
|
||||
whose `covered` ends an earlier merge of a repository.
|
||||
- [Issue 349](../04-ISSUES/349-a-merge-acted-on-late-superseded-the-merge-after-it-and-built-the-older-commit/00-report.md),
|
||||
whose rule decision 4 replaces.
|
||||
- To-be [47](../03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md) and
|
||||
[10](../03-DESIGN/01-to-be/10-delivery.md), amended with this record.
|
||||
@@ -5,8 +5,9 @@ code:
|
||||
- mesh-controller internal/builder
|
||||
- mesh-controller cmd/mesh-controller (build, build --behind, push, status)
|
||||
- mesh-controller internal/inventory/builds.go
|
||||
updated: 2026-10-06
|
||||
updated: 2026-10-10
|
||||
decisions:
|
||||
- 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md
|
||||
- 02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md
|
||||
- 02-DECISIONS/0090-a-failure-that-repeats-is-said-to-be-stuck.md
|
||||
- 02-DECISIONS/0082-the-registry-is-reached-by-name-and-trusted-by-the-overlay.md
|
||||
@@ -282,3 +283,9 @@ everything else while that module is held. A person can always deliver by hand.
|
||||
|
||||
The controller keeps the comparison, the planner, the gate, sending and rollback. `mesh-delivery` keeps the
|
||||
record and the order, and asks.
|
||||
|
||||
**Merges are walked in batches** (2026-10-10, by [ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)). The controller collects
|
||||
merges in a rolling merge window, 90 seconds by default and at most 10 minutes, and plans each batch once, as
|
||||
one walk with at most one commit per repository: the latest of its branch, which contains the earlier ones.
|
||||
One walk is open at a time, and a walk that started is never superseded. `plans` shows a batch while it
|
||||
assembles: what it holds and how long is left. To-be 47 says how a delivery follows its batch's walk.
|
||||
|
||||
@@ -2,8 +2,9 @@
|
||||
layer: to-be
|
||||
status: designed
|
||||
code: []
|
||||
updated: 2026-10-09
|
||||
updated: 2026-10-10
|
||||
decisions:
|
||||
- 02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md
|
||||
- 02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md
|
||||
- 02-DECISIONS/0250-a-merge-heard-only-from-the-buss-history-is-retired-and-owed-work-the-forge-cannot-do-ends.md
|
||||
- 02-DECISIONS/0249-a-delivery-groups-order-rules-have-a-precedence-and-a-declared-order-outranks-them-all.md
|
||||
@@ -21,6 +22,11 @@ delivery group is two or more deliveries that share a head branch name, delivere
|
||||
are owned by one module, `mesh-delivery`, which records every step and asks the controller for each act.**
|
||||
([ADR 0239](../../02-DECISIONS/0239-a-delivery-is-owned-by-the-mesh-delivery-module-and-runs-from-commit-to-delivered.md).)
|
||||
|
||||
**From its merge on, a delivery is delivered by the walk of its batch.** The controller collects merges in a
|
||||
rolling merge window and plans each batch once, as one walk with at most one commit per repository, one walk
|
||||
at a time. Every delivery the walk answers ends with it
|
||||
([ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md)).
|
||||
|
||||
This is the delivery of [to-be 10](10-delivery.md) carried out. To-be 10 says what is built: the comparison
|
||||
of source and artifacts decides, and nothing here changes that. This design says who answers for one
|
||||
commit's journey and in what order the steps are allowed.
|
||||
@@ -57,7 +63,9 @@ Its id is `<owner>/<repository>@<twelve characters of its commit>`. It holds:
|
||||
its diffset (what ADR 0238 called the "change plan", under the glossary's name);
|
||||
- its **state**, its **transitions** (the last hundred, each with when, the event, from, to and why), and its
|
||||
**machine steps** while delivering;
|
||||
- once merged, **the commit it landed on the trunk as**, and the walk that delivers it.
|
||||
- once merged, **the commit it landed on the trunk as**, and the walk that delivers it: the walk of the batch
|
||||
that answers its merge, which may carry a later commit of its repository that contains it (see *The merge
|
||||
window and the batch*).
|
||||
|
||||
## The state table
|
||||
|
||||
@@ -72,15 +80,15 @@ allows.
|
||||
| checked | a verdict arrived | 1 minute | nothing: it is decided at once |
|
||||
| ready | it passed; it waits to be merged | none | — |
|
||||
| rejected | it failed its check | none | — |
|
||||
| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a newer delivery took over its walk; `go` when its walk started |
|
||||
| published | merged on the trunk its modules follow, its walk opened and asking nothing; it waits for its turn or its word | 30 minutes | `superseded` when a walk that does not answer its merge took over its walk; `go` when its walk started |
|
||||
| held | it waits for a person; read again each tick, so it ends by itself when it moves nothing or a later merge covers it (ADR 0270) | 1 day | nothing: it is the operator's |
|
||||
| delivering | its walk runs; its builds are asked and registered tier by tier | 2 hours | `delivered`, `failed` or `superseded`, as the walk's record says |
|
||||
| delivered, failed, superseded, stopped, retired, covered | final | — | — |
|
||||
|
||||
The transitions are those of ADR 0239 decision 2 and [ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md). A delivery that is
|
||||
merged while not `ready` goes to `held`. Its check did not pass, and only a person decides that it goes on. A
|
||||
delivery merged with no walk opened for it within ten minutes is held too, saying so: nothing it moves follows
|
||||
that branch, or the merge was not heard. Nothing waits silently.
|
||||
delivery merged with no walk answering it within ten minutes is held too, with the true reason, and listed by
|
||||
`stalled` at once. Nothing waits silently.
|
||||
|
||||
**What needs no person ends by itself** (ADR 0270):
|
||||
|
||||
@@ -107,6 +115,44 @@ module and the build: `sent` (the first machine's send, or the rest's), `judging
|
||||
readings so far), `passed`, `failed`, `rolled-back`. They are read from the walk's own record in the
|
||||
controller's `plan-moved` event, never inferred from time.
|
||||
|
||||
## The merge window and the batch
|
||||
|
||||
*2026-10-10, by [ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md).*
|
||||
|
||||
- **The window.** The controller puts every merge into a branch a module follows into the open **batch**.
|
||||
Each merge restarts the **merge window**; it closes when none arrived for `merge-window` (90 seconds by
|
||||
default) or at the latest `merge-window-at-most` (10 minutes) after the batch's first merge. The batch, its
|
||||
merges and its times are kept in the controller's store and survive a restart.
|
||||
- **The cut.** When the window closes and no walk is open, the batch becomes one walk: one delivery plan for
|
||||
every commit in it, tiered along the module graph across repositories, with at most one commit per
|
||||
repository, the latest of its branch. The walk names every merge it answers: its own commits, and the
|
||||
earlier merges of a repository that its later commit contains.
|
||||
- **One walk at a time.** Merges that arrive while a walk is open join the next batch. A walk that started
|
||||
is never superseded. A walk still waiting for its word when the next batch is cut is folded into the new
|
||||
walk, which answers every merge the folded one answered.
|
||||
- **A late merge.** A merge heard after a later merge of its repository and branch was put in a walk is
|
||||
answered by that walk: the assembling batch, the open walk, or, when that walk was delivered, it ends
|
||||
`covered` at once.
|
||||
- **The ends.** A walk done ends every delivery it answers: `delivered` for its own commits, `covered` for
|
||||
an earlier merge of a repository it carried later, naming that merge. A walk that failed fails its own
|
||||
commits' deliveries and marks none delivered; each earlier merge it answered is then walked on its own
|
||||
commit, the newest first, one walk each, until one is delivered. A walk stopped stops them all.
|
||||
- **A group inside a walk.** The pairs of a delivery group's order (ADR 0249) are edges of the walk's
|
||||
tiers beside the module graph. mesh-delivery gives a walk its word only when every delivery it carries is
|
||||
at its turn.
|
||||
- **The bookkeeping.** mesh-delivery ties a walk to every live delivery whose merge it answers, follows a
|
||||
folded walk to the walk that took it over, and never ties a walk to a delivery that has ended. A waiting
|
||||
walk no live delivery holds is adopted as a delivery of its own, or listed by `stalled` at once.
|
||||
- **Shown.** `plans` lists the open batch first: "assembling: 13 s left (at the latest 00:31:09); grouped:
|
||||
novox/mesh-catalog@48bda475 (answers 553b7191), novox/mesh-controller@a6bc0931; plan not yet calculated",
|
||||
and a batch waiting behind an open walk names that walk. The controller says the batch on the bus as
|
||||
`plan-moved` in the state `assembling`, and mesh-delivery's `deliveries` and `show` say it for each merged
|
||||
delivery it holds.
|
||||
- **Bounds.** A batch still assembling a minute past `merge-window-at-most` with no walk open, and a batch
|
||||
waiting behind an open walk past that walk's bound, each raise a condition. A merged delivery that no walk
|
||||
answers within ten minutes is held with the true reason (the open walks of its repository and that none
|
||||
names its merge, or that the controller holds none) and listed by `stalled` at once.
|
||||
|
||||
## A delivery group
|
||||
|
||||
- **Membership**: the pull request's head branch name, across the mesh's repositories, among open pull
|
||||
@@ -161,7 +207,7 @@ In this order, each by its owner:
|
||||
| `groups` | every group with its members in order and its derived state |
|
||||
| `what-if` | the delivery plan a diffset would have, asked of the controller's planner, kept nowhere |
|
||||
| `table` | the state table and the machine-step table, with bounds and H2's repairs |
|
||||
| `stalled` | every delivery held past its bound, with the transition H2 may take |
|
||||
| `stalled` | every delivery held past its bound, with the transition H2 may take, and every waiting walk no live delivery holds |
|
||||
| `recheck` | a rejected or ready delivery back to proposed, with why |
|
||||
| `release` | a held delivery to delivering, with why: a person's word |
|
||||
| `stop` | any delivery not final to stopped, with why; its walk is ended through the controller |
|
||||
@@ -191,7 +237,8 @@ up.
|
||||
- Any other walk waits for `deliver` only while the seat has a holder on record. With none on record, the
|
||||
controller starts it as it always did.
|
||||
- A waiting walk is said by the controller whatever mesh-delivery says of itself: S16 raises
|
||||
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming `plans go <walk> --why` as the way on.
|
||||
`plan.<walk>.waiting` after 30 minutes, urgent after 4 hours, naming the merges the walk answers and
|
||||
`plans go <walk> --why` as the way on.
|
||||
A holder that is down is also D3's `holder-silent`; one that is up and never says go is said by S16 alone.
|
||||
|
||||
## The switch
|
||||
|
||||
+13
-3
@@ -3,7 +3,7 @@ status: located
|
||||
opened: 2026-10-10
|
||||
located-in: [mesh-catalog modules/mesh-delivery (holder.go deliveryWalking), mesh-controller cmd/mesh-controller (upgrades.go SourceMoved, release_plan.go laterOnTheBranch)]
|
||||
fixed-by:
|
||||
amended-design:
|
||||
amended-design: 03-DESIGN/01-to-be/47-delivery-from-commit-to-delivered.md
|
||||
---
|
||||
|
||||
# 362. Two catalogue merges seconds apart left their delivery waiting for a person
|
||||
@@ -22,8 +22,10 @@ walk waits for mesh-delivery's word — `plans go … --why …` starts it by ha
|
||||
|
||||
Both reasons were wrong. A walk had been opened, and it contained the held merge. Nothing gave it the word.
|
||||
The operator's change sat undelivered for 58 minutes, until the operator's hand-act `mesh-controller.plans
|
||||
go` started the walk. **No condition was raised to anyone in that time**: the hold is bounded at 24 hours
|
||||
before `mesh-delivery.stalled` lists it, and the walk's own wait raised nothing either.
|
||||
go` started the walk. **No condition asked anyone to act in that time**: the hold is bounded at 24 hours
|
||||
before `mesh-delivery.stalled` lists it, and the walk's own wait raised only a warning, at 00:51, which said
|
||||
there was nothing to do and pointed at the ended delivery (the diagnosis's second entry; this sentence said
|
||||
before that the walk's wait raised nothing).
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
@@ -55,3 +57,11 @@ disagree on what a walk's commit means:
|
||||
as a delivery of its own, so that something always holds a waiting walk.
|
||||
- A walk waiting for mesh-delivery's word past a bound, and a delivery held because no walk was opened,
|
||||
raise a condition to the operator instead of waiting silently (failure must be loud).
|
||||
|
||||
## Decided
|
||||
|
||||
The operator decided the model that removes this shape, not only its bookkeeping:
|
||||
[ADR 0276](../../02-DECISIONS/0276-merges-are-assembled-in-a-rolling-window-and-each-batch-is-delivered-by-one-walk-with-one-commit-per-repository.md). Merges are collected in a rolling merge window and each batch is walked once, with at most one
|
||||
commit per repository, one walk at a time. The bookkeeping that tied a walk to an ended delivery is repaired as
|
||||
well, and every wait of this shape has a bound and a condition. It is designed in to-be 47, *The merge window
|
||||
and the batch*.
|
||||
|
||||
+16
@@ -57,3 +57,19 @@ Two parts, each correct by its own rule, disagree on what a walk's commit means.
|
||||
349) plans an earlier merge at a later commit that contains it; mesh-delivery assumes a walk's commit is the
|
||||
merge commit of exactly one delivery. `located-in` names both. The fix is to make the walk say which merges
|
||||
it answers, or to make mesh-delivery match by containment, and to let no waiting walk be held by nothing.
|
||||
|
||||
## 2026-10-10 — what the controller's side said
|
||||
|
||||
The first entry says no condition reached the operator in the 58 minutes. The controller's condition history
|
||||
says more precisely what happened. S16 (a walk waiting for its delivery's word) raised
|
||||
`plan.plan-1791584475080493383.waiting` at 00:51:32, thirty minutes after the walk opened, as its bound says.
|
||||
It was a warning, its explanation was "Nothing for you to do … It becomes a question for you if it still waits
|
||||
after 4 hours", and its summary sent the reader to "`mesh-delivery.show` for the delivery that landed as
|
||||
48bda475", which is the dunst delivery the supersession had already ended. It cleared at 01:20:32, when the
|
||||
operator's `plans go` started the walk.
|
||||
|
||||
So the controller's side was not silent; it was mild and pointed at the wrong delivery. S16 cannot tell a
|
||||
group member waiting its turn from a walk nobody holds; only mesh-delivery knows that. That is why ADR 0276
|
||||
decision 9 has mesh-delivery list a waiting walk no live delivery holds through `stalled` at once, and has
|
||||
S16 name the merges a walk answers.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user