From 261bee1cab04a48e53d149a63a6c397ebf1537ad Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 9 Oct 2026 16:53:11 +0200 Subject: [PATCH] Issue 351: a held delivery whose module a backlog walk carried to every machine is never covered The catalogue's #125 stayed held 28 hours after ADR 0270 went live. The record covers a held merge when a later delivered walk sent its module everywhere, as the machine steps say; a build held back for a person and sent on the operator's word travels by a backlog walk, which gives one step per machine and whose end mesh-delivery read for nothing. Report, diagnosis located in mesh-delivery, the fix in mesh-catalog #163, and a progressive insight on ADR 0270's one-machine consequence, which this widens. --- ...f-and-a-hold-whose-reason-is-gone-lifts.md | 9 +++ .../00-report.md | 66 +++++++++++++++++ .../01-diagnosis.md | 73 +++++++++++++++++++ 3 files changed, 148 insertions(+) create mode 100644 04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md create mode 100644 04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md diff --git a/02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md b/02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md index 857992ef..8773bb34 100644 --- a/02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md +++ b/02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md @@ -153,6 +153,15 @@ The table of ADR 0239 decision 2 gains these rows: never say it reached every machine. mesh-delivery itself, on the control node alone, is such a module: a held merge of it stays held for a person, as before this record. Reading "every machine running it" from the first machines when they are all the machines would close it, and is left for a later change. + + > **Progressive insight — 2026-10-09.** *The gap was wider than one machine.* A build the mesh held back for + > a person and sent on the operator's word travels by a backlog walk, whose record gives one step per + > machine and no "every machine running it" or "the rest running it"; and that walk's last record keeps only + > the machines it is done on, which mesh-delivery read for nothing. So a held delivery whose module reached + > every machine that way was never covered either + > ([issue 351](../04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md)). + > Since its fix, a finished backlog walk passes the steps it carried, and a passed step on every machine the + > held delivery's plan sends the module to counts as sent everywhere. The one-machine case stays as stated. - **A group member waits longer, rather than being held too soon.** A composed check that never answers leaves the member in `published`. Its bound of 30 minutes lists it as stalled. That is the right owner for a check that does not answer. diff --git a/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md b/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md new file mode 100644 index 00000000..743eff70 --- /dev/null +++ b/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md @@ -0,0 +1,66 @@ +--- +status: located +opened: 2026-10-09 +located-in: [mesh-catalog modules/mesh-delivery (holder.go, creditBacklog and carries)] +fixed-by: mesh-catalog #163 +amended-design: +--- + +# 351. A held delivery whose module a backlog walk carried to every machine is never covered + +## Symptom + +On 2026-10-09 the catalogue's delivery of mesh-catalog #125 (the lighting module's account group, +[issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md)) +was still `held` 28 hours after its merge, raising `delivery..stalled` to the operator every hour. +mesh-delivery had run the build of [ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md) +(mesh-catalog #150) since 07:34 UTC that day, and its table (`mesh-delivery.table`) carried the `covered` +row. Seven other held deliveries ended `covered` in its first tick at 07:31 UTC, as the record promised. +This one did not, and the holder's journal held no line about it: the forge was never asked whether a +later merge contains it. + +What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `hand-acts`, `module list`): + +| time | what happened | +|---|---| +| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word | +| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop | +| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded | +| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) | +| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 | +| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours | + +No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever +move it again ([ADR 0267](../../02-DECISIONS/0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md)): +left as it is, the delivery stays held for ever. + +## Why it is an issue + +ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running +it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge +commit, which the forge says contains a6385479. The decision's question — does this change go on? — was +answered by the operator's own release of the build. Yet the record cannot end: + +- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what + its gate carried, and mesh-delivery credits the delivery that published each carried build one step per + machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing + from that: the steps stay `judging` for ever. +- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running + it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the + module sent everywhere. + +ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module +whose build reached its machines on the operator's word, after the mesh held it back, rather than by the +walk of the merge that built it. A held delivery in that case has no end but a person's wrong act +([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and +the condition it raises every hour is one nobody can clear correctly. + +## What was ruled out + +- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and + `gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128. +- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps. +- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is + rightly no coverer. +- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module + was built from, and is what ADR 0270 ruled out. diff --git a/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md b/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md new file mode 100644 index 00000000..51623a41 --- /dev/null +++ b/04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/01-diagnosis.md @@ -0,0 +1,73 @@ +# 351 — diagnosis + +**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`, +at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127, +`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`, +the forge's `gitea_contains`, and the holder's journal on the control node. + +**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a +`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives +the delivery whose merge commit each carried build was made from one step per machine: `judging` while the +gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk +holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`. +`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43 +and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the +module was not in that walk's tiers, and the backlog walk is a walk of its own.) + +**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a +later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only +on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running +it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is +found, and the forge is never asked — which is why the journal has no line. + +**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a +backlog walk that failed a gate ends `FAILED at tier … failed its gate on ` and puts the builds back +(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done, +with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of +the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis +is wrong there. + +**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb — +see the report. + +**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for +nothing, and `carries`, which counts no per-machine step. + +## The fix + +In mesh-catalog #163, one pull request, in `modules/mesh-delivery`: + +- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that + walk (`carried by `), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A + step it failed on, and a step another walk carried, are left as they are. +- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when + the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held + delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still + judging, one failed or one never sent leave the delivery held. +- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its + open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was + no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for + such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes + its record as said. + +Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own +walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's. +The record's consequence that named the one-machine gap gains a progressive insight naming this one. + +**How it is checked**, in `covered_test.go`: + +- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered; + still judging on one, failed on one, never sent to one, or a plan naming no machines stays held. +- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one + and leaves the other judging; a failed step stays failed; a step another walk carried is not touched. +- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held + while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the + module. + +All three fail on the catalogue's main before the change and pass after it. + +**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are +asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once +whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads +delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the +acceptance of this issue, read with `mesh-delivery.show` of #125.