2026-10-09 15:01:44 +00:00
3 changed files with 148 additions and 0 deletions
@@ -153,6 +153,15 @@ The table of ADR 0239 decision 2 gains these rows:
  never say it reached every machine. mesh-delivery itself, on the control node alone, is such a module: a
  held merge of it stays held for a person, as before this record. Reading "every machine running it" from
  the first machines when they are all the machines would close it, and is left for a later change.
+
+  > **Progressive insight — 2026-10-09.** *The gap was wider than one machine.* A build the mesh held back for
+  > a person and sent on the operator's word travels by a backlog walk, whose record gives one step per
+  > machine and no "every machine running it" or "the rest running it"; and that walk's last record keeps only
+  > the machines it is done on, which mesh-delivery read for nothing. So a held delivery whose module reached
+  > every machine that way was never covered either
+  > ([issue 351](../04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md)).
+  > Since its fix, a finished backlog walk passes the steps it carried, and a passed step on every machine the
+  > held delivery's plan sends the module to counts as sent everywhere. The one-machine case stays as stated.
 - **A group member waits longer, rather than being held too soon.** A composed check that never answers leaves
  the member in `published`. Its bound of 30 minutes lists it as stalled. That is the right owner for a check
  that does not answer.
@@ -0,0 +1,66 @@
+---
+status: located
+opened: 2026-10-09
+located-in: [mesh-catalog modules/mesh-delivery (holder.go, creditBacklog and carries)]
+fixed-by: mesh-catalog #163
+amended-design:
+---
+
+# 351. A held delivery whose module a backlog walk carried to every machine is never covered
+
+## Symptom
+
+On 2026-10-09 the catalogue's delivery of mesh-catalog #125 (the lighting module's account group,
+[issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md))
+was still `held` 28 hours after its merge, raising `delivery.<id>.stalled` to the operator every hour.
+mesh-delivery had run the build of [ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md)
+(mesh-catalog #150) since 07:34 UTC that day, and its table (`mesh-delivery.table`) carried the `covered`
+row. Seven other held deliveries ended `covered` in its first tick at 07:31 UTC, as the record promised.
+This one did not, and the holder's journal held no line about it: the forge was never asked whether a
+later merge contains it.
+
+What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `hand-acts`, `module list`):
+
+| time | what happened |
+|---|---|
+| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word |
+| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
+| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded |
+| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
+| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
+| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours |
+
+No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever
+move it again ([ADR 0267](../../02-DECISIONS/0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md)):
+left as it is, the delivery stays held for ever.
+
+## Why it is an issue
+
+ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running
+it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge
+commit, which the forge says contains a6385479. The decision's question — does this change go on? — was
+answered by the operator's own release of the build. Yet the record cannot end:
+
+- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what
+  its gate carried, and mesh-delivery credits the delivery that published each carried build one step per
+  machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing
+  from that: the steps stay `judging` for ever.
+- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running
+  it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the
+  module sent everywhere.
+
+ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module
+whose build reached its machines on the operator's word, after the mesh held it back, rather than by the
+walk of the merge that built it. A held delivery in that case has no end but a person's wrong act
+([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and
+the condition it raises every hour is one nobody can clear correctly.
+
+## What was ruled out
+
+- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and
+  `gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128.
+- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps.
+- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
+  rightly no coverer.
+- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module
+  was built from, and is what ADR 0270 ruled out.
@@ -0,0 +1,73 @@
+# 351 — diagnosis
+
+**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`,
+at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127,
+`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`,
+the forge's `gitea_contains`, and the holder's journal on the control node.
+
+**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a
+`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives
+the delivery whose merge commit each carried build was made from one step per machine: `judging` while the
+gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk
+holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`.
+`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43
+and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the
+module was not in that walk's tiers, and the backlog walk is a walk of its own.)
+
+**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a
+later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only
+on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running
+it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is
+found, and the forge is never asked — which is why the journal has no line.
+
+**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a
+backlog walk that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
+(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done,
+with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of
+the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis
+is wrong there.
+
+**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb —
+see the report.
+
+**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for
+nothing, and `carries`, which counts no per-machine step.
+
+## The fix
+
+In mesh-catalog #163, one pull request, in `modules/mesh-delivery`:
+
+- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that
+  walk (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
+  step it failed on, and a step another walk carried, are left as they are.
+- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when
+  the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
+  delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still
+  judging, one failed or one never sent leave the delivery held.
+- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its
+  open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was
+  no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for
+  such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
+  its record as said.
+
+Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own
+walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
+The record's consequence that named the one-machine gap gains a progressive insight naming this one.
+
+**How it is checked**, in `covered_test.go`:
+
+- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered;
+  still judging on one, failed on one, never sent to one, or a plan naming no machines stays held.
+- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one
+  and leaves the other judging; a failed step stays failed; a step another walk carried is not touched.
+- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held
+  while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the
+  module.
+
+All three fail on the catalogue's main before the change and pass after it.
+
+**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are
+asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once
+whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads
+delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the
+acceptance of this issue, read with `mesh-delivery.show` of #125.