Issue 351: a held delivery whose module a backlog walk carried to every machine is never covered
The catalogue's #125 stayed held 28 hours after ADR 0270 went live. The record covers a held merge when a later delivered walk sent its module everywhere, as the machine steps say; a build held back for a person and sent on the operator's word travels by a backlog walk, which gives one step per machine and whose end mesh-delivery read for nothing. Report, diagnosis located in mesh-delivery, the fix in mesh-catalog #163, and a progressive insight on ADR 0270's one-machine consequence, which this widens.
This commit is contained in:
+9
@@ -153,6 +153,15 @@ The table of ADR 0239 decision 2 gains these rows:
|
||||
never say it reached every machine. mesh-delivery itself, on the control node alone, is such a module: a
|
||||
held merge of it stays held for a person, as before this record. Reading "every machine running it" from
|
||||
the first machines when they are all the machines would close it, and is left for a later change.
|
||||
|
||||
> **Progressive insight — 2026-10-09.** *The gap was wider than one machine.* A build the mesh held back for
|
||||
> a person and sent on the operator's word travels by a backlog walk, whose record gives one step per
|
||||
> machine and no "every machine running it" or "the rest running it"; and that walk's last record keeps only
|
||||
> the machines it is done on, which mesh-delivery read for nothing. So a held delivery whose module reached
|
||||
> every machine that way was never covered either
|
||||
> ([issue 351](../04-ISSUES/351-a-held-delivery-whose-module-a-backlog-walk-carried-is-never-covered/00-report.md)).
|
||||
> Since its fix, a finished backlog walk passes the steps it carried, and a passed step on every machine the
|
||||
> held delivery's plan sends the module to counts as sent everywhere. The one-machine case stays as stated.
|
||||
- **A group member waits longer, rather than being held too soon.** A composed check that never answers leaves
|
||||
the member in `published`. Its bound of 30 minutes lists it as stalled. That is the right owner for a check
|
||||
that does not answer.
|
||||
|
||||
+66
@@ -0,0 +1,66 @@
|
||||
---
|
||||
status: located
|
||||
opened: 2026-10-09
|
||||
located-in: [mesh-catalog modules/mesh-delivery (holder.go, creditBacklog and carries)]
|
||||
fixed-by: mesh-catalog #163
|
||||
amended-design:
|
||||
---
|
||||
|
||||
# 351. A held delivery whose module a backlog walk carried to every machine is never covered
|
||||
|
||||
## Symptom
|
||||
|
||||
On 2026-10-09 the catalogue's delivery of mesh-catalog #125 (the lighting module's account group,
|
||||
[issue 316](../316-a-group-member-held-before-its-composed-check-answered-stayed-held-after-it-passed/00-report.md))
|
||||
was still `held` 28 hours after its merge, raising `delivery.<id>.stalled` to the operator every hour.
|
||||
mesh-delivery had run the build of [ADR 0270](../../02-DECISIONS/0270-a-held-delivery-that-moves-nothing-or-that-a-later-delivered-merge-contains-ends-by-itself-and-a-hold-whose-reason-is-gone-lifts.md)
|
||||
(mesh-catalog #150) since 07:34 UTC that day, and its table (`mesh-delivery.table`) carried the `covered`
|
||||
row. Seven other held deliveries ended `covered` in its first tick at 07:31 UTC, as the record promised.
|
||||
This one did not, and the holder's journal held no line about it: the forge was never asked whether a
|
||||
later merge contains it.
|
||||
|
||||
What the records say, in UTC (`mesh-delivery.show`, `mesh-controller.plans`, `hand-acts`, `module list`):
|
||||
|
||||
| time | what happened |
|
||||
|---|---|
|
||||
| 10-08 10:09 | #125 merged as a6385479 and held for its group's composed check (issue 316); its walk waited for its word |
|
||||
| 10-08 11:06 | three later catalogue merges took its walk over in turn ([issue 317](../317-a-held-deliverys-module-was-walked-by-a-later-merges-plan/00-report.md)); the last of them, #123's at 86077780, built the module and sent it to the laptop |
|
||||
| 10-08 11:21 | that gate failed: the module's unit was unhealthy on the laptop; the walk ended `failed`, then superseded |
|
||||
| 10-08 14:27 | #127 merged as bf542128; its walk built and delivered its own three modules, `done` at 14:34; the module was built from bf542128 too, and its moves were held back for a person ([issue 318](../318-one-modules-wait-for-a-new-login-held-every-other-modules-walk/00-report.md)) |
|
||||
| 10-08 14:43 | the operator released the held moves (`upgrade release-backlog`); a backlog walk sent the module to the laptop at 14:43 and the desktop at 14:46, judging each, and was `done` on both at 14:48 |
|
||||
| 10-09 14:40 | `module list`: the module built from bf542128, on the laptop and the desktop; `status` lists it as not behind its source; #127's delivery shows the module on both machines as `judging — carried by` that backlog walk, as it had for 24 hours |
|
||||
|
||||
No merge into the catalogue's trunk since #125 touched the module's source, so no later merge will ever
|
||||
move it again ([ADR 0267](../../02-DECISIONS/0267-an-artifact-is-rebuilt-only-when-its-build-source-changes-and-a-module-moves-only-along-build-dependencies-on-artifacts.md)):
|
||||
left as it is, the delivery stays held for ever.
|
||||
|
||||
## Why it is an issue
|
||||
|
||||
ADR 0270 covers a held merge when a later delivered merge's walk "sent the module to every machine running
|
||||
it, as its machine steps say". The change that runs on both machines is #125's, carried by #127's merge
|
||||
commit, which the forge says contains a6385479. The decision's question — does this change go on? — was
|
||||
answered by the operator's own release of the build. Yet the record cannot end:
|
||||
|
||||
- **A backlog walk's end is read by nothing.** While it judges, the controller's record of it names what
|
||||
its gate carried, and mesh-delivery credits the delivery that published each carried build one step per
|
||||
machine. Its last record keeps no gate, only the machines it is done on, and mesh-delivery reads nothing
|
||||
from that: the steps stay `judging` for ever.
|
||||
- **`covered` reads only a walk's own aggregate steps.** A walk's own record says "every machine running
|
||||
it" or "the rest running it"; a step per machine, which is all a backlog walk gives, never counts as the
|
||||
module sent everywhere.
|
||||
|
||||
ADR 0270 stated one such gap, for a module that runs on one machine only. This is a wider one: any module
|
||||
whose build reached its machines on the operator's word, after the mesh held it back, rather than by the
|
||||
walk of the merge that built it. A held delivery in that case has no end but a person's wrong act
|
||||
([issue 319](../319-a-held-delivery-a-later-commit-superseded-has-no-end-but-release/00-report.md)), and
|
||||
the condition it raises every hour is one nobody can clear correctly.
|
||||
|
||||
## What was ruled out
|
||||
|
||||
- A forge that could not answer: the holder's journal has no "not yet covered" line for it, and
|
||||
`gitea_contains` answers `contains: true` for a6385479 in both 86077780 and bf542128.
|
||||
- A coverer forgotten: a final delivery is kept 30 days, and #127's is read back whole, with its steps.
|
||||
- The walk that failed: #123's delivery is `failed` with the module's step `failed` on the laptop; it is
|
||||
rightly no coverer.
|
||||
- A person's `release` of #125 (the verb): it would walk a6385479, an older commit than the one the module
|
||||
was built from, and is what ADR 0270 ruled out.
|
||||
+73
@@ -0,0 +1,73 @@
|
||||
# 351 — diagnosis
|
||||
|
||||
**2026-10-09.** Read from mesh-delivery's code on the catalogue's main (`modules/mesh-delivery/cmd/mesh-delivery`,
|
||||
at the build of mesh-catalog #150 and #160), `mesh-delivery.show` of #125, #123, #124 and #127,
|
||||
`mesh-controller.delivery-walks` for #127's walk and for the backlog walk, `plans`, `hand-acts`, `module list`,
|
||||
the forge's `gitea_contains`, and the holder's journal on the control node.
|
||||
|
||||
**Where a backlog walk's steps come from.** `walkMoved` (holder.go) hands a walk whose record carries a
|
||||
`release` to `creditBacklog`, which reads the gate of its current record and, through `creditCarried`, gives
|
||||
the delivery whose merge commit each carried build was made from one step per machine: `judging` while the
|
||||
gate judges, `passed` or `failed` on its verdict. The controller's last record of a finished backlog walk
|
||||
holds no gate and no modules: `state: done`, `release.done: [the laptop, the desktop]`, `note: released on …`.
|
||||
`creditBacklog` reads nothing from `done`, so the two `judging` steps it credited to #127's delivery at 14:43
|
||||
and 14:46 UTC were never moved on. (The controller's own record of #127's walk never named the module: the
|
||||
module was not in that walk's tiers, and the backlog walk is a walk of its own.)
|
||||
|
||||
**Why `carries` refuses #127.** `coverOf` asks, for each module the held delivery's plan moves, whether a
|
||||
later delivery `carries` it. `carries` reads the candidate's steps for the module and sets "everywhere" only
|
||||
on a `sent` step whose machine is the walk's own aggregate, "every machine running it" or "the rest running
|
||||
it". #127's steps for the module are two per-machine steps, `judging`, so `carries` is false, no candidate is
|
||||
found, and the forge is never asked — which is why the journal has no line.
|
||||
|
||||
**What "done on a machine" means for a backlog walk.** Read from the controller's records, not its code: a
|
||||
backlog walk that failed a gate ends `FAILED at tier … failed its gate on <machine>` and puts the builds back
|
||||
(seen the same day for the mesh's own parts); one that ends `done` lists every machine in its order as done,
|
||||
with "skipped: none reported". So a machine in `done` is one the gate passed on. That reading is a fact of
|
||||
the records as they are; if the controller ever lists a machine as done without a gate's pass, this diagnosis
|
||||
is wrong there.
|
||||
|
||||
**Ruled out.** A forge that could not answer, a coverer forgotten, the failed walk, and the `release` verb —
|
||||
see the report.
|
||||
|
||||
**Located** in mesh-delivery: `creditBacklog`, which reads a finished backlog walk's `done` machines for
|
||||
nothing, and `carries`, which counts no per-machine step.
|
||||
|
||||
## The fix
|
||||
|
||||
In mesh-catalog #163, one pull request, in `modules/mesh-delivery`:
|
||||
|
||||
- **A finished backlog walk passes the steps it carried.** `creditReleased`: every step credited to that
|
||||
walk (`carried by <its id>`), on a machine it is done on, still `sent` or `judging`, moves to `passed`. A
|
||||
step it failed on, and a step another walk carried, are left as they are.
|
||||
- **`carries` reads per-machine steps against the held delivery's plan.** A module is sent everywhere when
|
||||
the walk's own aggregate step says so, as before, or when a `passed` step stands on every machine the held
|
||||
delivery's plan sends the module to (`machines[].receives`). A plan that names no machines, a machine still
|
||||
judging, one failed or one never sent leave the delivery held.
|
||||
- **A walk that carried a step still judging is asked for by its id.** The controller lists back only its
|
||||
open walks and the last fifty ended at a start; the backlog walk of this case, ended the day before, was
|
||||
no longer among them when read on 2026-10-09, so its end would never have been heard. The holder asks for
|
||||
such a walk by its id (`delivery-walks` with `plan`), at a start at once and then once an hour, and takes
|
||||
its record as said.
|
||||
|
||||
Nothing else in ADR 0270 changes: the later merge must still be delivered on its own passing check, by its own
|
||||
walk of its merge commit on the same trunk, and the forge must still say that commit contains the held one's.
|
||||
The record's consequence that named the one-machine gap gains a progressive insight naming this one.
|
||||
|
||||
**How it is checked**, in `covered_test.go`:
|
||||
|
||||
- `TestAModuleAReleaseCarriedToEveryMachineCovers`: passed on every machine the plan sends it to is covered;
|
||||
still judging on one, failed on one, never sent to one, or a plan naming no machines stays held.
|
||||
- `TestAFinishedReleaseMovesTheStepsItCarriedToPassed`: done on both passes both; done on one passes that one
|
||||
and leaves the other judging; a failed step stays failed; a step another walk carried is not touched.
|
||||
- `TestAHeldDeliveryWhoseModuleAReleaseCarriedIsCoveredOnceTheReleaseIsDone`: the day's case whole — held
|
||||
while the backlog walk judges, covered once it is done and read by its id, naming the later merge and the
|
||||
module.
|
||||
|
||||
All three fail on the catalogue's main before the change and pass after it.
|
||||
|
||||
**After the fix is delivered.** At the holder's first ticks with the new build, #127's two judging steps are
|
||||
asked for by the backlog walk's id and pass; #127 then carries the module for #125; the forge is asked once
|
||||
whether bf542128 contains a6385479, which it does; and #125 ends `covered` by #127. Its group then reads
|
||||
delivered, and the stalled condition clears at the self-check's next reading. Whether that happened is the
|
||||
acceptance of this issue, read with `mesh-delivery.show` of #125.
|
||||
Reference in New Issue
Block a user