From b409537dbc885825074d11eee92d4ba87263657f Mon Sep 17 00:00:00 2001 From: jochen Date: Thu, 8 Oct 2026 20:27:36 +0200 Subject: [PATCH] Issue 335: a build that passed its gate was failed and put back by the first machine's later reports Recorded so the gate's verdict is known to be final, and the replay R335 holds it. --- .../00-report.md | 53 ++++++++++++++++ .../01-diagnosis.md | 60 +++++++++++++++++++ 2 files changed, 113 insertions(+) create mode 100644 04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md create mode 100644 04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md diff --git a/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md b/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md new file mode 100644 index 00000000..97e327ec --- /dev/null +++ b/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/00-report.md @@ -0,0 +1,53 @@ +--- +status: located +opened: 2026-10-08 +located-in: [mesh-controller cmd/mesh-controller (release_plan.go, nextRollout and failFirstSend)] +fixed-by: mesh-controller#165, mesh-lab#72 +replay: R335 +amended-design: +--- + +# 335. A build that passed its gate was failed and put back by the first machine's later reports + +## What was observed + +On 2026-10-08 a person released a delivery of the catalogue that the group check (issue 316) had held. Its +walk had one tier with one module, `dunst`, which runs on the laptop and the workstation. The walk sent the +new build to the laptop first, and the gate passed it there: + +> dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent + +Twenty-eight minutes later the same walk said the opposite, about the same gate: + +> dunst failed its gate on the laptop in tier 0: healthy 3 times over 2m5s; put back dunst to ea462666 on +> the laptop, in one send; the workstation left as it was + +`plans` showed the gate as `passed` and the module as `failed its gate` on the same line. The controller +raised `build.dunst..rolled-back` and kept saying it every five minutes. The build was recorded as +failed at its gate, so nothing sends it again on its own, and the operator's new build (its font size and +width now settings) reached neither machine. + +## Why it is an issue + +A gate's pass is a verdict (ADR 0236): the first machine is judged once, and then the rest are sent. Here a +verdict was overturned by something that happened after it, with the pass's own words given as the +reason for a failure. That is a failure that says nothing true about why it happened. It also put back a +build that worked, recorded it as failed, and so blocked its delivery until a newer merge. + +## Fix + +The controller judged the first machine's report again on every step of the walk, also after the gate had +passed (see `01-diagnosis.md`). mesh-controller#165: + +- once a module's gate has passed, the first machine's later reports no longer decide anything about it: + the walk sends the rest; +- a module carried by another module's send takes over that pass before its step is read; +- a walk stopped after a module's gate passed never puts that module back, and says what stopped it. + +## How it is checked + +`TestReplay335` in the controller replays the live case with only what the controller had before the fix: +a passed gate on the laptop, the laptop's later report not about this build (or a failure of another +send), and the walk 30 minutes and 9 seconds after the first send. On the commit before the fix it fails +with "laptop did not report it applied within 30m0s", and the passed build reaches the put-back. It is R335 +in mesh-lab's replays register (mesh-lab#72), proved to fail before the fix and pass on it. diff --git a/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md b/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md new file mode 100644 index 00000000..860a2457 --- /dev/null +++ b/04-ISSUES/335-a-passed-gate-was-failed-by-the-first-machines-later-reports/01-diagnosis.md @@ -0,0 +1,60 @@ +# 335. Diagnosis + +## 2026-10-08 + +**The record.** A search of the records for "failed its gate", "rolled-back" and "gate passed rolled back" +found ADR 0236 (what a rollback is), issue 318 (a wait for a login failing a gate) and issue 279, none of +them about a gate that passed and was then failed. + +**The walk's own lines.** The controller's journal on the control-node said, for this walk (times UTC): + +| time | what the controller said | +|---|---| +| 16:44:45 | tier 0 built; sent dunst to the laptop first, the rest once its gate passes | +| 16:44:55 | the laptop applied what it was sent | +| 16:46:52 | dunst passed its gate on the laptop (healthy 3 times over 2m5s); the rest are sent | +| 16:46:53 | sending dunst to the workstation refused: a build of `build-agent` waits there for its gate, which another walk sends | +| 16:49:12 | the other walk (the controller's own merge, tier 1) sent `build-agent` to the control-node, the laptop and the workstation | +| 17:14:54 | dunst failed its gate on the laptop: healthy 3 times over 2m5s; put back on the laptop; the workstation left as it was | + +The laptop's node-engine logged `applied` at 16:44:55 and nothing more until 18:01:58. It did not report +on the 16:49:12 send for over an hour. Why it did not is not yet known; the laptop did not sleep until +18:01. That is a separate question, and this issue does not depend on its answer. + +**The arithmetic.** 17:14:54 is 30 minutes and 9 seconds after the first send at 16:44:45. The controller +waits 30 minutes (`planWaitBound`) for a first machine to report what it was sent. The failure came at that +bound and not at the gate's. + +**The code path.** In `advanceOnce` (`release_plan.go`), every step of a walk reads +`nextRollout(state, …)` before anything else, for every module in the tier not yet sent to the rest. For a +module sent first, `nextRollout` looks at the first machine's last report. If that report is not about the +declaration the machine was last sent (`Current` false), it waits; past the bound it answers +`failed: did not report it applied within 30m0s`. Nothing in it looked at the gate's verdict. + +After the pass at 16:46:52, the send to the rest was refused (the error returns, and the walk tries again on +the next step). At 16:49:12 the other walk's send made the laptop's last report no longer current. From +then on `nextRollout` waited for a report it would never get about this build. At the bound it failed, and +`advanceOnce` called `failFirstSend`. + +`failFirstSend` chooses between putting the module back or keeping it by the gate's `Failing` list. A passed +gate has an empty `Failing`, and the condition `len(g.Failing) == 0` sent it to `gateFailed`. `gateFailed` +does not overwrite a verdict already written, so it kept the pass's why ("healthy 3 times over 2m5s") as the +failure's reason. It then recorded the build as failed, restored the previous build and sent it to the +laptop. That is the line seen at 17:14:54. + +**Ruled out.** +- The gate itself (`judgeMoves`) decided `passed` and nothing changed that verdict. +- The person-wait of issue 318 (ADR 0254), `putBackBroken` and the carried-module paths: the plan had one + module, nothing was carried, and nothing was found broken. +- Sleep: the laptop did not sleep until 18:01, and the failure was at 17:14:54. +- The release itself (`mesh-delivery.release`): it let the walk go at 16:44, and the walk ran as any other. + +**Located.** mesh-controller, `cmd/mesh-controller/release_plan.go`: `nextRollout` judged a passed gate's +first machine again, and `failFirstSend` put back a module whose gate had passed. The path has been there +since the gate was added (ADR 0236). It shows only when the send to the rest does not happen on the same +step as the pass and another send reaches the first machine before the 30-minute bound. + +**What it needs to deliver dunst.** The build is recorded as failed at its gate, and the controller does +not send a build marked that way again on its own. The registered build is back at the one before. Once the +fix is delivered, a new build of dunst (`rebuild dunst`, or a newer merge of it) is judged at the gate +again and walked to both machines.