From 903340516cb28454f16c361d5ce9e5c0c87902fa Mon Sep 17 00:00:00 2001 From: jochen Date: Fri, 9 Oct 2026 17:15:43 +0200 Subject: [PATCH] Issue 352: a gate read another send's report, and a put-back chose a controller older than its store The 2026-10-09 16:19-16:36 sequence on the control node: a release's gate held the machine's report against a newer send another plan made, failed three healthy builds and put them back to a controller that could not read the store; that controller then passed the newer plan's gate from its own health. Measured and guessed apart; the fix in review. --- .../00-report.md | 46 ++++++++++ .../01-diagnosis.md | 86 +++++++++++++++++++ 2 files changed, 132 insertions(+) create mode 100644 04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md create mode 100644 04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md diff --git a/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md b/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md new file mode 100644 index 00000000..2bcceb5c --- /dev/null +++ b/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/00-report.md @@ -0,0 +1,46 @@ +--- +status: located +opened: 2026-10-09 +located-in: [mesh-controller cmd/mesh-controller (gate.go, release.go, release_plan.go, acting.go, push.go), mesh-controller internal/inventory, mesh-controller internal/catalogue] +fixed-by: +amended-design: +--- + +# 352. A gate read another send's report, and a put-back chose a controller older than its store + +## Symptom + +On 2026-10-09, local times on the control node (its controller's and node-engine's journals, and `plans`): + +- 16:19 a release sent the control node three core builds — the controller (2913c54c), the node-engine + (d8ff1540) and the tool runner (83c19941). 16:21:43 the control node applied them; 16:21:53 its + node-engine said each was healthy. The release's gate did not pass in the nine minutes that followed. +- 16:30:56 a merge of the controller (e6b00e2e) made a plan; 16:31:37 that plan sent the control node the + newer controller — a newer send to the same machine, while the release's gate was judging there; + 16:31:44 the serving controller was replaced by the newer one. +- 16:32:20 the new process judged the release's gate **failed**: "not healthy within 10m0s of its apply: + the control node has not reported on what it was sent" — and put all three builds back, on every + machine, in one send. The control node had reported them, ten minutes earlier. +- 16:32:24 the put-back controller (ef26d4cb) started and said it was behind its own row: it carried fewer + migrations than the store had applied, and served what it could. +- 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3 + times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent, + which the put-back had removed from the machine. The plan walked on, believing the controller delivered. + +## Why it is an issue + +Four rules did not hold. + +- **A gate judges its own send** (ADR 0236). The gate held the machine's report against the declaration + sent *last*, which another plan's send had just replaced; the report on the gate's own send read as + none. The report was never lost; it was compared with the wrong send. +- **A gate judges the build it sent.** The controller's gate read the lease holder's readiness and the + machine's report, and neither said *which* controller build was running. A put-back build passed the + gate of the build it had replaced. +- **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than + itself"). A controller older than the store's schema starts behind its own records. Nothing refused it. +- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to + the merge plan's send: a release keeps no module records, and the check for a walk in progress read + only those. + +The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md). diff --git a/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md b/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md new file mode 100644 index 00000000..89e50d15 --- /dev/null +++ b/04-ISSUES/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store/01-diagnosis.md @@ -0,0 +1,86 @@ +# 352. Diagnosis + +## 2026-10-09 + +**The record.** A search of the records for "has not reported on what it was sent" and "older than the +store" found nothing. Issue 335 (a passed gate failed by later reports) and issue 281 (a machine's condition +read as a module's) are the nearest; neither is this. + +### What is measured + +- **D1, the report compared with the wrong send.** The gate reads each machine's last report beside the + declaration the machine was *last sent* (`LastReports`: `Current` is "the report names the declaration + in the node's `sent` column"). The release's send wrote `sent`; the control node reported on it, applied; + the merge plan's send at 16:31:37 overwrote `sent`; from then the same report read as not current, and + the gate's wait ended at its bound with "has not reported on what it was sent". The controller restart + at 16:31:44 is in the timeline but is not the cause: the plan and the gate are kept in the store, and a + fresh process read the same `sent` and the same report. Nothing was lost across it. +- **D2, the build judged is not the build running.** `judgeHealth` for the controller asks: the lease is + held since after the send, and the holder says ready. The holder's `Build` is the link-time version + stamp, which the mesh's own builds do not set ("development build"), so it names no build. After the + put-back, the registered controller build, the lease holder and the machine's last send all were the + put-back build, and the newer plan's gate — comparing none of them with the build it sent — passed on + the put-back build's health. +- **D3, a put-back to a controller behind the store.** A failed gate restores the module's registered build + to the one the first machine ran before (`PreviousBuild`) and sends it. Nothing asked whether that + build can read the store as the failed build left it. The controller says so itself when it starts + ("this control plane is behind the mesh-controller row"), and serves anyway (ADR 0185). +- **The merge plan's send crossed the release's gate.** `walkedBy` finds a walk in progress by a plan's + module records (`p.Modules[module].First`). A release (the walk of the builds waiting for a gate) keeps its walk in its gate's carried moves, + not in module records, so the release's open gate on the control node held nothing back. And a plan's + own modules were never checked against another walk at all: the check ran over the moves a machine + had waiting, and skipped the plan's own. + +### What is guessed + +- Why the release's gate had not passed in the nine minutes before the merge is not established. The + node-engine said all three healthy at 16:21:53; the gate needs three passes forty seconds apart after a + two-minute settle, and the controller's own readiness (its self-check run once, `status` within ten + seconds). I think the controller's readiness came late — it was the build being judged — but I did not + read its lease's health at the time. + +## The fix + +In mesh-controller, pull request 185. + +- **A gate keeps what it sent, and reads the report against that.** Each gate keeps, per machine, the + declaration its send carried — its digest and its sequence (`PlanGate.Sent`, read from the node's + record right after the send). A report is on that send when it names that declaration, or one the + machine was sequenced after it. The gate's judging and the merge plan's first-machine wait both read + this; a gate kept before sends were kept reads the last send, as before. Kept with the plan in the + store, so a controller restarted mid-judging reads the same. +- **A gate judges only the build the machine was last sent.** Each gate judges what the machine was + last sent of the module (`sent_builds`) against the commit it judges. Another build there means the + judging is **superseded**: a new verdict, decided at once, with no verdict on the build, nothing marked, + nothing put back; a release so superseded ends, and what still waits is released again; a merge plan + so superseded ends, said, and the newer plan carries on. +- **The controller is never put back to a build older than the store's schema.** A controller is told + which build it is by its declaration (`MESH_CONTROLLER_VERSION`, the bundle's version, `${version}` + composed by the catalogue — now also inside a process's env), and records when it serves the highest + migration it carries, by that version (`controller_schema`, migration 0087). A failed controller gate + reads the record of the build it would put back: one that reaches less than the store has applied, or + never recorded what it reaches, is not put back; the current build is kept, and the condition + (`rollback-failed`, urgent) says why and that a person decides. The lease holder's `Build` is that + version too. +- **A walk of a module on a machine holds every other send of another build of it there.** `walkedBy` + reads a release's open gate as its walk, and a plan's own modules are held by it as any other move is: + a merge plan whose first machine is under a release's gate for the same module waits, said as + "waiting for … which release-… is walking". The same build to the same machine is not held: it is + already there. + +Not changed: whether a release's walk and a merge plan's walk of the core should be one walk (the walk +carrying over) is left to a design; serialising them is what this fixes. + +## How it is checked + +| Rule | Check | +|---|---| +| A newer send to the judged machine does not unreport the gate's | `TestReplay352` (the backlog fixture: the release sends, the machine reports, another send lands unreported, the gate still passes) — fails on the commit before the fix with "has not reported on what it was sent" | +| A send that moves the module supersedes the judging, puts nothing back | `TestReplay352`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule` | +| A report is held against the gate's own send, by digest or sequence | `TestAReportIsHeldAgainstTheGatesOwnSend`, inventory `TestASendAndAReportAreKnownByTheirDeclaration` | +| Only the build last sent is judged | `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent` | +| A walk holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` | +| No put-back to a controller behind the store, or one unproved | `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, inventory `TestASchemaReachIsKeptByBuild`, catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`, `TestTheHolderNamesTheBuildTheDeclarationToldIt` | +| Every condition of the change is tested | 31 mutations, each failing a test | + +A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).