Merge pull request 'Issue 352: a gate read another send's report, and a put-back chose a controller older than its store' (#252) from issues/352-a-gate-read-another-sends-report-and-a-put-back-chose-a-controller-older-than-its-store into main

2026-10-09 15:40:16 +00:00
parent 793e98d45a 903340516c
commit c613cf0b82
2 changed files with 132 additions and 0 deletions
@@ -0,0 +1,46 @@
+---
+status: located
+opened: 2026-10-09
+located-in: [mesh-controller cmd/mesh-controller (gate.go, release.go, release_plan.go, acting.go, push.go), mesh-controller internal/inventory, mesh-controller internal/catalogue]
+fixed-by:
+amended-design:
+---
+
+# 352. A gate read another send's report, and a put-back chose a controller older than its store
+
+## Symptom
+
+On 2026-10-09, local times on the control node (its controller's and node-engine's journals, and `plans`):
+
+- 16:19 a release sent the control node three core builds — the controller (2913c54c), the node-engine
+  (d8ff1540) and the tool runner (83c19941). 16:21:43 the control node applied them; 16:21:53 its
+  node-engine said each was healthy. The release's gate did not pass in the nine minutes that followed.
+- 16:30:56 a merge of the controller (e6b00e2e) made a plan; 16:31:37 that plan sent the control node the
+  newer controller — a newer send to the same machine, while the release's gate was judging there;
+  16:31:44 the serving controller was replaced by the newer one.
+- 16:32:20 the new process judged the release's gate **failed**: "not healthy within 10m0s of its apply:
+  the control node has not reported on what it was sent" — and put all three builds back, on every
+  machine, in one send. The control node had reported them, ten minutes earlier.
+- 16:32:24 the put-back controller (ef26d4cb) started and said it was behind its own row: it carried fewer
+  migrations than the store had applied, and served what it could.
+- 16:35:59 that older controller judged the newer plan's gate **passed** on the control node ("healthy 3
+  times over 4m17s") — from the health of the build it was itself, not of the build the plan had sent,
+  which the put-back had removed from the machine. The plan walked on, believing the controller delivered.
+
+## Why it is an issue
+
+Four rules did not hold.
+
+- **A gate judges its own send** (ADR 0236). The gate held the machine's report against the declaration
+  sent *last*, which another plan's send had just replaced; the report on the gate's own send read as
+  none. The report was never lost; it was compared with the wrong send.
+- **A gate judges the build it sent.** The controller's gate read the lease holder's readiness and the
+  machine's report, and neither said *which* controller build was running. A put-back build passed the
+  gate of the build it had replaced.
+- **A put-back puts back something that can run the mesh** (ADR 0236: "put back by something other than
+  itself"). A controller older than the store's schema starts behind its own records. Nothing refused it.
+- **One walk of a module on a machine at a time** (issue 281, ADR 0236). A release's walk was invisible to
+  the merge plan's send: a release keeps no module records, and the check for a walk in progress read
+  only those.
+
+The diagnosis and the fix are in [`01-diagnosis.md`](01-diagnosis.md).
@@ -0,0 +1,86 @@
+# 352. Diagnosis
+
+## 2026-10-09
+
+**The record.** A search of the records for "has not reported on what it was sent" and "older than the
+store" found nothing. Issue 335 (a passed gate failed by later reports) and issue 281 (a machine's condition
+read as a module's) are the nearest; neither is this.
+
+### What is measured
+
+- **D1, the report compared with the wrong send.** The gate reads each machine's last report beside the
+  declaration the machine was *last sent* (`LastReports`: `Current` is "the report names the declaration
+  in the node's `sent` column"). The release's send wrote `sent`; the control node reported on it, applied;
+  the merge plan's send at 16:31:37 overwrote `sent`; from then the same report read as not current, and
+  the gate's wait ended at its bound with "has not reported on what it was sent". The controller restart
+  at 16:31:44 is in the timeline but is not the cause: the plan and the gate are kept in the store, and a
+  fresh process read the same `sent` and the same report. Nothing was lost across it.
+- **D2, the build judged is not the build running.** `judgeHealth` for the controller asks: the lease is
+  held since after the send, and the holder says ready. The holder's `Build` is the link-time version
+  stamp, which the mesh's own builds do not set ("development build"), so it names no build. After the
+  put-back, the registered controller build, the lease holder and the machine's last send all were the
+  put-back build, and the newer plan's gate — comparing none of them with the build it sent — passed on
+  the put-back build's health.
+- **D3, a put-back to a controller behind the store.** A failed gate restores the module's registered build
+  to the one the first machine ran before (`PreviousBuild`) and sends it. Nothing asked whether that
+  build can read the store as the failed build left it. The controller says so itself when it starts
+  ("this control plane is behind the mesh-controller row"), and serves anyway (ADR 0185).
+- **The merge plan's send crossed the release's gate.** `walkedBy` finds a walk in progress by a plan's
+  module records (`p.Modules[module].First`). A release (the walk of the builds waiting for a gate) keeps its walk in its gate's carried moves,
+  not in module records, so the release's open gate on the control node held nothing back. And a plan's
+  own modules were never checked against another walk at all: the check ran over the moves a machine
+  had waiting, and skipped the plan's own.
+
+### What is guessed
+
+- Why the release's gate had not passed in the nine minutes before the merge is not established. The
+  node-engine said all three healthy at 16:21:53; the gate needs three passes forty seconds apart after a
+  two-minute settle, and the controller's own readiness (its self-check run once, `status` within ten
+  seconds). I think the controller's readiness came late — it was the build being judged — but I did not
+  read its lease's health at the time.
+
+## The fix
+
+In mesh-controller, pull request 185.
+
+- **A gate keeps what it sent, and reads the report against that.** Each gate keeps, per machine, the
+  declaration its send carried — its digest and its sequence (`PlanGate.Sent`, read from the node's
+  record right after the send). A report is on that send when it names that declaration, or one the
+  machine was sequenced after it. The gate's judging and the merge plan's first-machine wait both read
+  this; a gate kept before sends were kept reads the last send, as before. Kept with the plan in the
+  store, so a controller restarted mid-judging reads the same.
+- **A gate judges only the build the machine was last sent.** Each gate judges what the machine was
+  last sent of the module (`sent_builds`) against the commit it judges. Another build there means the
+  judging is **superseded**: a new verdict, decided at once, with no verdict on the build, nothing marked,
+  nothing put back; a release so superseded ends, and what still waits is released again; a merge plan
+  so superseded ends, said, and the newer plan carries on.
+- **The controller is never put back to a build older than the store's schema.** A controller is told
+  which build it is by its declaration (`MESH_CONTROLLER_VERSION`, the bundle's version, `${version}`
+  composed by the catalogue — now also inside a process's env), and records when it serves the highest
+  migration it carries, by that version (`controller_schema`, migration 0087). A failed controller gate
+  reads the record of the build it would put back: one that reaches less than the store has applied, or
+  never recorded what it reaches, is not put back; the current build is kept, and the condition
+  (`rollback-failed`, urgent) says why and that a person decides. The lease holder's `Build` is that
+  version too.
+- **A walk of a module on a machine holds every other send of another build of it there.** `walkedBy`
+  reads a release's open gate as its walk, and a plan's own modules are held by it as any other move is:
+  a merge plan whose first machine is under a release's gate for the same module waits, said as
+  "waiting for … which release-… is walking". The same build to the same machine is not held: it is
+  already there.
+
+Not changed: whether a release's walk and a merge plan's walk of the core should be one walk (the walk
+carrying over) is left to a design; serialising them is what this fixes.
+
+## How it is checked
+
+| Rule | Check |
+|---|---|
+| A newer send to the judged machine does not unreport the gate's | `TestReplay352` (the backlog fixture: the release sends, the machine reports, another send lands unreported, the gate still passes) — fails on the commit before the fix with "has not reported on what it was sent" |
+| A send that moves the module supersedes the judging, puts nothing back | `TestReplay352`, `TestAPlansJudgingIsSupersededByASendThatMovesItsModule` |
+| A report is held against the gate's own send, by digest or sequence | `TestAReportIsHeldAgainstTheGatesOwnSend`, inventory `TestASendAndAReportAreKnownByTheirDeclaration` |
+| Only the build last sent is judged | `TestAGateJudgesOnlyTheBuildTheMachineWasLastSent` |
+| A walk holds other sends of the module there, a release's too | `TestAWalkWaitsForAJudgingOfTheSameModuleOnThatMachine`, `TestAPlansFirstSendWaitsForAReleaseJudgingTheModuleThere` |
+| No put-back to a controller behind the store, or one unproved | `TestTheControllerIsNotPutBackToABuildOlderThanTheStoresSchema`, inventory `TestASchemaReachIsKeptByBuild`, catalogue `TestAProcessEnvCanNameTheBuildsOwnVersion`, `TestTheHolderNamesTheBuildTheDeclarationToldIt` |
+| Every condition of the change is tested | 31 mutations, each failing a test |
+
+A replay in mesh-lab's register is still owed before this core issue resolves (ADR 0237).