Issue 201: a push recreated the controller at a digest older than the seat row its successor wrote

2026-10-02 17:15:22 +02:00
parent 9ba4de5557
commit d0d5799884
1 changed files with 58 additions and 0 deletions
@@ -0,0 +1,58 @@
 ---
 status: open
 opened: 2026-10-02
 located-in:
  - mesh-controller
 fixed-by:
 amended-design:
 ---
 # 201 — A push recreated the controller at a digest older than the seat row its successor had written
 ## What was observed
 2026-10-02, two merges a minute apart on the control node: one to the controller, adding a verb to the
 controller seat's row; one to the host, adding a container field. Each made a plan. The controller's plan
 built and rolled the new controller, which started, widened its seat row with the new verb, and ran. The
 host's plan then pushed the control node with the controller digest it had recorded when it was made —
 the previous build — and recreated the controller container on it. The older binary read the row, found
 a verb it could not run, and refused to start:
 ```
 mesh-controller: the mesh-controller seat's row declares "command", which this control plane
 cannot run: "command" is not a verb the mesh-controller seat serves
 ```
 A crash loop followed for ten minutes: nothing answered on the bus, and no build was dispatched, since
 the controller is what fills the builder's queue. Recovery was the mesh's own binary run once from the
 newer image, outside the service, to push the control node again; the push sent the newer digest and
 the controller came up.
 ## Why it matters beyond this instance
 The row is the store's and the binary follows it ([ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md));
 a start-up check that refuses a row the binary cannot serve is right, and was built after the outage of
 2026-09-27 for exactly this reason. What is wrong is a plan sending a controller older than the one that
 wrote the row. A plan is made at a moment and sends what it recorded ([ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md));
 for every other module an older digest is a brief regression a later push corrects. For the controller
 it is the mesh losing its voice, and the correction needs a hand, because the thing that would correct
 it is the thing that is down. Two plans that overlap will happen again whenever two people merge within
 a minute.
 ## What a fix would have to do
 Either of two, and the first is the smaller:
 - A push never sends a controller digest older than the one the running controller is — the controller
  knows its own digest and refuses to downgrade itself through a plan, saying so in the plan's words.
 - Or the start-up check tolerates a row wider than the binary while a roll-out is in flight, and serves
  what it can. Weaker: it makes the row and the binary disagree on purpose, which is what the check
  exists to refuse.
 Until one is built: do not merge a controller change while another plan is rolling, and after merging
 one, wait for `node show` on the control node to report the new controller before merging anything else.
 ## References
 - [ADR 0154](../../02-DECISIONS/0154-the-meshs-own-verbs-are-the-controller-seats-tools.md), [ADR 0162](../../02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md)
 - mesh-controller `cmd/mesh-controller/seatverbs.go` (`seatToolHandlers`, the start-up check), `cmd/mesh-controller/push.go`