2.5 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | |
|---|---|---|---|---|---|
| resolved | 2026-10-02 |
|
mesh-controller PR |
204 — A controller handover re-sent every node a declaration composed from a stale view
What was observed
2026-10-02 21:29 UTC. Two machines were assigned the node tools runtime and had the console taken off them, and were pushed; the host on each applied it (the console removed, the runtime created and running). Two seconds later, on each, a second declaration arrived that undid it — the host's log:
23:29:18 created node-tools.runtime (node-tools): 239 file(s), running as node-tools.service
23:29:18 applied 569 resource(s)
23:29:20 removed node-tools.runtime (node-tools)
23:29:20 forgotten node-tools.interpreter (nodejs)
23:29:24 created mesh-console.needs-broker, created mesh-console.server (mesh-console)
The second declaration had the console assigned and the runtime absent: the assignments as they were a minute earlier. The controller's status knew of one send per machine, the person's. At that minute a plan from an unrelated merge was rolling a new controller build onto the control node, so an instance was starting while another was stopping. A third push by hand, two minutes later, restored both machines and nothing undid it again.
Which instance sent the stale declaration, and from what, is not established: the outgoing one on its way down, the incoming one at start-up before its view was current, or the rolling plan sending what it had composed when it was made — the same family as issue 201, where a plan sent a digest older than the one a successor had written.
Why it matters beyond this instance
A declaration the mesh sends is the mesh's word on what a machine should be; a machine applies it in full, including removing what it no longer names. A stale one is therefore not a no-op: it tears down whatever was assigned since, and the controller's own record does not show the send, so the next person reads "applied, current" over a machine that is wrong. Two people merging within a minute is ordinary, and a controller handover happens on every controller merge.
Questions HQ must answer
- Where does the stale view come from, and is every send recorded so status can show it?
- Should a declaration carry the assignment generation it was composed from, so a host refuses one older than the last it applied, as it already refuses a declaration it cannot verify?