2.9 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | ||
|---|---|---|---|---|---|---|
| resolved | 2026-10-01 |
|
mesh-controller PR 190 (fix/a-refused-membership-does-not-stop-the-controller) |
185 — A refused membership publish stops the controller
What was observed
At 13:17:22Z on 2026-10-01 the controller, acting on a build it had just taken in, sent the control
node a declaration and then issued that node's memberships. The server refused the first publish
(issue 183). From
that second on the controller heard nothing: the control node applied the declaration at 13:18 and
its report was never taken; two merges announced by the forge were not built; the heartbeats were
dropped by the bus as a slow consumer; the console's builds showed nothing new while the build
machine's own log showed builds done. The controller's seat verbs still answered, so status read
as quiet. It stayed so until the controller was replaced.
Why this is here
A stream publish waits for its acknowledgement for as long as its context lives, and a publish the server refuses is never acknowledged. The membership was published with the daemon's own context, which lives as long as the daemon, from inside the one loop that hears everything else. Two mechanisms built the day before met badly: the receive loop that acts on one message at a time (issue 184) and a publish that could wait for ever. The design let a refusal that is said in one log line become a controller that is deaf with no sign of it.
Resolved, 2026-10-01
Issuing one membership is bounded to ten seconds, and a push counts the memberships it could not issue, names the first failure, and stands: the declarations were sent and recorded before it, and every runtime without a membership serves the shape it derives (ADR 0160). The live controller was replaced by hand: the fix was built from the CLI inside the running container with a wait; the stuck daemon held the control node's advisory lock in the store, so the container was restarted to release it; the control node was pushed from the CLI and took the fixed controller; a second push from the fixed controller carried the broker's grant, and the broker reloaded. The push command itself issued no memberships — only the roll-out path did — which is mesh-controller PR 191.
How it is checked: a link test publishes a membership to a server that refuses it and returns within the bound; live, the controller's log after a push names the memberships it issued or could not, and keeps taking reports either way.