48 lines
2.9 KiB
Markdown
48 lines
2.9 KiB
Markdown
---
|
|
status: resolved
|
|
opened: 2026-10-01
|
|
located-in: [mesh-controller internal/link/bus.go (a membership published with the daemon's own context), mesh-controller cmd/mesh-controller/push.go (a push that returns on the first membership it cannot issue)]
|
|
fixed-by: mesh-controller PR 190 (fix/a-refused-membership-does-not-stop-the-controller)
|
|
amended-design: []
|
|
---
|
|
|
|
# 185 — A refused membership publish stops the controller
|
|
|
|
## What was observed
|
|
|
|
At 13:17:22Z on 2026-10-01 the controller, acting on a build it had just taken in, sent the control
|
|
node a declaration and then issued that node's memberships. The server refused the first publish
|
|
([issue 183](../183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md)). From
|
|
that second on the controller heard nothing: the control node applied the declaration at 13:18 and
|
|
its report was never taken; two merges announced by the forge were not built; the heartbeats were
|
|
dropped by the bus as a slow consumer; the console's `builds` showed nothing new while the build
|
|
machine's own log showed builds done. The controller's seat verbs still answered, so `status` read
|
|
as quiet. It stayed so until the controller was replaced.
|
|
|
|
## Why this is here
|
|
|
|
A stream publish waits for its acknowledgement for as long as its context lives, and a publish the
|
|
server refuses is never acknowledged. The membership was published with the daemon's own context,
|
|
which lives as long as the daemon, from inside the one loop that hears everything else. Two
|
|
mechanisms built the day before met badly: the receive loop that acts on one message at a time
|
|
([issue 184](../184-a-merge-announcement-blocks-the-controllers-receive-loop/00-report.md)) and a
|
|
publish that could wait for ever. The design let a refusal that is said in one log line become a
|
|
controller that is deaf with no sign of it.
|
|
|
|
## Resolved, 2026-10-01
|
|
|
|
Issuing one membership is bounded to ten seconds, and a push counts the memberships it could not
|
|
issue, names the first failure, and stands: the declarations were sent and recorded before it, and
|
|
every runtime without a membership serves the shape it derives
|
|
([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
|
|
The live controller was replaced by hand: the fix was built from the CLI inside the running
|
|
container with a wait; the stuck daemon held the control node's advisory lock in the store, so the
|
|
container was restarted to release it; the control node was pushed from the CLI and took the fixed
|
|
controller; a second push from the fixed controller carried the broker's grant, and the broker
|
|
reloaded. The push command itself issued no memberships — only the roll-out path did — which is
|
|
mesh-controller PR 191.
|
|
|
|
*How it is checked:* a link test publishes a membership to a server that refuses it and returns
|
|
within the bound; live, the controller's log after a push names the memberships it issued or could
|
|
not, and keeps taking reports either way.
|