Merge pull request 'Issue 185: a refused membership publish stopped the controller; 183 points to it' (#252) from fix/issue-185-a-refused-membership-stops-the-controller into main

2026-10-01 13:44:17 +00:00
parent 39f0d64840 2175d13935
commit 6b8fb562ce
2 changed files with 51 additions and 0 deletions
@@ -35,5 +35,9 @@ The controller's grant names `mesh.assignment.>`; the broker golden changed by t
 grant composed by its holder arrives late — the broker node is pushed after the controller rolls —
 so the roll-out is merge, push the broker node, then any push issues memberships.

+The refusal had a second consequence: published with the daemon's own context, the refused
+membership was waited on for ever and the controller went deaf —
+[issue 185](../185-a-refused-membership-publish-stops-the-controller/00-report.md).
+
 *How it is checked:* the broker golden carries the allow line; live, a runtime's log after the next
 push says it was issued a membership rather than that none exists.
@@ -0,0 +1,47 @@
+---
+status: resolved
+opened: 2026-10-01
+located-in: [mesh-controller internal/link/bus.go (a membership published with the daemon's own context), mesh-controller cmd/mesh-controller/push.go (a push that returns on the first membership it cannot issue)]
+fixed-by: mesh-controller PR 190 (fix/a-refused-membership-does-not-stop-the-controller)
+amended-design: []
+---
+
+# 185 — A refused membership publish stops the controller
+
+## What was observed
+
+At 13:17:22Z on 2026-10-01 the controller, acting on a build it had just taken in, sent the control
+node a declaration and then issued that node's memberships. The server refused the first publish
+([issue 183](../183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md)). From
+that second on the controller heard nothing: the control node applied the declaration at 13:18 and
+its report was never taken; two merges announced by the forge were not built; the heartbeats were
+dropped by the bus as a slow consumer; the console's `builds` showed nothing new while the build
+machine's own log showed builds done. The controller's seat verbs still answered, so `status` read
+as quiet. It stayed so until the controller was replaced.
+
+## Why this is here
+
+A stream publish waits for its acknowledgement for as long as its context lives, and a publish the
+server refuses is never acknowledged. The membership was published with the daemon's own context,
+which lives as long as the daemon, from inside the one loop that hears everything else. Two
+mechanisms built the day before met badly: the receive loop that acts on one message at a time
+([issue 184](../184-a-merge-announcement-blocks-the-controllers-receive-loop/00-report.md)) and a
+publish that could wait for ever. The design let a refusal that is said in one log line become a
+controller that is deaf with no sign of it.
+
+## Resolved, 2026-10-01
+
+Issuing one membership is bounded to ten seconds, and a push counts the memberships it could not
+issue, names the first failure, and stands: the declarations were sent and recorded before it, and
+every runtime without a membership serves the shape it derives
+([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)).
+The live controller was replaced by hand: the fix was built from the CLI inside the running
+container with a wait; the stuck daemon held the control node's advisory lock in the store, so the
+container was restarted to release it; the control node was pushed from the CLI and took the fixed
+controller; a second push from the fixed controller carried the broker's grant, and the broker
+reloaded. The push command itself issued no memberships — only the roll-out path did — which is
+mesh-controller PR 191.
+
+*How it is checked:* a link test publishes a membership to a server that refuses it and returns
+within the bound; live, the controller's log after a push names the memberships it issued or could
+not, and keeps taking reports either way.