diff --git a/04-ISSUES/183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md b/04-ISSUES/183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md index 86f453a..f0a4402 100644 --- a/04-ISSUES/183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md +++ b/04-ISSUES/183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md @@ -35,5 +35,9 @@ The controller's grant names `mesh.assignment.>`; the broker golden changed by t grant composed by its holder arrives late — the broker node is pushed after the controller rolls — so the roll-out is merge, push the broker node, then any push issues memberships. +The refusal had a second consequence: published with the daemon's own context, the refused +membership was waited on for ever and the controller went deaf — +[issue 185](../185-a-refused-membership-publish-stops-the-controller/00-report.md). + *How it is checked:* the broker golden carries the allow line; live, a runtime's log after the next push says it was issued a membership rather than that none exists. diff --git a/04-ISSUES/185-a-refused-membership-publish-stops-the-controller/00-report.md b/04-ISSUES/185-a-refused-membership-publish-stops-the-controller/00-report.md new file mode 100644 index 0000000..20947b0 --- /dev/null +++ b/04-ISSUES/185-a-refused-membership-publish-stops-the-controller/00-report.md @@ -0,0 +1,47 @@ +--- +status: resolved +opened: 2026-10-01 +located-in: [mesh-controller internal/link/bus.go (a membership published with the daemon's own context), mesh-controller cmd/mesh-controller/push.go (a push that returns on the first membership it cannot issue)] +fixed-by: mesh-controller PR 190 (fix/a-refused-membership-does-not-stop-the-controller) +amended-design: [] +--- + +# 185 — A refused membership publish stops the controller + +## What was observed + +At 13:17:22Z on 2026-10-01 the controller, acting on a build it had just taken in, sent the control +node a declaration and then issued that node's memberships. The server refused the first publish +([issue 183](../183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md)). From +that second on the controller heard nothing: the control node applied the declaration at 13:18 and +its report was never taken; two merges announced by the forge were not built; the heartbeats were +dropped by the bus as a slow consumer; the console's `builds` showed nothing new while the build +machine's own log showed builds done. The controller's seat verbs still answered, so `status` read +as quiet. It stayed so until the controller was replaced. + +## Why this is here + +A stream publish waits for its acknowledgement for as long as its context lives, and a publish the +server refuses is never acknowledged. The membership was published with the daemon's own context, +which lives as long as the daemon, from inside the one loop that hears everything else. Two +mechanisms built the day before met badly: the receive loop that acts on one message at a time +([issue 184](../184-a-merge-announcement-blocks-the-controllers-receive-loop/00-report.md)) and a +publish that could wait for ever. The design let a refusal that is said in one log line become a +controller that is deaf with no sign of it. + +## Resolved, 2026-10-01 + +Issuing one membership is bounded to ten seconds, and a push counts the memberships it could not +issue, names the first failure, and stands: the declarations were sent and recorded before it, and +every runtime without a membership serves the shape it derives +([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md)). +The live controller was replaced by hand: the fix was built from the CLI inside the running +container with a wait; the stuck daemon held the control node's advisory lock in the store, so the +container was restarted to release it; the control node was pushed from the CLI and took the fixed +controller; a second push from the fixed controller carried the broker's grant, and the broker +reloaded. The push command itself issued no memberships — only the roll-out path did — which is +mesh-controller PR 191. + +*How it is checked:* a link test publishes a membership to a server that refuses it and returns +within the bound; live, the controller's log after a push names the memberships it issued or could +not, and keeps taking reports either way.