A refused membership does not stop the controller #190

2026-10-01T13:30:18Z

mesh-admin commented

2026-10-01 13:30:18 +00:00

Live today: the daemon's receive loop has been stuck since 13:17:22Z. issueMemberships publishes each membership into the assignments stream with the daemon's own context; the server refused the publish (the grant, #189) and never acknowledged it, so the publish waits for ever, inside the loop that hears reports, build outcomes and merges. No report has been taken since; the host applied the 13:17 push at 13:18 and the controller never heard it; the merge announcements for mesh-tools #23 and #189 are queued behind it.

PublishMembership is bounded to ten seconds.
A push counts the memberships it could not issue, names the first failure, and stands: the declarations were sent and recorded before, and every runtime without a membership serves the derived shape (ADR 0160).

make check on the branch. Recovery of the live controller follows the one-shot recipe: build this from the CLI inside the container with --wait, push the control node from the CLI, then push again from the new daemon so the broker carries #189's grant.

Live today: the daemon's receive loop has been stuck since 13:17:22Z. `issueMemberships` publishes each membership into the assignments stream with the daemon's own context; the server refused the publish (the grant, #189) and never acknowledged it, so the publish waits for ever, inside the loop that hears reports, build outcomes and merges. No report has been taken since; the host applied the 13:17 push at 13:18 and the controller never heard it; the merge announcements for mesh-tools #23 and #189 are queued behind it. - `PublishMembership` is bounded to ten seconds. - A push counts the memberships it could not issue, names the first failure, and stands: the declarations were sent and recorded before, and every runtime without a membership serves the derived shape (ADR 0160). `make check` on the branch. Recovery of the live controller follows the one-shot recipe: build this from the CLI inside the container with `--wait`, push the control node from the CLI, then push again from the new daemon so the broker carries #189's grant.

mesh-admin added 1 commit 2026-10-01 13:30:18 +00:00

A refused membership does not stop the controller 05b90f966a

A stream publish waits for its acknowledgement as long as its context lives, and the server never
acknowledges a publish it refuses. Issuing memberships after a push used the daemon's own context, so
the one refused membership of 2026-10-01 (hq issue 183) held the controller's receive loop for good:
no report, no build outcome, no merge was heard until a restart (hq issue 185). Issuing one
membership is now bounded to ten seconds, and a push says how many could not be issued and stands —
the machines keep the shape they derive until the next push.

mesh-admin merged commit 69f559ab4c into main

2026-10-01 13:31:27 +00:00

mesh-admin deleted branch fix/a-refused-membership-does-not-stop-the-controller

2026-10-01 13:31:28 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: novox/mesh-controller#190