Merge pull request 'Issue 353: the controller asked before the bus held its grant' (#253) from issues/353-the-controller-asked-before-the-bus-held-its-grant into main

2026-10-09 17:57:38 +00:00
parent 0e7af9988a adbd57a1f9
commit fbaa8e16e6
1 changed files with 59 additions and 0 deletions
@@ -0,0 +1,59 @@
+---
+status: located
+opened: 2026-10-09
+located-in: [mesh-controller cmd/mesh-controller (asker.go, asker_wire.go), mesh-controller internal/broker (users.go, asksSeatOf)]
+fixed-by: mesh-controller pull request 186
+amended-design:
+replay-none: the whole of it is one process's record against one bus's user list; the controller's own test drives the asker with the bus's word held and withheld (TestNothingIsAskedWhileTheBusLacksTheControllersGrant), and a container runtime adds nothing to that
+---
+
+# 353. The controller asked before the bus held its grant
+
+## Symptom
+
+On 2026-10-09, local times on the control node, from its controller's journal (one process throughout):
+
+- 17:32 the router (the `operator-channel` holder) was unassigned, so that a merge gate of the catalogue had
+  nothing to send there; 17:52:54 a walk of another module pushed the control node and composed the bus's user
+  list **without** the controller's grant to ask: the grant is composed from a holder of the seat that is
+  *assigned*, and none was.
+- 17:54:05 the controller started asking (the router was assigned again by then, so its check of the record
+  said a router was here), and the bus refused its publish of the ask, on the same connection, seven times
+  until 17:56:05. Each refusal was followed by "the operator could not be asked about machine.…: context
+  deadline exceeded". The self-check raised `bus.controller.refused`.
+- 17:55:56 the push that put the router back on the control node sent it the user list **with** the grant; 17:56:05 the bus reloaded
+  it in place (the same second the watchdogs saw "jetstream not enabled", the reload's moment); 17:56:10 the
+  same connection asked successfully, and again at 18:03:53. The server applies a reloaded permission to a
+  standing connection: no reconnect was needed or made.
+- `bus.controller.refused` was raised at 17:54:06 and cleared by itself at 18:11:44, about 15 minutes after
+  the last refusal at 17:56:05 (the controller's condition history). The condition's evidence lists the same
+  refusal read again every 30 seconds until 18:10:35; the journal shows no refusal after 17:56:05, so those
+  are the signal reading its record again, not new refusals.
+
+## Why it is an issue
+
+- **The record said yes before the bus did.** The controller's grant to ask follows the router's assignment
+  ([ADR 0259](../../02-DECISIONS/0259-an-ask-is-a-message-on-the-bus-any-channel-answers-it-and-the-asker-acts-on-a-warrant.md)
+  §3: a caller-named verb is granted to the module that uses the seat), and reaches the bus only when the
+  machine holding the bus is next pushed. Between `assign` and `push`, and on any push of that machine made
+  while the router is unassigned, the controller believed its own record of the router and published into a
+  refusal. The composition is right; the asker trusted it before it was on the bus.
+- **The failure was loud in the wrong place.** Seven bus refusals and a warning about the bus, when the fact
+  was "the control node has not been pushed since the router was assigned", which the controller knows (its
+  push already judges whether the bus's machine is behind on the user list, to send it first).
+
+## Acceptance test (the operator's view)
+
+While the bus has not been given the controller's grant to ask, the controller asks nothing and the condition
+"Questions for you not delivered" says which push carries the grant; once that push lands, the questions go
+out, and the bus never refuses the controller's ask.
+
+## How it is checked
+
+- `TestNothingIsAskedWhileTheBusLacksTheControllersGrant` (mesh-controller `cmd/mesh-controller/asker_test.go`):
+  nothing is asked while the bus's machine was not sent the user list composed now, it is said once, the
+  conditions that need the operator are raised as undelivered naming the push, and they are asked once the bus
+  holds the grant.
+- Live: after the fix is rolled out, `assign` the router, read `mesh-controller.conditions` for
+  `asks-undelivered` naming the push, `push` the control node, and read that the asks go out with no
+  `bus.controller.refused` raised.