ADR 0160 built: both halves, and what the first roll-out taught; issues 183 (the controller's grant) and 184 (a merge blocks the receive loop)

2026-10-01 15:23:26 +02:00
parent 64acc94de8
commit eef03f2b08
3 changed files with 101 additions and 0 deletions
@@ -103,6 +103,25 @@ ask what to listen on. This record decides exactly that.
 | The console lists a stateful module on two machines once per machine, and composes no subject | the MCP conformance test |
 | Live: the store's databases asked of one named machine and through the seat, after a controller release and one push, with no module rebuilt | by hand |
 ## Built, 2026-10-01
 > **Progressive insight — 2026-10-01.** The decision stands; these are the facts of its building.
 - The controller's half: mesh-controller 188 — the membership, its subject, the assignments stream
  read directly, a module's account granted its own membership and nothing else of the stream, a
  membership published after each push.
 - The runtime's half: mesh-tools 23 — the one address derived, the membership read and followed,
  exactly the issued subjects served and re-served, the derived shape with a log line until one is
  issued, a seat's verbs implemented under the seat's name and never listed as the module's, the
  listing carrying subjects and the console composing none. A claim may now name the verbs it
  serves for its seat (mesh-controller 186), so a holder's own tools need not be the seat's.
 - What the first roll-out taught: the controller's own grant did not name the assignments it issues,
  so the first memberships were refused by the server and every runtime kept the derived shape —
  which is exactly the fallback this record asked for, and exactly why nobody noticed
  ([issue 183](../04-ISSUES/183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md)).
  The SDK's `invokeTool` still composes a subject; it reaches a membership through the runtime's
  broker, which does, so the caller-side rule is met there and not yet in the SDK's own words.
 ## References
 - [ADR 0159](0159-a-tool-call-names-the-machine-and-a-holder-serves-its-seats-verbs.md) — extended: the same facts, issued rather than derived
@@ -0,0 +1,39 @@
 ---
 status: resolved
 opened: 2026-10-01
 located-in: [mesh-controller internal/broker/nats.go (the controller's own publish grant)]
 fixed-by: mesh-controller PR 189 (fix/the-controller-may-publish-memberships)
 amended-design: []
 ---
 # 183 — The controller could not publish the memberships it issued
 ## What was observed
 The controller release that issues a membership per assignment ([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md))
 rolled onto the control node. After its first push the controller's log said, once:
 ```
 nats: permissions violation: Permissions Violation for Publish to "mesh.assignment.<node>.<module>"
 ```
 No membership reached the assignments stream. Nothing else changed: every runtime kept serving the
 shape it derives for itself, which is what the decision says happens while no membership is issued,
 and so the mesh looked healthy while the whole new mechanism was inert.
 ## Why this is here
 The controller composes every account's grant, its own included, and its own grant named the
 control, node and JetStream subjects and not the assignments it alone issues. A grant composed by its
 holder is checked by nothing but the server at publish time, and a refused publish is one log line
 that nothing reads. The fallback that makes the roll-out safe is the same thing that makes this
 failure silent.
 ## Resolved, 2026-10-01
 The controller's grant names `mesh.assignment.>`; the broker golden changed by that one line. A
 grant composed by its holder arrives late — the broker node is pushed after the controller rolls —
 so the roll-out is merge, push the broker node, then any push issues memberships.
 *How it is checked:* the broker golden carries the allow line; live, a runtime's log after the next
 push says it was issued a membership rather than that none exists.
@@ -0,0 +1,43 @@
 ---
 status: open
 opened: 2026-10-01
 located-in: [mesh-controller internal/link/serve.go (act handles one message at a time; sourceMoved waits for every build the merge asks)]
 fixed-by:
 amended-design: []
 ---
 # 184 — A merge announcement blocks the controller's receive loop
 ## What was observed
 The controller restarted at 12:53:32Z on 2026-10-01 and heard, among its first messages, a merge
 announcement for the catalogue that touched some forty modules. Until 13:17:16Z — twenty-four
 minutes — it took nothing else in: build outcomes that the build machine had announced and that
 the console's `builds` log showed as done sat unrecorded, so `builds` listed none of them and the
 modules stayed at their old versions; the node heartbeats were dropped by the bus as a slow consumer
 on `mesh.control.*.alive`, twice. When the merge handler returned, everything queued arrived at once
 and was taken in within a second.
 ## Why this is here
 The receive loop acts on one message at a time, which is the right discipline for a store the
 controller must write in order. Acting on a merge means asking builds and waiting for each outcome,
 minutes of work, and that wait happens inside the loop that would otherwise be hearing the outcomes
 of everything else. The bus keeps the merge message alive while the work runs — the fix for the
 earlier repeated-merge fault — so nothing is lost and nothing is redelivered, and nothing is heard
 either. The design permits the controller to go deaf for as long as a merge takes to build, and no
 status says so: the mesh reads as quiet, builds read as missing, and heartbeats read as a slow
 machine.
 ## What a fix needs to decide
 Whether a merge is work the loop dispatches and returns from — the builds asked, the outcomes taken
 in by the same `Built` handler every other outcome uses, since the stream already delivers them —
 or whether the loop runs more than one handler at a time with the store's ordering kept for the
 kinds that need it. The first is smaller and keeps one ordering. Either way, a controller that is
 busy should say so where `status` is read.
 *How this would be checked:* a controller test where a merge announcement that asks a slow build
 and a build outcome for another module arrive together, and the outcome is recorded before the
 build finishes; live, the controller's log during the next catalogue merge shows registrations
 interleaved with the merge's own.