ADR 0160 built: both halves, and what the first roll-out taught; issues 183 (the controller's grant) and 184 (a merge blocks the receive loop)

2026-10-01 15:23:26 +02:00
parent 64acc94de8
commit eef03f2b08
3 changed files with 101 additions and 0 deletions
@@ -0,0 +1,39 @@
+---
+status: resolved
+opened: 2026-10-01
+located-in: [mesh-controller internal/broker/nats.go (the controller's own publish grant)]
+fixed-by: mesh-controller PR 189 (fix/the-controller-may-publish-memberships)
+amended-design: []
+---
+
+# 183 — The controller could not publish the memberships it issued
+
+## What was observed
+
+The controller release that issues a membership per assignment ([ADR 0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md))
+rolled onto the control node. After its first push the controller's log said, once:
+
+```
+nats: permissions violation: Permissions Violation for Publish to "mesh.assignment.<node>.<module>"
+```
+
+No membership reached the assignments stream. Nothing else changed: every runtime kept serving the
+shape it derives for itself, which is what the decision says happens while no membership is issued,
+and so the mesh looked healthy while the whole new mechanism was inert.
+
+## Why this is here
+
+The controller composes every account's grant, its own included, and its own grant named the
+control, node and JetStream subjects and not the assignments it alone issues. A grant composed by its
+holder is checked by nothing but the server at publish time, and a refused publish is one log line
+that nothing reads. The fallback that makes the roll-out safe is the same thing that makes this
+failure silent.
+
+## Resolved, 2026-10-01
+
+The controller's grant names `mesh.assignment.>`; the broker golden changed by that one line. A
+grant composed by its holder arrives late — the broker node is pushed after the controller rolls —
+so the roll-out is merge, push the broker node, then any push issues memberships.
+
+*How it is checked:* the broker golden carries the allow line; live, a runtime's log after the next
+push says it was issued a membership rather than that none exists.
@@ -0,0 +1,43 @@
+---
+status: open
+opened: 2026-10-01
+located-in: [mesh-controller internal/link/serve.go (act handles one message at a time; sourceMoved waits for every build the merge asks)]
+fixed-by:
+amended-design: []
+---
+
+# 184 — A merge announcement blocks the controller's receive loop
+
+## What was observed
+
+The controller restarted at 12:53:32Z on 2026-10-01 and heard, among its first messages, a merge
+announcement for the catalogue that touched some forty modules. Until 13:17:16Z — twenty-four
+minutes — it took nothing else in: build outcomes that the build machine had announced and that
+the console's `builds` log showed as done sat unrecorded, so `builds` listed none of them and the
+modules stayed at their old versions; the node heartbeats were dropped by the bus as a slow consumer
+on `mesh.control.*.alive`, twice. When the merge handler returned, everything queued arrived at once
+and was taken in within a second.
+
+## Why this is here
+
+The receive loop acts on one message at a time, which is the right discipline for a store the
+controller must write in order. Acting on a merge means asking builds and waiting for each outcome,
+minutes of work, and that wait happens inside the loop that would otherwise be hearing the outcomes
+of everything else. The bus keeps the merge message alive while the work runs — the fix for the
+earlier repeated-merge fault — so nothing is lost and nothing is redelivered, and nothing is heard
+either. The design permits the controller to go deaf for as long as a merge takes to build, and no
+status says so: the mesh reads as quiet, builds read as missing, and heartbeats read as a slow
+machine.
+
+## What a fix needs to decide
+
+Whether a merge is work the loop dispatches and returns from — the builds asked, the outcomes taken
+in by the same `Built` handler every other outcome uses, since the stream already delivers them —
+or whether the loop runs more than one handler at a time with the store's ordering kept for the
+kinds that need it. The first is smaller and keeps one ordering. Either way, a controller that is
+busy should say so where `status` is read.
+
+*How this would be checked:* a controller test where a merge announcement that asks a slow build
+and a build outcome for another module arrive together, and the outcome is recorded before the
+build finishes; live, the controller's log during the next catalogue merge shows registrations
+interleaved with the merge's own.