2026-10-03 21:20:06 +00:00
1 changed files with 52 additions and 0 deletions
@@ -0,0 +1,52 @@
 ---
 status: located
 opened: 2026-10-03
 located-in:
  - mesh-tools
 fixed-by:
 amended-design:
 ---
 # 217 — A refused announcement took down every container's runtime, and the console with it
 ## What was observed
 2026-10-03, rolling out [ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md).
 The tool runtimes announced themselves by subscribing `$SRV.<verb>.>`; the grants composed for them
 allowed only `$SRV.<verb>` and the service's own name and instance. The bus refused the wildcard:
 ```
 Subscription Violation - User "novox.gitea", Subject "$SRV.PING.>"
 NatsError: 'Permissions Violation for Subscription to "$SRV.PING.>"'
 ```
 The TypeScript runtime the per-module containers run treats a refused subscription as fatal, so on
 the one machine that had received the new images nine containers crash-looped — gitea's runtime,
 postgres's, the catalogue, the vault, mongodb, mssql, keycloak, mailu, nextcloud. Gitea's tools went
 with them, which closed the usual path for merging the fix.
 Then the console stopped answering the controller's verbs, though the controller held every
 subscription: the console builds the list that tells a seat's verb from a module's tool by asking the
 controller *and* the catalogue, and gives up on both when the catalogue does not answer — so
 `mesh-controller.status` was asked of a module subject nobody serves.
 ## Why it matters beyond this instance
 Two properties, each worse than the mistake that exposed it:
 - **A runtime dies for an optional subscription.** Announcing is discovery; serving tools and running
  provisioners is the work. A refusal of the first should never stop the second.
 - **The console's view of the mesh's own verbs depended on a module.** The controller's verbs are how
  the operator repairs the mesh; they must not become unreachable because the catalogue is down.
 ## Diagnosis
 Owner mesh-tools. The wildcard is fixed on branch `fix/announce-only-what-the-grants-allow`: both
 runtimes subscribe exactly what the grants allow. The console that discovers from what announces itself
 (ADR 0195, 0197, on main) asks the bus and the controller, not the catalogue, which removes the second
 property once it is deployed. **Still to do:** the TypeScript runtime treats a refused announcement
 subscription as a logged warning, not as fatal; a test against a bus with real grants proves the
 announcement subscriptions are allowed for every principal kind.
 Recovered on the day without the forge's API: the toolchain images built by the controller straight
 from the fix branch, every module image rebuilt on them, and the machine pushed.