Issue 217: a refused announcement took down every container's runtime, and the console with it

2026-10-03 22:47:28 +02:00
parent d7bb24b181
commit 08cea893bd
1 changed files with 52 additions and 0 deletions
@@ -0,0 +1,52 @@
+---
+status: located
+opened: 2026-10-03
+located-in:
+  - mesh-tools
+fixed-by:
+amended-design:
+---
+
+# 217 — A refused announcement took down every container's runtime, and the console with it
+
+## What was observed
+
+2026-10-03, rolling out [ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md).
+The tool runtimes announced themselves by subscribing `$SRV.<verb>.>`; the grants composed for them
+allowed only `$SRV.<verb>` and the service's own name and instance. The bus refused the wildcard:
+
+```
+Subscription Violation - User "novox.gitea", Subject "$SRV.PING.>"
+NatsError: 'Permissions Violation for Subscription to "$SRV.PING.>"'
+```
+
+The TypeScript runtime the per-module containers run treats a refused subscription as fatal, so on
+the one machine that had received the new images nine containers crash-looped — gitea's runtime,
+postgres's, the catalogue, the vault, mongodb, mssql, keycloak, mailu, nextcloud. Gitea's tools went
+with them, which closed the usual path for merging the fix.
+
+Then the console stopped answering the controller's verbs, though the controller held every
+subscription: the console builds the list that tells a seat's verb from a module's tool by asking the
+controller *and* the catalogue, and gives up on both when the catalogue does not answer — so
+`mesh-controller.status` was asked of a module subject nobody serves.
+
+## Why it matters beyond this instance
+
+Two properties, each worse than the mistake that exposed it:
+
+- **A runtime dies for an optional subscription.** Announcing is discovery; serving tools and running
+  provisioners is the work. A refusal of the first should never stop the second.
+- **The console's view of the mesh's own verbs depended on a module.** The controller's verbs are how
+  the operator repairs the mesh; they must not become unreachable because the catalogue is down.
+
+## Diagnosis
+
+Owner mesh-tools. The wildcard is fixed on branch `fix/announce-only-what-the-grants-allow`: both
+runtimes subscribe exactly what the grants allow. The console that discovers from what announces itself
+(ADR 0195, 0197, on main) asks the bus and the controller, not the catalogue, which removes the second
+property once it is deployed. **Still to do:** the TypeScript runtime treats a refused announcement
+subscription as a logged warning, not as fatal; a test against a bus with real grants proves the
+announcement subscriptions are allowed for every principal kind.
+
+Recovered on the day without the forge's API: the toolchain images built by the controller straight
+from the fix branch, every module image rebuilt on them, and the machine pushed.