3.0 KiB
status, opened, located-in, fixed-by, amended-design
| status | opened | located-in | fixed-by | amended-design | |||
|---|---|---|---|---|---|---|---|
| resolved | 2026-10-03 |
|
|
217 — A refused announcement took down every container's runtime, and the console with it
What was observed
2026-10-03, rolling out ADR 0197.
The tool runtimes announced themselves by subscribing $SRV.<verb>.>; the grants composed for them
allowed only $SRV.<verb> and the service's own name and instance. The bus refused the wildcard:
Subscription Violation - User "novox.gitea", Subject "$SRV.PING.>"
NatsError: 'Permissions Violation for Subscription to "$SRV.PING.>"'
The TypeScript runtime the per-module containers run treats a refused subscription as fatal, so on the one machine that had received the new images nine containers crash-looped — gitea's runtime, postgres's, the catalogue, the vault, mongodb, mssql, keycloak, mailu, nextcloud. Gitea's tools went with them, which closed the usual path for merging the fix.
Then the console stopped answering the controller's verbs, though the controller held every
subscription: the console builds the list that tells a seat's verb from a module's tool by asking the
controller and the catalogue, and gives up on both when the catalogue does not answer — so
mesh-controller.status was asked of a module subject nobody serves.
Why it matters beyond this instance
Two properties, each worse than the mistake that exposed it:
- A runtime dies for an optional subscription. Announcing is discovery; serving tools and running provisioners is the work. A refusal of the first should never stop the second.
- The console's view of the mesh's own verbs depended on a module. The controller's verbs are how the operator repairs the mesh; they must not become unreachable because the catalogue is down.
Diagnosis
Owner mesh-tools. The wildcard is fixed on branch fix/announce-only-what-the-grants-allow: both
runtimes subscribe exactly what the grants allow. The console that discovers from what announces itself
(ADR 0195, 0197, on main) asks the bus and the controller, not the catalogue, which removes the second
property once it is deployed. Still to do: the TypeScript runtime treats a refused announcement
subscription as a logged warning, not as fatal; a test against a bus with real grants proves the
announcement subscriptions are allowed for every principal kind.
Recovered on the day without the forge's API: the toolchain images built by the controller straight from the fix branch, every module image rebuilt on them, and the machine pushed.
Resolved
A refused announcement is logged and the runtime serves on, and every runtime subscribes only the discovery subjects its grants allow. Proven 2026-10-04: after rolling out to every machine, no container restarts anywhere, and the discovery console lists no runtime as not answering. A refused tool subscription was made non-fatal the same way afterwards, under issue 218 (mesh-tools#46).