From 08cea893bd199bee89c5e05d4f42bd442b74ed00 Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 3 Oct 2026 22:47:28 +0200 Subject: [PATCH] Issue 217: a refused announcement took down every container's runtime, and the console with it --- .../00-report.md | 52 +++++++++++++++++++ 1 file changed, 52 insertions(+) create mode 100644 04-ISSUES/217-a-refused-announcement-took-down-every-containers-runtime/00-report.md diff --git a/04-ISSUES/217-a-refused-announcement-took-down-every-containers-runtime/00-report.md b/04-ISSUES/217-a-refused-announcement-took-down-every-containers-runtime/00-report.md new file mode 100644 index 0000000..fa602c8 --- /dev/null +++ b/04-ISSUES/217-a-refused-announcement-took-down-every-containers-runtime/00-report.md @@ -0,0 +1,52 @@ +--- +status: located +opened: 2026-10-03 +located-in: + - mesh-tools +fixed-by: +amended-design: +--- + +# 217 — A refused announcement took down every container's runtime, and the console with it + +## What was observed + +2026-10-03, rolling out [ADR 0197](../../02-DECISIONS/0197-every-tool-announces-itself-on-the-bus-in-the-nats-services-protocol.md). +The tool runtimes announced themselves by subscribing `$SRV..>`; the grants composed for them +allowed only `$SRV.` and the service's own name and instance. The bus refused the wildcard: + +``` +Subscription Violation - User "novox.gitea", Subject "$SRV.PING.>" +NatsError: 'Permissions Violation for Subscription to "$SRV.PING.>"' +``` + +The TypeScript runtime the per-module containers run treats a refused subscription as fatal, so on +the one machine that had received the new images nine containers crash-looped — gitea's runtime, +postgres's, the catalogue, the vault, mongodb, mssql, keycloak, mailu, nextcloud. Gitea's tools went +with them, which closed the usual path for merging the fix. + +Then the console stopped answering the controller's verbs, though the controller held every +subscription: the console builds the list that tells a seat's verb from a module's tool by asking the +controller *and* the catalogue, and gives up on both when the catalogue does not answer — so +`mesh-controller.status` was asked of a module subject nobody serves. + +## Why it matters beyond this instance + +Two properties, each worse than the mistake that exposed it: + +- **A runtime dies for an optional subscription.** Announcing is discovery; serving tools and running + provisioners is the work. A refusal of the first should never stop the second. +- **The console's view of the mesh's own verbs depended on a module.** The controller's verbs are how + the operator repairs the mesh; they must not become unreachable because the catalogue is down. + +## Diagnosis + +Owner mesh-tools. The wildcard is fixed on branch `fix/announce-only-what-the-grants-allow`: both +runtimes subscribe exactly what the grants allow. The console that discovers from what announces itself +(ADR 0195, 0197, on main) asks the bus and the controller, not the catalogue, which removes the second +property once it is deployed. **Still to do:** the TypeScript runtime treats a refused announcement +subscription as a logged warning, not as fatal; a test against a bus with real grants proves the +announcement subscriptions are allowed for every principal kind. + +Recovered on the day without the forge's API: the toolchain images built by the controller straight +from the fix branch, every module image rebuilt on them, and the machine pushed.