From bb8075b37bc1a7f232d9534f8b4032ed8eaea056 Mon Sep 17 00:00:00 2001 From: jochen Date: Wed, 14 Oct 2026 08:42:54 +0200 Subject: [PATCH] Give ADR 0357 a home in design 45: the bus module says every user's slow consumer, and S9 raises it (issue 270) --- .../45-a-core-that-cannot-fail-silently.md | 31 ++++++++++++++----- 1 file changed, 23 insertions(+), 8 deletions(-) diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index 2bee12f4..2aee0d1b 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -2,8 +2,9 @@ layer: to-be status: in-progress code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab] -updated: 2026-10-12 +updated: 2026-10-14 decisions: + - "ADR 0357" - "ADR 0325" - "ADR 0323" - "ADR 0318" @@ -259,7 +260,7 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age | S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — | | S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — | | S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — | -| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` | +| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted; every user's connection the server closed as a slow consumer or for breaking the bus's rules (ADR 0357) | bus server's advisory subjects; the controller's connection; the bus module's watch of its server (`bus.slow-consumer`, `bus.connection-closed`) | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `connection-closed` (one per bus user), `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` | | S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — | | S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — | | S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — | @@ -274,10 +275,24 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age The bus advisories (S9) cost one read-only subscription: the server already publishes them. The controller translates each into a condition naming the thing in the mesh's words, as the refused reply of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer -is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a -system account the controller hears a slow consumer and a refused subject for its own connection only -([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md), -open question 2). +is said only for one the mesh names and expects, and clears when it exists again. + +**Every user's slow consumer, from the bus module (ADR 0357).** The server closes every client it judges +a slow consumer and keeps its user and reason in its record of closed connections; it publishes neither +in the mesh's account. So the `nats` module's tool bundle reads that record on its own machine every 30 +seconds and publishes `bus.slow-consumer` for each connection closed as a slow consumer, and +`bus.connection-closed` for each the server closed for something its client did against the bus's +rules or limits — naming the bus user, the connection and the reason, never a credential. A closure by +the client, by the network or by the server's own shutdown is not said: D15 says a user dropped in a +loop. A closure — its user, connection id and opening time — is said at least once: again when publishing +it failed, and again for the last hour when the bundle starts. The controller's watchdogs read the last +hour of both events back from the events stream on every tick, count each closure once, and raise one +condition per bus user and kind, naming the module and its machine; it clears an hour after the newest +closure. A closure is a condition at most about a minute after the server closed it. + +**A refused subject is still heard for the controller's own connection only.** The server tells the +client and writes its log, and says it nowhere else — not in its monitoring endpoint and not to a system +account (issue 702, open). ## 4. The self-check: `doctor` (rule 6) @@ -787,10 +802,10 @@ has a week of entries; the durations are recorded for every machine. | Repository | Delivers | |---|---| -| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table | +| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9), and every user's slow consumer and closed connection read back from the bus module's events (ADR 0357); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table | | `mesh-host` | the heartbeat carries its interval; D1's validator published as a package the controller imports | | mesh-tools | the tool runner's heartbeat (S11) | -| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) | +| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5); the `nats` module's watch of its server's closed connections, publishing `bus.slow-consumer` and `bus.connection-closed` (S9, ADR 0357) | | mesh-lab | R9: each signal suppressed in turn | **Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a