Merge pull request 'To-be 45: the bus module says every user's slow consumer, and S9 raises it (issue 270, ADR 0357)' (#352) from design/270-the-bus-module-says-every-users-slow-consumer into main

2026-10-14 07:00:14 +00:00
parent 40a39c8602 bb8075b37b
commit dcb5d5ce38
1 changed files with 23 additions and 8 deletions
@@ -2,8 +2,9 @@
 layer: to-be
 status: in-progress
 code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
-updated: 2026-10-12
+updated: 2026-10-14
 decisions:
+  - "ADR 0357"
  - "ADR 0325"
  - "ADR 0323"
  - "ADR 0318"
@@ -259,7 +260,7 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
 | S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
 | S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — |
 | S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
-| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
+| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted; every user's connection the server closed as a slow consumer or for breaking the bus's rules (ADR 0357) | bus server's advisory subjects; the controller's connection; the bus module's watch of its server (`bus.slow-consumer`, `bus.connection-closed`) | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `connection-closed` (one per bus user), `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
 | S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
 | S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
 | S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
@@ -274,10 +275,24 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
 The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
 controller translates each into a condition naming the thing in the mesh's words, as the refused reply
 of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer
-is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a
-system account the controller hears a slow consumer and a refused subject for its own connection only
-([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md),
-open question 2).
+is said only for one the mesh names and expects, and clears when it exists again.
+
+**Every user's slow consumer, from the bus module (ADR 0357).** The server closes every client it judges
+a slow consumer and keeps its user and reason in its record of closed connections; it publishes neither
+in the mesh's account. So the `nats` module's tool bundle reads that record on its own machine every 30
+seconds and publishes `bus.slow-consumer` for each connection closed as a slow consumer, and
+`bus.connection-closed` for each the server closed for something its client did against the bus's
+rules or limits — naming the bus user, the connection and the reason, never a credential. A closure by
+the client, by the network or by the server's own shutdown is not said: D15 says a user dropped in a
+loop. A closure — its user, connection id and opening time — is said at least once: again when publishing
+it failed, and again for the last hour when the bundle starts. The controller's watchdogs read the last
+hour of both events back from the events stream on every tick, count each closure once, and raise one
+condition per bus user and kind, naming the module and its machine; it clears an hour after the newest
+closure. A closure is a condition at most about a minute after the server closed it.
+
+**A refused subject is still heard for the controller's own connection only.** The server tells the
+client and writes its log, and says it nowhere else — not in its monitoring endpoint and not to a system
+account (issue 702, open).

 ## 4. The self-check: `doctor` (rule 6)

@@ -787,10 +802,10 @@ has a week of entries; the durations are recorded for every machine.

 | Repository | Delivers |
 |---|---|
-| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
+| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9), and every user's slow consumer and closed connection read back from the bus module's events (ADR 0357); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
 | `mesh-host` | the heartbeat carries its interval; D1's validator published as a package the controller imports |
 | mesh-tools | the tool runner's heartbeat (S11) |
-| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) |
+| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5); the `nats` module's watch of its server's closed connections, publishing `bus.slow-consumer` and `bus.connection-closed` (S9, ADR 0357) |
 | mesh-lab | R9: each signal suppressed in turn |

 **Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a