Give ADR 0357 a home in design 45: the bus module says every user's slow consumer, and S9 raises it (issue 270)
mesh/merge-gate head bb8075b37bc1 merged into trunk 40a39c860278 — pass: the change touches no module of the mesh's graph
mesh/repo-check head bb8075b37bc1 merged into trunk 40a39c860278 — pass: its merge-check.sh passed
mesh/delivery delivered

This commit is contained in:
jochen
2026-10-14 08:42:54 +02:00
parent 40a39c8602
commit bb8075b37b
@@ -2,8 +2,9 @@
layer: to-be
status: in-progress
code: [mesh-controller, mesh-host, mesh-tools, mesh-catalog, mesh-sdk, mesh-lab]
updated: 2026-10-12
updated: 2026-10-14
decisions:
- "ADR 0357"
- "ADR 0325"
- "ADR 0323"
- "ADR 0318"
@@ -259,7 +260,7 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
| S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
| S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; `doctor` 3 min; others 10 min | `call-hung` | warning | — |
| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted; every user's connection the server closed as a slow consumer or for breaking the bus's rules (ADR 0357) | bus server's advisory subjects; the controller's connection; the bus module's watch of its server (`bus.slow-consumer`, `bus.connection-closed`) | any | any occurrence; a message given up on for as long as `DEAD_LETTERS` holds it (ADR 0264) | `slow-consumer`, `connection-closed` (one per bus user), `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
| S11 | tool runner heartbeat | tool runner | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
| S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
@@ -274,10 +275,24 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
controller translates each into a condition naming the thing in the mesh's words, as the refused reply
of issue 265 is translated today. An advisory clears after an hour without another; a deleted consumer
is said only for one the mesh names and expects, and clears when it exists again. Until the bus has a
system account the controller hears a slow consumer and a refused subject for its own connection only
([issue 270](../../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md),
open question 2).
is said only for one the mesh names and expects, and clears when it exists again.
**Every user's slow consumer, from the bus module (ADR 0357).** The server closes every client it judges
a slow consumer and keeps its user and reason in its record of closed connections; it publishes neither
in the mesh's account. So the `nats` module's tool bundle reads that record on its own machine every 30
seconds and publishes `bus.slow-consumer` for each connection closed as a slow consumer, and
`bus.connection-closed` for each the server closed for something its client did against the bus's
rules or limits — naming the bus user, the connection and the reason, never a credential. A closure by
the client, by the network or by the server's own shutdown is not said: D15 says a user dropped in a
loop. A closure — its user, connection id and opening time — is said at least once: again when publishing
it failed, and again for the last hour when the bundle starts. The controller's watchdogs read the last
hour of both events back from the events stream on every tick, count each closure once, and raise one
condition per bus user and kind, naming the module and its machine; it clears an hour after the newest
closure. A closure is a condition at most about a minute after the server closed it.
**A refused subject is still heard for the controller's own connection only.** The server tells the
client and writes its log, and says it nowhere else — not in its monitoring endpoint and not to a system
account (issue 702, open).
## 4. The self-check: `doctor` (rule 6)
@@ -787,10 +802,10 @@ has a week of entries; the durations are recorded for every machine.
| Repository | Delivers |
|---|---|
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S11 and S13 with bounds set from Phase 0's durations (S12 is Phase 2's, S14 Phase 5's); the bus advisories subscribed and translated (S9), and every user's slow consumer and closed connection read back from the bus module's events (ADR 0357); `doctor` with D1–D4, D6–D10, DW and its heartbeat (§4; D5 is Phase 2's); `status` led by open conditions; the test generated from the signals table |
| `mesh-host` | the heartbeat carries its interval; D1's validator published as a package the controller imports |
| mesh-tools | the tool runner's heartbeat (S11) |
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) |
| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5); the `nats` module's watch of its server's closed connections, publishing `bus.slow-consumer` and `bus.connection-closed` (S9, ADR 0357) |
| mesh-lab | R9: each signal suppressed in turn |
**Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a