diff --git a/04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md b/04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md new file mode 100644 index 0000000..ea101f2 --- /dev/null +++ b/04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md @@ -0,0 +1,69 @@ +--- +status: located +opened: 2026-10-06 +located-in: [mesh-controller] +fixed-by: +amended-design: +--- + +# 270. Phase 1 watches signals that come later, and hears only the controller's own bus faults + +## Symptom + +Found building Phase 1 of [to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) +on 2026-10-06. The phase table gives the controller "watchdogs for S1–S10 and S12–S14" and "`doctor` +with D1–D10". Four of those cannot be built in Phase 1, because what they watch does not exist until a +later phase, and one is built to half of what the signals table says: + +| Row | What the design asks | Why it cannot be done in Phase 1 | +|---|---|---| +| S12 *the controller lease renewed* | a watchdog on the lease's renewal | the lease is Phase 2 (§6); there is nothing renewed | +| S14 *facts snapshot exported* | a watchdog on a daily export | the facts snapshot is Phase 5 (§9); nothing exports one | +| D5 *exactly one lease holder, no stale epoch* | a probe of the lease | Phase 2, as S12 | +| S13 *stale refusals, naming the writer* | more than 5 refusals from one writer in 5 min | the node-engine refuses a stale declaration today, but no declaration carries its writer's epoch until Phase 2: the watchdog counts refusals per **machine** and cannot name the writer | +| S9 *slow consumer, permission violation* | every principal's | the bus has one account and no system account, and the server says these of other principals only to a system account: the controller hears them for **its own connection** (the client library is told) and the account-wide JetStream advisories (maximum deliveries, consumer deleted) — not a module's slow consumer or a module refused a subject | + +Built as each phase's dependency allows, the rows are in the compiled signals table and the probe +registry marked deferred, each with its reason and phase; the test generated from the table refuses a +deferred row that is watched or a watched row without a suppression. Nothing is silent about them — +but the design says Phase 1 delivers them, and a reader of the design believes it. + +## Decided while building, and not written in the design + +1. **The condition history is its own bucket**, `mesh-controller_condition-history`, kept ninety days. + A bucket has one age for every key; the open conditions must have none, or a condition open longer + than the history would vanish while still true. The writers table names only + `mesh-controller_conditions`. +2. **The condition events carry the condition at the top level**, with `event`, `at`, `change` + (raised, reopened, severity, resolver, silenced, silence-ended, cleared), `why`, `was`, `cleared` + and `show` beside it — the shape the operator-channel's holder was written against. A silence and + its ending are `condition-changed`; a reopening within ten minutes is `condition-raised` with change + `reopened`. The self-check's heartbeat is the seat's event `doctor-heartbeat`. +3. **A key's last token is the kind's short word** where §2's examples give one — + `provider....failing` for `provider-failing`, `core.controller.deaf` — and + the kind elsewhere. A key is opaque to every reader; the kind is the field. +4. **A probe the registry did not have, DW: the watchdogs ran within three ticks.** Rule 6 has the + self-check watched from a second machine; the watchdogs themselves had no watcher. The self-check + watches them, and they raise S10 when the self-check stops. +5. **A deleted consumer is said only for a consumer the mesh names** (the controller's, a machine's + declaration consumer, `_`, a seat's worker), and only while the mesh expects it and it + is missing. Every watch of a bucket and every read-back of a stream makes and deletes a consumer of + its own, and the server advises each deletion — five a tick, found running the controller against + a real bus. +6. **Bounds the design leaves to the build**, provisional like the rest: S6, no build timeout is + declared, so max(20 min, 3 × the p90 of measured builds), one hour while nothing is measured; S7, + per verb (push, rotate and command 30 min; assign and unassign 15 min; doctor 3 min; others 10); + S9, an advisory clears after an hour with no other; D6, a consumer more than 1000 messages from its + stream's head is behind. +7. **ADR 0224's standing moved into the store without importing what the old table held.** A + provider says its failing word again every fifteen minutes; the table `provider_standing` is no + longer read or written and is left in place, because dropping it is a removal of data and that is + the operator's word to give. + +## Open questions + +1. Does Phase 1's table move S12, D5 to Phase 2 and S14 to Phase 5, and S13's naming of the writer to + Phase 2? (The build assumes yes.) +2. Does the bus gain a system account — or the controller read the server's monitoring endpoint — so + S9 hears every principal's slow consumer and refused subject? Either changes the foundation's shape. +3. Do decisions 1–6 above go into §1–§4 as written, through an amendment of to-be 45?