Issue 187: the mesh tells nobody when it stops working #256
@@ -0,0 +1,61 @@
|
||||
---
|
||||
status: open
|
||||
opened: 2026-10-01
|
||||
located-in: []
|
||||
fixed-by:
|
||||
amended-design: []
|
||||
---
|
||||
|
||||
# 187 — The mesh tells nobody when it stops working
|
||||
|
||||
## What was observed
|
||||
|
||||
One day, 2026-10-01, and five faults, each found by a person reading a container's log hours after
|
||||
it began, and each invisible to every surface the mesh offers:
|
||||
|
||||
- The controller's receive loop was blocked for twenty-four minutes by a merge handler, then for
|
||||
nineteen minutes by a publish the bus had refused ([184](../184-a-merge-announcement-blocks-the-controllers-receive-loop/00-report.md),
|
||||
[185](../185-a-refused-membership-publish-stops-the-controller/00-report.md)). Throughout, `status`
|
||||
answered and read as quiet, `builds` listed what it had, the console answered every tool. Nothing
|
||||
said *the controller has taken nothing in since 13:17*.
|
||||
- The build machine dropped twenty-six of forty-three asks ([186](../186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md)).
|
||||
Nothing counts asks against builds; the queue read as empty; the loss was inferred two hours later
|
||||
from a wave that would not finish.
|
||||
- The controller's own grant refused every membership it published ([183](../183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md)),
|
||||
and then every module on the new runtime was refused its one read of the stream. Both were one
|
||||
`Publish Violation` line each in the bus's log, which nothing in the mesh reads.
|
||||
- The bus dropped the machines' heartbeats as a slow consumer, twice, and said so to the controller's
|
||||
log only.
|
||||
|
||||
In every case the designed fallback held — runtimes served the derived shape, a push later carried
|
||||
what an earlier one had not — which is why the mesh kept working and why nobody was told.
|
||||
|
||||
## Why this is here
|
||||
|
||||
The repository's own rule is that a rule states how it is checked, and every record here does. But
|
||||
the checks are tests and `status`, both asked by a person. The mesh has no account of its own
|
||||
liveness: whether the controller is hearing, whether the queue is moving, whether the bus is
|
||||
refusing what the mesh composed, how old each machine's last report is. A fault that leaves the
|
||||
fallbacks standing is a fault nobody learns about until it compounds, and today three of them
|
||||
compounded into an evening of reading logs. This is the design permitting a failure to be silent,
|
||||
which is the first line of what belongs here.
|
||||
|
||||
## What a decision would settle
|
||||
|
||||
- **What the mesh observes about itself.** At least: the receive loop's last message taken and its
|
||||
age; asks against builds, with the oldest unbuilt ask's age; publishes the bus refused and
|
||||
subscriptions it dropped, read from the bus rather than from a log; each machine's last report
|
||||
and heartbeat age; a module whose runtime says it serves the derived shape.
|
||||
- **Where it says so.** As events on the bus under the controller's seat, so a log viewer and a
|
||||
notifier are consumers and not special cases; and in `status`, which must go red for any of them
|
||||
rather than listing only machines that are behind.
|
||||
- **Who is told.** A channel a person actually reads — the mesh already has modules that send
|
||||
mail and messages — chosen once, with a rule for what interrupts a person and what waits for
|
||||
`status`.
|
||||
- **What is not a monitor.** Nothing here is a dashboard product the mesh adopts; it is the mesh
|
||||
stating facts about itself, the way [ADR 0134](../../02-DECISIONS/0134-the-mesh-says-what-it-applied.md)
|
||||
made it state what it did.
|
||||
|
||||
*How this would be checked:* a controller test where the loop is held and `status` goes red
|
||||
naming the age; a test where an ask is unbuilt past a bound and `builds` says so; live, the next
|
||||
fault of today's kinds reaches a person before a person reaches the log.
|
||||
Reference in New Issue
Block a user