Issue 187: the mesh tells nobody when it stops working #256
@@ -0,0 +1,61 @@
|
|||||||
|
---
|
||||||
|
status: open
|
||||||
|
opened: 2026-10-01
|
||||||
|
located-in: []
|
||||||
|
fixed-by:
|
||||||
|
amended-design: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# 187 — The mesh tells nobody when it stops working
|
||||||
|
|
||||||
|
## What was observed
|
||||||
|
|
||||||
|
One day, 2026-10-01, and five faults, each found by a person reading a container's log hours after
|
||||||
|
it began, and each invisible to every surface the mesh offers:
|
||||||
|
|
||||||
|
- The controller's receive loop was blocked for twenty-four minutes by a merge handler, then for
|
||||||
|
nineteen minutes by a publish the bus had refused ([184](../184-a-merge-announcement-blocks-the-controllers-receive-loop/00-report.md),
|
||||||
|
[185](../185-a-refused-membership-publish-stops-the-controller/00-report.md)). Throughout, `status`
|
||||||
|
answered and read as quiet, `builds` listed what it had, the console answered every tool. Nothing
|
||||||
|
said *the controller has taken nothing in since 13:17*.
|
||||||
|
- The build machine dropped twenty-six of forty-three asks ([186](../186-a-release-across-repositories-is-an-order-in-a-persons-head/00-report.md)).
|
||||||
|
Nothing counts asks against builds; the queue read as empty; the loss was inferred two hours later
|
||||||
|
from a wave that would not finish.
|
||||||
|
- The controller's own grant refused every membership it published ([183](../183-the-controller-could-not-publish-the-memberships-it-issued/00-report.md)),
|
||||||
|
and then every module on the new runtime was refused its one read of the stream. Both were one
|
||||||
|
`Publish Violation` line each in the bus's log, which nothing in the mesh reads.
|
||||||
|
- The bus dropped the machines' heartbeats as a slow consumer, twice, and said so to the controller's
|
||||||
|
log only.
|
||||||
|
|
||||||
|
In every case the designed fallback held — runtimes served the derived shape, a push later carried
|
||||||
|
what an earlier one had not — which is why the mesh kept working and why nobody was told.
|
||||||
|
|
||||||
|
## Why this is here
|
||||||
|
|
||||||
|
The repository's own rule is that a rule states how it is checked, and every record here does. But
|
||||||
|
the checks are tests and `status`, both asked by a person. The mesh has no account of its own
|
||||||
|
liveness: whether the controller is hearing, whether the queue is moving, whether the bus is
|
||||||
|
refusing what the mesh composed, how old each machine's last report is. A fault that leaves the
|
||||||
|
fallbacks standing is a fault nobody learns about until it compounds, and today three of them
|
||||||
|
compounded into an evening of reading logs. This is the design permitting a failure to be silent,
|
||||||
|
which is the first line of what belongs here.
|
||||||
|
|
||||||
|
## What a decision would settle
|
||||||
|
|
||||||
|
- **What the mesh observes about itself.** At least: the receive loop's last message taken and its
|
||||||
|
age; asks against builds, with the oldest unbuilt ask's age; publishes the bus refused and
|
||||||
|
subscriptions it dropped, read from the bus rather than from a log; each machine's last report
|
||||||
|
and heartbeat age; a module whose runtime says it serves the derived shape.
|
||||||
|
- **Where it says so.** As events on the bus under the controller's seat, so a log viewer and a
|
||||||
|
notifier are consumers and not special cases; and in `status`, which must go red for any of them
|
||||||
|
rather than listing only machines that are behind.
|
||||||
|
- **Who is told.** A channel a person actually reads — the mesh already has modules that send
|
||||||
|
mail and messages — chosen once, with a rule for what interrupts a person and what waits for
|
||||||
|
`status`.
|
||||||
|
- **What is not a monitor.** Nothing here is a dashboard product the mesh adopts; it is the mesh
|
||||||
|
stating facts about itself, the way [ADR 0134](../../02-DECISIONS/0134-the-mesh-says-what-it-applied.md)
|
||||||
|
made it state what it did.
|
||||||
|
|
||||||
|
*How this would be checked:* a controller test where the loop is held and `status` goes red
|
||||||
|
naming the age; a test where an ask is unbuilt past a bound and `builds` says so; live, the next
|
||||||
|
fault of today's kinds reaches a person before a person reaches the log.
|
||||||
Reference in New Issue
Block a user