hq/01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md

# 01 — The evidence, classified by class of failure

Every issue report opened between 2026-09-30 and 2026-10-06 that bears on the mesh's core — the
controller, the machine host, the bus, the console, the build path — read in full and classified by
**the class of failure**, not by the component it was found in. A component view says "fix the host";
a class view says "the same thing is wrong in four places", which is what a principle is for.

## The count

| | |
|---|---|
| issue reports opened 2026-10-01 to 2026-10-06 | **92** (about fifteen a day) |
| of those (and a few from the days before), classified below as core failures | **48** distinct issues |
| classified in more than one class | 17 of 48 |
| a fault that **came back** after a fix of the same symptom | 4 chains: 200 → 265, 230 → 264, 257 → 261 → 267, 175 → 184 → 248 |
| noticed because a person or an agent looked — at a stalled plan, a wrong outcome, a journal, a test run by hand, a review | **48 of 48** |
| of those, the mesh's own answer carried the fact for whoever asked (a refusal, a `status` line, a push's output) | 4 (233, 244, 259, 263) |
| raised by the mesh to anyone, unasked | **0 of 48** |

The recurrences matter most. Each fix was correct for its instance and left the class standing, so the
same symptom came back through a different door days later. That is the measurement behind the
operator's mandate: point fixes are converging on the instances, not on the class.

## The classes

Nine classes, as the mandate frames them. The table under each is the evidence; *detected* is the time
from the fault's start to the moment anyone knew; *noticed by* is how.

### (a) Races: concurrent actors without an ordering

Two actors act on the same thing, and the order of arrival — not an explicit order — decides the outcome.

| Issue | The two actors | Detected | Noticed by |
|---|---|---|---|
| 204 | an outgoing and an incoming controller both sent declarations | 2 min | a person saw a module undone |
| 201 | a plan's push carried a controller digest older than its successor had written | 10 min crash loop | a person, nothing answered |
| 214 | the controller rebuilding itself; the outcome reached the old one or neither | 27 min | a person asked the plan twice |
| 219 | an older build finishing later replaced a newer one | hours | a person reading builds |
| 234 | seven declarations arrived during an eleven-minute apply; the newest was composed from a stale view and undeclared four modules | 8 min of removals | a person, the modules were gone |
| 254 | three plans for three merges ran at once, each asking the same builds | hours | a person, one plan stuck "building" |
| 256 | a first machine's report landed between a module's two sends and read as stale | 7 min | a person |
| 257, 261 | the machine's five-minute reconcile and a delivery took the apply lock in the wrong order | 5 min (257), 29 s visible undo (261) | a person |
| 267 | a reconcile's report queued behind a delivery's apply overtook it at the controller | until a hand push | a person |
| 265 | a push reloaded the bus's permissions while its own answer was still owed | 54 of 103 pushes over two days | a person, "did not answer in time" |

**What they share.** Every one is a receiver that kept *the last thing written* rather than *the
newest thing by an explicit order*. Declarations carry a sequence (issue 107); **reports do not**
(267 says so: "the report does not carry the declaration's sequence number, so the digest decides").
Plans are ordered by when they were made only since ADR 0218. Builds are ordered since issue 219.
Controllers have no epoch, so two instances can both act (204). Ordering was added one message kind at
a time, each after a race in it was seen.

### (b) Commands silently ignored, arguments dropped

| Issue | What was dropped | Effect |
|---|---|---|
| 244 | the console removed `node` from every mesh-seat verb's schema and call | `plan` could only refuse; **`push <one machine>` arrived empty and pushed every machine** |
| 259 | a named push's flush sent every machine a build a policy held back | a fault met on every machine at once, not one |
| 202 | a module whose setting was unset was *left out* of the machine | the resolver vanished from a declaration, no error |
| 188 | a refusal inside "who is on the network" dropped a machine | 40 min, every symptom pointed elsewhere |
| 231 | a misspelled placeholder written to a file as literal text | passes every check |
| 241 | an unreadable contributions file read as "nobody asks" | **seven databases dropped and recreated empty** |
| 255 | the journal verb read nothing and said "-- No entries --" | a refusal that reads as a quiet service |
| 246 | a runtime that answered late was treated as absent | the console said modules "run nowhere" |

**What they share.** A receiver that could not tell *nothing was asked* from *something was lost on the
way*, and chose a default. In 241 and 244 the default was the most destructive reading available.

### (c) Outcomes not fed back to the caller

| Issue | What the caller was told | What happened |
|---|---|---|
| 200, 265 | "did not answer in time" | the push ran; the answer was refused by the bus |
| 176 | the console's build tool neither waits nor registers | — |
| 229 | `plans` answers once in prose; nothing waits for a plan | an agent went round the mesh with `curl` |
| 230, 264 | a host stood aside for its successor and its report was cancelled | the plan waited for ever, reading `late: false` |
| 186 | the build machine dropped 26 of 43 asks; nothing counts asks against outcomes | inferred two hours later |
| 237 | `assign` answered "held" and "does not resolve for lack of it" in one breath | a person or agent would loop |

Since 2026-10-06 the controller answers within ten seconds and keeps every call's outcome under an id
(`calls`, issue 265). Read live the same night: **that log holds the last hundred calls in the
controller's memory**, so a controller restart — which every merge to the controller's own repository
causes — forgets every outcome it held. And `status`, a read-only verb, took **18 seconds** to answer,
twice in a row, so even the health question is answered only through the "still running, ask `calls`"
path.

### (d) Failures visible only as log lines

| Issue | Where it was said | For how long |
|---|---|---|
| 179 (recurred) | the identity provider's journal, every five seconds | **23 hours**, about 31 000 refused logins |
| 184 | the controller's log: slow consumer, heartbeats dropped | 24 min deaf |
| 187 | five faults in one day, each found by reading a container's log hours later | hours each |
| 183, 217, 265 | a `Permissions Violation` line from the bus client library | days |
| 233 | `status` said `refused`, correctly; nothing said it had lasted | 1.5 h |
| 243 | nothing: machines silently ignored lower licence generations | until a login waited three minutes |
| 248 | the controller's event loop stopped logging at 15:17 | hours; "status showed every plan done" |
| 266 | nothing: a merge was skipped by the bus | **23 unmatched merges over three days** |
| 238 | a ban list of 400 entries | the operator's own address banned for four weeks |

`status` printed its all-well sentence through 179, 248 and 266. ADR 0224 made the first of those break
it. The other two have no signal that `status` reads.

### (e) State that two writers own

| Issue | The two writers |
|---|---|
| 190, 222 | the runtime's configuration written by modules that are not the runtime, and by the controller |
| 201 | the controller seat's row written by a successor, read by a predecessor pushed back in |
| 239 | two definitions, in two repositories, held one module name |
| 245 | "behind" answered by a commit comparison beside the plan that already knows |
| 257, 261, 267 | the machine's state written by both the delivery and the five-minute reconcile |
| 250 | a merge announced by the forge's tool and by its poll |
| 179 | the identity provider's admin password: the mesh minted one, the database kept another |

The operator's direction on 245 is the principle in their own words: *"a second answer to the same
question is how the two came to disagree."*

### (f) Manual repair needed

Counted from the reports' own "what unblocked it" sections:

| Repair by hand | Issues | Times |
|---|---|---|
| a push by hand to unstick a plan waiting on a report | 230, 257, 264, 267 | at least 4 |
| a controller restart to recreate a missing object or let go of a held message | 208, 248 | 2 |
| a one-off program run as the controller, outside the service | 201, 248 | 2 |
| the identity provider's admin reset through its bootstrap command | 179 | 2 |
| a kept file restored on a machine by hand | 233 | 1 |
| a stuck plan closed by hand | 214, 254 | 2 |
| a ban lifted by hand | 238 | 1 |
| a consumer remade from now | 248 | 1 |

ADR 0224 (*detected automatically, repaired where safe, loud where not*) is the first rule that turns
one of these into a mechanism. 248's `broker consumer-reset` and 254's `plans close` turned two into
verbs a person runs. Every other row is still a hand on a machine.

### (g) Self-upgrade fragility

The core updates itself: the controller rebuilds and replaces itself, the host delivers its own
successor (ADR 0141), the runtime and the console are modules, and the bus is a module on the control
node.

| Issue | What the self-upgrade broke |
|---|---|
| 201 | the controller pushed back to an older build than its own successor's row |
| 204 | two controllers both sending during a handover |
| 213, 223 | the controller ran as a container, and a new mesh installed it so |
| 214 | the plan that rebuilds the controller lost track of it |
| 230, 264 | the host that stands aside loses the report of the apply that delivered it (fixed twice) |
| 245 | a rebuild of everything replaced the bus's container: **every runtime lost the bus for a minute** |
| 248, 266 | a controller restart is where merges go missing: most of 266's 23 lie in such windows |
| 217 | a refused announcement crash-looped nine runtimes, closing the path that would merge the fix |

A machine's first-in-line rollout (ADR 0218) protects modules. It does not protect the core from itself:
the health a plan waits for is "reported applied", which a controller that cannot plan, a host that
cannot report or a bus that drops messages can each satisfy. Nothing rolls back. The recovery in 201
was the mesh's own binary run by hand from the newer image.

### (h) Checks that pass in CI and fail live

| Issue | The environmental fact the check did not have |
|---|---|
| 177 | the store-backed tests are skipped by the quick check, and the mesh runs none of a module's tests |
| 236 | the host's declaration validation is not run by the catalogue check |
| 262 | musl takes an NXDOMAIN for IPv6 as final; glibc does not |
| 263 | the real machine names make a consumer's identity 23–26 characters against a bound of 20 |
| 202 | the controller's test against the real catalogue, run by nobody until that day |
| 228 | the host's removal has no case for a `user` — found by a review, not a test |

Each check was right about the world it was given. None of them was given the mesh's world: its machine
names, its catalogue, its host's validation, its C libraries.

### (i) Third-party bugs

| Issue | |
|---|---|
| 266 | the bus server's 2.10 release skips messages on a consumer with several filter subjects |
| 265 | a reload of the bus's authorization forgets every reply permission already granted (documented server behaviour, read from its source) |
| 262 | a resolver answering NXDOMAIN where NODATA is correct, met by musl's stricter reading |

The lesson of 266 is not "upgrade the bus" — it is that nothing compared *what was announced* with
*what was acted on*, so a dependency's bug was silent for three days. A defence in depth (watch the
outcome, not the transport) would have caught it whichever layer was wrong.

## What would have prevented or caught each class

| Class | Would have been prevented by | Would have been caught by |
|---|---|---|
| (a) races | every message ordered by its writer, stale refused by every receiver | a lab replay of the interleaving |
| (b) dropped | refusing an unknown or unreadable input by name | a schema walk over every verb |
| (c) no feedback | answer at once with an id; outcome kept durably | a watchdog on "asked and never finished" |
| (d) logs only | — | a condition in `status` and a notification |
| (e) two writers | one writer per piece of state | a registry of writers checked at composition |
| (f) manual repair | a healer for every repair done twice | a counter of hand acts |
| (g) self-upgrade | one machine first, health-gated, rolled back | a lab upgrade with a deliberately broken build |
| (h) CI vs live | checks fed the real mesh's facts | the same, before merge |
| (i) third party | pinning and testing the version that runs | an end-to-end count of announced vs acted |

The two columns are the principles of [02](02-principles.md). Read by count, **the "caught by" column
is the cheapest and widest**: a watchdog and a condition would have shortened most of the 48 from "a
person noticed" to minutes, whatever the cause.