hq/01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md

01 — The evidence, classified by class of failure

Every issue report opened between 2026-09-30 and 2026-10-06 that bears on the mesh's core — the controller, the machine host, the bus, the console, the build path — read in full and classified by the class of failure, not by the component it was found in. A component view says "fix the host"; a class view says "the same thing is wrong in four places", which is what a principle is for.

The count

issue reports opened 2026-10-01 to 2026-10-06	92 (about fifteen a day)
of those (and a few from the days before), classified below as core failures	48 distinct issues
classified in more than one class	17 of 48
a fault that came back after a fix of the same symptom	4 chains: 200 → 265, 230 → 264, 257 → 261 → 267, 175 → 184 → 248
noticed because a person or an agent looked — at a stalled plan, a wrong outcome, a journal, a test run by hand, a review	48 of 48
of those, the mesh's own answer carried the fact for whoever asked (a refusal, a status line, a push's output)	4 (233, 244, 259, 263)
raised by the mesh to anyone, unasked	0 of 48

The recurrences matter most. Each fix was correct for its instance and left the class standing, so the same symptom came back through a different door days later. That is the measurement behind the operator's mandate: point fixes are converging on the instances, not on the class.

The classes

Nine classes, as the mandate frames them. The table under each is the evidence; detected is the time from the fault's start to the moment anyone knew; noticed by is how.

(a) Races: concurrent actors without an ordering

Two actors act on the same thing, and the order of arrival — not an explicit order — decides the outcome.

Issue	The two actors	Detected	Noticed by
204	an outgoing and an incoming controller both sent declarations	2 min	a person saw a module undone
201	a plan's push carried a controller digest older than its successor had written	10 min crash loop	a person, nothing answered
214	the controller rebuilding itself; the outcome reached the old one or neither	27 min	a person asked the plan twice
219	an older build finishing later replaced a newer one	hours	a person reading builds
234	seven declarations arrived during an eleven-minute apply; the newest was composed from a stale view and undeclared four modules	8 min of removals	a person, the modules were gone
254	three plans for three merges ran at once, each asking the same builds	hours	a person, one plan stuck "building"
256	a first machine's report landed between a module's two sends and read as stale	7 min	a person
257, 261	the machine's five-minute reconcile and a delivery took the apply lock in the wrong order	5 min (257), 29 s visible undo (261)	a person
267	a reconcile's report queued behind a delivery's apply overtook it at the controller	until a hand push	a person
265	a push reloaded the bus's permissions while its own answer was still owed	54 of 103 pushes over two days	a person, "did not answer in time"

What they share. Every one is a receiver that kept the last thing written rather than the newest thing by an explicit order. Declarations carry a sequence (issue 107); reports do not (267 says so: "the report does not carry the declaration's sequence number, so the digest decides"). Plans are ordered by when they were made only since ADR 0218. Builds are ordered since issue 219. Controllers have no epoch, so two instances can both act (204). Ordering was added one message kind at a time, each after a race in it was seen.

(b) Commands silently ignored, arguments dropped

Issue	What was dropped	Effect
244	the console removed node from every mesh-seat verb's schema and call	plan could only refuse; push <one machine> arrived empty and pushed every machine
259	a named push's flush sent every machine a build a policy held back	a fault met on every machine at once, not one
202	a module whose setting was unset was left out of the machine	the resolver vanished from a declaration, no error
188	a refusal inside "who is on the network" dropped a machine	40 min, every symptom pointed elsewhere
231	a misspelled placeholder written to a file as literal text	passes every check
241	an unreadable contributions file read as "nobody asks"	seven databases dropped and recreated empty
255	the journal verb read nothing and said "-- No entries --"	a refusal that reads as a quiet service
246	a runtime that answered late was treated as absent	the console said modules "run nowhere"

What they share. A receiver that could not tell nothing was asked from something was lost on the way, and chose a default. In 241 and 244 the default was the most destructive reading available.

(c) Outcomes not fed back to the caller

Issue	What the caller was told	What happened
200, 265	"did not answer in time"	the push ran; the answer was refused by the bus
176	the console's build tool neither waits nor registers	—
229	plans answers once in prose; nothing waits for a plan	an agent went round the mesh with curl
230, 264	a host stood aside for its successor and its report was cancelled	the plan waited for ever, reading late: false
186	the build machine dropped 26 of 43 asks; nothing counts asks against outcomes	inferred two hours later
237	assign answered "held" and "does not resolve for lack of it" in one breath	a person or agent would loop

Since 2026-10-06 the controller answers within ten seconds and keeps every call's outcome under an id (calls, issue 265). Read live the same night: that log holds the last hundred calls in the controller's memory, so a controller restart — which every merge to the controller's own repository causes — forgets every outcome it held. And status, a read-only verb, took 18 seconds to answer, twice in a row, so even the health question is answered only through the "still running, ask calls" path.

(d) Failures visible only as log lines

Issue	Where it was said	For how long
179 (recurred)	the identity provider's journal, every five seconds	23 hours, about 31 000 refused logins
184	the controller's log: slow consumer, heartbeats dropped	24 min deaf
187	five faults in one day, each found by reading a container's log hours later	hours each
183, 217, 265	a Permissions Violation line from the bus client library	days
233	status said refused, correctly; nothing said it had lasted	1.5 h
243	nothing: machines silently ignored lower licence generations	until a login waited three minutes
248	the controller's event loop stopped logging at 15:17	hours; "status showed every plan done"
266	nothing: a merge was skipped by the bus	23 unmatched merges over three days
238	a ban list of 400 entries	the operator's own address banned for four weeks

status printed its all-well sentence through 179, 248 and 266. ADR 0224 made the first of those break it. The other two have no signal that status reads.

(e) State that two writers own

Issue	The two writers
190, 222	the runtime's configuration written by modules that are not the runtime, and by the controller
201	the controller seat's row written by a successor, read by a predecessor pushed back in
239	two definitions, in two repositories, held one module name
245	"behind" answered by a commit comparison beside the plan that already knows
257, 261, 267	the machine's state written by both the delivery and the five-minute reconcile
250	a merge announced by the forge's tool and by its poll
179	the identity provider's admin password: the mesh minted one, the database kept another

The operator's direction on 245 is the principle in their own words: "a second answer to the same question is how the two came to disagree."

(f) Manual repair needed

Counted from the reports' own "what unblocked it" sections:

Repair by hand	Issues	Times
a push by hand to unstick a plan waiting on a report	230, 257, 264, 267	at least 4
a controller restart to recreate a missing object or let go of a held message	208, 248	2
a one-off program run as the controller, outside the service	201, 248	2
the identity provider's admin reset through its bootstrap command	179	2
a kept file restored on a machine by hand	233	1
a stuck plan closed by hand	214, 254	2
a ban lifted by hand	238	1
a consumer remade from now	248	1

ADR 0224 (detected automatically, repaired where safe, loud where not) is the first rule that turns one of these into a mechanism. 248's broker consumer-reset and 254's plans close turned two into verbs a person runs. Every other row is still a hand on a machine.

(g) Self-upgrade fragility

The core updates itself: the controller rebuilds and replaces itself, the host delivers its own successor (ADR 0141), the runtime and the console are modules, and the bus is a module on the control node.

Issue	What the self-upgrade broke
201	the controller pushed back to an older build than its own successor's row
204	two controllers both sending during a handover
213, 223	the controller ran as a container, and a new mesh installed it so
214	the plan that rebuilds the controller lost track of it
230, 264	the host that stands aside loses the report of the apply that delivered it (fixed twice)
245	a rebuild of everything replaced the bus's container: every runtime lost the bus for a minute
248, 266	a controller restart is where merges go missing: most of 266's 23 lie in such windows
217	a refused announcement crash-looped nine runtimes, closing the path that would merge the fix

A machine's first-in-line rollout (ADR 0218) protects modules. It does not protect the core from itself: the health a plan waits for is "reported applied", which a controller that cannot plan, a host that cannot report or a bus that drops messages can each satisfy. Nothing rolls back. The recovery in 201 was the mesh's own binary run by hand from the newer image.

(h) Checks that pass in CI and fail live

Issue	The environmental fact the check did not have
177	the store-backed tests are skipped by the quick check, and the mesh runs none of a module's tests
236	the host's declaration validation is not run by the catalogue check
262	musl takes an NXDOMAIN for IPv6 as final; glibc does not
263	the real machine names make a consumer's identity 23–26 characters against a bound of 20
202	the controller's test against the real catalogue, run by nobody until that day
228	the host's removal has no case for a user — found by a review, not a test

Each check was right about the world it was given. None of them was given the mesh's world: its machine names, its catalogue, its host's validation, its C libraries.

(i) Third-party bugs

Issue
266	the bus server's 2.10 release skips messages on a consumer with several filter subjects
265	a reload of the bus's authorization forgets every reply permission already granted (documented server behaviour, read from its source)
262	a resolver answering NXDOMAIN where NODATA is correct, met by musl's stricter reading

The lesson of 266 is not "upgrade the bus" — it is that nothing compared what was announced with what was acted on, so a dependency's bug was silent for three days. A defence in depth (watch the outcome, not the transport) would have caught it whichever layer was wrong.

What would have prevented or caught each class

Class	Would have been prevented by	Would have been caught by
(a) races	every message ordered by its writer, stale refused by every receiver	a lab replay of the interleaving
(b) dropped	refusing an unknown or unreadable input by name	a schema walk over every verb
(c) no feedback	answer at once with an id; outcome kept durably	a watchdog on "asked and never finished"
(d) logs only	—	a condition in status and a notification
(e) two writers	one writer per piece of state	a registry of writers checked at composition
(f) manual repair	a healer for every repair done twice	a counter of hand acts
(g) self-upgrade	one machine first, health-gated, rolled back	a lab upgrade with a deliberately broken build
(h) CI vs live	checks fed the real mesh's facts	the same, before merge
(i) third party	pinning and testing the version that runs	an end-to-end count of announced vs acted

The two columns are the principles of 02. Read by count, the "caught by" column is the cheapest and widest: a watchdog and a condition would have shortened most of the 48 from "a person noticed" to minutes, whatever the cause.