Files
hq/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md
T
jochen 0f163a3e7c Research 031 graduates: ADR 0227 and to-be 45
The operator approved the conclusion: the core must say when it is wrong
instead of waiting to be looked at. Record the nine rules with their
checks, and specify the parts and the six phases a build can take.
2026-10-06 02:31:21 +02:00

4.1 KiB

status, initiated, touches, became
status initiated touches became
graduated 2026-10-06
00-META/how-we-build.md
01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md
01-RESEARCH/028-the-meshs-output-channel/00-overview.md
01-RESEARCH/019-a-warm-twin-of-the-running-mesh/00-overview.md
02-DECISIONS/0141-the-host-delivers-its-own-successor.md
02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md
02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md
02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md
03-DESIGN/01-to-be/06-the-controller.md
03-DESIGN/01-to-be/09-the-node-lifecycle.md
03-DESIGN/01-to-be/25-the-bus-on-nats.md
03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md
02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md

031 — A core that cannot fail silently

What. The principles the mesh's core must hold — the controller, the machine host, the bus, the console and the path a change takes through them — so that it is fully diagnosable, monitors itself, heals what it knows how to heal, and upgrades itself without a person standing by. And the mechanisms and the order in which to build them.

Why. The operator, 2026-10-06: "I still notice a lot of race issues, and commands being ignored, or no feedback, no logs, no monitoring. Our mesh core setup must be fully diagnosable, with active monitoring, self-healing, self-upgradeable, self-monitoring. The core principles must be very sturdy, no ambiguities, clear plan of execution, fail-proof setup."

The record bears it out. In the six days to 2026-10-06, 92 issue reports were opened. Of the 48 read here as core failures, every one was noticed because a person or an agent looked, and none was raised by the mesh unasked. Four faults came back through a different door after their first fix, because each fix closed an instance and left its class open. One merge was skipped by the bus, and twenty-three over three days have no matching action; a provider failed for twenty-three hours with only its own journal saying so; seven databases were dropped on one unreadable file.

What it touches. The controller (its verbs, status, plans, a lease), the host (its apply and report), the bus (its advisories and its upgrade), the console, the build path, the output channel of research 028, and the self-healing intent of research 017, which this effort extends from the loops that converge modules to the core that runs those loops. 017 deferred heartbeats, conditions and advisories until the bus was NATS; it is now.

Documents.

  • 01 — The evidence: 48 issues classified by class of failure (races, dropped commands, no feedback, logs only, two writers, manual repair, self-upgrade, CI-versus-live, third party), with time to detect and how each was noticed.
  • 02 — Principles: nine, each with what exists, what is missing and how it is checked; and the candidates weighed and not kept.
  • 03 — Mechanisms and roadmap: conditions, watchdogs from a signals table, a self-check (doctor), the output channel, healers with a hand-act log, a controller lease and report sequences, staged core upgrades with rollback, a facts snapshot for merge checks, a lab replay of every incident; five phases, ordered by risk removed; the five largest risks today.

Graduated 2026-10-06. The operator approved the conclusion the same day ("do the research and implement it"). The nine principles and the plan became ADR 0227; the mechanisms, the tables and the six phases became to-be 45. The two measurements owed — the signals table's bounds and a week of the hand-act log — are Phase 0's work there, not preconditions of the decision.