jochen
0f163a3e7c
The operator approved the conclusion: the core must say when it is wrong instead of waiting to be looked at. Record the nine rules with their checks, and specify the parts and the six phases a build can take.
67 lines
4.1 KiB
Markdown
67 lines
4.1 KiB
Markdown
---
|
|
status: graduated
|
|
initiated: 2026-10-06
|
|
touches:
|
|
- 00-META/how-we-build.md
|
|
- 01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md
|
|
- 01-RESEARCH/028-the-meshs-output-channel/00-overview.md
|
|
- 01-RESEARCH/019-a-warm-twin-of-the-running-mesh/00-overview.md
|
|
- 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
|
|
- 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md
|
|
- 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md
|
|
- 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md
|
|
- 03-DESIGN/01-to-be/06-the-controller.md
|
|
- 03-DESIGN/01-to-be/09-the-node-lifecycle.md
|
|
- 03-DESIGN/01-to-be/25-the-bus-on-nats.md
|
|
- 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md
|
|
- 04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md
|
|
became:
|
|
- 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
|
|
- 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
|
|
---
|
|
|
|
# 031 — A core that cannot fail silently
|
|
|
|
**What.** The principles the mesh's core must hold — the controller, the machine host, the bus, the
|
|
console and the path a change takes through them — so that it is fully diagnosable, monitors itself,
|
|
heals what it knows how to heal, and upgrades itself without a person standing by. And the mechanisms
|
|
and the order in which to build them.
|
|
|
|
**Why.** The operator, 2026-10-06: *"I still notice a lot of race issues, and commands being ignored, or
|
|
no feedback, no logs, no monitoring. Our mesh core setup must be fully diagnosable, with active
|
|
monitoring, self-healing, self-upgradeable, self-monitoring. The core principles must be very sturdy, no
|
|
ambiguities, clear plan of execution, fail-proof setup."*
|
|
|
|
The record bears it out. In the six days to 2026-10-06, 92 issue reports were opened. Of the 48 read here
|
|
as core failures, **every one was noticed because a person or an agent looked**, and **none was raised by
|
|
the mesh unasked**. Four faults came back through a different door after their first fix, because each
|
|
fix closed an instance and left its class open. One merge was skipped by the bus, and twenty-three over three
|
|
days have no matching action; a provider failed for twenty-three hours with only its own
|
|
journal saying so; seven databases were dropped on one unreadable file.
|
|
|
|
**What it touches.** The controller (its verbs, `status`, plans, a lease), the host (its apply and
|
|
report), the bus (its advisories and its upgrade), the console, the build path, the output channel of
|
|
research 028, and the self-healing intent of research 017, which this effort extends from the loops that
|
|
converge modules to the core that runs those loops. 017 deferred heartbeats, conditions and advisories
|
|
until the bus was NATS; it is now.
|
|
|
|
**Documents.**
|
|
|
|
- [01 — The evidence](01-evidence.md): 48 issues classified by class of failure (races, dropped
|
|
commands, no feedback, logs only, two writers, manual repair, self-upgrade, CI-versus-live, third
|
|
party), with time to detect and how each was noticed.
|
|
- [02 — Principles](02-principles.md): nine, each with what exists, what is missing and **how it is
|
|
checked**; and the candidates weighed and not kept.
|
|
- [03 — Mechanisms and roadmap](03-mechanisms-and-roadmap.md): conditions, watchdogs from a signals table,
|
|
a self-check (`doctor`), the output channel, healers with a hand-act log, a controller lease and report
|
|
sequences, staged core upgrades with rollback, a facts snapshot for merge checks, a lab replay of every
|
|
incident; five phases, ordered by risk removed; the five largest risks today.
|
|
|
|
**Graduated 2026-10-06.** The operator approved the conclusion the same day (*"do the research and
|
|
implement it"*). The nine principles and the plan became
|
|
[ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md);
|
|
the mechanisms, the tables and the six phases became
|
|
[to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md). The two measurements owed
|
|
— the signals table's bounds and a week of the hand-act log — are Phase 0's work there, not
|
|
preconditions of the decision.
|