hq/01-RESEARCH/031-a-core-that-cannot-fail-silently/02-principles.md

02 — Principles for the core, each with how it is checked

Nine principles. Each is stated as a rule a reviewer can refuse a change against, carries the classes of 01 it answers, says what exists already, and says how it is checked — the repository's own rule (how-we-build §5): a rule that states no check is indistinguishable from a wrong one.

They extend, not replace, the six of research 017 (a loop compares with what is; healing is the ordinary path again; a repair never destroys; nothing fails silently; what cannot be fixed goes to an agent; correctness, not only liveness). Those are about the loops that converge modules. These are about the core that runs those loops: the controller, the host, the bus, the console and the path a change takes through them. 017 deferred heartbeats, conditions and advisories until the bus was NATS. It is now, so that deferral has expired.

The core, for this document: the controller, the machine host and its launcher, the bus server, the tool runtime and the console, the build seat, and the forge's announcer of merges. Everything a change passes through before a module's own code runs.

---

P1 — One writer per piece of state

Every piece of state the mesh keeps has exactly one writer, named. Anyone else who wants it changed asks that writer; nobody writes beside it, and nobody computes a second answer to a question it already answers.

• Answers: (e), most of (a). Issues 190, 201, 204, 222, 239, 245, 250, 257/261/267.
• Exists: the controller is the only writer of stream definitions (to-be 25); ADR 0222 §3 (the controller writes no file a seat's holder owns); the collision check at composition (no two modules declare one path, unit, name or package); the operator's direction on 245.
• Missing: a single writer for a machine's applied state (the delivery and the five-minute reconcile both apply and both report — three issues in two days); a single controller (two instances can both act during a handover, 204: nothing holds a lease); a single announcer per event kind (250 was found by counting duplicates by hand).
• How it is checked:
  1. A writers table — state kind, its writer, where it is kept — is part of the to-be design, and a test in each core repository asserts that the code paths that write each kind are the one named (by a lint over the store's write calls and the bus subjects each component publishes on, the latter read from the grants the controller composes: a subject two components may publish on is refused at composition unless the table says it is shared).
  2. Live: the controller holds a lease (a key-value entry with a revision) and every message it sends carries the lease's epoch; a host refuses a declaration from an older epoch and says so. A probe (03, the self-check) asserts one lease holder and no message from a stale epoch in the last interval.

P2 — Everything that changes state carries its writer's order, and every receiver refuses what is older

Declarations, reports, plans, builds, calls and announcements each carry (writer, epoch, sequence). Every receiver keeps the highest it has accepted per writer, and refuses — with a line in the mesh's own words and a counter — anything older. Arrival order never decides.

• Answers: (a). Issues 201, 204, 214, 219, 234, 256, 257, 261, 264, 267.
• Exists: a declaration carries a sequence and a host keeps the newest (issue 107, to-be 25 §3); a newer build wins over an older one finishing later (219); a newer merge's plan supersedes an older one (ADR 0218 §3); a report about a declaration the mesh has moved past no longer replaces the stored account (267).
• Missing: a report carries no sequence — the digest last recorded as sent decides, which is why each of 256, 257 and 267 needed its own rule. No epoch on the controller. A plan's state carries no revision, so two instances can both advance it (214).
• How it is checked:
  1. A contract test per message kind, in the receiver's repository: deliver n, then n−1; the state names n and a refusal is counted. Deliver from epoch e−1 after e: refused. A new message kind without such a test fails a check that lists every subject the component consumes against the tests that name it.
  2. Live: the refusals counter is a signal (P5): zero is normal, a burst is a condition naming the writer that sent stale.

P3 — Every command is acknowledged at once, and its outcome is kept where it can be read later

A call is answered within a bound the caller can rely on — with its result, or with an id. Its outcome is kept durably, outlives the process that ran it, and can be read by id or waited on. Nothing is fired and forgotten, and no answer depends on what the command does to the transport carrying it.

• Answers: (c). Issues 176, 186, 200, 229, 230, 237, 264, 265.
• Exists: since 265, a seat's call answers within ten seconds or says "running" with an id, push answers before it acts, and calls keeps the last hundred calls and their answers; the build path says "asked, not waited for" and ADR 0219 §3 makes every cancel/kill leave an outcome.
• Missing: calls lives in the controller's memory, so a controller restart forgets every outcome — and the controller restarts on every merge to its own repository. Nothing waits on a plan (229). Observed the night this effort began: the read-only status took 18 s, so the health question itself is answered through the "still running" path.
• How it is checked:
  1. A test that walks every verb the controller announces: each answers within AnswerWithin, with a result or an id (already partly built for 265).
  2. A test that restarts the controller between a call and the read of its outcome: the outcome is still there.
  3. Live: a probe calls status and asserts it answers in full within the bound; a call running for longer than its verb's declared bound is a condition (P5).

P4 — Nothing is dropped silently: an input that is unknown, unreadable or unmet is refused by name

A receiver that cannot read, place or honour an input refuses it and says which, where and why. It never substitutes a default — above all never "empty" — for "I could not tell". An unknown argument, key, placeholder, seat or subject is refused, naming it.

• Answers: (b). Issues 188, 202, 231, 241, 244, 246, 255, 259.
• Exists: the host's parser refuses unknown keys (ADR 0007, 0045); an undeclared setting or endpoint is refused (ADR 0164, 0138); a verb takes only its declared arguments (ADR 0154) and, since 244, the controller and the console refuse an undeclared one by name; a seat the mesh does not answer is refused by name (ADR 0222 §1); ADR 0219 §3, "nothing dropped is silent".
• Missing: the rule is in a dozen records and in no principle, so each new reader re-meets it. The destructive form — an unreadable input read as "nothing asked", then acted on (241) — has no general guard.
• How it is checked:
  1. Per component, a test that feeds each input reader an unreadable, malformed and foreign input and asserts a refusal, never an empty result. A reader whose error path returns an empty value fails a lint that the core repositories run (the shape is mechanical: an error branch that returns the zero value of a collection).
  2. Schema walks for every verb (244's tests) and every placeholder namespace (231).
  3. Destructive deltas are braked: a reconcile that would withdraw more than a bound of what it holds (one consumer, one module, a fraction set per provider) stops and raises a condition instead (P7). Checked by a test that empties the input and asserts nothing is withdrawn.

P5 — Every expected signal has a watchdog: absence is itself a condition

Every signal the core expects on a cadence or after an act — a heartbeat, a report after a send, a plan's progress, a build's outcome after its ask, the controller's event loop taking something in, a provider's standing, an announcement turning into an action — has a declared bound. Silence past the bound is raised as a condition, naming what was expected, from whom, since when.

• Answers: (c), (d), (i). Issues 179, 184, 186, 187, 230, 243, 248, 257, 264, 266, 267.
• Exists: a machine's "last heard from — out of touch N m" in node show; ADR 0090's stuck machine (three identical reports); ADR 0224's provider standing, shown with its silence after thirty minutes; 266's catch-up of merges not acted on after ten minutes (the first true announced-versus-acted watchdog); ADR 0162's bound on a plan, which 230 found never fires (late: false for ever).
• Missing: a list of the signals, their bounds and their owners; the plan bound working; the event loop's last-taken age (187); asks counted against outcomes (186); the bus's own advisories (slow consumer, maximum deliveries, permission violations) read as observations instead of log lines.
• How it is checked:
  1. A signals table — signal, emitter, cadence or trigger, bound, condition raised — kept in the to-be design, and compiled into the controller. A test generated from it suppresses each signal in turn and asserts the named condition is raised within its bound and cleared when the signal returns.
  2. Live: the self-check (P6) reports, for every row, the age of the newest signal, so a row that never fires is itself visible.

P6 — The mesh checks itself continuously, against live facts, and says what it found outward

The invariants the design states are probed against the running mesh on a schedule, not only in unit tests. A violation is a condition — durable, with since-when, evidence and who can resolve it — shown in status and sent to the operator through the output channel. The checker's own heartbeat is watched from somewhere it does not run.

• Answers: (d), (h). Issues 177, 187, 238, 245, 253, 262.
• Exists: status itself (assembled from reports, as how-we-build §5 requires); 017's condition shape; 028's output seat, researched only; individual live checks done by hand (262: "the live check stays by hand, on each resolver"); 253's measurement before the collector's first run — the one time in the window a check ran before the damage.
• Missing: a scheduled runner, a condition store, a channel out, and a watcher's watcher.
• How it is checked:
  1. Every invariant in the to-be design that names a live check is a probe in the self-check's registry; a check over the design documents counts invariants with a stated live probe against those without, and the number may only go down.
  2. The self-check publishes a heartbeat; a second machine's watcher raises "the self-check is silent" through a channel that does not depend on the control node.
  3. Live, once: a deliberately broken invariant on a lab mesh appears in status and as a notification within one probe interval.

P7 — A known failure heals itself, under a brake, and every repair is said

A failure that has been repaired by hand twice is a failure the mesh must repair itself: by running the ordinary path again (017 P2), never by destroying (017 P3), with a budget and a back-off, and with one line and one event saying what it did and why. When the budget is spent, or the only repair destroys, it is a condition and a notification, not a retry.

• Answers: (f). Issues 179, 208, 214, 230, 233, 248, 254, 257, 264, 267.
• Exists: ADR 0224 §5, detected automatically, repaired where safe, loud where not, applied to the identity provider's admin; the host's launcher rolls back once to known-good and then halts (ADR 0141); the provisioner's holds re-provisions what a backend lost (017/02); broker consumer-reset and plans close as verbs.
• Missing: the general mechanism. The most frequent hand act in the window — a push by hand to unstick a plan waiting on a report — has no healer: the controller could ask the machine to report again (the machine knows what it applied) before waiting longer.
• How it is checked:
  1. Every hand act on the core is done through a verb that records it (who, what, why) — the hand-act log. Its count per week is a reported number; an act recorded twice for the same cause is a condition asking for a healer.
  2. Each healer ships with a test that induces its failure, asserts the repair and the event, and asserts the brake after the budget.

P8 — The core upgrades itself one machine at a time, health-gated, and rolls back on its own

A new controller, host, runtime or bus reaches one machine first; it is healthy only when the self-check's probes for that component pass there (not merely when it "reported applied"); the rest follow only then. A component that does not become healthy within its bound is rolled back to the last known good by something other than itself, and the rollback is said. The component being replaced is never the only witness of its successor's success.

• Answers: (g). Issues 201, 204, 213, 214, 217, 230, 245, 248, 264, 266.
• Exists: ADR 0218's one machine first, for modules, with "applied and current" as the gate; ADR 0141/0142's side-by-side host versions, known-good and the launcher's single rollback; ADR 0185's controller serving what it can when it is behind its seat's row; 264's report kept on disk across the hand-over.
• Missing: a health definition per core component; a gate stronger than "reported"; rollback for the controller, the runtime and the bus; a lease hand-over between controllers (P1); a planned, rehearsed path for the bus, which is still one process on one machine and whose next upgrade (266: 2.10 → 2.11) is one-way.
• How it is checked:
  1. Lab: a deliberately broken build of each core component (one that starts and does nothing; one that crashes; one that cannot reach the bus) is merged on a lab mesh. Each is rolled back without a hand, the mesh ends on the previous build, and a condition and a notification say so.
  2. Live: every core rollout leaves a record — first machine, health verdict, time to verdict, rolled back or not — readable through plans.

P9 — A check is fed the real mesh's facts before a change is merged

A check whose verdict depends on the environment — names and their lengths, the machines that exist, the catalogue as it is, the host's validation, the C library, the server versions — runs against the mesh's real facts, exported and anonymised, before merge. A dependency's version that the mesh runs is the version its tests run.

• Answers: (h), (i). Issues 177, 202, 228, 236, 262, 263, 266.
• Exists: 266's TestTheImageIsTheServerTestedHere (the bus image's release equals the tested server's); ADR 0223's composition test that renders the resolver's machine list; 202's test against the real catalogue (run by hand).
• Missing: the export of facts; a merge gate that composes every real machine's declaration with the change and runs the host's validation over it (which would have refused 236, 263 and 202 in their own pull requests); a resolver test under musl as well as glibc.
• How it is checked:
  1. The controller exports a facts snapshot (machines, names, assignments, seats, catalogue commit — no secrets, no addresses) daily; the core repositories' merge check composes every machine from it with the change applied and runs the host's validation; a pull request that makes any machine fail to compose or validate fails its check, naming the machine's role and the module.
  2. The snapshot's age is a signal (P5).

---

Candidates weighed and not kept as principles

• "Every invariant has a live probe, not only a unit test" — merged into P6; it is how P6 is built.
• "Environment-dependent checks run against the real mesh's facts" — kept as P9; the third-party case (i) folded into it, because pinning and testing the version that runs is the same act.
• "Self-healing is the default" — kept as P7 but narrowed to known failures, those repaired by hand twice. A default of healing everything heals what is not understood, which is how a repair destroys (241's reconcile was, in its own terms, healing).
• "No loop blocks on long work" (184, 248, 175) — not a separate principle: a blocked loop is a signal gone silent (P5, the loop's last-taken age) and a design defect each owner fixes; stating it as a principle adds a rule with no mesh-wide check.
• "Clear plan of execution" from the mandate — not a principle about the mesh; it is the roadmap in 03, and each phase there states its own verification.

How the principles relate

P2 and P1 prevent the races. P4 prevents the silent drops. P3, P5 and P6 catch whatever the first three miss, at the cost of minutes, not hours. P7 and P8 repair. P9 moves the catching before merge. The order of the roadmap follows from that: catching first, because it is cheapest and covers every class, including the ones nobody has met yet.