hq/01-RESEARCH/031-a-core-that-cannot-fail-silently/02-principles.md

# 02 — Principles for the core, each with how it is checked

Nine principles. Each is stated as a rule a reviewer can refuse a change against, carries the classes of
[01](01-evidence.md) it answers, says what exists already, and says **how it is checked** — the
repository's own rule ([how-we-build §5](../../00-META/how-we-build.md)): a rule that states no check is
indistinguishable from a wrong one.

They extend, not replace, the six of [research 017](../017-a-mesh-that-heals-itself/01-the-intended-behaviour.md)
(a loop compares with what is; healing is the ordinary path again; a repair never destroys; nothing fails
silently; what cannot be fixed goes to an agent; correctness, not only liveness). Those are about the
loops that converge modules. These are about **the core that runs those loops**: the controller, the
host, the bus, the console and the path a change takes through them. 017 deferred heartbeats, conditions
and advisories until the bus was NATS. It is now, so that deferral has expired.

**The core**, for this document: the controller, the machine host and its launcher, the bus server, the
tool runtime and the console, the build seat, and the forge's announcer of merges. Everything a change
passes through before a module's own code runs.

---

## P1 — One writer per piece of state

Every piece of state the mesh keeps has exactly one writer, named. Anyone else who wants it changed asks
that writer; nobody writes beside it, and nobody computes a second answer to a question it already
answers.

- **Answers:** (e), most of (a). Issues 190, 201, 204, 222, 239, 245, 250, 257/261/267.
- **Exists:** the controller is the only writer of stream definitions (to-be 25); ADR 0222 §3 (the
  controller writes no file a seat's holder owns); the collision check at composition (no two modules
  declare one path, unit, name or package); the operator's direction on 245.
- **Missing:** a single writer for a *machine's applied state* (the delivery and the five-minute
  reconcile both apply and both report — three issues in two days); a single *controller* (two instances
  can both act during a handover, 204: nothing holds a lease); a single announcer per event kind (250
  was found by counting duplicates by hand).
- **How it is checked:**
  1. A **writers table** — state kind, its writer, where it is kept — is part of the to-be design, and a
     test in each core repository asserts that the code paths that write each kind are the one named
     (by a lint over the store's write calls and the bus subjects each component publishes on, the
     latter read from the grants the controller composes: a subject two components may publish on is
     refused at composition unless the table says it is shared).
  2. **Live:** the controller holds a **lease** (a key-value entry with a revision) and every message it
     sends carries the lease's epoch; a host refuses a declaration from an older epoch and says so. A
     probe ([03](03-mechanisms-and-roadmap.md), the self-check) asserts one lease holder and no message
     from a stale epoch in the last interval.

## P2 — Everything that changes state carries its writer's order, and every receiver refuses what is older

Declarations, reports, plans, builds, calls and announcements each carry `(writer, epoch, sequence)`.
Every receiver keeps the highest it has accepted per writer, and **refuses** — with a line in the mesh's
own words and a counter — anything older. Arrival order never decides.

- **Answers:** (a). Issues 201, 204, 214, 219, 234, 256, 257, 261, 264, 267.
- **Exists:** a declaration carries a sequence and a host keeps the newest (issue 107, to-be 25 §3); a
  newer build wins over an older one finishing later (219); a newer merge's plan supersedes an older
  one (ADR 0218 §3); a report about a declaration the mesh has moved past no longer replaces the stored
  account (267).
- **Missing:** a **report carries no sequence** — the digest last recorded as sent decides, which is why
  each of 256, 257 and 267 needed its own rule. No epoch on the controller. A plan's state carries no
  revision, so two instances can both advance it (214).
- **How it is checked:**
  1. A contract test per message kind, in the receiver's repository: deliver `n`, then `n−1`; the state
     names `n` and a refusal is counted. Deliver from epoch `e−1` after `e`: refused. A new message kind
     without such a test fails a check that lists every subject the component consumes against the
     tests that name it.
  2. **Live:** the refusals counter is a signal (P5): zero is normal, a burst is a condition naming the
     writer that sent stale.

## P3 — Every command is acknowledged at once, and its outcome is kept where it can be read later

A call is answered within a bound the caller can rely on — with its result, or with an id. Its outcome is
kept **durably**, outlives the process that ran it, and can be read by id or waited on. Nothing is fired
and forgotten, and no answer depends on what the command does to the transport carrying it.

- **Answers:** (c). Issues 176, 186, 200, 229, 230, 237, 264, 265.
- **Exists:** since 265, a seat's call answers within ten seconds or says "running" with an id, `push`
  answers before it acts, and `calls` keeps the last hundred calls and their answers; the build path
  says "asked, not waited for" and ADR 0219 §3 makes every cancel/kill leave an outcome.
- **Missing:** `calls` lives in the controller's memory, so **a controller restart forgets every
  outcome** — and the controller restarts on every merge to its own repository. Nothing waits on a plan
  (229). Observed the night this effort began: the read-only `status` took 18 s, so the health question
  itself is answered through the "still running" path.
- **How it is checked:**
  1. A test that walks every verb the controller announces: each answers within `AnswerWithin`, with a
     result or an id (already partly built for 265).
  2. A test that restarts the controller between a call and the read of its outcome: the outcome is
     still there.
  3. **Live:** a probe calls `status` and asserts it answers *in full* within the bound; a call
     `running` for longer than its verb's declared bound is a condition (P5).

## P4 — Nothing is dropped silently: an input that is unknown, unreadable or unmet is refused by name

A receiver that cannot read, place or honour an input refuses it and says which, where and why. It never
substitutes a default — above all never "empty" — for "I could not tell". An unknown argument, key,
placeholder, seat or subject is refused, naming it.

- **Answers:** (b). Issues 188, 202, 231, 241, 244, 246, 255, 259.
- **Exists:** the host's parser refuses unknown keys (ADR 0007, 0045); an undeclared setting or endpoint
  is refused (ADR 0164, 0138); a verb takes only its declared arguments (ADR 0154) and, since 244, the
  controller and the console refuse an undeclared one by name; a seat the mesh does not answer is
  refused by name (ADR 0222 §1); ADR 0219 §3, "nothing dropped is silent".
- **Missing:** the rule is in a dozen records and in no principle, so each new reader re-meets it. The
  destructive form — **an unreadable input read as "nothing asked", then acted on** (241) — has no
  general guard.
- **How it is checked:**
  1. Per component, a test that feeds each input reader an unreadable, malformed and foreign input and
     asserts a refusal, never an empty result. A reader whose error path returns an empty value fails a
     lint that the core repositories run (the shape is mechanical: an error branch that returns the
     zero value of a collection).
  2. Schema walks for every verb (244's tests) and every placeholder namespace (231).
  3. **Destructive deltas are braked**: a reconcile that would withdraw more than a bound of what it
     holds (one consumer, one module, a fraction set per provider) stops and raises a condition instead
     (P7). Checked by a test that empties the input and asserts nothing is withdrawn.

## P5 — Every expected signal has a watchdog: absence is itself a condition

Every signal the core expects on a cadence or after an act — a heartbeat, a report after a send, a
plan's progress, a build's outcome after its ask, the controller's event loop taking something in, a
provider's standing, an announcement turning into an action — has a declared bound. Silence past the
bound is raised as a condition, naming what was expected, from whom, since when.

- **Answers:** (c), (d), (i). Issues 179, 184, 186, 187, 230, 243, 248, 257, 264, 266, 267.
- **Exists:** a machine's "last heard from — out of touch N m" in `node show`; ADR 0090's stuck machine
  (three identical reports); ADR 0224's provider standing, shown with its silence after thirty minutes;
  266's catch-up of merges not acted on after ten minutes (the first true *announced-versus-acted*
  watchdog); ADR 0162's bound on a plan, which 230 found never fires (`late: false` for ever).
- **Missing:** a list of the signals, their bounds and their owners; the plan bound working; the event
  loop's last-taken age (187); asks counted against outcomes (186); the bus's own advisories (slow
  consumer, maximum deliveries, permission violations) read as observations instead of log lines.
- **How it is checked:**
  1. A **signals table** — signal, emitter, cadence or trigger, bound, condition raised — kept in the
     to-be design, and compiled into the controller. A test generated from it suppresses each signal in
     turn and asserts the named condition is raised within its bound and cleared when the signal
     returns.
  2. **Live:** the self-check (P6) reports, for every row, the age of the newest signal, so a row that
     never fires is itself visible.

## P6 — The mesh checks itself continuously, against live facts, and says what it found outward

The invariants the design states are probed **against the running mesh** on a schedule, not only in unit
tests. A violation is a **condition** — durable, with since-when, evidence and who can resolve it —
shown in `status` and **sent to the operator** through the output channel. The checker's own heartbeat is
watched from somewhere it does not run.

- **Answers:** (d), (h). Issues 177, 187, 238, 245, 253, 262.
- **Exists:** `status` itself (assembled from reports, as how-we-build §5 requires); 017's *condition*
  shape; 028's output seat, researched only; individual live checks done by hand (262: "the live check
  stays by hand, on each resolver"); 253's measurement before the collector's first run — the one time
  in the window a check ran before the damage.
- **Missing:** a scheduled runner, a condition store, a channel out, and a watcher's watcher.
- **How it is checked:**
  1. Every invariant in the to-be design that names a live check is a **probe** in the self-check's
     registry; a check over the design documents counts invariants with a stated live probe against
     those without, and the number may only go down.
  2. The self-check publishes a heartbeat; a second machine's watcher raises "the self-check is silent"
     through a channel that does not depend on the control node.
  3. **Live, once:** a deliberately broken invariant on a lab mesh appears in `status` and as a
     notification within one probe interval.

## P7 — A known failure heals itself, under a brake, and every repair is said

A failure that has been repaired by hand twice is a failure the mesh must repair itself: by running the
ordinary path again (017 P2), never by destroying (017 P3), with a budget and a back-off, and with one
line and one event saying what it did and why. When the budget is spent, or the only repair destroys,
it is a condition and a notification, not a retry.

- **Answers:** (f). Issues 179, 208, 214, 230, 233, 248, 254, 257, 264, 267.
- **Exists:** ADR 0224 §5, *detected automatically, repaired where safe, loud where not*, applied to the
  identity provider's admin; the host's launcher rolls back once to known-good and then halts (ADR
  0141); the provisioner's `holds` re-provisions what a backend lost (017/02); `broker consumer-reset`
  and `plans close` as verbs.
- **Missing:** the general mechanism. The most frequent hand act in the window — **a push by hand to
  unstick a plan waiting on a report** — has no healer: the controller could ask the machine to report
  again (the machine knows what it applied) before waiting longer.
- **How it is checked:**
  1. Every hand act on the core is done through a verb that records it (who, what, why) — the **hand-act
     log**. Its count per week is a reported number; an act recorded twice for the same cause is a
     condition asking for a healer.
  2. Each healer ships with a test that induces its failure, asserts the repair and the event, and
     asserts the brake after the budget.

## P8 — The core upgrades itself one machine at a time, health-gated, and rolls back on its own

A new controller, host, runtime or bus reaches one machine first; it is **healthy** only when the
self-check's probes for that component pass there (not merely when it "reported applied"); the rest
follow only then. A component that does not become healthy within its bound is rolled back to the last
known good **by something other than itself**, and the rollback is said. The component being replaced is
never the only witness of its successor's success.

- **Answers:** (g). Issues 201, 204, 213, 214, 217, 230, 245, 248, 264, 266.
- **Exists:** ADR 0218's one machine first, for modules, with "applied and current" as the gate; ADR
  0141/0142's side-by-side host versions, known-good and the launcher's single rollback; ADR 0185's
  controller serving what it can when it is behind its seat's row; 264's report kept on disk across the
  hand-over.
- **Missing:** a health definition per core component; a gate stronger than "reported"; rollback for
  the controller, the runtime and the bus; a lease hand-over between controllers (P1); a planned,
  rehearsed path for the bus, which is still one process on one machine and whose next upgrade (266:
  2.10 → 2.11) is one-way.
- **How it is checked:**
  1. **Lab:** a deliberately broken build of each core component (one that starts and does nothing; one
     that crashes; one that cannot reach the bus) is merged on a lab mesh. Each is rolled back without a
     hand, the mesh ends on the previous build, and a condition and a notification say so.
  2. **Live:** every core rollout leaves a record — first machine, health verdict, time to verdict,
     rolled back or not — readable through `plans`.

## P9 — A check is fed the real mesh's facts before a change is merged

A check whose verdict depends on the environment — names and their lengths, the machines that exist,
the catalogue as it is, the host's validation, the C library, the server versions — runs against **the
mesh's real facts**, exported and anonymised, before merge. A dependency's version that the mesh runs is
the version its tests run.

- **Answers:** (h), (i). Issues 177, 202, 228, 236, 262, 263, 266.
- **Exists:** 266's `TestTheImageIsTheServerTestedHere` (the bus image's release equals the tested
  server's); ADR 0223's composition test that renders the resolver's machine list; 202's test against the real
  catalogue (run by hand).
- **Missing:** the export of facts; a merge gate that composes every real machine's declaration with the
  change and runs the host's validation over it (which would have refused 236, 263 and 202 in their own
  pull requests); a resolver test under musl as well as glibc.
- **How it is checked:**
  1. The controller exports a **facts snapshot** (machines, names, assignments, seats, catalogue
     commit — no secrets, no addresses) daily; the core repositories' merge check composes every machine
     from it with the change applied and runs the host's validation; a pull request that makes any
     machine fail to compose or validate fails its check, naming the machine's role and the module.
  2. The snapshot's age is a signal (P5).

---

## Candidates weighed and not kept as principles

- **"Every invariant has a live probe, not only a unit test"** — merged into P6; it is how P6 is built.
- **"Environment-dependent checks run against the real mesh's facts"** — kept as P9; the third-party
  case (i) folded into it, because pinning and testing the version that runs is the same act.
- **"Self-healing is the default"** — kept as P7 but narrowed to *known* failures, those repaired by
  hand twice. A default of healing everything heals what is not understood, which is how a repair
  destroys (241's reconcile was, in its own terms, healing).
- **"No loop blocks on long work"** (184, 248, 175) — not a separate principle: a blocked loop is a
  signal gone silent (P5, the loop's last-taken age) and a design defect each owner fixes; stating it
  as a principle adds a rule with no mesh-wide check.
- **"Clear plan of execution"** from the mandate — not a principle about the mesh; it is the roadmap in
  [03](03-mechanisms-and-roadmap.md), and each phase there states its own verification.

## How the principles relate

P2 and P1 **prevent** the races. P4 **prevents** the silent drops. P3, P5 and P6 **catch** whatever the
first three miss, at the cost of minutes, not hours. P7 and P8 **repair**. P9 **moves** the catching
before merge. The order of the roadmap follows from that: catching first, because it is cheapest and
covers every class, including the ones nobody has met yet.