From 97ff655e3442a363e51f437b975d5dfaed4aa555 Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 6 Oct 2026 02:09:55 +0200 Subject: [PATCH 1/2] Research 031: open the effort on a core that cannot fail silently The operator's mandate: races, ignored commands and silence keep recurring. Classifies 48 recent issues by class, proposes nine checkable principles and a phased roadmap, for review before anything graduates. --- .../00-overview.md | 62 +++++ .../01-evidence.md | 202 ++++++++++++++ .../02-principles.md | 247 ++++++++++++++++++ .../03-mechanisms-and-roadmap.md | 221 ++++++++++++++++ 4 files changed, 732 insertions(+) create mode 100644 01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md create mode 100644 01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md create mode 100644 01-RESEARCH/031-a-core-that-cannot-fail-silently/02-principles.md create mode 100644 01-RESEARCH/031-a-core-that-cannot-fail-silently/03-mechanisms-and-roadmap.md diff --git a/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md b/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md new file mode 100644 index 0000000..5bd3977 --- /dev/null +++ b/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md @@ -0,0 +1,62 @@ +--- +status: active +initiated: 2026-10-06 +touches: + - 00-META/how-we-build.md + - 01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md + - 01-RESEARCH/028-the-meshs-output-channel/00-overview.md + - 01-RESEARCH/019-a-warm-twin-of-the-running-mesh/00-overview.md + - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md + - 02-DECISIONS/0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md + - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md + - 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md + - 03-DESIGN/01-to-be/06-the-controller.md + - 03-DESIGN/01-to-be/09-the-node-lifecycle.md + - 03-DESIGN/01-to-be/25-the-bus-on-nats.md + - 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md + - 04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md +became: [] +--- + +# 031 — A core that cannot fail silently + +**What.** The principles the mesh's core must hold — the controller, the machine host, the bus, the +console and the path a change takes through them — so that it is fully diagnosable, monitors itself, +heals what it knows how to heal, and upgrades itself without a person standing by. And the mechanisms +and the order in which to build them. + +**Why.** The operator, 2026-10-06: *"I still notice a lot of race issues, and commands being ignored, or +no feedback, no logs, no monitoring. Our mesh core setup must be fully diagnosable, with active +monitoring, self-healing, self-upgradeable, self-monitoring. The core principles must be very sturdy, no +ambiguities, clear plan of execution, fail-proof setup."* + +The record bears it out. In the six days to 2026-10-06, 92 issue reports were opened. Of the 48 read here +as core failures, **every one was noticed because a person or an agent looked**, and **none was raised by +the mesh unasked**. Four faults came back through a different door after their first fix, because each +fix closed an instance and left its class open. One merge was skipped by the bus, and twenty-three over three +days have no matching action; a provider failed for twenty-three hours with only its own +journal saying so; seven databases were dropped on one unreadable file. + +**What it touches.** The controller (its verbs, `status`, plans, a lease), the host (its apply and +report), the bus (its advisories and its upgrade), the console, the build path, the output channel of +research 028, and the self-healing intent of research 017, which this effort extends from the loops that +converge modules to the core that runs those loops. 017 deferred heartbeats, conditions and advisories +until the bus was NATS; it is now. + +**Documents.** + +- [01 — The evidence](01-evidence.md): 48 issues classified by class of failure (races, dropped + commands, no feedback, logs only, two writers, manual repair, self-upgrade, CI-versus-live, third + party), with time to detect and how each was noticed. +- [02 — Principles](02-principles.md): nine, each with what exists, what is missing and **how it is + checked**; and the candidates weighed and not kept. +- [03 — Mechanisms and roadmap](03-mechanisms-and-roadmap.md): conditions, watchdogs from a signals table, + a self-check (`doctor`), the output channel, healers with a hand-act log, a controller lease and report + sequences, staged core upgrades with rollback, a facts snapshot for merge checks, a lab replay of every + incident; five phases, ordered by risk removed; the five largest risks today. + +**Next.** The operator reviews the principles and the order of the phases. Graduation would be one +record for the principles (likely amending how-we-build's non-negotiables), and to-be designs for the +condition store with its signals table, and for staged core upgrades. Two measurements are owed before +then: the bounds in the signals table, measured on the live mesh; and a week of the hand-act log, as the +baseline for the self-healing phase. diff --git a/01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md b/01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md new file mode 100644 index 0000000..bdf09e6 --- /dev/null +++ b/01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md @@ -0,0 +1,202 @@ +# 01 — The evidence, classified by class of failure + +Every issue report opened between 2026-09-30 and 2026-10-06 that bears on the mesh's core — the +controller, the machine host, the bus, the console, the build path — read in full and classified by +**the class of failure**, not by the component it was found in. A component view says "fix the host"; +a class view says "the same thing is wrong in four places", which is what a principle is for. + +## The count + +| | | +|---|---| +| issue reports opened 2026-10-01 to 2026-10-06 | **92** (about fifteen a day) | +| of those (and a few from the days before), classified below as core failures | **48** distinct issues | +| classified in more than one class | 17 of 48 | +| a fault that **came back** after a fix of the same symptom | 4 chains: 200 → 265, 230 → 264, 257 → 261 → 267, 175 → 184 → 248 | +| noticed because a person or an agent looked — at a stalled plan, a wrong outcome, a journal, a test run by hand, a review | **48 of 48** | +| of those, the mesh's own answer carried the fact for whoever asked (a refusal, a `status` line, a push's output) | 4 (233, 244, 259, 263) | +| raised by the mesh to anyone, unasked | **0 of 48** | + +The recurrences matter most. Each fix was correct for its instance and left the class standing, so the +same symptom came back through a different door days later. That is the measurement behind the +operator's mandate: point fixes are converging on the instances, not on the class. + +## The classes + +Nine classes, as the mandate frames them. The table under each is the evidence; *detected* is the time +from the fault's start to the moment anyone knew; *noticed by* is how. + +### (a) Races: concurrent actors without an ordering + +Two actors act on the same thing, and the order of arrival — not an explicit order — decides the outcome. + +| Issue | The two actors | Detected | Noticed by | +|---|---|---|---| +| 204 | an outgoing and an incoming controller both sent declarations | 2 min | a person saw a module undone | +| 201 | a plan's push carried a controller digest older than its successor had written | 10 min crash loop | a person, nothing answered | +| 214 | the controller rebuilding itself; the outcome reached the old one or neither | 27 min | a person asked the plan twice | +| 219 | an older build finishing later replaced a newer one | hours | a person reading builds | +| 234 | seven declarations arrived during an eleven-minute apply; the newest was composed from a stale view and undeclared four modules | 8 min of removals | a person, the modules were gone | +| 254 | three plans for three merges ran at once, each asking the same builds | hours | a person, one plan stuck "building" | +| 256 | a first machine's report landed between a module's two sends and read as stale | 7 min | a person | +| 257, 261 | the machine's five-minute reconcile and a delivery took the apply lock in the wrong order | 5 min (257), 29 s visible undo (261) | a person | +| 267 | a reconcile's report queued behind a delivery's apply overtook it at the controller | until a hand push | a person | +| 265 | a push reloaded the bus's permissions while its own answer was still owed | 54 of 103 pushes over two days | a person, "did not answer in time" | + +**What they share.** Every one is a receiver that kept *the last thing written* rather than *the +newest thing by an explicit order*. Declarations carry a sequence (issue 107); **reports do not** +(267 says so: "the report does not carry the declaration's sequence number, so the digest decides"). +Plans are ordered by when they were made only since ADR 0218. Builds are ordered since issue 219. +Controllers have no epoch, so two instances can both act (204). Ordering was added one message kind at +a time, each after a race in it was seen. + +### (b) Commands silently ignored, arguments dropped + +| Issue | What was dropped | Effect | +|---|---|---| +| 244 | the console removed `node` from every mesh-seat verb's schema and call | `plan` could only refuse; **`push ` arrived empty and pushed every machine** | +| 259 | a named push's flush sent every machine a build a policy held back | a fault met on every machine at once, not one | +| 202 | a module whose setting was unset was *left out* of the machine | the resolver vanished from a declaration, no error | +| 188 | a refusal inside "who is on the network" dropped a machine | 40 min, every symptom pointed elsewhere | +| 231 | a misspelled placeholder written to a file as literal text | passes every check | +| 241 | an unreadable contributions file read as "nobody asks" | **seven databases dropped and recreated empty** | +| 255 | the journal verb read nothing and said "-- No entries --" | a refusal that reads as a quiet service | +| 246 | a runtime that answered late was treated as absent | the console said modules "run nowhere" | + +**What they share.** A receiver that could not tell *nothing was asked* from *something was lost on the +way*, and chose a default. In 241 and 244 the default was the most destructive reading available. + +### (c) Outcomes not fed back to the caller + +| Issue | What the caller was told | What happened | +|---|---|---| +| 200, 265 | "did not answer in time" | the push ran; the answer was refused by the bus | +| 176 | the console's build tool neither waits nor registers | — | +| 229 | `plans` answers once in prose; nothing waits for a plan | an agent went round the mesh with `curl` | +| 230, 264 | a host stood aside for its successor and its report was cancelled | the plan waited for ever, reading `late: false` | +| 186 | the build machine dropped 26 of 43 asks; nothing counts asks against outcomes | inferred two hours later | +| 237 | `assign` answered "held" and "does not resolve for lack of it" in one breath | a person or agent would loop | + +Since 2026-10-06 the controller answers within ten seconds and keeps every call's outcome under an id +(`calls`, issue 265). Read live the same night: **that log holds the last hundred calls in the +controller's memory**, so a controller restart — which every merge to the controller's own repository +causes — forgets every outcome it held. And `status`, a read-only verb, took **18 seconds** to answer, +twice in a row, so even the health question is answered only through the "still running, ask `calls`" +path. + +### (d) Failures visible only as log lines + +| Issue | Where it was said | For how long | +|---|---|---| +| 179 (recurred) | the identity provider's journal, every five seconds | **23 hours**, about 31 000 refused logins | +| 184 | the controller's log: slow consumer, heartbeats dropped | 24 min deaf | +| 187 | five faults in one day, each found by reading a container's log hours later | hours each | +| 183, 217, 265 | a `Permissions Violation` line from the bus client library | days | +| 233 | `status` said `refused`, correctly; nothing said it had lasted | 1.5 h | +| 243 | nothing: machines silently ignored lower licence generations | until a login waited three minutes | +| 248 | the controller's event loop stopped logging at 15:17 | hours; "status showed every plan done" | +| 266 | nothing: a merge was skipped by the bus | **23 unmatched merges over three days** | +| 238 | a ban list of 400 entries | the operator's own address banned for four weeks | + +`status` printed its all-well sentence through 179, 248 and 266. ADR 0224 made the first of those break +it. The other two have no signal that `status` reads. + +### (e) State that two writers own + +| Issue | The two writers | +|---|---| +| 190, 222 | the runtime's configuration written by modules that are not the runtime, and by the controller | +| 201 | the controller seat's row written by a successor, read by a predecessor pushed back in | +| 239 | two definitions, in two repositories, held one module name | +| 245 | "behind" answered by a commit comparison beside the plan that already knows | +| 257, 261, 267 | the machine's state written by both the delivery and the five-minute reconcile | +| 250 | a merge announced by the forge's tool and by its poll | +| 179 | the identity provider's admin password: the mesh minted one, the database kept another | + +The operator's direction on 245 is the principle in their own words: *"a second answer to the same +question is how the two came to disagree."* + +### (f) Manual repair needed + +Counted from the reports' own "what unblocked it" sections: + +| Repair by hand | Issues | Times | +|---|---|---| +| a push by hand to unstick a plan waiting on a report | 230, 257, 264, 267 | at least 4 | +| a controller restart to recreate a missing object or let go of a held message | 208, 248 | 2 | +| a one-off program run as the controller, outside the service | 201, 248 | 2 | +| the identity provider's admin reset through its bootstrap command | 179 | 2 | +| a kept file restored on a machine by hand | 233 | 1 | +| a stuck plan closed by hand | 214, 254 | 2 | +| a ban lifted by hand | 238 | 1 | +| a consumer remade from now | 248 | 1 | + +ADR 0224 (*detected automatically, repaired where safe, loud where not*) is the first rule that turns +one of these into a mechanism. 248's `broker consumer-reset` and 254's `plans close` turned two into +verbs a person runs. Every other row is still a hand on a machine. + +### (g) Self-upgrade fragility + +The core updates itself: the controller rebuilds and replaces itself, the host delivers its own +successor (ADR 0141), the runtime and the console are modules, and the bus is a module on the control +node. + +| Issue | What the self-upgrade broke | +|---|---| +| 201 | the controller pushed back to an older build than its own successor's row | +| 204 | two controllers both sending during a handover | +| 213, 223 | the controller ran as a container, and a new mesh installed it so | +| 214 | the plan that rebuilds the controller lost track of it | +| 230, 264 | the host that stands aside loses the report of the apply that delivered it (fixed twice) | +| 245 | a rebuild of everything replaced the bus's container: **every runtime lost the bus for a minute** | +| 248, 266 | a controller restart is where merges go missing: most of 266's 23 lie in such windows | +| 217 | a refused announcement crash-looped nine runtimes, closing the path that would merge the fix | + +A machine's first-in-line rollout (ADR 0218) protects modules. It does not protect the core from itself: +the health a plan waits for is "reported applied", which a controller that cannot plan, a host that +cannot report or a bus that drops messages can each satisfy. Nothing rolls back. The recovery in 201 +was the mesh's own binary run by hand from the newer image. + +### (h) Checks that pass in CI and fail live + +| Issue | The environmental fact the check did not have | +|---|---| +| 177 | the store-backed tests are skipped by the quick check, and the mesh runs none of a module's tests | +| 236 | the host's declaration validation is not run by the catalogue check | +| 262 | musl takes an NXDOMAIN for IPv6 as final; glibc does not | +| 263 | the real machine names make a consumer's identity 23–26 characters against a bound of 20 | +| 202 | the controller's test against the real catalogue, run by nobody until that day | +| 228 | the host's removal has no case for a `user` — found by a review, not a test | + +Each check was right about the world it was given. None of them was given the mesh's world: its machine +names, its catalogue, its host's validation, its C libraries. + +### (i) Third-party bugs + +| Issue | | +|---|---| +| 266 | the bus server's 2.10 release skips messages on a consumer with several filter subjects | +| 265 | a reload of the bus's authorization forgets every reply permission already granted (documented server behaviour, read from its source) | +| 262 | a resolver answering NXDOMAIN where NODATA is correct, met by musl's stricter reading | + +The lesson of 266 is not "upgrade the bus" — it is that nothing compared *what was announced* with +*what was acted on*, so a dependency's bug was silent for three days. A defence in depth (watch the +outcome, not the transport) would have caught it whichever layer was wrong. + +## What would have prevented or caught each class + +| Class | Would have been prevented by | Would have been caught by | +|---|---|---| +| (a) races | every message ordered by its writer, stale refused by every receiver | a lab replay of the interleaving | +| (b) dropped | refusing an unknown or unreadable input by name | a schema walk over every verb | +| (c) no feedback | answer at once with an id; outcome kept durably | a watchdog on "asked and never finished" | +| (d) logs only | — | a condition in `status` and a notification | +| (e) two writers | one writer per piece of state | a registry of writers checked at composition | +| (f) manual repair | a healer for every repair done twice | a counter of hand acts | +| (g) self-upgrade | one machine first, health-gated, rolled back | a lab upgrade with a deliberately broken build | +| (h) CI vs live | checks fed the real mesh's facts | the same, before merge | +| (i) third party | pinning and testing the version that runs | an end-to-end count of announced vs acted | + +The two columns are the principles of [02](02-principles.md). Read by count, **the "caught by" column +is the cheapest and widest**: a watchdog and a condition would have shortened most of the 48 from "a +person noticed" to minutes, whatever the cause. diff --git a/01-RESEARCH/031-a-core-that-cannot-fail-silently/02-principles.md b/01-RESEARCH/031-a-core-that-cannot-fail-silently/02-principles.md new file mode 100644 index 0000000..92e6613 --- /dev/null +++ b/01-RESEARCH/031-a-core-that-cannot-fail-silently/02-principles.md @@ -0,0 +1,247 @@ +# 02 — Principles for the core, each with how it is checked + +Nine principles. Each is stated as a rule a reviewer can refuse a change against, carries the classes of +[01](01-evidence.md) it answers, says what exists already, and says **how it is checked** — the +repository's own rule ([how-we-build §5](../../00-META/how-we-build.md)): a rule that states no check is +indistinguishable from a wrong one. + +They extend, not replace, the six of [research 017](../017-a-mesh-that-heals-itself/01-the-intended-behaviour.md) +(a loop compares with what is; healing is the ordinary path again; a repair never destroys; nothing fails +silently; what cannot be fixed goes to an agent; correctness, not only liveness). Those are about the +loops that converge modules. These are about **the core that runs those loops**: the controller, the +host, the bus, the console and the path a change takes through them. 017 deferred heartbeats, conditions +and advisories until the bus was NATS. It is now, so that deferral has expired. + +**The core**, for this document: the controller, the machine host and its launcher, the bus server, the +tool runtime and the console, the build seat, and the forge's announcer of merges. Everything a change +passes through before a module's own code runs. + +--- + +## P1 — One writer per piece of state + +Every piece of state the mesh keeps has exactly one writer, named. Anyone else who wants it changed asks +that writer; nobody writes beside it, and nobody computes a second answer to a question it already +answers. + +- **Answers:** (e), most of (a). Issues 190, 201, 204, 222, 239, 245, 250, 257/261/267. +- **Exists:** the controller is the only writer of stream definitions (to-be 25); ADR 0222 §3 (the + controller writes no file a seat's holder owns); the collision check at composition (no two modules + declare one path, unit, name or package); the operator's direction on 245. +- **Missing:** a single writer for a *machine's applied state* (the delivery and the five-minute + reconcile both apply and both report — three issues in two days); a single *controller* (two instances + can both act during a handover, 204: nothing holds a lease); a single announcer per event kind (250 + was found by counting duplicates by hand). +- **How it is checked:** + 1. A **writers table** — state kind, its writer, where it is kept — is part of the to-be design, and a + test in each core repository asserts that the code paths that write each kind are the one named + (by a lint over the store's write calls and the bus subjects each component publishes on, the + latter read from the grants the controller composes: a subject two components may publish on is + refused at composition unless the table says it is shared). + 2. **Live:** the controller holds a **lease** (a key-value entry with a revision) and every message it + sends carries the lease's epoch; a host refuses a declaration from an older epoch and says so. A + probe ([03](03-mechanisms-and-roadmap.md), the self-check) asserts one lease holder and no message + from a stale epoch in the last interval. + +## P2 — Everything that changes state carries its writer's order, and every receiver refuses what is older + +Declarations, reports, plans, builds, calls and announcements each carry `(writer, epoch, sequence)`. +Every receiver keeps the highest it has accepted per writer, and **refuses** — with a line in the mesh's +own words and a counter — anything older. Arrival order never decides. + +- **Answers:** (a). Issues 201, 204, 214, 219, 234, 256, 257, 261, 264, 267. +- **Exists:** a declaration carries a sequence and a host keeps the newest (issue 107, to-be 25 §3); a + newer build wins over an older one finishing later (219); a newer merge's plan supersedes an older + one (ADR 0218 §3); a report about a declaration the mesh has moved past no longer replaces the stored + account (267). +- **Missing:** a **report carries no sequence** — the digest last recorded as sent decides, which is why + each of 256, 257 and 267 needed its own rule. No epoch on the controller. A plan's state carries no + revision, so two instances can both advance it (214). +- **How it is checked:** + 1. A contract test per message kind, in the receiver's repository: deliver `n`, then `n−1`; the state + names `n` and a refusal is counted. Deliver from epoch `e−1` after `e`: refused. A new message kind + without such a test fails a check that lists every subject the component consumes against the + tests that name it. + 2. **Live:** the refusals counter is a signal (P5): zero is normal, a burst is a condition naming the + writer that sent stale. + +## P3 — Every command is acknowledged at once, and its outcome is kept where it can be read later + +A call is answered within a bound the caller can rely on — with its result, or with an id. Its outcome is +kept **durably**, outlives the process that ran it, and can be read by id or waited on. Nothing is fired +and forgotten, and no answer depends on what the command does to the transport carrying it. + +- **Answers:** (c). Issues 176, 186, 200, 229, 230, 237, 264, 265. +- **Exists:** since 265, a seat's call answers within ten seconds or says "running" with an id, `push` + answers before it acts, and `calls` keeps the last hundred calls and their answers; the build path + says "asked, not waited for" and ADR 0219 §3 makes every cancel/kill leave an outcome. +- **Missing:** `calls` lives in the controller's memory, so **a controller restart forgets every + outcome** — and the controller restarts on every merge to its own repository. Nothing waits on a plan + (229). Observed the night this effort began: the read-only `status` took 18 s, so the health question + itself is answered through the "still running" path. +- **How it is checked:** + 1. A test that walks every verb the controller announces: each answers within `AnswerWithin`, with a + result or an id (already partly built for 265). + 2. A test that restarts the controller between a call and the read of its outcome: the outcome is + still there. + 3. **Live:** a probe calls `status` and asserts it answers *in full* within the bound; a call + `running` for longer than its verb's declared bound is a condition (P5). + +## P4 — Nothing is dropped silently: an input that is unknown, unreadable or unmet is refused by name + +A receiver that cannot read, place or honour an input refuses it and says which, where and why. It never +substitutes a default — above all never "empty" — for "I could not tell". An unknown argument, key, +placeholder, seat or subject is refused, naming it. + +- **Answers:** (b). Issues 188, 202, 231, 241, 244, 246, 255, 259. +- **Exists:** the host's parser refuses unknown keys (ADR 0007, 0045); an undeclared setting or endpoint + is refused (ADR 0164, 0138); a verb takes only its declared arguments (ADR 0154) and, since 244, the + controller and the console refuse an undeclared one by name; a seat the mesh does not answer is + refused by name (ADR 0222 §1); ADR 0219 §3, "nothing dropped is silent". +- **Missing:** the rule is in a dozen records and in no principle, so each new reader re-meets it. The + destructive form — **an unreadable input read as "nothing asked", then acted on** (241) — has no + general guard. +- **How it is checked:** + 1. Per component, a test that feeds each input reader an unreadable, malformed and foreign input and + asserts a refusal, never an empty result. A reader whose error path returns an empty value fails a + lint that the core repositories run (the shape is mechanical: an error branch that returns the + zero value of a collection). + 2. Schema walks for every verb (244's tests) and every placeholder namespace (231). + 3. **Destructive deltas are braked**: a reconcile that would withdraw more than a bound of what it + holds (one consumer, one module, a fraction set per provider) stops and raises a condition instead + (P7). Checked by a test that empties the input and asserts nothing is withdrawn. + +## P5 — Every expected signal has a watchdog: absence is itself a condition + +Every signal the core expects on a cadence or after an act — a heartbeat, a report after a send, a +plan's progress, a build's outcome after its ask, the controller's event loop taking something in, a +provider's standing, an announcement turning into an action — has a declared bound. Silence past the +bound is raised as a condition, naming what was expected, from whom, since when. + +- **Answers:** (c), (d), (i). Issues 179, 184, 186, 187, 230, 243, 248, 257, 264, 266, 267. +- **Exists:** a machine's "last heard from — out of touch N m" in `node show`; ADR 0090's stuck machine + (three identical reports); ADR 0224's provider standing, shown with its silence after thirty minutes; + 266's catch-up of merges not acted on after ten minutes (the first true *announced-versus-acted* + watchdog); ADR 0162's bound on a plan, which 230 found never fires (`late: false` for ever). +- **Missing:** a list of the signals, their bounds and their owners; the plan bound working; the event + loop's last-taken age (187); asks counted against outcomes (186); the bus's own advisories (slow + consumer, maximum deliveries, permission violations) read as observations instead of log lines. +- **How it is checked:** + 1. A **signals table** — signal, emitter, cadence or trigger, bound, condition raised — kept in the + to-be design, and compiled into the controller. A test generated from it suppresses each signal in + turn and asserts the named condition is raised within its bound and cleared when the signal + returns. + 2. **Live:** the self-check (P6) reports, for every row, the age of the newest signal, so a row that + never fires is itself visible. + +## P6 — The mesh checks itself continuously, against live facts, and says what it found outward + +The invariants the design states are probed **against the running mesh** on a schedule, not only in unit +tests. A violation is a **condition** — durable, with since-when, evidence and who can resolve it — +shown in `status` and **sent to the operator** through the output channel. The checker's own heartbeat is +watched from somewhere it does not run. + +- **Answers:** (d), (h). Issues 177, 187, 238, 245, 253, 262. +- **Exists:** `status` itself (assembled from reports, as how-we-build §5 requires); 017's *condition* + shape; 028's output seat, researched only; individual live checks done by hand (262: "the live check + stays by hand, on each resolver"); 253's measurement before the collector's first run — the one time + in the window a check ran before the damage. +- **Missing:** a scheduled runner, a condition store, a channel out, and a watcher's watcher. +- **How it is checked:** + 1. Every invariant in the to-be design that names a live check is a **probe** in the self-check's + registry; a check over the design documents counts invariants with a stated live probe against + those without, and the number may only go down. + 2. The self-check publishes a heartbeat; a second machine's watcher raises "the self-check is silent" + through a channel that does not depend on the control node. + 3. **Live, once:** a deliberately broken invariant on a lab mesh appears in `status` and as a + notification within one probe interval. + +## P7 — A known failure heals itself, under a brake, and every repair is said + +A failure that has been repaired by hand twice is a failure the mesh must repair itself: by running the +ordinary path again (017 P2), never by destroying (017 P3), with a budget and a back-off, and with one +line and one event saying what it did and why. When the budget is spent, or the only repair destroys, +it is a condition and a notification, not a retry. + +- **Answers:** (f). Issues 179, 208, 214, 230, 233, 248, 254, 257, 264, 267. +- **Exists:** ADR 0224 §5, *detected automatically, repaired where safe, loud where not*, applied to the + identity provider's admin; the host's launcher rolls back once to known-good and then halts (ADR + 0141); the provisioner's `holds` re-provisions what a backend lost (017/02); `broker consumer-reset` + and `plans close` as verbs. +- **Missing:** the general mechanism. The most frequent hand act in the window — **a push by hand to + unstick a plan waiting on a report** — has no healer: the controller could ask the machine to report + again (the machine knows what it applied) before waiting longer. +- **How it is checked:** + 1. Every hand act on the core is done through a verb that records it (who, what, why) — the **hand-act + log**. Its count per week is a reported number; an act recorded twice for the same cause is a + condition asking for a healer. + 2. Each healer ships with a test that induces its failure, asserts the repair and the event, and + asserts the brake after the budget. + +## P8 — The core upgrades itself one machine at a time, health-gated, and rolls back on its own + +A new controller, host, runtime or bus reaches one machine first; it is **healthy** only when the +self-check's probes for that component pass there (not merely when it "reported applied"); the rest +follow only then. A component that does not become healthy within its bound is rolled back to the last +known good **by something other than itself**, and the rollback is said. The component being replaced is +never the only witness of its successor's success. + +- **Answers:** (g). Issues 201, 204, 213, 214, 217, 230, 245, 248, 264, 266. +- **Exists:** ADR 0218's one machine first, for modules, with "applied and current" as the gate; ADR + 0141/0142's side-by-side host versions, known-good and the launcher's single rollback; ADR 0185's + controller serving what it can when it is behind its seat's row; 264's report kept on disk across the + hand-over. +- **Missing:** a health definition per core component; a gate stronger than "reported"; rollback for + the controller, the runtime and the bus; a lease hand-over between controllers (P1); a planned, + rehearsed path for the bus, which is still one process on one machine and whose next upgrade (266: + 2.10 → 2.11) is one-way. +- **How it is checked:** + 1. **Lab:** a deliberately broken build of each core component (one that starts and does nothing; one + that crashes; one that cannot reach the bus) is merged on a lab mesh. Each is rolled back without a + hand, the mesh ends on the previous build, and a condition and a notification say so. + 2. **Live:** every core rollout leaves a record — first machine, health verdict, time to verdict, + rolled back or not — readable through `plans`. + +## P9 — A check is fed the real mesh's facts before a change is merged + +A check whose verdict depends on the environment — names and their lengths, the machines that exist, +the catalogue as it is, the host's validation, the C library, the server versions — runs against **the +mesh's real facts**, exported and anonymised, before merge. A dependency's version that the mesh runs is +the version its tests run. + +- **Answers:** (h), (i). Issues 177, 202, 228, 236, 262, 263, 266. +- **Exists:** 266's `TestTheImageIsTheServerTestedHere` (the bus image's release equals the tested + server's); ADR 0223's composition test that renders the resolver's machine list; 202's test against the real + catalogue (run by hand). +- **Missing:** the export of facts; a merge gate that composes every real machine's declaration with the + change and runs the host's validation over it (which would have refused 236, 263 and 202 in their own + pull requests); a resolver test under musl as well as glibc. +- **How it is checked:** + 1. The controller exports a **facts snapshot** (machines, names, assignments, seats, catalogue + commit — no secrets, no addresses) daily; the core repositories' merge check composes every machine + from it with the change applied and runs the host's validation; a pull request that makes any + machine fail to compose or validate fails its check, naming the machine's role and the module. + 2. The snapshot's age is a signal (P5). + +--- + +## Candidates weighed and not kept as principles + +- **"Every invariant has a live probe, not only a unit test"** — merged into P6; it is how P6 is built. +- **"Environment-dependent checks run against the real mesh's facts"** — kept as P9; the third-party + case (i) folded into it, because pinning and testing the version that runs is the same act. +- **"Self-healing is the default"** — kept as P7 but narrowed to *known* failures, those repaired by + hand twice. A default of healing everything heals what is not understood, which is how a repair + destroys (241's reconcile was, in its own terms, healing). +- **"No loop blocks on long work"** (184, 248, 175) — not a separate principle: a blocked loop is a + signal gone silent (P5, the loop's last-taken age) and a design defect each owner fixes; stating it + as a principle adds a rule with no mesh-wide check. +- **"Clear plan of execution"** from the mandate — not a principle about the mesh; it is the roadmap in + [03](03-mechanisms-and-roadmap.md), and each phase there states its own verification. + +## How the principles relate + +P2 and P1 **prevent** the races. P4 **prevents** the silent drops. P3, P5 and P6 **catch** whatever the +first three miss, at the cost of minutes, not hours. P7 and P8 **repair**. P9 **moves** the catching +before merge. The order of the roadmap follows from that: catching first, because it is cheapest and +covers every class, including the ones nobody has met yet. diff --git a/01-RESEARCH/031-a-core-that-cannot-fail-silently/03-mechanisms-and-roadmap.md b/01-RESEARCH/031-a-core-that-cannot-fail-silently/03-mechanisms-and-roadmap.md new file mode 100644 index 0000000..ebc3f58 --- /dev/null +++ b/01-RESEARCH/031-a-core-that-cannot-fail-silently/03-mechanisms-and-roadmap.md @@ -0,0 +1,221 @@ +# 03 — Mechanisms and a phased roadmap + +The principles of [02](02-principles.md) need few new things. Most of the parts exist in some form; +what is missing is the connective tissue that makes a fact the mesh already has reach someone without +being asked. This document names the mechanisms, then orders them into phases **by risk removed per unit +of effort**, each phase with deliverables and a verification that says it is done. + +Effort is given in **focused working days** of one agent-and-operator pair, at the pace the record shows +(a located issue to a merged fix in under a day is common). The figures are for ordering, not promises. + +## The mechanisms + +### M1 — Conditions (P5, P6) + +017's *condition*, built: a durable fact about something the mesh owns — what is wrong, since when, the +evidence, what was tried, who can resolve it, and whether it may clear itself. Kept in a key-value +bucket the controller writes (one writer, P1), keyed by subject (`plan/`, `machine/`, +`provider//`, `core/`). Raised and cleared by observation only; a person +can **silence** one for a stated time, never resolve it. `status` becomes, first, the list of open +conditions; the all-well sentence is "no open conditions". ADR 0224's provider standing is the first +condition kind and moves into it unchanged. + +### M2 — Watchdogs from a signals table (P5) + +One table, compiled into the controller, of every signal the core expects. Its first rows, each from an +issue in [01](01-evidence.md): + +| Signal | Bound (to be measured, then set) | Condition raised | Issue | +|---|---|---|---| +| machine heartbeat | 3 × interval | machine silent (asleep is a declared state, ADR 0211) | 187 | +| report after a send | the machine's last apply duration × 3, at least 2 min | sent, not reported — then **ask the machine to report again** (M5) | 230, 257, 264, 267 | +| plan tier progress | per tier, from build and apply durations | plan stalled at tier N, waiting on X | 214, 230, 254 | +| controller event loop took something | 2 min while the stream has pending | controller deaf | 184, 248 | +| merge announced → plan made or "nothing reads it" | 10 min (exists, 266) | merge never acted on | 248, 266 | +| build asked → outcome | build's own declared timeout | ask lost | 186 | +| call `running` → finished | the verb's declared bound | call hung | 265 | +| provider standing repeated | 30 min (exists, ADR 0224) | provider silent | 179 | +| bus advisories: slow consumer, maximum deliveries, permission violation | any | bus refused or dropped X for Y | 183, 187, 217, 265 | +| self-check heartbeat | 2 × its interval, watched from a second machine | the watcher is silent | — | +| facts snapshot age | 2 days | merge checks run on stale facts | 263 | + +The bus advisories are the cheapest row: the server already publishes them on its system subjects, and +the controller only has to subscribe (read-only) and translate each into the mesh's words, naming the +call or the consumer, as 265 now does for a refused reply. + +### M3 — The self-check: `doctor` (P6) + +A controller verb, `doctor`, and the same code run every few minutes by the controller itself. Each run +executes the **probe registry** — the live form of the design's invariants — and raises or clears +conditions. First probes, each an invariant that a person checked by hand in the window: + +- every machine's declaration composes, and every host would accept it (236, 263); +- every holder of the mesh's resolver answers a machine name for IPv4 and NODATA for IPv6 (262); +- every seat on record has a live holder that answers (208, 218); +- every kept archive is held by a manifest (253 — the controller's collection command already reports it); +- exactly one controller holds the lease (P1); +- every durable consumer's position is near its stream's head (248's replay, 266's skip); +- no address the mesh owns is in a ban list (238); +- `status` answers in full within its bound (P3). + +`doctor` with no argument answers the last run's verdict at once (P3) and, with `run`, runs now under an +id. Its own heartbeat is a signal (M2), watched from a second machine. + +### M4 — The output channel (P6) + +[Research 028](../028-the-meshs-output-channel/00-overview.md)'s seat, built minimally first: **one** +channel the operator chose (028 records it), plus the desktop notifier where the operator is. A condition +is sent when raised, once more if it lasts past a bound, and when it clears. Deduplicated by the +condition's key. The watcher's watcher (028's open question) is the second-machine watchdog of M3, +sending through a channel that does not pass through the control node. + +### M5 — Healers (P7) + +A healer is a registered response to one condition kind: its repair (the ordinary path again), its +budget, its brake, and the event it emits. First healers, all from hand acts in [01](01-evidence.md) §(f): + +| Condition | Repair | Brake | +|---|---|---| +| sent, not reported | ask the machine to report what it last applied (it keeps it since 264); if that names another declaration, send again | twice, then condition | +| plan stalled on a superseded or finished wait | close the plan with its note (`plans close`, done by the mesh) | once | +| seat holder without its worker | raise the seat's objects again (208) | once per holder | +| consumer far behind on a history stream | `broker consumer-reset` (248) — **only** for the consumers the table marks resettable | once, then condition | +| provider admin refuses the mesh's secret | ADR 0224 §5 (exists) | exists | + +And the **hand-act log**: every repair a person makes on the core goes through a verb that records who, +what and why. Its weekly count is the measure of P7. + +### M6 — Order and epochs (P1, P2) + +- The controller takes a **lease** in a key-value bucket before it acts and renews it; its revision is + the **epoch** every declaration and plan write carries. A starting controller waits for the lease; + the outgoing one stops sending when it loses it. That closes 204 and makes 201/214 detectable. +- A **report carries the sequence** of the declaration it is about; the controller keeps the highest + per machine and refuses older accounts by sequence, not by a digest lookup (256, 257, 267 become one + rule). +- **The host has one apply queue.** A delivery and the reconcile are two reasons to enqueue the same + act; the queue applies the newest declaration once, and makes one report (257/261/267 become + impossible rather than handled). +- Every receiver's refusal of something stale is a counted line (P2's live check). + +### M7 — Staged, reversible core upgrades (P8) + +- **A health definition per core component**, written as probes in M3's registry: the controller + answers `status` in bound and holds the lease; the host has reported its current declaration; the + runtime has announced and answers a PING; the bus has every stream and every durable consumer and + passes a request/reply round trip. +- **The gate:** ADR 0218's first machine is judged by those probes, not by "applied". +- **Rollback by a witness that is not the new build:** the host's launcher for the host (exists); for + the controller, the previous controller's process kept installed beside it, re-started by the host + when the new one does not take the lease in bound; for the runtime, the host's known-good the same + way. Each rollback is a condition, so it is said. +- **The bus is planned, not rolled.** A bus upgrade is a declared maintenance step: streams snapshotted, + the step announced as a condition while it runs, every consumer's position checked after (M3). The + one-way 2.10 → 2.11 upgrade of 266 is the first. Whether the bus should become a cluster of three so + that it can be upgraded live is a question for its own effort. + +### M8 — The facts snapshot and the merge gate (P9) + +The controller exports a facts snapshot (machines by role and name length, assignments, seats, catalogue +commit) to a place the build seat reads. The core repositories' and the catalogue's merge checks compose +every machine with the change and run the host's validation. The resolver module's tests run under musl +and glibc. A bus, store or library version the mesh runs is the one its tests run (266's pattern, +generalised). + +### M9 — The lab replay (all) + +[Research 019](../019-a-warm-twin-of-the-running-mesh/00-overview.md)'s warm twin, or a throwaway lab of +three containers, running **scripted replays of each incident** in the window: a reconcile due during a +push; a host self-update during its report; a bus reload during a call; a consumer with several filters +under mixed traffic; a missing consumer made with the server's default; an unreadable contributions +file; a controller rebuilding itself mid-plan; two controllers at once. Each replay asserts the +principle's outcome (refused stale, condition raised, healed, rolled back). They run on every merge to a +core repository. + +--- + +## The roadmap + +Ordered by **risk removed per day**. Detection comes first because it covers every class at once, +including the ones not met yet; prevention second; repair and staging third; the pre-merge and lab work +last because they are larger and pay off over months. + +### Phase 0 — Finish what is in flight (2–3 days) + +- Land and roll out the located fixes: 244, 264, 265, 266 (including the bus's planned 2.11 upgrade, done + as M7's first planned bus step), 267, 257/261. +- Make `calls` durable (a key-value bucket, bounded by count and age), and bring `status` inside its own + answer bound. +- Start the **hand-act log** now, before anything else, so every later phase is measured against a + baseline. +- **Done when:** the four located core issues resolve with their live checks; a controller restart + keeps `calls`; `status` answers in full within ten seconds. + +### Phase 1 — The mesh says when it is wrong (5–8 days) + +- M1 conditions (ADR 0224's standing moved into them); M2 watchdogs for the first eight rows; the bus + advisories subscribed and translated; M3 `doctor` with the first probes; M4 with one channel and the + second-machine watcher. +- **Done when:** on a lab mesh, suppressing each signal in the table raises its condition within its + bound and sends a notification; clearing it clears both. Live: a week of conditions read back, every + one either real or a bound corrected. +- **Risk removed:** every class in [01](01-evidence.md) moves from "noticed by a person, hours later" to + "said by the mesh, minutes later". + +### Phase 2 — Order and one writer (5–7 days) + +- M6: the controller's lease and epoch; a report's sequence; the host's single apply queue; stale + refusals counted. +- The writers table and the signals table written into a to-be design, with their compile-time checks. +- P4's lint for empty-on-error readers in the core repositories, and the destructive-delta brake in the + provisioner harness. +- **Done when:** the lab replays of 204, 257/261/267 and 241 end with "refused stale", "one report" and + "withdrawal braked" respectively; the contract test for every consumed subject exists. +- **Risk removed:** class (a), the largest by count, and the destructive half of (b). + +### Phase 3 — Healers (3–5 days) + +- M5's first healers and the rule that a repair done by hand twice asks for one (from the hand-act log). +- **Done when:** a lab mesh recovers from each induced failure in the healers table with no hand, says so, + and brakes after its budget. Live: a week in which the hand-act log has no repeat. + +### Phase 4 — Core upgrades that roll back (8–12 days) + +- M7: health definitions; the gate; rollback for the controller and the runtime; the bus as a planned + step. +- **Done when:** on a lab mesh, a deliberately broken build of the controller, the host and the runtime + is each rolled back without a hand, the mesh ends on the previous build, and the rollback is a + condition and a notification. Live: the next three core rollouts each record a health verdict. +- **Risk removed:** class (g) — the failures that take the control path itself down. + +### Phase 5 — Checks before merge, and the replay suite (8–12 days, then ongoing) + +- M8: the facts snapshot and the compose-and-validate merge gate; the libc matrix; versions tested as + run. +- M9: every incident in the window as a scripted replay, run on every core merge; each new core issue + adds its replay as its "how it is checked". +- **Done when:** the replays of 236, 262, 263 and 266 fail on the commit before their fix and pass after; + a new core issue cannot resolve without a replay or a stated reason why none is possible. + +**Total, roughly six to eight weeks of focused work**, with the first visible change — the mesh saying +when it is wrong — inside the first two. + +## The five largest risks today + +Ranked by likelihood × damage, from the window's evidence and the mesh as read the night this effort +began: + +1. **A stall nobody is told about.** A plan, a merge or a report that stops is noticed only by someone + looking (230, 248, 257, 264, 266, 267; issue 187 still open). Every release goes through a plan. +2. **A destructive act on a misread input.** 241 dropped seven databases on one unreadable file; 234 + undeclared four modules from a stale composition; 245's advice rebuilt everything and cut the bus; + 253's collector would have deleted every archive. There is no general brake on a large withdrawal. +3. **The core replacing itself with nothing to roll it back.** A controller build that starts but cannot + plan (201, 214) halts every later change, because the controller is what plans the fix. Only the + host has a launcher rollback. +4. **The bus as a single, un-upgradeable process.** It runs a release that skips messages (266), its + reload drops owed replies (265), replacing it cuts every machine off (245), and its next upgrade is + one-way and needs a restart. +5. **Concurrent actors with no lease or report order.** Several agent sessions, plans and reconciles act + at once; on the night this began, four named pushes, one per machine, started within two seconds. The digests + catch most of it now, one rule per message kind; the next message kind will not have one. From 0f163a3e7cb88cbd90a3129f60f0b3d03d83a9a0 Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 6 Oct 2026 02:31:21 +0200 Subject: [PATCH 2/2] Research 031 graduates: ADR 0227 and to-be 45 The operator approved the conclusion: the core must say when it is wrong instead of waiting to be looked at. Record the nine rules with their checks, and specify the parts and the six phases a build can take. --- .../00-overview.md | 18 +- ...cked-and-is-built-to-them-in-six-phases.md | 217 +++++++++ 02-DECISIONS/README.md | 1 + .../45-a-core-that-cannot-fail-silently.md | 436 ++++++++++++++++++ 03-DESIGN/01-to-be/README.md | 1 + 5 files changed, 666 insertions(+), 7 deletions(-) create mode 100644 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md create mode 100644 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md diff --git a/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md b/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md index 5bd3977..03dbed1 100644 --- a/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md +++ b/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md @@ -1,5 +1,5 @@ --- -status: active +status: graduated initiated: 2026-10-06 touches: - 00-META/how-we-build.md @@ -15,7 +15,9 @@ touches: - 03-DESIGN/01-to-be/25-the-bus-on-nats.md - 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md - 04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md -became: [] +became: + - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md + - 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md --- # 031 — A core that cannot fail silently @@ -55,8 +57,10 @@ until the bus was NATS; it is now. sequences, staged core upgrades with rollback, a facts snapshot for merge checks, a lab replay of every incident; five phases, ordered by risk removed; the five largest risks today. -**Next.** The operator reviews the principles and the order of the phases. Graduation would be one -record for the principles (likely amending how-we-build's non-negotiables), and to-be designs for the -condition store with its signals table, and for staged core upgrades. Two measurements are owed before -then: the bounds in the signals table, measured on the live mesh; and a week of the hand-act log, as the -baseline for the self-healing phase. +**Graduated 2026-10-06.** The operator approved the conclusion the same day (*"do the research and +implement it"*). The nine principles and the plan became +[ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md); +the mechanisms, the tables and the six phases became +[to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md). The two measurements owed +— the signals table's bounds and a week of the hand-act log — are Phase 0's work there, not +preconditions of the decision. diff --git a/02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md b/02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md new file mode 100644 index 0000000..78ae1c4 --- /dev/null +++ b/02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md @@ -0,0 +1,217 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-06 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md +--- + +# 227. The core holds nine rules, each checked, and is built to them in six phases + +## Context + +**The core is the controller, the node-engine and its launcher, the bus server, the node tools and the +console, the build seat, and the forge's announcer of merges** — everything a change passes through +before a module's own code runs. On 2026-10-06 the operator asked for it to be *"fully diagnosable, with +active monitoring, self-healing, self-upgradeable, self-monitoring … very sturdy, no ambiguities, clear +plan of execution, fail-proof setup"*, and approved [research 031](../01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md)'s +conclusion the same day: *"do the research and implement it"*. + +The evidence is [031/01](../01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md): + +- **92 issue reports in six days; 48 of them core failures.** Every one of the 48 was noticed because a + person or an agent looked. **None was raised by the mesh unasked.** In four the mesh's own answer + carried the fact for whoever asked; in none did it tell anybody. +- **Four faults came back through a different door after their first fix** (200 → 265, 230 → 264, + 257 → 261 → 267, 175 → 184 → 248): each fix closed an instance and left the class open. +- **The classes, by count:** races between actors with no explicit order (10 issues); commands or + arguments dropped with a default chosen in their place (8, two of them destructive — + [241](../04-ISSUES/241-one-unreadable-grants-file-dropped-every-database-on-the-control-node/00-report.md) + dropped seven databases on one unreadable file, + [244](../04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/00-report.md) + pushed every machine when one was named); outcomes never fed back (8); failures visible only as log + lines (9, the longest twenty-three hours); state with two writers (7); repairs by hand (at least + fifteen acts, the commonest a push by hand to unstick a plan waiting on a report); self-upgrade + breaking the core (10); checks that pass in CI and fail on the mesh's real facts (6); third-party + faults nobody compared against outcomes (3, one of them + [266](../04-ISSUES/266-a-merge-on-the-bus-was-never-handed-to-the-controller/00-report.md): + 23 merges unacted on over three days). +- **The parts mostly exist, one rule per message kind.** Declarations carry a sequence; reports do not. + Builds and plans are ordered; controllers have no epoch. `calls` keeps outcomes, in memory, and a + controller restart — every merge to its own repository — forgets them. ADR 0224 made one failure + kind a problem `status` reports and repairs where safe; nothing generalises it. + +**Checked against GENESIS.** The mission's core value *failure must be loud — prefer failing to lying* +is the rule this record enforces on the core. The context says *human agents are few, often one, and +usually asleep; anything requiring a human to notice it will be noticed late* — the 48-of-48 count is +that sentence measured. The effect promises that *the mesh notices when something is wrong before you +do* and asks *with the context, not a log line*. Nothing in 031 conflicts with GENESIS; the effort is +the gap between the effect and the as-is, counted. + +## Considered Options + +1. **Keep fixing issues one at a time.** Rejected: that is what the window did, and four classes + recurred through a different door. Point fixes converge on instances, not classes, and the next + message kind or the next reader of an unreadable file starts from nothing. +2. **Monitoring from outside: a metrics stack with alert rules** (an exporter per component, a time + series store, an alert manager). Rejected: it observes symptoms the core would still keep to itself + (a report never sent is not a metric), it is a second answer to questions the controller already + answers (how-we-build §5: a report is read from the system), it adds three services to a mesh with one + operator, and it repairs nothing. The mesh's facts are in the controller; the watcher belongs there, + with one watcher of the watcher outside it. +3. **Prevention first: order and one writer before anything else.** Rejected as the *first* step, kept + as the second. Prevention covers the classes already met; detection covers every class including + those not met yet, and is cheaper per day. Phase 1 makes the mesh say when it is wrong; Phase 2 + removes the largest class. +4. **Heal everything by default.** Rejected: a default of healing heals what is not understood, which is + how a repair destroys — 241's reconcile was, in its own terms, healing. Healing is narrowed to + *known* failures, ones repaired by hand twice, under a brake. +5. **A three-server bus cluster, so the bus can be upgraded live.** Not decided here: it is a change of + the foundation's shape and needs its own effort. Until then a bus upgrade is a planned, announced + step (rule 8). +6. **The lab as the place every core change is proven.** Rejected by [ADR 0149](0149-the-live-mesh-is-the-test-bed.md), + which stands: the live mesh is the test bed. Lab replays are kept for what must not be done to the + live mesh on purpose — two controllers at once, a deliberately broken controller build, a suppressed + signal — and each rule still has a live check. +7. **Nine principles as stated, the mechanisms M1–M9 and the phased roadmap of 031/03.** Chosen. + +## Decision + +**The nine rules below hold for the core. A change to a core repository is refused in review if it +breaks one, and each rule is checked as its row in *How it is checked* says.** They extend, and do not +replace, [research 017](../01-RESEARCH/017-a-mesh-that-heals-itself/01-the-intended-behaviour.md)'s +six for the loops that converge modules. + +1. **One writer per piece of state.** Every kind of state the core keeps has exactly one writer, named + in the writers table of [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md). + Anyone else asks that writer. Nobody computes a second answer to a question it already answers. The + controller holds a **lease**; only the holder acts. +2. **Everything that changes state carries its writer's order, and every receiver refuses what is + older.** Declarations, reports, plans, builds, calls and announcements carry writer, epoch and + sequence. A receiver keeps the highest it accepted per writer and refuses anything older, with a line + in the mesh's words and a counter. Arrival order never decides. +3. **Every command is answered at once, and its outcome is kept where it can be read later.** Within + the verb's declared bound, with a result or a call id. The outcome is durable: it outlives the process + that ran it, and is read or waited on by id. No answer depends on what the command does to the + transport carrying it. +4. **Nothing is dropped silently.** An input that is unknown, unreadable or unmet is refused by name — + which input, where, why. A default is never substituted for *I could not tell*, above all never + "empty". A reconcile that would withdraw more than a bound of what it holds stops and raises a + condition instead. +5. **Every expected signal has a watchdog; absence is a condition.** Every signal the core expects on + a cadence or after an act has a row in the signals table: emitter, trigger, bound, condition raised. + Silence past the bound is a condition naming what was expected, from whom, since when. +6. **The mesh checks itself continuously against live facts and says what it found outward.** The + design's invariants are probes run on a schedule against the running mesh (`doctor`). A violation is + a **condition** — durable, with since-when, evidence and who can resolve it — shown in `status` and + sent to the operator through the output channel. The checker's heartbeat is watched from a machine + that is not the control node, through a channel that does not pass through it. +7. **A known failure heals itself, under a brake, and every repair is said.** A failure repaired by + hand twice gets a healer: the ordinary path again, never a destructive act, with a budget and a + back-off, and one event saying what it did. A spent budget, or a repair that could only destroy, is + a condition, not a retry. Every repair a person makes on the core goes through a verb that records + who, what and why (the **hand-act log**). +8. **The core upgrades itself one machine at a time, health-gated, and rolls back on its own.** A new + controller, node-engine or node tools build is judged on its first machine by that component's + health probes, not by "reported applied". One not healthy within its bound is rolled back to the + last known good **by something other than itself**, and the rollback is a condition. The component + being replaced is never the only witness of its successor. A bus upgrade is a planned, announced + maintenance step, never a plain rollout. +9. **A check is fed the real mesh's facts before a change is merged.** A check whose verdict depends + on the environment runs against a facts snapshot exported by the controller — every machine + composed with the change and validated by the node-engine's validator — and a dependency's version + the mesh runs is the version its tests run. + +**The plan of execution is the six phases of to-be 45**, in that order, each ending at its own *done +when*: + +| Phase | What it delivers | Rules | +|---|---|---| +| 0 — finish what is in flight | the located core fixes rolled out; `calls` durable; `status` inside its bound; the hand-act log; the bus's planned upgrade as the first maintenance step; the durations the bounds are measured from | 3, 7, 8 | +| 1 — the mesh says when it is wrong | conditions; watchdogs from the signals table; the bus's advisories; `doctor`; the output channel in its minimal form and the second-machine watcher | 5, 6 | +| 2 — order and one writer | the controller's lease and epoch; a report's sequence; one apply queue on every machine; stale refusals counted; the writers table enforced; the empty-on-error lint; the withdrawal brake | 1, 2, 4 | +| 3 — healers | the first healers, each braked; a repeated hand act asks for one | 7 | +| 4 — core upgrades that roll back | a health definition per core component; the gate; rollback by a witness; the bus as a planned step | 8 | +| 5 — checks before merge, and replays | the facts snapshot and the compose-and-validate merge gate; versions tested as run; every core incident a replay | 9, and all | + +**The output channel is built in the smallest form research 028 allows.** One mesh seat, +`operator-channel`, accepting `notify` as a work queue and holding its open messages in its own state +([028 Q1](../01-RESEARCH/028-the-meshs-output-channel/03-open-questions.md), option a); the controller +emits condition events and the holder decides what is sent (028 Q5, option a); the Telegram channel the +operator required and the desktop notifier as the two channels; deduplicated by the condition's key; no +answering back except through the mesh's own verbs; a message carries roles and words, never an +address, a path or a secret, refused by the holder otherwise (028 Q8). Routing by presence, quiet hours, +answering back and the external dead-man service stay open in 028, whose graduation amends to-be 45. + +**ADR 0224's provider standing becomes the first condition kind**, unchanged in what it says and +when; its storage moves into the condition store. + +## Consequences + +- **`status` changes meaning.** It becomes, first, the list of open conditions; the all-well sentence + is "no open conditions", silenced ones included. A silenced condition is still open; silence stops + only its messages, for a stated time, with a reason, recorded as a hand act. Nobody resolves a + condition by hand: it clears when observation says so. +- **The bounds are measured, not guessed.** The signals table's first bounds are provisional; Phase 0 + records the durations they are set from, and Phase 1's first live week corrects every bound that + raised a condition that was not real. A corrected bound is a change to the table, reviewed like code. +- **The controller's restart stops being a forgetting.** Calls, conditions, the hand-act log and the + lease live in key-value buckets ([ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)); + a new holder of the lease marks a call the old one left running as abandoned, which is said. +- **The node-engine gains an order it did not have.** One apply queue replaces the delivery path and + the five-minute reconcile as two appliers; a report carries the declaration's sequence; a declaration + from an older controller epoch is refused. Issues 257, 261 and 267 become impossible rather than + handled. The controller–node-engine wire changes, so the rollout order is the node-engine first + (it accepts both shapes), the controller second. +- **More is checked before merge, and merges get slower.** The compose-and-validate gate runs every + machine through the validator on every core and catalogue merge. That is minutes, against the hours + each of 202, 236 and 263 cost. +- **A third party now carries operational words.** Telegram is not end-to-end encrypted for bots, so the + holder's content rule is the only thing between a condition's evidence and someone else's server. It + is enforced by the holder and tested there, not trusted to each source. +- **The watcher outside the control node holds the channel's secret.** That is one more machine with a + bot token, accepted for the one case nothing else covers: the control node or the bus is what failed. +- **The roadmap is six to eight weeks of focused work.** The first visible change — the mesh saying + when it is wrong — is inside the first two. Until Phase 2 lands, races are still caught by watchdogs, + not prevented. +- **The rules are not yet in how-we-build.** They are this record's until they are carried into + how-we-build §2 through playbook 05 (constitution sync), which is a separate change. + +## How it is checked + +| Rule | Checked by | From | +|---|---|---| +| 1. one writer | the writers table in to-be 45; a test per core repository that the code paths writing each kind are the named writer's; the controller refusing, at composition, a bus subject two components may publish on unless the table says it is shared; live, the `doctor` probe *exactly one lease holder, no stale-epoch message in the last interval* | Phase 2 | +| 2. order | a contract test per consumed message kind in the receiver's repository (deliver *n*, then *n−1*: refused and counted; epoch *e−1* after *e*: refused); a check listing every consumed subject against the tests that name it; live, the stale-refusals signal | Phase 2 | +| 3. answered, kept | a test walking every verb the controller announces (answers within its bound, with a result or an id); a test restarting the controller between a call and the read of its outcome; live, the `doctor` probe *status answers in full within ten seconds* and the *call hung* signal | Phase 0 | +| 4. nothing dropped | per component, a test feeding each input reader an unreadable, malformed and foreign input (a refusal, never an empty result); a lint in each core repository refusing an error branch that returns an empty collection; schema walks over every verb and placeholder namespace; a test emptying a provider's input and asserting nothing is withdrawn | Phase 2 | +| 5. watchdogs | a test generated from the signals table that suppresses each signal in turn and asserts its condition is raised within its bound and cleared when it returns; live, `doctor` reporting the age of the newest signal of every row | Phase 1 | +| 6. self-check, outward | a check over the to-be designs counting invariants with a live probe against those without, which may only go down; the second-machine watcher raising *self-check silent* when the controller is stopped; once, on a lab mesh, a broken invariant appearing in `status` and as a message within one probe interval | Phase 1 | +| 7. healers | a test per healer inducing its failure, asserting the repair, the event and the brake after the budget; the hand-act log's weekly count in `status`; a cause recorded twice raising *healer wanted* | Phase 3 | +| 8. staged upgrades | on a lab mesh, a broken build of the controller, the node-engine and the node tools (one that starts and does nothing, one that crashes, one that cannot reach the bus) each rolled back with no hand, ending on the previous build, said as a condition and a message; live, every core rollout's record (first machine, verdict, time to verdict, rolled back or not) readable through `plans` | Phase 4 | +| 9. real facts | the merge gate in mesh-controller, mesh-host and mesh-catalog failing a change that makes any machine of the snapshot fail to compose or validate, naming the machine's role and the module; a test that a pinned dependency's version equals the version in the snapshot; the replays of 236, 262, 263 and 266 failing on the commit before their fix | Phase 5 | +| the plan | each phase's *done when* in to-be 45, recorded in that design when met, with the design's status following | each phase | + +## References + +- [Research 031](../01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md) — the evidence, + the principles with the candidates not kept, the mechanisms and the roadmap. +- [To-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — the design this record + authorises. +- [Research 017](../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md) — the condition, and + the six principles for the loops; [research 028](../01-RESEARCH/028-the-meshs-output-channel/00-overview.md) + — the output channel, of which the minimal form is taken here. +- [ADR 0224](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md) — + *detected automatically, repaired where safe, loud where not*, generalised here. +- [ADR 0141](0141-the-host-delivers-its-own-successor.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md), + [ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md), + [ADR 0149](0149-the-live-mesh-is-the-test-bed.md), [ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md). +- Issues [187](../04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md), + [204](../04-ISSUES/204-a-controller-handover-re-sent-every-node-a-stale-declaration/00-report.md), + [241](../04-ISSUES/241-one-unreadable-grants-file-dropped-every-database-on-the-control-node/00-report.md), + [248](../04-ISSUES/248-the-controllers-event-consumer-replayed-a-week-and-held-every-merge-behind-it/00-report.md), + [264](../04-ISSUES/264-a-self-updating-engine-lost-the-report-of-the-apply-that-delivered-it/00-report.md), + [265](../04-ISSUES/265-a-push-outlived-its-caller-and-its-answer-was-refused/00-report.md), + [266](../04-ISSUES/266-a-merge-on-the-bus-was-never-handed-to-the-controller/00-report.md), + [267](../04-ISSUES/267-a-reconciles-report-overtook-the-apply-that-followed-it/00-report.md). diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 3a52a78..0ad0f32 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -199,6 +199,7 @@ python3 00-META/checks/index.py fail if stale - **0221** — [A push sends no build a policy or a plan holds back, except to the machine it names](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md) - **0222** — [A module is told where a mesh seat's holder is reached, and the controller writes no file a seat's holder owns](0222-a-module-is-told-where-a-mesh-seats-holder-is-reached-and-the-controller-writes-no-file-a-seats-holder-owns.md) - **0224** — [A provider that keeps failing a consumer is a problem the controller reports](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md) +- **0227** — [The core holds nine rules, each checked, and is built to them in six phases](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md new file mode 100644 index 0000000..4f70660 --- /dev/null +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -0,0 +1,436 @@ +--- +layer: to-be +status: designed +code: [] +updated: 2026-10-06 +decisions: + - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md + - 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md + - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md + - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md + - 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md +--- + +# 45 — A core that cannot fail silently + +**The core says when it is wrong, refuses what is stale or unreadable, repairs what it knows how to +repair, replaces itself one machine at a time with something other than itself watching, and is checked +against the mesh's real facts before a change merges** ([ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md), +from [research 031](../../01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md)). + +**The core**, here: the controller, the node-engine and its launcher, the bus server, the node tools and +the console, the build seat, and the forge's announcer of merges. + +This document is the build's specification. §1–§9 are the parts; §10 is the order they are built in, +each phase with what it delivers, in which repository, and when it is done. Every bound marked +*provisional* is set from the durations Phase 0 records and corrected in Phase 1's first live week. + +## The parts, and how a fact reaches the operator + +``` + signals ──────────► watchdogs ─┐ ┌─► status (open conditions first) + (heartbeats, reports, │ │ + plan progress, advisories) ├─► CONDITION STORE ───┼─► condition events ─► operator-channel ─► Telegram + doctor probes ─────────────────┤ (controller is │ holder desktop notifier + (live invariants, every 5 min) │ its only writer) └─► healers ─► repair, braked ─► event + events (provider failing, …) ──┘ │ + └─► hand-act log (people) + + second machine: watcher ── hears doctor's heartbeat? ── no, past bound ──► Telegram directly (not the bus) +``` + +Owning repositories: **mesh-controller** (the condition store, watchdogs, `doctor`, healers, the lease, +calls, the hand-act log, the facts snapshot), **mesh-host** (the node-engine: the apply queue, report +order, epoch refusal, the `report` verb, rollback witnessing), **mesh-tools** (the node tools and the +console: their heartbeat, passing every argument, health answers), **mesh-catalog** (the +`operator-channel` seat's holder and channels, the watcher, the provider brake, the catalogue's merge +gate), **mesh-lab** (the replays and the induced-failure scenarios). + +--- + +## 1. The writers table (rule 1) + +Every kind of state the core keeps, and the one component that writes it. Anyone else asks. + +| State | Writer | Kept in | Others | +|---|---|---|---| +| a machine's declaration | controller (lease holder) | the bus, last per subject | read | +| a machine's applied state and its report | the node-engine's apply queue (§6) | the machine; the report on the bus | the reconcile and a delivery *enqueue*, never apply | +| the controller lease | the controller instance holding it | key-value `mesh-controller_lease` | a candidate waits | +| plans and their tiers | controller (lease holder), compare-and-set on the plan's revision | the controller's store | read through `plans` | +| conditions | controller | key-value `mesh-controller_conditions` | raise or clear only through observations the controller reads | +| calls and their outcomes | controller | key-value `mesh-controller_calls` | read by id | +| the hand-act log | controller, through the verbs that act | key-value `mesh-controller_hand-acts` | — | +| stream definitions and bus permissions | controller | the bus | — | +| builds and their outcomes | the build seat's holder | its own state | the controller asks | +| a merge announced | **one** announcer per forge (the hook, or the poll when the hook is absent — never both) | the bus | — | +| a provider's standing | the provider | the provider's events | the controller keeps the newest word as a condition | +| the operator-channel's open messages | the seat's holder | its own key-value state | — | +| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads | + +A bus subject two components may publish on is refused when the controller composes grants, unless +this table marks it shared. Changing a writer is a change to this table, through a decision. + +## 2. The condition store (rules 5, 6) + +**A condition is a durable fact about something the mesh owns that is wrong.** It is raised and cleared +by observation only. + +**Where.** A key-value bucket, `mesh-controller_conditions`, written by the controller alone, one entry +per open condition. Every transition — raised, changed, silenced, cleared — is also appended to a +history kept ninety days, read through `conditions history`. + +**The key** names the thing and the kind, so the same fault said again is the same condition: +`..`, where scope is one of `machine`, `plan`, `call`, `build`, `merge`, `provider`, +`seat`, `bus`, `core`, `probe`, `mesh`. Examples of the shape: `machine..silent`, +`plan..stalled`, `provider....failing`, `core.controller..rolled-back`, +`bus..slow-consumer`. + +**The fields:** + +| Field | Holds | +|---|---| +| kind | the condition kind, from the signals table, the probe registry or an event kind | +| subject | scope, id, and the machine it concerns when there is one | +| severity | `urgent` (needs the operator now) or `warning` (when they can) — two levels, no more | +| summary | one line in the mesh's words | +| evidence | the newest observations, at most ten, each with its time | +| source | the signals-table row, probe or event that raised it | +| raised, last observed | times; and how many observations since raised | +| tried | each healer attempt: when, what, outcome | +| resolver | `self` (clears on observation), `healer:`, `operator`, or `agent` | +| silenced | until when, by whom, why — empty when not silenced | +| epoch | the controller lease epoch that last wrote it | + +**The life of one:** + +``` + (absent) ──observation past bound──► OPEN ──healer tries──► OPEN (tried += …) + │ ▲ │ budget spent + │ └─ seen again ◄───────┘──► OPEN, resolver: operator, severity: urgent + │ + silence (verb, ≤ 7 days, reason, hand act) ──► OPEN, silenced (no messages) + │ + observation says resolved ──► CLEARED: entry removed, transition kept in history +``` + +- **Nobody resolves a condition by hand.** It clears when the signal returns or the probe passes. +- **A condition cleared and raised again within ten minutes** reopens with its count increased; it is + not a new message. +- **The verbs:** `conditions` (open ones, filtered by scope, severity or machine), `conditions show + `, `conditions silence --for --why `, `conditions history`. +- **The events**, emitted by the controller on its own subjects: `condition-raised`, + `condition-changed` (severity or resolver changed; not every observation), `condition-cleared`. The + operator-channel's holder and any other surface consume them; the controller learns nothing about + telling. +- **`status`** lists open conditions first, urgent before warning, oldest first, silenced ones marked + with their expiry. The all-well sentence requires none open, silenced included. +- **ADR 0224's provider standing** is the first kind: `provider-failing`, raised by the provider's + event, cleared by its recovery, its silence after thirty minutes the row S8 below. + +## 3. The signals table (rule 5) + +Compiled into the controller. A watchdog per row; the test generated from the table suppresses each +signal and asserts its condition. `doctor signals` shows, for every row, the age of its newest signal. + +| Row | Signal | Emitter | Trigger | Bound | Condition kind | Severity | Healer | +|---|---|---|---|---|---|---|---| +| S1 | machine heartbeat | node-engine | its interval | 3 × interval, *provisional* | `silent` — not raised while the machine has declared itself asleep or shut down (ADR 0211) | warning; urgent after 30 min for the control node | — | +| S2 | report after a send | node-engine | each declaration sent | max(2 min, 3 × that machine's last apply duration), *provisional* | `sent-not-reported` | warning | H1 | +| S3 | plan tier progress | controller's plan | each tier entered | from the tier's build and apply durations, *provisional* | `stalled` (names the tier and what it waits on) | warning | H2 | +| S4 | the controller's event loop takes a message | controller | while its consumer has pending messages | 2 min | `controller-deaf` | urgent | — | +| S5 | a merge announced becomes a plan or *nothing reads it* | announcer → controller | each merge | 10 min (exists, issue 266) | `merge-not-acted` | urgent | — | +| S6 | a build asked → its outcome | build seat | each ask | the build's declared timeout | `ask-lost` | warning | — | +| S7 | a call running → finished | controller | each call | the verb's declared bound | `call-hung` | warning | — | +| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — | +| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H4 for `slow-consumer` on a resettable consumer | +| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — | +| S11 | node tools heartbeat | node tools | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — | +| S12 | the controller lease renewed | controller | every 5 s | 15 s | `lease-lost` | urgent | — | +| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer) | warning | — | +| S14 | facts snapshot exported | controller | daily | 2 days | `facts-stale` | warning | — | +| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days | `healer-wanted` | warning | — | + +The bus advisories (S9) cost one read-only subscription: the server already publishes them. The +controller translates each into a condition naming the thing in the mesh's words, as the refused reply +of issue 265 is translated today. + +## 4. The self-check: `doctor` (rule 6) + +A **probe registry** in the controller: each probe is a live invariant of a design, with an id, an +interval, a timeout and the condition kind it raises. The controller runs the registry every five +minutes; each probe has thirty seconds. A probe that errors or times out raises `probe-failed` for +itself — an unanswered probe is never a pass. + +| Probe | Asserts | From | +|---|---|---| +| D1 | every machine's declaration composes, and passes the node-engine's validation (the validator is a package of mesh-host the controller and the merge gate import — one validator) | 236, 263 | +| D2 | every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6 | 262 | +| D3 | every seat on record has a live holder that answers | 208, 218 | +| D4 | every kept archive is held by a manifest | 253 | +| D5 | exactly one lease holder; no message from a stale epoch in the last interval | 204 | +| D6 | every durable consumer exists with its definition and is near its stream's head | 248, 266 | +| D7 | every stream the controller defines exists with its definition | 208 | +| D8 | no address the mesh owns is in a ban list | 238 | +| D9 | `status` answers in full within ten seconds | 265 | +| D10 | every machine runs the node-engine and node tools builds its plan says, or is inside a plan's window | version split | +| H-* | the health probes of §8, run for every core component on every machine | rule 8 | + +- **`doctor`** answers the last run's verdict at once: per probe, pass, fail or failed-to-run, and age. + **`doctor run`** runs now under a call id. **`doctor probes`** lists the registry; **`doctor + signals`** the table's ages. +- **Every run ends with a heartbeat** event carrying the run's id and counts. That is S10. +- **The registry is the design's live form.** A check over the to-be designs counts invariants that + name a probe against those that do not; the number without may only go down. + +## 5. The output channel, minimal form (rule 6) + +From [research 028](../../01-RESEARCH/028-the-meshs-output-channel/00-overview.md), the smallest form +that works; its open questions stay open and its graduation amends this section. + +- **One mesh seat, `operator-channel`**, held once. It **accepts** `notify` as a work queue, so a + message waits for a holder; it keeps its **open messages** in its own key-value state, so a restart + forgets nothing; it **serves** `open` and `history`. +- **The holder consumes the controller's condition events** and decides what is sent. The controller + calls nobody. +- **Two channels**, each a module contributing itself to the seat: **Telegram** (a bot to the + operator's chat; its token and chat id are the channel module's secrets) and the **desktop + notifier** of the machine the operator is at. +- **A message** is: the condition's key, its subject (a machine's role, a module, a plan), kind, + severity, the one-line summary, since when, and the verb that shows more. **Deduplicated by the key.** +- **When:** on `condition-raised`; once more if still open after 1 hour (urgent) or 12 hours + (warning); on `condition-cleared`, by editing the first message where the channel can. A silenced + condition sends nothing. Urgent goes to both channels; warning to the desktop notifier when the + operator's session is there, otherwise to Telegram. +- **Rate:** at most twenty messages an hour; the excess is folded into one message naming them all. +- **What may leave the mesh:** roles and words. A message carrying an address, a path or anything + shaped like a secret is refused by the holder and raises `channel-refused` instead. +- **No answering back** in this form; acknowledging is `conditions silence`, through the mesh. +- **The watcher's watcher.** A module, `mesh-watcher`, assigned to one machine that is not the control + node, holds the Telegram channel's secret too. It listens for the self-check heartbeat (S10) and for + the bus itself; when either has been silent past its bound it sends to Telegram **directly over + HTTPS, not through the bus**, and says so again when they return. It is the only sender that does not + pass through the control node. + +## 6. Order: lease, epoch, report sequence, one apply queue (rules 1, 2) + +**The lease.** A controller instance acts — sends a declaration, writes a plan, a condition or a call +— only while it holds the key `holder` in `mesh-controller_lease`: written with compare-and-set, a +fifteen-second time to live, renewed every five seconds. The **epoch** is the bucket revision at which +it was taken. A starting controller waits for the key to be absent or expired. One that fails a renewal +stops acting at once and exits, so its service manager restarts it as a waiting candidate. + +``` + controller A (epoch 41) ──renew──renew──╳ (renewal refused)──► stops sending, exits + controller B ──wait──────────────take (epoch 57)──► acts; marks A's running calls abandoned + node-engine accepts 41 … then 57; refuses anything from 41 after 57, counted, reported +``` + +**What carries the order:** + +| Message | Carries | The receiver keeps, per writer | Refuses | +|---|---|---|---| +| declaration | epoch, sequence | highest epoch, then highest sequence | older epoch; same epoch, lower sequence | +| report | machine, the declaration's epoch and sequence it is about, the node-engine's own report sequence (kept on disk, increasing across restarts and self-updates) | highest declaration sequence, then report sequence | an account of an older declaration; an older report | +| plan write | the plan's revision, the epoch | — (compare-and-set) | a write against a revision already moved | +| build outcome | the build's ask id and order | newest per module | an older build finishing later (exists, 219) | +| call | call id, epoch | — | a finish from an epoch that is not the holder's, recorded as abandoned | +| merge announcement | forge, repository, commit, the announcer's sequence | newest per repository | a duplicate (one announcer) | + +**Every refusal** is one line in the receiver's log in the mesh's words, a counter, and — from the +node-engine — a report naming the refused declaration, so the controller sees it. The counter feeds +S13. + +**One apply queue on every machine.** The node-engine has one worker that applies; a delivery, the +five-minute reconcile, a self-update hand-over and a manual `apply` are four reasons to **enqueue**. +The queue holds at most one pending request, coalesced. When the worker starts it takes the newest +declaration held at that moment, applies it once, and sends one report naming the sequence it applied. +Nothing else in the node-engine applies. + +**The `report` verb.** The node-engine answers, on request, the report of the last declaration it +applied, from what it keeps on disk. It is what healer H1 asks. + +**Durable calls.** Every call's record — verb, arguments with secrets removed, caller, started, epoch, +state (`running`, `finished`, `failed`, `abandoned`), the answer bounded in size, finished — lives in +`mesh-controller_calls`, the last thousand or fourteen days. A new lease holder marks the previous +holder's running calls `abandoned`, which is said. + +## 7. Healers and the hand-act log (rule 7) + +**A healer** is a registered response to one condition kind: its repair (the ordinary path again), its +budget, its back-off, its brake, and the `healer-acted` event naming the condition, the act and the +outcome. A healer may not withdraw, delete or recreate data; such a repair is a condition for the +operator. + +| Healer | Condition | Repair | Budget, then | +|---|---|---|---| +| H1 | `sent-not-reported` | ask the machine's node-engine for `report`; if it names an older declaration, send the current one again | twice per condition, then resolver `operator`, urgent | +| H2 | `stalled` on a wait that is superseded or already finished | close the plan with its note, as `plans close` does | once | +| H3 | a seat holder without its worker (D3) | raise the seat's objects again (208) | once per holder per hour | +| H4 | `slow-consumer` far behind on a stream marked *resettable* in the stream table | `broker consumer-reset` (248) | once a day per consumer, then condition | +| H5 | `provider-failing` with the administrator refusing the mesh's secret | ADR 0224 §5 (exists, in the module) | as ADR 0224 | + +**The hand-act log.** Every verb that repairs by hand — a named `push` outside a plan, `plans close`, +`broker consumer-reset`, `conditions silence`, and `hand-act record` for an act done outside the mesh — +takes a required `--why` and writes an entry: who, which verb and arguments, why, when, the condition +key it addresses if any, and a **cause** (the condition kind, or a word the person gives). `status` +shows the week's count. A cause recorded twice within fourteen days raises `healer-wanted` (S15). + +## 8. Staged core upgrades and rollback (rule 8) + +**Health, per core component**, as probes in the registry (§4): + +| Component | Healthy when | Witness that rolls it back | +|---|---|---| +| controller | holds the lease within 60 s of starting; `status` answers in full within 10 s; `doctor` ran once | the node-engine on the control node, which keeps the previous controller build installed beside the new one and reads the lease bucket | +| node-engine | has reported its current declaration under its own build | its launcher (ADR 0141), which keeps the known-good | +| node tools | announced, and answer a ping within 5 s | the node-engine, which keeps the known-good | +| bus | every stream and durable consumer present (D6, D7); a request/reply round trip from every machine | none — a planned step, below | + +**The gate.** ADR 0218's first machine for a core component is judged by that component's health +probes passing three consecutive times over two minutes, within ten minutes of the apply. Only then do +the other machines follow. "Reported applied" is not enough. + +``` + merge ─► plan ─► first machine applies new build ─► health probes ×3 within 10 min? + ├─ yes ─► the rest follow, one tier at a time + └─ no ──► witness restores previous build + ─► condition core...rolled-back (urgent) + ─► plan halts that component, records why +``` + +- **Every core rollout leaves a record** in its plan: component, first machine, from and to build, + verdict, time to verdict, rolled back or not. Read through `plans`. +- **A rolled-back build is not retried** by the same plan. A newer merge makes a new plan. + +**The bus is a planned step.** A bus upgrade is a maintenance step a person starts through the +controller: the streams are snapshotted; a `bus-maintenance` condition is open for the step's duration; +the bus is replaced; afterwards D6, D7 and the round trip must pass, or the step is reported failed and +the snapshot is the way back. A step whose new version cannot be reverted (the bus's 2.10 → 2.11 is one) +says so before it starts and runs only on a person's explicit word, recorded as a hand act. Whether the +bus becomes a cluster that can be upgraded live is left to its own effort. + +## 9. Before merge: facts and replays (rule 9) + +**The facts snapshot.** The controller exports daily, and after any change of machines, assignments or +seats: every machine (a stable pseudonym of the same length as its name, its role, operating system, +C library, architecture, node-engine and node tools builds), assignments, settings keys and their +non-secret values, seats and their holders, the catalogue commit, and the versions the mesh runs of +the bus server, the store and the node-engine. No secret and no address: an address is replaced by one +from a documentation range. It is kept in the artifact store as `facts/latest`, where the build seat +reads it. + +**The merge gate.** In mesh-controller, mesh-host and mesh-catalog, a check composes every machine of +the snapshot with the change applied and runs the node-engine's validator over each. A change that +makes any machine fail to compose or validate fails its check, naming the machine's role and the +module. The resolver module's tests run under both C libraries the snapshot lists. A test in each core +repository asserts that a pinned dependency's version equals the version the snapshot says runs. + +**The replays.** mesh-lab carries a scripted scenario for each core incident, asserting the rule's +outcome, run on every merge to mesh-controller, mesh-host and mesh-tools: + +| Replay | Incident | Asserts | +|---|---|---| +| R1 | two controllers at once (204) | the second waits; nothing from the stale epoch is applied | +| R2 | a reconcile due during a push (257, 261, 267) | one apply, one report, the newest sequence | +| R3 | the node-engine self-updates during its report (230, 264) | the report arrives under the new build | +| R4 | the bus's authorization reloads during a call (265) | the call's outcome is readable by id | +| R5 | a consumer with several filter subjects under mixed traffic (266) | no announcement is skipped; S5 fires if one is | +| R6 | an unreadable contributions file (241) | refused by name; nothing withdrawn; a condition | +| R7 | the controller rebuilds itself mid-plan (214) | the plan continues under the new epoch | +| R8 | a broken controller, node-engine and node tools build | each rolled back with no hand; condition and message | +| R9 | each signal of §3 suppressed | its condition within its bound, cleared on return | + +A new core issue resolves with its replay added, or with a stated reason none is possible. The live +mesh stays the test bed ([ADR 0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md)): a +replay covers what must not be done to it on purpose, and every rule keeps a live check. + +--- + +## 10. The phases + +Ordered by risk removed per day: detection first, because it covers every class including those not +met yet. A phase is done when its *done when* holds; this section records the date when it does. + +### Phase 0 — Finish what is in flight + +| Repository | Delivers | +|---|---| +| mesh-controller | the located fixes of 244, 265, 266, 267 rolled out; `calls` moved into `mesh-controller_calls`; `status` answered from a summary the event loop keeps current, inside ten seconds; the hand-act log with `--why` on the repairing verbs and `hand-act record`; recording the durations the bounds come from (apply duration per machine, heartbeat gaps, plan tier durations, build durations) | +| mesh-host | the fixes of 264 and 257/261 rolled out to every machine | +| mesh-tools | the console passing a mesh seat's `node` (244) rolled out | +| mesh-catalog | the bus's 2.10 → 2.11 upgrade, done as the first planned bus step by hand (snapshot, announced, checked after), recorded as a hand act | + +**Done when:** the four located core issues resolve with their live checks; a controller restart keeps +every call's outcome; `status` answers in full within ten seconds five times in a row; the hand-act log +has a week of entries; the durations are recorded for every machine. + +### Phase 1 — The mesh says when it is wrong + +| Repository | Delivers | +|---|---| +| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S10 and S12–S14 with bounds set from Phase 0's durations; the bus advisories subscribed and translated (S9); `doctor` with D1–D10 and its heartbeat (§4); `status` led by open conditions; the test generated from the signals table | +| mesh-host | the heartbeat carries its interval; D1's validator published as a package the controller imports | +| mesh-tools | the node tools' heartbeat (S11) | +| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) | +| mesh-lab | R9: each signal suppressed in turn | + +**Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a +message; restoring it clears both. Stopping the controller makes the watcher send *self-check silent* +within twice the self-check's interval. Live: a week of conditions read back, every one real or its +bound corrected in the table. + +### Phase 2 — Order and one writer + +| Repository | Delivers | +|---|---| +| mesh-controller | the lease and epoch (§6); plans written by compare-and-set; a report kept by sequence; abandoned calls marked; S12, S13, D5; the writers table enforced at grant composition; contract tests for every consumed subject and the check listing them; the empty-on-error lint | +| mesh-host | one apply queue; the report sequence kept on disk; epoch refusal reported; the `report` verb; the contract tests and lint | +| mesh-tools | refusing an unreadable or unknown input by name; the lint | +| mesh-catalog | the withdrawal brake in the providers' loop: a reconcile that would withdraw more than one consumer, or a set fraction, stops and raises a condition | +| mesh-lab | R1, R2, R6 | + +**Done when:** R1 ends with nothing from the stale epoch applied, R2 with one report of the newest +sequence, R6 with the withdrawal braked; every consumed subject has its contract test. + +### Phase 3 — Healers + +| Repository | Delivers | +|---|---| +| mesh-controller | the healer registry and H1–H4, each braked and said; S15 | +| mesh-lab | an induced failure per healer | + +**Done when:** a lab mesh recovers from each induced failure with no hand, says so, and brakes after +its budget. Live: a week with no cause repeated in the hand-act log. + +### Phase 4 — Core upgrades that roll back + +| Repository | Delivers | +|---|---| +| mesh-controller | the health probes of §8; the gate on a core component's first machine; the rollout record in the plan; the bus maintenance step as a verb | +| mesh-host | keeping the previous controller and node tools builds; restoring one when its health is not met in bound, watching the lease bucket for the controller | +| mesh-tools | answering the health ping | +| mesh-lab | R3, R7, R8 | + +**Done when:** on a lab mesh, a broken build of the controller, the node-engine and the node tools — +one that starts and does nothing, one that crashes, one that cannot reach the bus — is each rolled +back with no hand, the mesh ends on the previous build, and a condition and a message say so. Live: the +next three core rollouts each record a health verdict. + +### Phase 5 — Checks before merge, and the replays + +| Repository | Delivers | +|---|---| +| mesh-controller | the facts snapshot and S14 | +| mesh-controller, mesh-host, mesh-catalog | the compose-and-validate merge gate; versions tested as run; the resolver's tests under both C libraries | +| mesh-lab | R4, R5 and the rest of the window's incidents; running the replays on every core merge | + +**Done when:** the replays of 236, 262, 263 and 266 fail on the commit before their fix and pass after; +a new core issue cannot resolve without a replay or a stated reason. + +## What is not decided here + +- The bus as a cluster of three, to upgrade it live. +- Routing by presence, quiet hours, answering back through a channel, and an external dead-man + service — research 028. +- A condition that needs judgement handed to an agent as work — research 017. diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index ab52764..dbac323 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -45,6 +45,7 @@ document is written and this one's status becomes `implemented`. | [`38-building-the-operators-machine.md`](38-building-the-operators-machine.md) | **In progress.** The work of design 37 as packages: the runtime serves many modules, the controller composes one per node, the console becomes its serving mode, the packet filter moves first, then the shell and the service manager — tested on the live mesh by the operator's decision | [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md), [0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md), [0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md) | | [`41-the-shell-and-the-accounts-environment.md`](41-the-shell-and-the-accounts-environment.md) | **In progress.** The shell and the account's environment as modules: an environment module every module contributes variables and `PATH` entries to, shell code contributed to the login shell in named slots, the prompt and plugins as modules, the host giving a login back, and the service manager's module finished | [ADR 0203](../../02-DECISIONS/0203-the-accounts-environment-is-one-modules-and-every-module-contributes-to-it.md), [ADR 0204](../../02-DECISIONS/0204-a-module-contributes-shell-code-to-the-login-shell-in-named-slots.md), [ADR 0205](../../02-DECISIONS/0205-software-the-distribution-does-not-package-ships-as-a-pinned-archive-of-the-module.md) | | [`42-the-machines-modules-in-order.md`](42-the-machines-modules-in-order.md) | **In progress.** The order the machines' modules of research 026 and 027 are built and rolled out: every machine's first (sudo, localization, time sync, pacman, logrotate, avahi, systemd, docker, `~/.ssh`, scripts, kernel), then both workstations', then one machine model's; each proven on one workstation before the rest | [ADR 0173](../../02-DECISIONS/0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md), [ADR 0182](../../02-DECISIONS/0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md), [ADR 0205](../../02-DECISIONS/0205-software-the-distribution-does-not-package-ships-as-a-pinned-archive-of-the-module.md) | +| [`45-a-core-that-cannot-fail-silently.md`](45-a-core-that-cannot-fail-silently.md) | **Designed.** The core says when it is wrong, refuses what is stale or unreadable, heals what it knows, upgrades one machine at a time with a witness that rolls it back, and is checked against the real mesh before merge: the writers and signals tables, the condition store, `doctor`, the minimal output channel, healers, the lease and report order, staged upgrades, the facts snapshot and replays, in six phases | [ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md), [ADR 0224](../../02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md), [ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md) | ## Not yet written