From 0f163a3e7cb88cbd90a3129f60f0b3d03d83a9a0 Mon Sep 17 00:00:00 2001 From: jochen Date: Tue, 6 Oct 2026 02:31:21 +0200 Subject: [PATCH] Research 031 graduates: ADR 0227 and to-be 45 The operator approved the conclusion: the core must say when it is wrong instead of waiting to be looked at. Record the nine rules with their checks, and specify the parts and the six phases a build can take. --- .../00-overview.md | 18 +- ...cked-and-is-built-to-them-in-six-phases.md | 217 +++++++++ 02-DECISIONS/README.md | 1 + .../45-a-core-that-cannot-fail-silently.md | 436 ++++++++++++++++++ 03-DESIGN/01-to-be/README.md | 1 + 5 files changed, 666 insertions(+), 7 deletions(-) create mode 100644 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md create mode 100644 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md diff --git a/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md b/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md index 5bd3977..03dbed1 100644 --- a/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md +++ b/01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md @@ -1,5 +1,5 @@ --- -status: active +status: graduated initiated: 2026-10-06 touches: - 00-META/how-we-build.md @@ -15,7 +15,9 @@ touches: - 03-DESIGN/01-to-be/25-the-bus-on-nats.md - 03-DESIGN/01-to-be/30-the-mesh-updates-itself-on-a-push.md - 04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md -became: [] +became: + - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md + - 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md --- # 031 — A core that cannot fail silently @@ -55,8 +57,10 @@ until the bus was NATS; it is now. sequences, staged core upgrades with rollback, a facts snapshot for merge checks, a lab replay of every incident; five phases, ordered by risk removed; the five largest risks today. -**Next.** The operator reviews the principles and the order of the phases. Graduation would be one -record for the principles (likely amending how-we-build's non-negotiables), and to-be designs for the -condition store with its signals table, and for staged core upgrades. Two measurements are owed before -then: the bounds in the signals table, measured on the live mesh; and a week of the hand-act log, as the -baseline for the self-healing phase. +**Graduated 2026-10-06.** The operator approved the conclusion the same day (*"do the research and +implement it"*). The nine principles and the plan became +[ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md); +the mechanisms, the tables and the six phases became +[to-be 45](../../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md). The two measurements owed +— the signals table's bounds and a week of the hand-act log — are Phase 0's work there, not +preconditions of the decision. diff --git a/02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md b/02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md new file mode 100644 index 0000000..78ae1c4 --- /dev/null +++ b/02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md @@ -0,0 +1,217 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-06 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md +--- + +# 227. The core holds nine rules, each checked, and is built to them in six phases + +## Context + +**The core is the controller, the node-engine and its launcher, the bus server, the node tools and the +console, the build seat, and the forge's announcer of merges** — everything a change passes through +before a module's own code runs. On 2026-10-06 the operator asked for it to be *"fully diagnosable, with +active monitoring, self-healing, self-upgradeable, self-monitoring … very sturdy, no ambiguities, clear +plan of execution, fail-proof setup"*, and approved [research 031](../01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md)'s +conclusion the same day: *"do the research and implement it"*. + +The evidence is [031/01](../01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md): + +- **92 issue reports in six days; 48 of them core failures.** Every one of the 48 was noticed because a + person or an agent looked. **None was raised by the mesh unasked.** In four the mesh's own answer + carried the fact for whoever asked; in none did it tell anybody. +- **Four faults came back through a different door after their first fix** (200 → 265, 230 → 264, + 257 → 261 → 267, 175 → 184 → 248): each fix closed an instance and left the class open. +- **The classes, by count:** races between actors with no explicit order (10 issues); commands or + arguments dropped with a default chosen in their place (8, two of them destructive — + [241](../04-ISSUES/241-one-unreadable-grants-file-dropped-every-database-on-the-control-node/00-report.md) + dropped seven databases on one unreadable file, + [244](../04-ISSUES/244-a-verb-whose-schema-is-empty-cannot-be-called-through-the-console/00-report.md) + pushed every machine when one was named); outcomes never fed back (8); failures visible only as log + lines (9, the longest twenty-three hours); state with two writers (7); repairs by hand (at least + fifteen acts, the commonest a push by hand to unstick a plan waiting on a report); self-upgrade + breaking the core (10); checks that pass in CI and fail on the mesh's real facts (6); third-party + faults nobody compared against outcomes (3, one of them + [266](../04-ISSUES/266-a-merge-on-the-bus-was-never-handed-to-the-controller/00-report.md): + 23 merges unacted on over three days). +- **The parts mostly exist, one rule per message kind.** Declarations carry a sequence; reports do not. + Builds and plans are ordered; controllers have no epoch. `calls` keeps outcomes, in memory, and a + controller restart — every merge to its own repository — forgets them. ADR 0224 made one failure + kind a problem `status` reports and repairs where safe; nothing generalises it. + +**Checked against GENESIS.** The mission's core value *failure must be loud — prefer failing to lying* +is the rule this record enforces on the core. The context says *human agents are few, often one, and +usually asleep; anything requiring a human to notice it will be noticed late* — the 48-of-48 count is +that sentence measured. The effect promises that *the mesh notices when something is wrong before you +do* and asks *with the context, not a log line*. Nothing in 031 conflicts with GENESIS; the effort is +the gap between the effect and the as-is, counted. + +## Considered Options + +1. **Keep fixing issues one at a time.** Rejected: that is what the window did, and four classes + recurred through a different door. Point fixes converge on instances, not classes, and the next + message kind or the next reader of an unreadable file starts from nothing. +2. **Monitoring from outside: a metrics stack with alert rules** (an exporter per component, a time + series store, an alert manager). Rejected: it observes symptoms the core would still keep to itself + (a report never sent is not a metric), it is a second answer to questions the controller already + answers (how-we-build §5: a report is read from the system), it adds three services to a mesh with one + operator, and it repairs nothing. The mesh's facts are in the controller; the watcher belongs there, + with one watcher of the watcher outside it. +3. **Prevention first: order and one writer before anything else.** Rejected as the *first* step, kept + as the second. Prevention covers the classes already met; detection covers every class including + those not met yet, and is cheaper per day. Phase 1 makes the mesh say when it is wrong; Phase 2 + removes the largest class. +4. **Heal everything by default.** Rejected: a default of healing heals what is not understood, which is + how a repair destroys — 241's reconcile was, in its own terms, healing. Healing is narrowed to + *known* failures, ones repaired by hand twice, under a brake. +5. **A three-server bus cluster, so the bus can be upgraded live.** Not decided here: it is a change of + the foundation's shape and needs its own effort. Until then a bus upgrade is a planned, announced + step (rule 8). +6. **The lab as the place every core change is proven.** Rejected by [ADR 0149](0149-the-live-mesh-is-the-test-bed.md), + which stands: the live mesh is the test bed. Lab replays are kept for what must not be done to the + live mesh on purpose — two controllers at once, a deliberately broken controller build, a suppressed + signal — and each rule still has a live check. +7. **Nine principles as stated, the mechanisms M1–M9 and the phased roadmap of 031/03.** Chosen. + +## Decision + +**The nine rules below hold for the core. A change to a core repository is refused in review if it +breaks one, and each rule is checked as its row in *How it is checked* says.** They extend, and do not +replace, [research 017](../01-RESEARCH/017-a-mesh-that-heals-itself/01-the-intended-behaviour.md)'s +six for the loops that converge modules. + +1. **One writer per piece of state.** Every kind of state the core keeps has exactly one writer, named + in the writers table of [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md). + Anyone else asks that writer. Nobody computes a second answer to a question it already answers. The + controller holds a **lease**; only the holder acts. +2. **Everything that changes state carries its writer's order, and every receiver refuses what is + older.** Declarations, reports, plans, builds, calls and announcements carry writer, epoch and + sequence. A receiver keeps the highest it accepted per writer and refuses anything older, with a line + in the mesh's words and a counter. Arrival order never decides. +3. **Every command is answered at once, and its outcome is kept where it can be read later.** Within + the verb's declared bound, with a result or a call id. The outcome is durable: it outlives the process + that ran it, and is read or waited on by id. No answer depends on what the command does to the + transport carrying it. +4. **Nothing is dropped silently.** An input that is unknown, unreadable or unmet is refused by name — + which input, where, why. A default is never substituted for *I could not tell*, above all never + "empty". A reconcile that would withdraw more than a bound of what it holds stops and raises a + condition instead. +5. **Every expected signal has a watchdog; absence is a condition.** Every signal the core expects on + a cadence or after an act has a row in the signals table: emitter, trigger, bound, condition raised. + Silence past the bound is a condition naming what was expected, from whom, since when. +6. **The mesh checks itself continuously against live facts and says what it found outward.** The + design's invariants are probes run on a schedule against the running mesh (`doctor`). A violation is + a **condition** — durable, with since-when, evidence and who can resolve it — shown in `status` and + sent to the operator through the output channel. The checker's heartbeat is watched from a machine + that is not the control node, through a channel that does not pass through it. +7. **A known failure heals itself, under a brake, and every repair is said.** A failure repaired by + hand twice gets a healer: the ordinary path again, never a destructive act, with a budget and a + back-off, and one event saying what it did. A spent budget, or a repair that could only destroy, is + a condition, not a retry. Every repair a person makes on the core goes through a verb that records + who, what and why (the **hand-act log**). +8. **The core upgrades itself one machine at a time, health-gated, and rolls back on its own.** A new + controller, node-engine or node tools build is judged on its first machine by that component's + health probes, not by "reported applied". One not healthy within its bound is rolled back to the + last known good **by something other than itself**, and the rollback is a condition. The component + being replaced is never the only witness of its successor. A bus upgrade is a planned, announced + maintenance step, never a plain rollout. +9. **A check is fed the real mesh's facts before a change is merged.** A check whose verdict depends + on the environment runs against a facts snapshot exported by the controller — every machine + composed with the change and validated by the node-engine's validator — and a dependency's version + the mesh runs is the version its tests run. + +**The plan of execution is the six phases of to-be 45**, in that order, each ending at its own *done +when*: + +| Phase | What it delivers | Rules | +|---|---|---| +| 0 — finish what is in flight | the located core fixes rolled out; `calls` durable; `status` inside its bound; the hand-act log; the bus's planned upgrade as the first maintenance step; the durations the bounds are measured from | 3, 7, 8 | +| 1 — the mesh says when it is wrong | conditions; watchdogs from the signals table; the bus's advisories; `doctor`; the output channel in its minimal form and the second-machine watcher | 5, 6 | +| 2 — order and one writer | the controller's lease and epoch; a report's sequence; one apply queue on every machine; stale refusals counted; the writers table enforced; the empty-on-error lint; the withdrawal brake | 1, 2, 4 | +| 3 — healers | the first healers, each braked; a repeated hand act asks for one | 7 | +| 4 — core upgrades that roll back | a health definition per core component; the gate; rollback by a witness; the bus as a planned step | 8 | +| 5 — checks before merge, and replays | the facts snapshot and the compose-and-validate merge gate; versions tested as run; every core incident a replay | 9, and all | + +**The output channel is built in the smallest form research 028 allows.** One mesh seat, +`operator-channel`, accepting `notify` as a work queue and holding its open messages in its own state +([028 Q1](../01-RESEARCH/028-the-meshs-output-channel/03-open-questions.md), option a); the controller +emits condition events and the holder decides what is sent (028 Q5, option a); the Telegram channel the +operator required and the desktop notifier as the two channels; deduplicated by the condition's key; no +answering back except through the mesh's own verbs; a message carries roles and words, never an +address, a path or a secret, refused by the holder otherwise (028 Q8). Routing by presence, quiet hours, +answering back and the external dead-man service stay open in 028, whose graduation amends to-be 45. + +**ADR 0224's provider standing becomes the first condition kind**, unchanged in what it says and +when; its storage moves into the condition store. + +## Consequences + +- **`status` changes meaning.** It becomes, first, the list of open conditions; the all-well sentence + is "no open conditions", silenced ones included. A silenced condition is still open; silence stops + only its messages, for a stated time, with a reason, recorded as a hand act. Nobody resolves a + condition by hand: it clears when observation says so. +- **The bounds are measured, not guessed.** The signals table's first bounds are provisional; Phase 0 + records the durations they are set from, and Phase 1's first live week corrects every bound that + raised a condition that was not real. A corrected bound is a change to the table, reviewed like code. +- **The controller's restart stops being a forgetting.** Calls, conditions, the hand-act log and the + lease live in key-value buckets ([ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md)); + a new holder of the lease marks a call the old one left running as abandoned, which is said. +- **The node-engine gains an order it did not have.** One apply queue replaces the delivery path and + the five-minute reconcile as two appliers; a report carries the declaration's sequence; a declaration + from an older controller epoch is refused. Issues 257, 261 and 267 become impossible rather than + handled. The controller–node-engine wire changes, so the rollout order is the node-engine first + (it accepts both shapes), the controller second. +- **More is checked before merge, and merges get slower.** The compose-and-validate gate runs every + machine through the validator on every core and catalogue merge. That is minutes, against the hours + each of 202, 236 and 263 cost. +- **A third party now carries operational words.** Telegram is not end-to-end encrypted for bots, so the + holder's content rule is the only thing between a condition's evidence and someone else's server. It + is enforced by the holder and tested there, not trusted to each source. +- **The watcher outside the control node holds the channel's secret.** That is one more machine with a + bot token, accepted for the one case nothing else covers: the control node or the bus is what failed. +- **The roadmap is six to eight weeks of focused work.** The first visible change — the mesh saying + when it is wrong — is inside the first two. Until Phase 2 lands, races are still caught by watchdogs, + not prevented. +- **The rules are not yet in how-we-build.** They are this record's until they are carried into + how-we-build §2 through playbook 05 (constitution sync), which is a separate change. + +## How it is checked + +| Rule | Checked by | From | +|---|---|---| +| 1. one writer | the writers table in to-be 45; a test per core repository that the code paths writing each kind are the named writer's; the controller refusing, at composition, a bus subject two components may publish on unless the table says it is shared; live, the `doctor` probe *exactly one lease holder, no stale-epoch message in the last interval* | Phase 2 | +| 2. order | a contract test per consumed message kind in the receiver's repository (deliver *n*, then *n−1*: refused and counted; epoch *e−1* after *e*: refused); a check listing every consumed subject against the tests that name it; live, the stale-refusals signal | Phase 2 | +| 3. answered, kept | a test walking every verb the controller announces (answers within its bound, with a result or an id); a test restarting the controller between a call and the read of its outcome; live, the `doctor` probe *status answers in full within ten seconds* and the *call hung* signal | Phase 0 | +| 4. nothing dropped | per component, a test feeding each input reader an unreadable, malformed and foreign input (a refusal, never an empty result); a lint in each core repository refusing an error branch that returns an empty collection; schema walks over every verb and placeholder namespace; a test emptying a provider's input and asserting nothing is withdrawn | Phase 2 | +| 5. watchdogs | a test generated from the signals table that suppresses each signal in turn and asserts its condition is raised within its bound and cleared when it returns; live, `doctor` reporting the age of the newest signal of every row | Phase 1 | +| 6. self-check, outward | a check over the to-be designs counting invariants with a live probe against those without, which may only go down; the second-machine watcher raising *self-check silent* when the controller is stopped; once, on a lab mesh, a broken invariant appearing in `status` and as a message within one probe interval | Phase 1 | +| 7. healers | a test per healer inducing its failure, asserting the repair, the event and the brake after the budget; the hand-act log's weekly count in `status`; a cause recorded twice raising *healer wanted* | Phase 3 | +| 8. staged upgrades | on a lab mesh, a broken build of the controller, the node-engine and the node tools (one that starts and does nothing, one that crashes, one that cannot reach the bus) each rolled back with no hand, ending on the previous build, said as a condition and a message; live, every core rollout's record (first machine, verdict, time to verdict, rolled back or not) readable through `plans` | Phase 4 | +| 9. real facts | the merge gate in mesh-controller, mesh-host and mesh-catalog failing a change that makes any machine of the snapshot fail to compose or validate, naming the machine's role and the module; a test that a pinned dependency's version equals the version in the snapshot; the replays of 236, 262, 263 and 266 failing on the commit before their fix | Phase 5 | +| the plan | each phase's *done when* in to-be 45, recorded in that design when met, with the design's status following | each phase | + +## References + +- [Research 031](../01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md) — the evidence, + the principles with the candidates not kept, the mechanisms and the roadmap. +- [To-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — the design this record + authorises. +- [Research 017](../01-RESEARCH/017-a-mesh-that-heals-itself/00-overview.md) — the condition, and + the six principles for the loops; [research 028](../01-RESEARCH/028-the-meshs-output-channel/00-overview.md) + — the output channel, of which the minimal form is taken here. +- [ADR 0224](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md) — + *detected automatically, repaired where safe, loud where not*, generalised here. +- [ADR 0141](0141-the-host-delivers-its-own-successor.md), [ADR 0162](0162-a-merge-produces-a-tiered-plan-the-mesh-keeps.md), + [ADR 0218](0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md), + [ADR 0149](0149-the-live-mesh-is-the-test-bed.md), [ADR 0201](0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md). +- Issues [187](../04-ISSUES/187-the-mesh-tells-nobody-when-it-stops-working/00-report.md), + [204](../04-ISSUES/204-a-controller-handover-re-sent-every-node-a-stale-declaration/00-report.md), + [241](../04-ISSUES/241-one-unreadable-grants-file-dropped-every-database-on-the-control-node/00-report.md), + [248](../04-ISSUES/248-the-controllers-event-consumer-replayed-a-week-and-held-every-merge-behind-it/00-report.md), + [264](../04-ISSUES/264-a-self-updating-engine-lost-the-report-of-the-apply-that-delivered-it/00-report.md), + [265](../04-ISSUES/265-a-push-outlived-its-caller-and-its-answer-was-refused/00-report.md), + [266](../04-ISSUES/266-a-merge-on-the-bus-was-never-handed-to-the-controller/00-report.md), + [267](../04-ISSUES/267-a-reconciles-report-overtook-the-apply-that-followed-it/00-report.md). diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 3a52a78..0ad0f32 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -199,6 +199,7 @@ python3 00-META/checks/index.py fail if stale - **0221** — [A push sends no build a policy or a plan holds back, except to the machine it names](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md) - **0222** — [A module is told where a mesh seat's holder is reached, and the controller writes no file a seat's holder owns](0222-a-module-is-told-where-a-mesh-seats-holder-is-reached-and-the-controller-writes-no-file-a-seats-holder-owns.md) - **0224** — [A provider that keeps failing a consumer is a problem the controller reports](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md) +- **0227** — [The core holds nine rules, each checked, and is built to them in six phases](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md new file mode 100644 index 0000000..4f70660 --- /dev/null +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -0,0 +1,436 @@ +--- +layer: to-be +status: designed +code: [] +updated: 2026-10-06 +decisions: + - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md + - 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md + - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md + - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md + - 02-DECISIONS/0201-a-module-keeps-its-current-state-in-key-value-buckets-it-declares-and-reaches-through-the-runtime.md +--- + +# 45 — A core that cannot fail silently + +**The core says when it is wrong, refuses what is stale or unreadable, repairs what it knows how to +repair, replaces itself one machine at a time with something other than itself watching, and is checked +against the mesh's real facts before a change merges** ([ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md), +from [research 031](../../01-RESEARCH/031-a-core-that-cannot-fail-silently/00-overview.md)). + +**The core**, here: the controller, the node-engine and its launcher, the bus server, the node tools and +the console, the build seat, and the forge's announcer of merges. + +This document is the build's specification. §1–§9 are the parts; §10 is the order they are built in, +each phase with what it delivers, in which repository, and when it is done. Every bound marked +*provisional* is set from the durations Phase 0 records and corrected in Phase 1's first live week. + +## The parts, and how a fact reaches the operator + +``` + signals ──────────► watchdogs ─┐ ┌─► status (open conditions first) + (heartbeats, reports, │ │ + plan progress, advisories) ├─► CONDITION STORE ───┼─► condition events ─► operator-channel ─► Telegram + doctor probes ─────────────────┤ (controller is │ holder desktop notifier + (live invariants, every 5 min) │ its only writer) └─► healers ─► repair, braked ─► event + events (provider failing, …) ──┘ │ + └─► hand-act log (people) + + second machine: watcher ── hears doctor's heartbeat? ── no, past bound ──► Telegram directly (not the bus) +``` + +Owning repositories: **mesh-controller** (the condition store, watchdogs, `doctor`, healers, the lease, +calls, the hand-act log, the facts snapshot), **mesh-host** (the node-engine: the apply queue, report +order, epoch refusal, the `report` verb, rollback witnessing), **mesh-tools** (the node tools and the +console: their heartbeat, passing every argument, health answers), **mesh-catalog** (the +`operator-channel` seat's holder and channels, the watcher, the provider brake, the catalogue's merge +gate), **mesh-lab** (the replays and the induced-failure scenarios). + +--- + +## 1. The writers table (rule 1) + +Every kind of state the core keeps, and the one component that writes it. Anyone else asks. + +| State | Writer | Kept in | Others | +|---|---|---|---| +| a machine's declaration | controller (lease holder) | the bus, last per subject | read | +| a machine's applied state and its report | the node-engine's apply queue (§6) | the machine; the report on the bus | the reconcile and a delivery *enqueue*, never apply | +| the controller lease | the controller instance holding it | key-value `mesh-controller_lease` | a candidate waits | +| plans and their tiers | controller (lease holder), compare-and-set on the plan's revision | the controller's store | read through `plans` | +| conditions | controller | key-value `mesh-controller_conditions` | raise or clear only through observations the controller reads | +| calls and their outcomes | controller | key-value `mesh-controller_calls` | read by id | +| the hand-act log | controller, through the verbs that act | key-value `mesh-controller_hand-acts` | — | +| stream definitions and bus permissions | controller | the bus | — | +| builds and their outcomes | the build seat's holder | its own state | the controller asks | +| a merge announced | **one** announcer per forge (the hook, or the poll when the hook is absent — never both) | the bus | — | +| a provider's standing | the provider | the provider's events | the controller keeps the newest word as a condition | +| the operator-channel's open messages | the seat's holder | its own key-value state | — | +| the facts snapshot | controller | the artifact store, `facts/latest` | the build seat reads | + +A bus subject two components may publish on is refused when the controller composes grants, unless +this table marks it shared. Changing a writer is a change to this table, through a decision. + +## 2. The condition store (rules 5, 6) + +**A condition is a durable fact about something the mesh owns that is wrong.** It is raised and cleared +by observation only. + +**Where.** A key-value bucket, `mesh-controller_conditions`, written by the controller alone, one entry +per open condition. Every transition — raised, changed, silenced, cleared — is also appended to a +history kept ninety days, read through `conditions history`. + +**The key** names the thing and the kind, so the same fault said again is the same condition: +`..`, where scope is one of `machine`, `plan`, `call`, `build`, `merge`, `provider`, +`seat`, `bus`, `core`, `probe`, `mesh`. Examples of the shape: `machine..silent`, +`plan..stalled`, `provider....failing`, `core.controller..rolled-back`, +`bus..slow-consumer`. + +**The fields:** + +| Field | Holds | +|---|---| +| kind | the condition kind, from the signals table, the probe registry or an event kind | +| subject | scope, id, and the machine it concerns when there is one | +| severity | `urgent` (needs the operator now) or `warning` (when they can) — two levels, no more | +| summary | one line in the mesh's words | +| evidence | the newest observations, at most ten, each with its time | +| source | the signals-table row, probe or event that raised it | +| raised, last observed | times; and how many observations since raised | +| tried | each healer attempt: when, what, outcome | +| resolver | `self` (clears on observation), `healer:`, `operator`, or `agent` | +| silenced | until when, by whom, why — empty when not silenced | +| epoch | the controller lease epoch that last wrote it | + +**The life of one:** + +``` + (absent) ──observation past bound──► OPEN ──healer tries──► OPEN (tried += …) + │ ▲ │ budget spent + │ └─ seen again ◄───────┘──► OPEN, resolver: operator, severity: urgent + │ + silence (verb, ≤ 7 days, reason, hand act) ──► OPEN, silenced (no messages) + │ + observation says resolved ──► CLEARED: entry removed, transition kept in history +``` + +- **Nobody resolves a condition by hand.** It clears when the signal returns or the probe passes. +- **A condition cleared and raised again within ten minutes** reopens with its count increased; it is + not a new message. +- **The verbs:** `conditions` (open ones, filtered by scope, severity or machine), `conditions show + `, `conditions silence --for --why `, `conditions history`. +- **The events**, emitted by the controller on its own subjects: `condition-raised`, + `condition-changed` (severity or resolver changed; not every observation), `condition-cleared`. The + operator-channel's holder and any other surface consume them; the controller learns nothing about + telling. +- **`status`** lists open conditions first, urgent before warning, oldest first, silenced ones marked + with their expiry. The all-well sentence requires none open, silenced included. +- **ADR 0224's provider standing** is the first kind: `provider-failing`, raised by the provider's + event, cleared by its recovery, its silence after thirty minutes the row S8 below. + +## 3. The signals table (rule 5) + +Compiled into the controller. A watchdog per row; the test generated from the table suppresses each +signal and asserts its condition. `doctor signals` shows, for every row, the age of its newest signal. + +| Row | Signal | Emitter | Trigger | Bound | Condition kind | Severity | Healer | +|---|---|---|---|---|---|---|---| +| S1 | machine heartbeat | node-engine | its interval | 3 × interval, *provisional* | `silent` — not raised while the machine has declared itself asleep or shut down (ADR 0211) | warning; urgent after 30 min for the control node | — | +| S2 | report after a send | node-engine | each declaration sent | max(2 min, 3 × that machine's last apply duration), *provisional* | `sent-not-reported` | warning | H1 | +| S3 | plan tier progress | controller's plan | each tier entered | from the tier's build and apply durations, *provisional* | `stalled` (names the tier and what it waits on) | warning | H2 | +| S4 | the controller's event loop takes a message | controller | while its consumer has pending messages | 2 min | `controller-deaf` | urgent | — | +| S5 | a merge announced becomes a plan or *nothing reads it* | announcer → controller | each merge | 10 min (exists, issue 266) | `merge-not-acted` | urgent | — | +| S6 | a build asked → its outcome | build seat | each ask | the build's declared timeout | `ask-lost` | warning | — | +| S7 | a call running → finished | controller | each call | the verb's declared bound | `call-hung` | warning | — | +| S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — | +| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H4 for `slow-consumer` on a resettable consumer | +| S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — | +| S11 | node tools heartbeat | node tools | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — | +| S12 | the controller lease renewed | controller | every 5 s | 15 s | `lease-lost` | urgent | — | +| S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer) | warning | — | +| S14 | facts snapshot exported | controller | daily | 2 days | `facts-stale` | warning | — | +| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days | `healer-wanted` | warning | — | + +The bus advisories (S9) cost one read-only subscription: the server already publishes them. The +controller translates each into a condition naming the thing in the mesh's words, as the refused reply +of issue 265 is translated today. + +## 4. The self-check: `doctor` (rule 6) + +A **probe registry** in the controller: each probe is a live invariant of a design, with an id, an +interval, a timeout and the condition kind it raises. The controller runs the registry every five +minutes; each probe has thirty seconds. A probe that errors or times out raises `probe-failed` for +itself — an unanswered probe is never a pass. + +| Probe | Asserts | From | +|---|---|---| +| D1 | every machine's declaration composes, and passes the node-engine's validation (the validator is a package of mesh-host the controller and the merge gate import — one validator) | 236, 263 | +| D2 | every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6 | 262 | +| D3 | every seat on record has a live holder that answers | 208, 218 | +| D4 | every kept archive is held by a manifest | 253 | +| D5 | exactly one lease holder; no message from a stale epoch in the last interval | 204 | +| D6 | every durable consumer exists with its definition and is near its stream's head | 248, 266 | +| D7 | every stream the controller defines exists with its definition | 208 | +| D8 | no address the mesh owns is in a ban list | 238 | +| D9 | `status` answers in full within ten seconds | 265 | +| D10 | every machine runs the node-engine and node tools builds its plan says, or is inside a plan's window | version split | +| H-* | the health probes of §8, run for every core component on every machine | rule 8 | + +- **`doctor`** answers the last run's verdict at once: per probe, pass, fail or failed-to-run, and age. + **`doctor run`** runs now under a call id. **`doctor probes`** lists the registry; **`doctor + signals`** the table's ages. +- **Every run ends with a heartbeat** event carrying the run's id and counts. That is S10. +- **The registry is the design's live form.** A check over the to-be designs counts invariants that + name a probe against those that do not; the number without may only go down. + +## 5. The output channel, minimal form (rule 6) + +From [research 028](../../01-RESEARCH/028-the-meshs-output-channel/00-overview.md), the smallest form +that works; its open questions stay open and its graduation amends this section. + +- **One mesh seat, `operator-channel`**, held once. It **accepts** `notify` as a work queue, so a + message waits for a holder; it keeps its **open messages** in its own key-value state, so a restart + forgets nothing; it **serves** `open` and `history`. +- **The holder consumes the controller's condition events** and decides what is sent. The controller + calls nobody. +- **Two channels**, each a module contributing itself to the seat: **Telegram** (a bot to the + operator's chat; its token and chat id are the channel module's secrets) and the **desktop + notifier** of the machine the operator is at. +- **A message** is: the condition's key, its subject (a machine's role, a module, a plan), kind, + severity, the one-line summary, since when, and the verb that shows more. **Deduplicated by the key.** +- **When:** on `condition-raised`; once more if still open after 1 hour (urgent) or 12 hours + (warning); on `condition-cleared`, by editing the first message where the channel can. A silenced + condition sends nothing. Urgent goes to both channels; warning to the desktop notifier when the + operator's session is there, otherwise to Telegram. +- **Rate:** at most twenty messages an hour; the excess is folded into one message naming them all. +- **What may leave the mesh:** roles and words. A message carrying an address, a path or anything + shaped like a secret is refused by the holder and raises `channel-refused` instead. +- **No answering back** in this form; acknowledging is `conditions silence`, through the mesh. +- **The watcher's watcher.** A module, `mesh-watcher`, assigned to one machine that is not the control + node, holds the Telegram channel's secret too. It listens for the self-check heartbeat (S10) and for + the bus itself; when either has been silent past its bound it sends to Telegram **directly over + HTTPS, not through the bus**, and says so again when they return. It is the only sender that does not + pass through the control node. + +## 6. Order: lease, epoch, report sequence, one apply queue (rules 1, 2) + +**The lease.** A controller instance acts — sends a declaration, writes a plan, a condition or a call +— only while it holds the key `holder` in `mesh-controller_lease`: written with compare-and-set, a +fifteen-second time to live, renewed every five seconds. The **epoch** is the bucket revision at which +it was taken. A starting controller waits for the key to be absent or expired. One that fails a renewal +stops acting at once and exits, so its service manager restarts it as a waiting candidate. + +``` + controller A (epoch 41) ──renew──renew──╳ (renewal refused)──► stops sending, exits + controller B ──wait──────────────take (epoch 57)──► acts; marks A's running calls abandoned + node-engine accepts 41 … then 57; refuses anything from 41 after 57, counted, reported +``` + +**What carries the order:** + +| Message | Carries | The receiver keeps, per writer | Refuses | +|---|---|---|---| +| declaration | epoch, sequence | highest epoch, then highest sequence | older epoch; same epoch, lower sequence | +| report | machine, the declaration's epoch and sequence it is about, the node-engine's own report sequence (kept on disk, increasing across restarts and self-updates) | highest declaration sequence, then report sequence | an account of an older declaration; an older report | +| plan write | the plan's revision, the epoch | — (compare-and-set) | a write against a revision already moved | +| build outcome | the build's ask id and order | newest per module | an older build finishing later (exists, 219) | +| call | call id, epoch | — | a finish from an epoch that is not the holder's, recorded as abandoned | +| merge announcement | forge, repository, commit, the announcer's sequence | newest per repository | a duplicate (one announcer) | + +**Every refusal** is one line in the receiver's log in the mesh's words, a counter, and — from the +node-engine — a report naming the refused declaration, so the controller sees it. The counter feeds +S13. + +**One apply queue on every machine.** The node-engine has one worker that applies; a delivery, the +five-minute reconcile, a self-update hand-over and a manual `apply` are four reasons to **enqueue**. +The queue holds at most one pending request, coalesced. When the worker starts it takes the newest +declaration held at that moment, applies it once, and sends one report naming the sequence it applied. +Nothing else in the node-engine applies. + +**The `report` verb.** The node-engine answers, on request, the report of the last declaration it +applied, from what it keeps on disk. It is what healer H1 asks. + +**Durable calls.** Every call's record — verb, arguments with secrets removed, caller, started, epoch, +state (`running`, `finished`, `failed`, `abandoned`), the answer bounded in size, finished — lives in +`mesh-controller_calls`, the last thousand or fourteen days. A new lease holder marks the previous +holder's running calls `abandoned`, which is said. + +## 7. Healers and the hand-act log (rule 7) + +**A healer** is a registered response to one condition kind: its repair (the ordinary path again), its +budget, its back-off, its brake, and the `healer-acted` event naming the condition, the act and the +outcome. A healer may not withdraw, delete or recreate data; such a repair is a condition for the +operator. + +| Healer | Condition | Repair | Budget, then | +|---|---|---|---| +| H1 | `sent-not-reported` | ask the machine's node-engine for `report`; if it names an older declaration, send the current one again | twice per condition, then resolver `operator`, urgent | +| H2 | `stalled` on a wait that is superseded or already finished | close the plan with its note, as `plans close` does | once | +| H3 | a seat holder without its worker (D3) | raise the seat's objects again (208) | once per holder per hour | +| H4 | `slow-consumer` far behind on a stream marked *resettable* in the stream table | `broker consumer-reset` (248) | once a day per consumer, then condition | +| H5 | `provider-failing` with the administrator refusing the mesh's secret | ADR 0224 §5 (exists, in the module) | as ADR 0224 | + +**The hand-act log.** Every verb that repairs by hand — a named `push` outside a plan, `plans close`, +`broker consumer-reset`, `conditions silence`, and `hand-act record` for an act done outside the mesh — +takes a required `--why` and writes an entry: who, which verb and arguments, why, when, the condition +key it addresses if any, and a **cause** (the condition kind, or a word the person gives). `status` +shows the week's count. A cause recorded twice within fourteen days raises `healer-wanted` (S15). + +## 8. Staged core upgrades and rollback (rule 8) + +**Health, per core component**, as probes in the registry (§4): + +| Component | Healthy when | Witness that rolls it back | +|---|---|---| +| controller | holds the lease within 60 s of starting; `status` answers in full within 10 s; `doctor` ran once | the node-engine on the control node, which keeps the previous controller build installed beside the new one and reads the lease bucket | +| node-engine | has reported its current declaration under its own build | its launcher (ADR 0141), which keeps the known-good | +| node tools | announced, and answer a ping within 5 s | the node-engine, which keeps the known-good | +| bus | every stream and durable consumer present (D6, D7); a request/reply round trip from every machine | none — a planned step, below | + +**The gate.** ADR 0218's first machine for a core component is judged by that component's health +probes passing three consecutive times over two minutes, within ten minutes of the apply. Only then do +the other machines follow. "Reported applied" is not enough. + +``` + merge ─► plan ─► first machine applies new build ─► health probes ×3 within 10 min? + ├─ yes ─► the rest follow, one tier at a time + └─ no ──► witness restores previous build + ─► condition core...rolled-back (urgent) + ─► plan halts that component, records why +``` + +- **Every core rollout leaves a record** in its plan: component, first machine, from and to build, + verdict, time to verdict, rolled back or not. Read through `plans`. +- **A rolled-back build is not retried** by the same plan. A newer merge makes a new plan. + +**The bus is a planned step.** A bus upgrade is a maintenance step a person starts through the +controller: the streams are snapshotted; a `bus-maintenance` condition is open for the step's duration; +the bus is replaced; afterwards D6, D7 and the round trip must pass, or the step is reported failed and +the snapshot is the way back. A step whose new version cannot be reverted (the bus's 2.10 → 2.11 is one) +says so before it starts and runs only on a person's explicit word, recorded as a hand act. Whether the +bus becomes a cluster that can be upgraded live is left to its own effort. + +## 9. Before merge: facts and replays (rule 9) + +**The facts snapshot.** The controller exports daily, and after any change of machines, assignments or +seats: every machine (a stable pseudonym of the same length as its name, its role, operating system, +C library, architecture, node-engine and node tools builds), assignments, settings keys and their +non-secret values, seats and their holders, the catalogue commit, and the versions the mesh runs of +the bus server, the store and the node-engine. No secret and no address: an address is replaced by one +from a documentation range. It is kept in the artifact store as `facts/latest`, where the build seat +reads it. + +**The merge gate.** In mesh-controller, mesh-host and mesh-catalog, a check composes every machine of +the snapshot with the change applied and runs the node-engine's validator over each. A change that +makes any machine fail to compose or validate fails its check, naming the machine's role and the +module. The resolver module's tests run under both C libraries the snapshot lists. A test in each core +repository asserts that a pinned dependency's version equals the version the snapshot says runs. + +**The replays.** mesh-lab carries a scripted scenario for each core incident, asserting the rule's +outcome, run on every merge to mesh-controller, mesh-host and mesh-tools: + +| Replay | Incident | Asserts | +|---|---|---| +| R1 | two controllers at once (204) | the second waits; nothing from the stale epoch is applied | +| R2 | a reconcile due during a push (257, 261, 267) | one apply, one report, the newest sequence | +| R3 | the node-engine self-updates during its report (230, 264) | the report arrives under the new build | +| R4 | the bus's authorization reloads during a call (265) | the call's outcome is readable by id | +| R5 | a consumer with several filter subjects under mixed traffic (266) | no announcement is skipped; S5 fires if one is | +| R6 | an unreadable contributions file (241) | refused by name; nothing withdrawn; a condition | +| R7 | the controller rebuilds itself mid-plan (214) | the plan continues under the new epoch | +| R8 | a broken controller, node-engine and node tools build | each rolled back with no hand; condition and message | +| R9 | each signal of §3 suppressed | its condition within its bound, cleared on return | + +A new core issue resolves with its replay added, or with a stated reason none is possible. The live +mesh stays the test bed ([ADR 0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md)): a +replay covers what must not be done to it on purpose, and every rule keeps a live check. + +--- + +## 10. The phases + +Ordered by risk removed per day: detection first, because it covers every class including those not +met yet. A phase is done when its *done when* holds; this section records the date when it does. + +### Phase 0 — Finish what is in flight + +| Repository | Delivers | +|---|---| +| mesh-controller | the located fixes of 244, 265, 266, 267 rolled out; `calls` moved into `mesh-controller_calls`; `status` answered from a summary the event loop keeps current, inside ten seconds; the hand-act log with `--why` on the repairing verbs and `hand-act record`; recording the durations the bounds come from (apply duration per machine, heartbeat gaps, plan tier durations, build durations) | +| mesh-host | the fixes of 264 and 257/261 rolled out to every machine | +| mesh-tools | the console passing a mesh seat's `node` (244) rolled out | +| mesh-catalog | the bus's 2.10 → 2.11 upgrade, done as the first planned bus step by hand (snapshot, announced, checked after), recorded as a hand act | + +**Done when:** the four located core issues resolve with their live checks; a controller restart keeps +every call's outcome; `status` answers in full within ten seconds five times in a row; the hand-act log +has a week of entries; the durations are recorded for every machine. + +### Phase 1 — The mesh says when it is wrong + +| Repository | Delivers | +|---|---| +| mesh-controller | the condition store, its verbs, history and events (§2); ADR 0224's standing moved into it; watchdogs for S1–S10 and S12–S14 with bounds set from Phase 0's durations; the bus advisories subscribed and translated (S9); `doctor` with D1–D10 and its heartbeat (§4); `status` led by open conditions; the test generated from the signals table | +| mesh-host | the heartbeat carries its interval; D1's validator published as a package the controller imports | +| mesh-tools | the node tools' heartbeat (S11) | +| mesh-catalog | the `operator-channel` seat and its holder; the Telegram channel and the desktop notifier contributing to it; `mesh-watcher` on a machine other than the control node (§5) | +| mesh-lab | R9: each signal suppressed in turn | + +**Done when:** on a lab mesh, suppressing each signal raises its condition within its bound and sends a +message; restoring it clears both. Stopping the controller makes the watcher send *self-check silent* +within twice the self-check's interval. Live: a week of conditions read back, every one real or its +bound corrected in the table. + +### Phase 2 — Order and one writer + +| Repository | Delivers | +|---|---| +| mesh-controller | the lease and epoch (§6); plans written by compare-and-set; a report kept by sequence; abandoned calls marked; S12, S13, D5; the writers table enforced at grant composition; contract tests for every consumed subject and the check listing them; the empty-on-error lint | +| mesh-host | one apply queue; the report sequence kept on disk; epoch refusal reported; the `report` verb; the contract tests and lint | +| mesh-tools | refusing an unreadable or unknown input by name; the lint | +| mesh-catalog | the withdrawal brake in the providers' loop: a reconcile that would withdraw more than one consumer, or a set fraction, stops and raises a condition | +| mesh-lab | R1, R2, R6 | + +**Done when:** R1 ends with nothing from the stale epoch applied, R2 with one report of the newest +sequence, R6 with the withdrawal braked; every consumed subject has its contract test. + +### Phase 3 — Healers + +| Repository | Delivers | +|---|---| +| mesh-controller | the healer registry and H1–H4, each braked and said; S15 | +| mesh-lab | an induced failure per healer | + +**Done when:** a lab mesh recovers from each induced failure with no hand, says so, and brakes after +its budget. Live: a week with no cause repeated in the hand-act log. + +### Phase 4 — Core upgrades that roll back + +| Repository | Delivers | +|---|---| +| mesh-controller | the health probes of §8; the gate on a core component's first machine; the rollout record in the plan; the bus maintenance step as a verb | +| mesh-host | keeping the previous controller and node tools builds; restoring one when its health is not met in bound, watching the lease bucket for the controller | +| mesh-tools | answering the health ping | +| mesh-lab | R3, R7, R8 | + +**Done when:** on a lab mesh, a broken build of the controller, the node-engine and the node tools — +one that starts and does nothing, one that crashes, one that cannot reach the bus — is each rolled +back with no hand, the mesh ends on the previous build, and a condition and a message say so. Live: the +next three core rollouts each record a health verdict. + +### Phase 5 — Checks before merge, and the replays + +| Repository | Delivers | +|---|---| +| mesh-controller | the facts snapshot and S14 | +| mesh-controller, mesh-host, mesh-catalog | the compose-and-validate merge gate; versions tested as run; the resolver's tests under both C libraries | +| mesh-lab | R4, R5 and the rest of the window's incidents; running the replays on every core merge | + +**Done when:** the replays of 236, 262, 263 and 266 fail on the commit before their fix and pass after; +a new core issue cannot resolve without a replay or a stated reason. + +## What is not decided here + +- The bus as a cluster of three, to upgrade it live. +- Routing by presence, quiet hours, answering back through a channel, and an external dead-man + service — research 028. +- A condition that needs judgement handed to an agent as work — research 017. diff --git a/03-DESIGN/01-to-be/README.md b/03-DESIGN/01-to-be/README.md index ab52764..dbac323 100644 --- a/03-DESIGN/01-to-be/README.md +++ b/03-DESIGN/01-to-be/README.md @@ -45,6 +45,7 @@ document is written and this one's status becomes `implemented`. | [`38-building-the-operators-machine.md`](38-building-the-operators-machine.md) | **In progress.** The work of design 37 as packages: the runtime serves many modules, the controller composes one per node, the console becomes its serving mode, the packet filter moves first, then the shell and the service manager — tested on the live mesh by the operator's decision | [ADR 0175](../../02-DECISIONS/0175-one-tool-runtime-per-node-serves-every-modules-tools-on-the-host-side.md), [0160](../../02-DECISIONS/0160-the-mesh-issues-an-assignments-subjects-and-a-runtime-serves-what-it-is-issued.md), [0149](../../02-DECISIONS/0149-the-live-mesh-is-the-test-bed.md) | | [`41-the-shell-and-the-accounts-environment.md`](41-the-shell-and-the-accounts-environment.md) | **In progress.** The shell and the account's environment as modules: an environment module every module contributes variables and `PATH` entries to, shell code contributed to the login shell in named slots, the prompt and plugins as modules, the host giving a login back, and the service manager's module finished | [ADR 0203](../../02-DECISIONS/0203-the-accounts-environment-is-one-modules-and-every-module-contributes-to-it.md), [ADR 0204](../../02-DECISIONS/0204-a-module-contributes-shell-code-to-the-login-shell-in-named-slots.md), [ADR 0205](../../02-DECISIONS/0205-software-the-distribution-does-not-package-ships-as-a-pinned-archive-of-the-module.md) | | [`42-the-machines-modules-in-order.md`](42-the-machines-modules-in-order.md) | **In progress.** The order the machines' modules of research 026 and 027 are built and rolled out: every machine's first (sudo, localization, time sync, pacman, logrotate, avahi, systemd, docker, `~/.ssh`, scripts, kernel), then both workstations', then one machine model's; each proven on one workstation before the rest | [ADR 0173](../../02-DECISIONS/0173-the-operators-machine-is-the-meshs-and-a-module-is-what-it-declares.md), [ADR 0182](../../02-DECISIONS/0182-inside-a-home-the-mesh-owns-what-it-places-and-holds-the-rest-as-found.md), [ADR 0205](../../02-DECISIONS/0205-software-the-distribution-does-not-package-ships-as-a-pinned-archive-of-the-module.md) | +| [`45-a-core-that-cannot-fail-silently.md`](45-a-core-that-cannot-fail-silently.md) | **Designed.** The core says when it is wrong, refuses what is stale or unreadable, heals what it knows, upgrades one machine at a time with a witness that rolls it back, and is checked against the real mesh before merge: the writers and signals tables, the condition store, `doctor`, the minimal output channel, healers, the lease and report order, staged upgrades, the facts snapshot and replays, in six phases | [ADR 0227](../../02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md), [ADR 0224](../../02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md), [ADR 0218](../../02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md) | ## Not yet written