diff --git a/02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md b/02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md new file mode 100644 index 0000000..9913d4b --- /dev/null +++ b/02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md @@ -0,0 +1,158 @@ +--- +topic: the mesh +status: accepted +date: 2026-10-06 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md +--- + +# 231. A healer acts on what observation raised, and only observation says it worked + +## Context + +Phase 3 of [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — *healers* — was +built on 2026-10-06 in the controller and the node-engine. §7 of the design gives each healer a row: +the condition, the repair, the budget and what happens then. Building them met questions the rows do +not answer, and each was answered in code. This record is those answers. + +Five could not be left to taste: + +- **Where a budget is counted.** A budget kept in the controller's memory is reset by every restart, + and a controller restarting in a loop is exactly when a healer must not start counting again. A + budget kept in the condition is lost when the condition clears and reopens — which a repair that + half-works makes happen every few minutes. +- **When an act has failed.** A repair is made in a moment; whether it worked is known only when the + watchdog or the probe that raised the condition looks again — thirty seconds for a watchdog, five + minutes for a probe. +- **What a healer may send.** H1's repair is "send the current declaration again", which is what a + named push does — and a named push sends a machine the builds a policy or a plan is holding back + ([ADR 0221](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md)). + A person naming a machine means it; a healer does not. +- **What triggers H4.** The design names S9's `slow-consumer`. The controller hears a slow consumer for + its own connection only until the bus has a system account + ([issue 270](../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md)), + and a slow *connection* is not a consumer a reset repairs. What 248 met — a durable consumer a week + behind its stream — is what D6 finds. +- **How the machine is asked to report.** The design gives the node-engine a `report` verb. A machine's + bus user may not answer anybody's inbox, and granting it that is a second channel out of every + machine. + +## Considered Options + +1. **Budgets in memory, success when the repair returns.** Rejected: a restart resets every budget, and + "the act returned" is the healer's opinion of its own work — the second answer to a question + rule 1 says has one. +2. **Budgets in a bucket on the bus.** Rejected: the bus's user list must grant a new bucket before the + first heal could be counted, so the first controller with healers would heal uncounted on the live + mesh until the bus's machine was pushed. +3. **Budgets in the controller's store, each act begun before it is made; success only as the + condition's clearing; H4 on D6's own kind; the machine answering through its ordinary report.** + Chosen. + +## Decision + +**A healer is a row of the healer registry**, compiled into the controller: the condition kinds it +answers, its repair, its budget (acts against one *budget key* within a window), its **settle** (how long +after an act the observation is given before the next act or the escalation), what happens when the +budget is spent, where the repair runs, and the event each act is said as. A test generated from the +registry holds every row to a kind the mesh raises — a row of the signals table, a probe of the +self-check, a provider's event — a budget, a settle inside its window, and its event, and every healer +the controller runs to an induced failure that sees it act and brake. + +**The healers of Phase 3:** + +| Healer | Answers | Repair | Budget, settle | +|---|---|---|---| +| H1 | `sent-not-reported` (S2) | ask the machine's node-engine to report again; if it does not then report the declaration it was sent, or says nothing within 45 s, send it again — never moving a build a policy or a plan holds back | 2 per condition in 6 h; 3 min | +| H2 | `stalled` (S3) on a wait that is **superseded** (a newer plan of the same repository and branch exists) or **finished** (every module of every tier built or failed, every one that rolls out sent) | close the plan with its note — `superseded`, naming the newer plan, or `done`; what it asked still builds | 1 per plan in 24 h; 2 min | +| H3 | `holder-silent` (D3), `consumer-lost` (D6, S9) | assert the bus's streams, consumers and seat workers — the assertion every send makes (issue 208) | 1 per object in 1 h; 6 min | +| H4 | `consumer-behind` (D6) on a consumer the stream table marks resettable | `broker consumer-reset` | 1 per consumer in 24 h; 6 min | +| H5 | `provider-failing`, the identity provider's administrator refusing the mesh's secret | the provider's own repair (ADR 0224 §5), registered and not run by the controller | the provider's | + +- **A condition a healer does not apply to is left alone.** H2 passes over a plan waiting on something + still to come; H4 over any consumer not marked resettable. Passing over is not an attempt: no budget, + no escalation, said once in the controller's journal. **Only the controller's own events consumer is + marked resettable**, with why in the table: what a reset drops is caught up — a merge by the catch-up + pass that reads the forge (issue 266), a build's outcome from the build records (issue 214), a + provider's failing word said again within a quarter of an hour (ADR 0224). A module's consumer is + never: nothing would catch up for it. +- **A send by a healer is a push that did not name the machine** (ADR 0221): a machine whose + composition would move a held build is not sent again, and the attempt says so and why. +- **D6's far-behind finding has its own kind, `consumer-behind`**, so H4 answers it and nothing a reset + cannot repair; a probe's registry row names the kinds its findings carry besides its own. + +**Every act is begun in the controller's store before it is made**, under the lease and carrying its +epoch, and finished after it: `acted`, `failed`, or `escalated`. The budgets and the brake are counted +from there, so a controller dying mid-act has still spent it, and one restarting in a loop resets +nothing. Only the controller holding the lease heals; one serving without it (S12) heals nothing — a +repair is the one act that can always wait. + +**Success is never a healer's to say.** An act is kept in its condition's `tried` as `healer Hn`, the +resolver becomes `healer:Hn`, and the condition stays open until the watchdog or probe that raised it +no longer observes it. A condition cleared and reopened within ten minutes carries what was tried. **A +spent budget** — the budget's acts made, the last one's settle past, the condition still open — hands +the condition to the operator: resolver `operator`, severity `urgent`, an attempt saying what was tried. +An observation does not lower an escalated condition's severity again, and no healer touches it until it +clears. + +**Every act is said** as the controller seat's event `healer-acted`: the healer, the condition and its +kind, the act, the outcome, where the budget stands, the epoch, and the verb that shows more. A heal is +never written to the hand-act log, which is how S15 tells a repair the mesh made from one a person had +to. `healers` lists the registry, the acts lately and the brake; `status` counts the week's heals. + +**The mesh-wide brake.** Twelve acts in an hour, all healers together, and every healer stops: the +urgent condition `mesh.healers.braked` names which healer acted on what, and the brake holds until an +hour after the last act. A healer looping is then at most a dozen acts, said, never the incident. +Escalations are sayings, not acts, and are not counted. + +**The `report` verb, as built.** The controller asks on `mesh.node..ask.report`, on core NATS, +which each machine's bus user may subscribe to for its own name only. The node-engine enqueues a +reconcile — the one apply queue's ordinary act — whose account of the declaration it keeps is said +whether or not it is news; a delivery waiting meanwhile is applied and reported instead. **The answer is +the machine's ordinary report** on its own report subject: nothing new is published and nobody's inbox +is answered. A node-engine older than the verb hears nothing, and H1 then sends again, as a person did. + +**S15, as built.** The watchdog reads the hand-act log's fortnight every half minute; a cause recorded +twice within fourteen days raises `mesh.hand-acts..healer-wanted`, naming the acts, who and why — +and, where a healer answers that cause, that it was not enough. It clears when fewer than two acts of +that cause remain within the fortnight. + +## Consequences + +- **The controller, the bus's user list and the node-engines roll out in any order**: a machine whose node-engine cannot hear the question is sent again instead, and a + bus whose user list does not yet grant `healer-acted` refuses only the event — the act and `tried` still + say it. A push of the machine holding the bus ends that. +- **H1 sends what a push of the machine would, minus what is held back.** A send that went unreported + because the machine holds a held build is not repaired by a healer: its two attempts say why, and the + operator's `push ` is still the way, recorded as a hand act. +- **Twelve an hour is a guess**, like Phase 1's first bounds: corrected from the live week, in the + registry, reviewed like code. +- **A heal that half-works costs a few acts, then a person.** The settle and the budget make the + slowest probe the pace: H3 and H4 act at most once per object an hour and a day. +- **Not built:** the induced failures on a lab mesh (mesh-lab); a healer for the commonest remaining hand + acts that have none — a ban lifted, a kept file restored — which S15 will now ask for by name. + +## References + +- [To-be 45 §7](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — the design this record + details. +- [Research 031, evidence §(f)](../01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md) — the + hand acts the healers replace. +- [Issue 208](../04-ISSUES/208-a-seats-worker-is-made-only-when-the-controller-starts/00-report.md) — H3 as + the send's own assertion, for D3 and D6 alike. +- mesh-controller and mesh-host, branch `feat/a-core-that-cannot-fail-silently-phase-3`. + +## How it is checked + +| What | Checked by | +|---|---| +| every healer answers a raised kind, with a budget, a brake, an event and an induced failure | `TestEveryHealerAnswersAKindTheMeshRaisesWithABudgetABrakeAndItsEvent` | +| H1 asks, sends again, keeps `tried`, says `healer-acted`, escalates and stops | `TestH1AsksAMachineToReportAndSendsItAgain` | +| H2 closes a superseded plan and leaves one still waiting | `TestH2ClosesAPlanAnotherHasTakenOver` | +| H3 against a real bus: repaired, cleared by the probe, escalated after its budget | `TestNatsH3AssertsAMissingConsumerAgainAndBrakesAfterItsBudget` | +| H4 against a real bus: reset, cleared by the probe, and the controller still hears | `TestNatsH4ResetsTheControllersEventsConsumerAndItStillDelivers`, `TestH4ResetsOnlyWhatTheTableMarksResettable` | +| the brake, and no heal without the lease | `TestTheBrakeStopsEveryHealerAndSaysSo`, `TestNoHealerActsWithoutTheLease` | +| the `report` verb | mesh-host `internal/link/asked_test.go`, against a real bus; the grant in `TestNatsAskToReportReachesTheMachine` | +| S15 | the signals table's generated test, and `TestARepeatedHandActNamesItsCauseAndItsHealer` | +| live | `healers`, each condition's `tried`, and the hand-act log's weekly count in `status` | diff --git a/02-DECISIONS/README.md b/02-DECISIONS/README.md index 7bd46c4..dc037a7 100644 --- a/02-DECISIONS/README.md +++ b/02-DECISIONS/README.md @@ -201,6 +201,7 @@ python3 00-META/checks/index.py fail if stale - **0224** — [A provider that keeps failing a consumer is a problem the controller reports](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md) - **0227** — [The core holds nine rules, each checked, and is built to them in six phases](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md) - **0229** — [The core's order is a lease the store remembers, and an epoch a machine is sent once it reads one](0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md) +- **0231** — [A healer acts on what observation raised, and only observation says it worked](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md) ### Its tiers, from the bottom up diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index 5fce151..6451218 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -6,6 +6,7 @@ updated: 2026-10-06 decisions: - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md - 02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md + - 02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md - 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md @@ -62,6 +63,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone | conditions | controller | key-value `mesh-controller_conditions`, and their history `mesh-controller_condition-history` | raise or clear only through observations the controller reads | | calls and their outcomes | controller | key-value `mesh-controller_calls` | read by id | | the hand-act log | controller, through the verbs that act | key-value `mesh-controller_hand-acts` | — | +| the healers' acts and their brake | controller (lease holder), each act begun before it is made | the controller's store | read through `healers`; each act said as `healer-acted` and in its condition's `tried` | | stream definitions and bus permissions | controller | the bus | — | | builds and their outcomes | the build seat's holder | its own state | the controller asks | | a merge announced | **one** announcer per forge (the hook, or the poll when the hook is absent — never both) | the bus | — | @@ -159,13 +161,13 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age | S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — | | S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; doctor 3 min; others 10 min | `call-hung` | warning | — | | S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — | -| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H4 for `slow-consumer` on a resettable consumer | +| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` | | S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — | | S11 | node tools heartbeat | node tools | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — | | S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — | | S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — | | S14 | facts snapshot exported | controller | daily | 2 days (Phase 5) | `facts-stale` | warning | — | -| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days | `healer-wanted` | warning | — | +| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — | The bus advisories (S9) cost one read-only subscription: the server already publishes them. The controller translates each into a condition naming the thing in the mesh's words, as the refused reply @@ -186,10 +188,10 @@ itself — an unanswered probe is never a pass. |---|---|---| | D1 | every machine's declaration composes, and passes the node-engine's validation (the validator is a package of mesh-host the controller and the merge gate import — one validator) | 236, 263 | | D2 | every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6 | 262 | -| D3 | every seat on record has a live holder that answers | 208, 218 | +| D3 | every seat on record has a live holder that answers (`holder-silent`, H3) | 208, 218 | | D4 | every kept archive is held by a manifest | 253 | | D5 | exactly one lease holder — the key names this controller at the epoch it acts under, and the controller's record holds no other epoch open; no message from a stale epoch refused in the last interval | 204 | -| D6 | every durable consumer exists with its definition and is near its stream's head (within 1000 messages, *provisional*) | 248, 266 | +| D6 | every durable consumer exists with its definition and is near its stream's head (within 1000 messages, *provisional*); a missing one is `consumer-lost` (H3), one far behind `consumer-behind` (H4) | 248, 266 | | D7 | every stream the controller defines exists with its definition | 208 | | D8 | no address the mesh owns is in a ban list | 238 | | D9 | `status` answers in full within ten seconds | 265 | @@ -294,7 +296,10 @@ declaration held at that moment, applies it once, and sends one report naming th Nothing else in the node-engine applies. **The `report` verb.** The node-engine answers, on request, the report of the last declaration it -applied, from what it keeps on disk. It is what healer H1 asks. +applied, from what it keeps on disk. It is what healer H1 asks. As built ([ADR 0231](../../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)): +asked on `mesh.node..ask.report`, its own only, the node-engine enqueues a reconcile whose +account is said whether or not it is news — a delivery waiting is applied and reported instead — and the +answer is its ordinary report; nobody's inbox is answered. **Durable calls.** Every call's record — verb, arguments with secrets removed, caller, started, epoch, state (`running`, `finished`, `failed`, `abandoned`), the answer bounded in size, finished — lives in @@ -310,12 +315,21 @@ operator. | Healer | Condition | Repair | Budget, then | |---|---|---|---| -| H1 | `sent-not-reported` | ask the machine's node-engine for `report`; if it names an older declaration, send the current one again | twice per condition, then resolver `operator`, urgent | -| H2 | `stalled` on a wait that is superseded or already finished | close the plan with its note, as `plans close` does | once | -| H3 | a seat holder without its worker (D3) | raise the seat's objects again (208) | once per holder per hour | -| H4 | `slow-consumer` far behind on a stream marked *resettable* in the stream table | `broker consumer-reset` (248) | once a day per consumer, then condition | +| H1 | `sent-not-reported` | ask the machine's node-engine for `report`; if it does not then report the declaration it was sent, send it again — never moving a build a policy or a plan holds back | twice per condition in 6 h, then resolver `operator`, urgent | +| H2 | `stalled` on a wait that is superseded (a newer plan of its repository and branch) or already finished (every module built or failed, every one that rolls out sent) | close the plan with its note, as `plans close` does: `superseded` or `done` | once | +| H3 | a seat holder that does not answer (D3, `holder-silent`), or a consumer the mesh expects missing (D6, S9, `consumer-lost`) | assert the bus's objects again — the assertion every send makes (208) | once per object per hour | +| H4 | `consumer-behind` (D6) on a consumer the stream table marks *resettable* — the controller's own events consumer alone | `broker consumer-reset` (248) | once a day per consumer, then condition | | H5 | `provider-failing` with the administrator refusing the mesh's secret | ADR 0224 §5 (exists, in the module) | as ADR 0224 | +As built ([ADR 0231](../../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)): the registry is compiled into the controller with a test generated +from it; each act is begun in the controller's store before it is made, under the lease, and the +budgets are counted from there; a healer that does not apply to a condition leaves it alone; **only +observation says a repair worked** — the act is kept in the condition's `tried` as `healer Hn` and said +as `healer-acted`, and the condition clears when what raised it no longer sees it; a spent budget hands it +to the operator, urgent, and no healer touches it again. **The mesh-wide brake:** twelve acts in an hour, +all healers together, stop every healer — `mesh.healers.braked`, urgent — until an hour after the last. +A heal is never a hand act. `healers` lists the registry, the acts and the brake. + **The hand-act log.** Every verb that repairs by hand — a named `push` outside a plan, `plans close`, `broker consumer-reset`, `conditions silence`, and `hand-act record` for an act done outside the mesh — takes a required `--why` and writes an entry: who, which verb and arguments, why, when, the condition @@ -457,6 +471,13 @@ brake is said in its journal only. **Done when:** a lab mesh recovers from each induced failure with no hand, says so, and brakes after its budget. Live: a week with no cause repeated in the hand-act log. +**Built 2026-10-06** ([ADR 0231](../../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)): the healer registry +and H1–H4 in the controller, H5 registered as the provider's own; the mesh-wide brake; `healer-acted`; +`healers`; S15 live; the node-engine's `report` verb. Each healer's induced failure runs in the +controller's tests against a real store, and H3 and H4 against a real bus: induced, repaired, cleared by +the probe's own next run, and handed to the operator after the budget. **Not yet:** the induced failures +on a lab mesh (mesh-lab), and so the phase's *done when*; the live week. + ### Phase 4 — Core upgrades that roll back | Repository | Delivers |