Merge pull request 'ADR 0231: a healer acts on what observation raised, and only observation says it worked' (#136) from feat/a-core-that-cannot-fail-silently-phase-3 into main

02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md

@@ -0,0 +1,158 @@
+---
+topic: the mesh
+status: accepted
+date: 2026-10-06
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
+---
+
+# 231. A healer acts on what observation raised, and only observation says it worked
+
+## Context
+
+Phase 3 of [to-be 45](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — *healers* — was
+built on 2026-10-06 in the controller and the node-engine. §7 of the design gives each healer a row:
+the condition, the repair, the budget and what happens then. Building them met questions the rows do
+not answer, and each was answered in code. This record is those answers.
+
+Five could not be left to taste:
+
+- **Where a budget is counted.** A budget kept in the controller's memory is reset by every restart,
+  and a controller restarting in a loop is exactly when a healer must not start counting again. A
+  budget kept in the condition is lost when the condition clears and reopens — which a repair that
+  half-works makes happen every few minutes.
+- **When an act has failed.** A repair is made in a moment; whether it worked is known only when the
+  watchdog or the probe that raised the condition looks again — thirty seconds for a watchdog, five
+  minutes for a probe.
+- **What a healer may send.** H1's repair is "send the current declaration again", which is what a
+  named push does — and a named push sends a machine the builds a policy or a plan is holding back
+  ([ADR 0221](0221-a-push-sends-no-build-a-policy-or-a-plan-holds-back-except-to-the-machine-it-names.md)).
+  A person naming a machine means it; a healer does not.
+- **What triggers H4.** The design names S9's `slow-consumer`. The controller hears a slow consumer for
+  its own connection only until the bus has a system account
+  ([issue 270](../04-ISSUES/270-phase-1-watches-signals-that-come-later-and-hears-only-the-controllers-faults/00-report.md)),
+  and a slow *connection* is not a consumer a reset repairs. What 248 met — a durable consumer a week
+  behind its stream — is what D6 finds.
+- **How the machine is asked to report.** The design gives the node-engine a `report` verb. A machine's
+  bus user may not answer anybody's inbox, and granting it that is a second channel out of every
+  machine.
+
+## Considered Options
+
+1. **Budgets in memory, success when the repair returns.** Rejected: a restart resets every budget, and
+   "the act returned" is the healer's opinion of its own work — the second answer to a question
+   rule 1 says has one.
+2. **Budgets in a bucket on the bus.** Rejected: the bus's user list must grant a new bucket before the
+   first heal could be counted, so the first controller with healers would heal uncounted on the live
+   mesh until the bus's machine was pushed.
+3. **Budgets in the controller's store, each act begun before it is made; success only as the
+   condition's clearing; H4 on D6's own kind; the machine answering through its ordinary report.**
+   Chosen.
+
+## Decision
+
+**A healer is a row of the healer registry**, compiled into the controller: the condition kinds it
+answers, its repair, its budget (acts against one *budget key* within a window), its **settle** (how long
+after an act the observation is given before the next act or the escalation), what happens when the
+budget is spent, where the repair runs, and the event each act is said as. A test generated from the
+registry holds every row to a kind the mesh raises — a row of the signals table, a probe of the
+self-check, a provider's event — a budget, a settle inside its window, and its event, and every healer
+the controller runs to an induced failure that sees it act and brake.
+
+**The healers of Phase 3:**
+
+| Healer | Answers | Repair | Budget, settle |
+|---|---|---|---|
+| H1 | `sent-not-reported` (S2) | ask the machine's node-engine to report again; if it does not then report the declaration it was sent, or says nothing within 45 s, send it again — never moving a build a policy or a plan holds back | 2 per condition in 6 h; 3 min |
+| H2 | `stalled` (S3) on a wait that is **superseded** (a newer plan of the same repository and branch exists) or **finished** (every module of every tier built or failed, every one that rolls out sent) | close the plan with its note — `superseded`, naming the newer plan, or `done`; what it asked still builds | 1 per plan in 24 h; 2 min |
+| H3 | `holder-silent` (D3), `consumer-lost` (D6, S9) | assert the bus's streams, consumers and seat workers — the assertion every send makes (issue 208) | 1 per object in 1 h; 6 min |
+| H4 | `consumer-behind` (D6) on a consumer the stream table marks resettable | `broker consumer-reset` | 1 per consumer in 24 h; 6 min |
+| H5 | `provider-failing`, the identity provider's administrator refusing the mesh's secret | the provider's own repair (ADR 0224 §5), registered and not run by the controller | the provider's |
+
+- **A condition a healer does not apply to is left alone.** H2 passes over a plan waiting on something
+  still to come; H4 over any consumer not marked resettable. Passing over is not an attempt: no budget,
+  no escalation, said once in the controller's journal. **Only the controller's own events consumer is
+  marked resettable**, with why in the table: what a reset drops is caught up — a merge by the catch-up
+  pass that reads the forge (issue 266), a build's outcome from the build records (issue 214), a
+  provider's failing word said again within a quarter of an hour (ADR 0224). A module's consumer is
+  never: nothing would catch up for it.
+- **A send by a healer is a push that did not name the machine** (ADR 0221): a machine whose
+  composition would move a held build is not sent again, and the attempt says so and why.
+- **D6's far-behind finding has its own kind, `consumer-behind`**, so H4 answers it and nothing a reset
+  cannot repair; a probe's registry row names the kinds its findings carry besides its own.
+
+**Every act is begun in the controller's store before it is made**, under the lease and carrying its
+epoch, and finished after it: `acted`, `failed`, or `escalated`. The budgets and the brake are counted
+from there, so a controller dying mid-act has still spent it, and one restarting in a loop resets
+nothing. Only the controller holding the lease heals; one serving without it (S12) heals nothing — a
+repair is the one act that can always wait.
+
+**Success is never a healer's to say.** An act is kept in its condition's `tried` as `healer Hn`, the
+resolver becomes `healer:Hn`, and the condition stays open until the watchdog or probe that raised it
+no longer observes it. A condition cleared and reopened within ten minutes carries what was tried. **A
+spent budget** — the budget's acts made, the last one's settle past, the condition still open — hands
+the condition to the operator: resolver `operator`, severity `urgent`, an attempt saying what was tried.
+An observation does not lower an escalated condition's severity again, and no healer touches it until it
+clears.
+
+**Every act is said** as the controller seat's event `healer-acted`: the healer, the condition and its
+kind, the act, the outcome, where the budget stands, the epoch, and the verb that shows more. A heal is
+never written to the hand-act log, which is how S15 tells a repair the mesh made from one a person had
+to. `healers` lists the registry, the acts lately and the brake; `status` counts the week's heals.
+
+**The mesh-wide brake.** Twelve acts in an hour, all healers together, and every healer stops: the
+urgent condition `mesh.healers.braked` names which healer acted on what, and the brake holds until an
+hour after the last act. A healer looping is then at most a dozen acts, said, never the incident.
+Escalations are sayings, not acts, and are not counted.
+
+**The `report` verb, as built.** The controller asks on `mesh.node.<machine>.ask.report`, on core NATS,
+which each machine's bus user may subscribe to for its own name only. The node-engine enqueues a
+reconcile — the one apply queue's ordinary act — whose account of the declaration it keeps is said
+whether or not it is news; a delivery waiting meanwhile is applied and reported instead. **The answer is
+the machine's ordinary report** on its own report subject: nothing new is published and nobody's inbox
+is answered. A node-engine older than the verb hears nothing, and H1 then sends again, as a person did.
+
+**S15, as built.** The watchdog reads the hand-act log's fortnight every half minute; a cause recorded
+twice within fourteen days raises `mesh.hand-acts.<cause>.healer-wanted`, naming the acts, who and why —
+and, where a healer answers that cause, that it was not enough. It clears when fewer than two acts of
+that cause remain within the fortnight.
+
+## Consequences
+
+- **The controller, the bus's user list and the node-engines roll out in any order**: a machine whose node-engine cannot hear the question is sent again instead, and a
+  bus whose user list does not yet grant `healer-acted` refuses only the event — the act and `tried` still
+  say it. A push of the machine holding the bus ends that.
+- **H1 sends what a push of the machine would, minus what is held back.** A send that went unreported
+  because the machine holds a held build is not repaired by a healer: its two attempts say why, and the
+  operator's `push <machine>` is still the way, recorded as a hand act.
+- **Twelve an hour is a guess**, like Phase 1's first bounds: corrected from the live week, in the
+  registry, reviewed like code.
+- **A heal that half-works costs a few acts, then a person.** The settle and the budget make the
+  slowest probe the pace: H3 and H4 act at most once per object an hour and a day.
+- **Not built:** the induced failures on a lab mesh (mesh-lab); a healer for the commonest remaining hand
+  acts that have none — a ban lifted, a kept file restored — which S15 will now ask for by name.
+
+## References
+
+- [To-be 45 §7](../03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md) — the design this record
+  details.
+- [Research 031, evidence §(f)](../01-RESEARCH/031-a-core-that-cannot-fail-silently/01-evidence.md) — the
+  hand acts the healers replace.
+- [Issue 208](../04-ISSUES/208-a-seats-worker-is-made-only-when-the-controller-starts/00-report.md) — H3 as
+  the send's own assertion, for D3 and D6 alike.
+- mesh-controller and mesh-host, branch `feat/a-core-that-cannot-fail-silently-phase-3`.
+
+## How it is checked
+
+| What | Checked by |
+|---|---|
+| every healer answers a raised kind, with a budget, a brake, an event and an induced failure | `TestEveryHealerAnswersAKindTheMeshRaisesWithABudgetABrakeAndItsEvent` |
+| H1 asks, sends again, keeps `tried`, says `healer-acted`, escalates and stops | `TestH1AsksAMachineToReportAndSendsItAgain` |
+| H2 closes a superseded plan and leaves one still waiting | `TestH2ClosesAPlanAnotherHasTakenOver` |
+| H3 against a real bus: repaired, cleared by the probe, escalated after its budget | `TestNatsH3AssertsAMissingConsumerAgainAndBrakesAfterItsBudget` |
+| H4 against a real bus: reset, cleared by the probe, and the controller still hears | `TestNatsH4ResetsTheControllersEventsConsumerAndItStillDelivers`, `TestH4ResetsOnlyWhatTheTableMarksResettable` |
+| the brake, and no heal without the lease | `TestTheBrakeStopsEveryHealerAndSaysSo`, `TestNoHealerActsWithoutTheLease` |
+| the `report` verb | mesh-host `internal/link/asked_test.go`, against a real bus; the grant in `TestNatsAskToReportReachesTheMachine` |
+| S15 | the signals table's generated test, and `TestARepeatedHandActNamesItsCauseAndItsHealer` |
+| live | `healers`, each condition's `tried`, and the hand-act log's weekly count in `status` |

02-DECISIONS/README.md

@@ -201,6 +201,7 @@ python3 00-META/checks/index.py            fail if stale
 - **0224** — [A provider that keeps failing a consumer is a problem the controller reports](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md)
 - **0227** — [The core holds nine rules, each checked, and is built to them in six phases](0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md)
 - **0229** — [The core's order is a lease the store remembers, and an epoch a machine is sent once it reads one](0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md)
+- **0231** — [A healer acts on what observation raised, and only observation says it worked](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)
 
 ### Its tiers, from the bottom up
 

03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md

@@ -6,6 +6,7 @@ updated: 2026-10-06
 decisions:
   - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md
   - 02-DECISIONS/0229-the-cores-order-is-a-lease-the-store-remembers-and-an-epoch-a-machine-is-sent-once-it-reads-one.md
+  - 02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md
   - 02-DECISIONS/0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md
   - 02-DECISIONS/0218-a-plan-sends-grants-before-code-rolls-out-one-machine-first-and-a-newer-merge-takes-over-an-older-plan.md
   - 02-DECISIONS/0141-the-host-delivers-its-own-successor.md
@@ -62,6 +63,7 @@ Every kind of state the core keeps, and the one component that writes it. Anyone
 | conditions | controller | key-value `mesh-controller_conditions`, and their history `mesh-controller_condition-history` | raise or clear only through observations the controller reads |
 | calls and their outcomes | controller | key-value `mesh-controller_calls` | read by id |
 | the hand-act log | controller, through the verbs that act | key-value `mesh-controller_hand-acts` | — |
+| the healers' acts and their brake | controller (lease holder), each act begun before it is made | the controller's store | read through `healers`; each act said as `healer-acted` and in its condition's `tried` |
 | stream definitions and bus permissions | controller | the bus | — |
 | builds and their outcomes | the build seat's holder | its own state | the controller asks |
 | a merge announced | **one** announcer per forge (the hook, or the poll when the hook is absent — never both) | the bus | — |
@@ -159,13 +161,13 @@ signal and asserts its condition. `doctor signals` shows, for every row, the age
 | S6 | a build asked → its outcome | build seat | each ask | max(20 min, 3 × the p90 of measured builds), 1 h while nothing is measured, *provisional* (the build seat declares no timeout) | `ask-lost` | warning | — |
 | S7 | a call running → finished | controller | each call | the verb's bound: push, rotate and command 30 min; assign and unassign 15 min; doctor 3 min; others 10 min | `call-hung` | warning | — |
 | S8 | a provider's failing word repeated | provider | every 15 min while failing (ADR 0224) | 30 min | `provider-silent` | warning | — |
-| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H4 for `slow-consumer` on a resettable consumer |
+| S9 | bus advisories: slow consumer, maximum deliveries, permission violation, consumer deleted | bus server's system subjects | any | any occurrence | `slow-consumer`, `max-deliveries`, `refused`, `consumer-lost` — each naming the call, consumer or module in the mesh's words | warning | H3 for `consumer-lost` |
 | S10 | the self-check's heartbeat | controller's `doctor` | every run | 2 × its interval, watched **from the second machine** (§5) | `self-check-silent` | urgent | — |
 | S11 | node tools heartbeat | node tools | its interval | 3 × interval, *provisional* | `tools-silent` | warning | — |
 | S12 | the controller lease renewed | controller | every 5 s | 15 s; a holder that lost the lease or was found expired, and a lease bucket found raised again from nothing, said for an hour; a controller serving without the lease, while it does | `lease-lost` | urgent | — |
 | S13 | stale refusals | every receiver (rule 2) | each refusal | more than 5 from one writer in 5 min | `stale-writer` (names the writer: the controller epoch a refused declaration claimed, with its instance and how its lease ended; or the machine whose older accounts the controller refused) | warning | — |
 | S14 | facts snapshot exported | controller | daily | 2 days (Phase 5) | `facts-stale` | warning | — |
-| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days | `healer-wanted` | warning | — |
+| S15 | a hand act with a cause already recorded | hand-act log | each act | the second within 14 days; clears when fewer than two remain within 14 days | `healer-wanted` (names the cause, and the healer that was not enough where one answers it) | warning | — |
 
 The bus advisories (S9) cost one read-only subscription: the server already publishes them. The
 controller translates each into a condition naming the thing in the mesh's words, as the refused reply
@@ -186,10 +188,10 @@ itself — an unanswered probe is never a pass.
 |---|---|---|
 | D1 | every machine's declaration composes, and passes the node-engine's validation (the validator is a package of mesh-host the controller and the merge gate import — one validator) | 236, 263 |
 | D2 | every holder of the mesh's resolver answers a machine name for IPv4, and NODATA for IPv6 | 262 |
-| D3 | every seat on record has a live holder that answers | 208, 218 |
+| D3 | every seat on record has a live holder that answers (`holder-silent`, H3) | 208, 218 |
 | D4 | every kept archive is held by a manifest | 253 |
 | D5 | exactly one lease holder — the key names this controller at the epoch it acts under, and the controller's record holds no other epoch open; no message from a stale epoch refused in the last interval | 204 |
-| D6 | every durable consumer exists with its definition and is near its stream's head (within 1000 messages, *provisional*) | 248, 266 |
+| D6 | every durable consumer exists with its definition and is near its stream's head (within 1000 messages, *provisional*); a missing one is `consumer-lost` (H3), one far behind `consumer-behind` (H4) | 248, 266 |
 | D7 | every stream the controller defines exists with its definition | 208 |
 | D8 | no address the mesh owns is in a ban list | 238 |
 | D9 | `status` answers in full within ten seconds | 265 |
@@ -294,7 +296,10 @@ declaration held at that moment, applies it once, and sends one report naming th
 Nothing else in the node-engine applies.
 
 **The `report` verb.** The node-engine answers, on request, the report of the last declaration it
-applied, from what it keeps on disk. It is what healer H1 asks.
+applied, from what it keeps on disk. It is what healer H1 asks. As built ([ADR 0231](../../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)):
+asked on `mesh.node.<machine>.ask.report`, its own only, the node-engine enqueues a reconcile whose
+account is said whether or not it is news — a delivery waiting is applied and reported instead — and the
+answer is its ordinary report; nobody's inbox is answered.
 
 **Durable calls.** Every call's record — verb, arguments with secrets removed, caller, started, epoch,
 state (`running`, `finished`, `failed`, `abandoned`), the answer bounded in size, finished — lives in
@@ -310,12 +315,21 @@ operator.
 
 | Healer | Condition | Repair | Budget, then |
 |---|---|---|---|
-| H1 | `sent-not-reported` | ask the machine's node-engine for `report`; if it names an older declaration, send the current one again | twice per condition, then resolver `operator`, urgent |
-| H2 | `stalled` on a wait that is superseded or already finished | close the plan with its note, as `plans close` does | once |
-| H3 | a seat holder without its worker (D3) | raise the seat's objects again (208) | once per holder per hour |
-| H4 | `slow-consumer` far behind on a stream marked *resettable* in the stream table | `broker consumer-reset` (248) | once a day per consumer, then condition |
+| H1 | `sent-not-reported` | ask the machine's node-engine for `report`; if it does not then report the declaration it was sent, send it again — never moving a build a policy or a plan holds back | twice per condition in 6 h, then resolver `operator`, urgent |
+| H2 | `stalled` on a wait that is superseded (a newer plan of its repository and branch) or already finished (every module built or failed, every one that rolls out sent) | close the plan with its note, as `plans close` does: `superseded` or `done` | once |
+| H3 | a seat holder that does not answer (D3, `holder-silent`), or a consumer the mesh expects missing (D6, S9, `consumer-lost`) | assert the bus's objects again — the assertion every send makes (208) | once per object per hour |
+| H4 | `consumer-behind` (D6) on a consumer the stream table marks *resettable* — the controller's own events consumer alone | `broker consumer-reset` (248) | once a day per consumer, then condition |
 | H5 | `provider-failing` with the administrator refusing the mesh's secret | ADR 0224 §5 (exists, in the module) | as ADR 0224 |
 
+As built ([ADR 0231](../../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)): the registry is compiled into the controller with a test generated
+from it; each act is begun in the controller's store before it is made, under the lease, and the
+budgets are counted from there; a healer that does not apply to a condition leaves it alone; **only
+observation says a repair worked** — the act is kept in the condition's `tried` as `healer Hn` and said
+as `healer-acted`, and the condition clears when what raised it no longer sees it; a spent budget hands it
+to the operator, urgent, and no healer touches it again. **The mesh-wide brake:** twelve acts in an hour,
+all healers together, stop every healer — `mesh.healers.braked`, urgent — until an hour after the last.
+A heal is never a hand act. `healers` lists the registry, the acts and the brake.
+
 **The hand-act log.** Every verb that repairs by hand — a named `push` outside a plan, `plans close`,
 `broker consumer-reset`, `conditions silence`, and `hand-act record` for an act done outside the mesh —
 takes a required `--why` and writes an entry: who, which verb and arguments, why, when, the condition
@@ -457,6 +471,13 @@ brake is said in its journal only.
 **Done when:** a lab mesh recovers from each induced failure with no hand, says so, and brakes after
 its budget. Live: a week with no cause repeated in the hand-act log.
 
+**Built 2026-10-06** ([ADR 0231](../../02-DECISIONS/0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)): the healer registry
+and H1–H4 in the controller, H5 registered as the provider's own; the mesh-wide brake; `healer-acted`;
+`healers`; S15 live; the node-engine's `report` verb. Each healer's induced failure runs in the
+controller's tests against a real store, and H3 and H4 against a real bus: induced, repaired, cleared by
+the probe's own next run, and handed to the operator after the budget. **Not yet:** the induced failures
+on a lab mesh (mesh-lab), and so the phase's *done when*; the live week.
+
 ### Phase 4 — Core upgrades that roll back
 
 | Repository | Delivers |