ADR 0280: the resolver restarts itself when it stops answering, by the operator's word (issue 348)

ADR 0240 rule 6 left a restart on health to a record of its own. The resolver's
failure cuts its machine off by name, so no healer in the controller can reach
it; the module repairs itself, braked, and says it as its health. To-be 48 and
ADR 0240 point to the exception.

02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md

@@ -181,6 +181,12 @@ gate waits rather than fails. What a consumer finds while its provider is health
 reaches a person through the operator's conversation, or a healer; a healer that restarts on health is its
 own decision.
 
+> **The mechanism changed — 2026-10-10, by ADR 0280.** Rule 6 still stands for the node-engine and for
+> every module but one: the dnsmasq module restarts its own resolver when the resolver stops answering,
+> braked and said as its health, by the operator's word
+> ([ADR 0280](0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md)).
+> Any other restart on health still needs its own record.
+
 **7. A declaration is proved before it is trusted.** `module check` refuses a `health` field that names an
 endpoint the module does not declare, an interval, timeout or count outside its bounds, or a tool the module
 does not serve. A bed from mesh-lab, run by the catalogue's check on the build seat, starts every resource

02-DECISIONS/0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md

@@ -0,0 +1,106 @@
+---
+topic: what runs on it
+status: accepted
+date: 2026-10-10
+deciders: jochen
+reconstructed: false
+extends: 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md
+---
+
+# 280. The resolver restarts itself when it stops answering, braked, and says it as its health
+
+## Context
+
+[ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) rule 6 says nothing is
+restarted for being unhealthy, and that "a healer that restarts on health is its own decision"; its
+options left that to "a record of its own under ADR 0231". This is that record, for one module.
+
+[Issue 348](../04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/00-report.md):
+on 2026-10-09 the mesh's resolver on the anchor refused every question on its private address for
+twenty minutes (10:57:57 to 11:18:12 UTC) while its unit stayed active and its journal said nothing.
+Loopback still answered. The certificate authority failed its health check from 10:59:37 UTC, and builds
+sent in that window were blamed for a failure that was the resolver's. Two facts shape the answer:
+
+- **The cause is a known failure of the resolver itself.** dnsmasq 2.93 closes every listener when an
+  address re-read fails while a bridge is torn down; reproduced in a network namespace, 53 closings in
+  120 teardowns. The catalogue's fix binds each address once at start (`bind-interfaces`), which removes
+  the known cause, but a resolver that stops answering for another reason looks the same.
+- **A resolver that stops answering can cut its machine off from the rest of the mesh by name.** Then
+  nothing outside the machine reaches it to repair it: not the gate's rollback
+  ([ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)),
+  and not a healer in the controller
+  ([ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)),
+  which acts by sending to the machine over the bus.
+
+## Considered Options
+
+1. *Keep rule 6 and let the condition reach the operator.* Rejected: the operator is often asleep or away,
+   and twenty minutes of a dead resolver on the anchor stalls every machine that resolves through it. The
+   restart is what the operator did by hand on 2026-10-09.
+2. *A healer row in the controller's registry (ADR 0231) that restarts the unit.* Rejected for this case:
+   the repair must reach a machine whose name lookups are failing, and the controller reaches a machine
+   only through the bus. It remains the right shape for a repair that does not cut its own path.
+3. **The module repairs its own resolver, as ADR 0224 §5 lets a provider repair a known failure:
+   braked, observed by its next look, and said as its health.** Chosen.
+
+## Decision
+
+**The dnsmasq module restarts its own resolver when the resolver stops answering, and only then.** Its
+long-running watch asks the machine's own name on every listen address, over UDP and TCP, every 15
+seconds. After two failing looks in a row it restarts the resolver's unit. It restarts only a unit that is
+running; a stopped or failed unit is the service manager's and the operator's.
+
+**The repair is braked.** At most one restart every two minutes and three within an hour. The budget is
+kept in the watch's state on the machine, so a restart of the watch itself does not reset it. When the
+budget is spent the watch restarts nothing more until the oldest restart is an hour old, and says why.
+
+**Only the next look says whether it worked** (ADR 0231's rule, applied in the module). A restart helped
+only when a look within a minute after it answered. The module's declared health check (`dnsmasq_health`,
+ADR 0240 rule 2) reads the watch: a restart that helped leaves the module healthy; one that did not, or a
+spent budget, is unhealthy, which the node-engine states and the controller raises as
+`module.dnsmasq.<machine>.unhealthy`. The operator hears of it then.
+
+**Every restart is said**, as the module's `restarted` event and in the watch's state, which
+`dnsmasq_health` shows.
+
+**The operator's word.** Rule 6 and the catalogue change (mesh-catalog pull request 161) both said this
+departure needed the operator's word or a healer record. The operator gave it on 2026-10-10 by approving
+the merge, knowing that it makes the resolver restart itself.
+
+**This is one module's exception, not a change to rule 6.** Any other module that wants to restart on
+health needs its own record, or a healer row under ADR 0231.
+
+## Consequences
+
+- A resolver that stops answering is back within about 30 seconds plus its start time, instead of after a
+  person notices it.
+- **A module's own repair is not a healer.** It is not counted by the controller's mesh-wide brake, and
+  it is not listed by the controller's `healers` verb. Its own brake (three an hour) is what bounds it,
+  and its `restarted` event and health state are where it is seen.
+- A restart hides a failure for as long as it helps. The evidence of each one is kept in the event and
+  the watch's state, so a resolver that is restarted every hour is still visible there. Nothing raises a
+  condition for repeated restarts that each helped; that is not built, and the operator hears only of a
+  restart that did not help.
+- The rule 6 test in the node-engine still holds: the node-engine restarts nothing. The restart is the
+  module's, made through the service manager on its own machine.
+
+## References
+
+- [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) rule 6, which this
+  record makes one exception to.
+- [ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md) — the
+  rules a repair keeps: braked, and only observation says it worked.
+- [ADR 0224](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md) §5 —
+  a module repairing a known failure of its own.
+- [Issue 348](../04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/00-report.md).
+- mesh-catalog pull request 161, branch `fix/the-resolver-answers-or-is-restarted`, `modules/dnsmasq`.
+
+## How it is checked
+
+| What | Checked by |
+|---|---|
+| two failing looks restart the unit, within the gap and the hourly budget | `TestTheWatchRestartsOnASecondFailingLookWithinItsBudget` (mesh-catalog `modules/dnsmasq`) |
+| a restart helped only when a look soon after it answered; health says whether it did | `TestTheWatchRestartsAResolverThatStoppedAnsweringAndHealthSaysWhetherItHelped`, `TestARestartHelpedOnlyWhenTheResolverAnsweredSoonAfterIt` |
+| only a running unit is restarted | `TestTheWatchRestartsOnlyAUnitThatRuns` |
+| the budget survives the watch's own restart | `TestTheWatchLoopKeepsItsBudgetAcrossItsOwnRestart` |
+| live | `dnsmasq_health` on each machine that runs the resolver: its restarts within the hour and why |