From c58f3dc765a42a39bb18b5e54ee663be752d4f8b Mon Sep 17 00:00:00 2001 From: jochen Date: Sat, 10 Oct 2026 17:47:49 +0200 Subject: [PATCH] ADR 0280: the resolver restarts itself when it stops answering, by the operator's word (issue 348) ADR 0240 rule 6 left a restart on health to a record of its own. The resolver's failure cuts its machine off by name, so no healer in the controller can reach it; the module repairs itself, braked, and says it as its health. To-be 48 and ADR 0240 point to the exception. --- ...s-healthy-and-the-node-engine-judges-it.md | 6 + ...answering-braked-and-said-as-its-health.md | 106 ++++++++++++++++++ .../48-a-module-says-how-it-is-healthy.md | 11 +- 3 files changed, 121 insertions(+), 2 deletions(-) create mode 100644 02-DECISIONS/0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md diff --git a/02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md b/02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md index 5b1ab8be..6ed47d57 100644 --- a/02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md +++ b/02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md @@ -181,6 +181,12 @@ gate waits rather than fails. What a consumer finds while its provider is health reaches a person through the operator's conversation, or a healer; a healer that restarts on health is its own decision. +> **The mechanism changed — 2026-10-10, by ADR 0280.** Rule 6 still stands for the node-engine and for +> every module but one: the dnsmasq module restarts its own resolver when the resolver stops answering, +> braked and said as its health, by the operator's word +> ([ADR 0280](0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md)). +> Any other restart on health still needs its own record. + **7. A declaration is proved before it is trusted.** `module check` refuses a `health` field that names an endpoint the module does not declare, an interval, timeout or count outside its bounds, or a tool the module does not serve. A bed from mesh-lab, run by the catalogue's check on the build seat, starts every resource diff --git a/02-DECISIONS/0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md b/02-DECISIONS/0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md new file mode 100644 index 00000000..cec37bc6 --- /dev/null +++ b/02-DECISIONS/0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md @@ -0,0 +1,106 @@ +--- +topic: what runs on it +status: accepted +date: 2026-10-10 +deciders: jochen +reconstructed: false +extends: 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md +--- + +# 280. The resolver restarts itself when it stops answering, braked, and says it as its health + +## Context + +[ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) rule 6 says nothing is +restarted for being unhealthy, and that "a healer that restarts on health is its own decision"; its +options left that to "a record of its own under ADR 0231". This is that record, for one module. + +[Issue 348](../04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/00-report.md): +on 2026-10-09 the mesh's resolver on the anchor refused every question on its private address for +twenty minutes (10:57:57 to 11:18:12 UTC) while its unit stayed active and its journal said nothing. +Loopback still answered. The certificate authority failed its health check from 10:59:37 UTC, and builds +sent in that window were blamed for a failure that was the resolver's. Two facts shape the answer: + +- **The cause is a known failure of the resolver itself.** dnsmasq 2.93 closes every listener when an + address re-read fails while a bridge is torn down; reproduced in a network namespace, 53 closings in + 120 teardowns. The catalogue's fix binds each address once at start (`bind-interfaces`), which removes + the known cause, but a resolver that stops answering for another reason looks the same. +- **A resolver that stops answering can cut its machine off from the rest of the mesh by name.** Then + nothing outside the machine reaches it to repair it: not the gate's rollback + ([ADR 0236](0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md)), + and not a healer in the controller + ([ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md)), + which acts by sending to the machine over the bus. + +## Considered Options + +1. *Keep rule 6 and let the condition reach the operator.* Rejected: the operator is often asleep or away, + and twenty minutes of a dead resolver on the anchor stalls every machine that resolves through it. The + restart is what the operator did by hand on 2026-10-09. +2. *A healer row in the controller's registry (ADR 0231) that restarts the unit.* Rejected for this case: + the repair must reach a machine whose name lookups are failing, and the controller reaches a machine + only through the bus. It remains the right shape for a repair that does not cut its own path. +3. **The module repairs its own resolver, as ADR 0224 §5 lets a provider repair a known failure: + braked, observed by its next look, and said as its health.** Chosen. + +## Decision + +**The dnsmasq module restarts its own resolver when the resolver stops answering, and only then.** Its +long-running watch asks the machine's own name on every listen address, over UDP and TCP, every 15 +seconds. After two failing looks in a row it restarts the resolver's unit. It restarts only a unit that is +running; a stopped or failed unit is the service manager's and the operator's. + +**The repair is braked.** At most one restart every two minutes and three within an hour. The budget is +kept in the watch's state on the machine, so a restart of the watch itself does not reset it. When the +budget is spent the watch restarts nothing more until the oldest restart is an hour old, and says why. + +**Only the next look says whether it worked** (ADR 0231's rule, applied in the module). A restart helped +only when a look within a minute after it answered. The module's declared health check (`dnsmasq_health`, +ADR 0240 rule 2) reads the watch: a restart that helped leaves the module healthy; one that did not, or a +spent budget, is unhealthy, which the node-engine states and the controller raises as +`module.dnsmasq..unhealthy`. The operator hears of it then. + +**Every restart is said**, as the module's `restarted` event and in the watch's state, which +`dnsmasq_health` shows. + +**The operator's word.** Rule 6 and the catalogue change (mesh-catalog pull request 161) both said this +departure needed the operator's word or a healer record. The operator gave it on 2026-10-10 by approving +the merge, knowing that it makes the resolver restart itself. + +**This is one module's exception, not a change to rule 6.** Any other module that wants to restart on +health needs its own record, or a healer row under ADR 0231. + +## Consequences + +- A resolver that stops answering is back within about 30 seconds plus its start time, instead of after a + person notices it. +- **A module's own repair is not a healer.** It is not counted by the controller's mesh-wide brake, and + it is not listed by the controller's `healers` verb. Its own brake (three an hour) is what bounds it, + and its `restarted` event and health state are where it is seen. +- A restart hides a failure for as long as it helps. The evidence of each one is kept in the event and + the watch's state, so a resolver that is restarted every hour is still visible there. Nothing raises a + condition for repeated restarts that each helped; that is not built, and the operator hears only of a + restart that did not help. +- The rule 6 test in the node-engine still holds: the node-engine restarts nothing. The restart is the + module's, made through the service manager on its own machine. + +## References + +- [ADR 0240](0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md) rule 6, which this + record makes one exception to. +- [ADR 0231](0231-a-healer-acts-on-what-observation-raised-and-only-observation-says-it-worked.md) — the + rules a repair keeps: braked, and only observation says it worked. +- [ADR 0224](0224-a-provider-that-keeps-failing-a-consumer-is-a-problem-the-controller-reports.md) §5 — + a module repairing a known failure of its own. +- [Issue 348](../04-ISSUES/348-a-resolver-that-stopped-answering-ran-on-unnoticed-and-builds-sent-after-it-were-blamed/00-report.md). +- mesh-catalog pull request 161, branch `fix/the-resolver-answers-or-is-restarted`, `modules/dnsmasq`. + +## How it is checked + +| What | Checked by | +|---|---| +| two failing looks restart the unit, within the gap and the hourly budget | `TestTheWatchRestartsOnASecondFailingLookWithinItsBudget` (mesh-catalog `modules/dnsmasq`) | +| a restart helped only when a look soon after it answered; health says whether it did | `TestTheWatchRestartsAResolverThatStoppedAnsweringAndHealthSaysWhetherItHelped`, `TestARestartHelpedOnlyWhenTheResolverAnsweredSoonAfterIt` | +| only a running unit is restarted | `TestTheWatchRestartsOnlyAUnitThatRuns` | +| the budget survives the watch's own restart | `TestTheWatchLoopKeepsItsBudgetAcrossItsOwnRestart` | +| live | `dnsmasq_health` on each machine that runs the resolver: its restarts within the hour and why | diff --git a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md index 8985b1df..e3169c99 100644 --- a/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md +++ b/03-DESIGN/01-to-be/48-a-module-says-how-it-is-healthy.md @@ -2,7 +2,7 @@ layer: to-be status: in-progress code: [mesh-host, mesh-controller, mesh-lab, mesh-catalog] -updated: 2026-10-08 +updated: 2026-10-10 decisions: - 02-DECISIONS/0254-a-wait-for-a-persons-new-login-is-not-a-gates-failure-and-each-module-of-a-send-keeps-its-own-pass.md - 02-DECISIONS/0240-a-module-says-how-it-is-healthy-and-the-node-engine-judges-it.md @@ -10,6 +10,7 @@ decisions: - 02-DECISIONS/0247-a-machine-with-a-vpn-client-routes-names-by-domain-through-a-resolver-of-its-own.md - 02-DECISIONS/0236-a-build-is-judged-on-its-first-machine-and-put-back-by-something-other-than-itself-and-so-it-rolls-out-unattended.md - 02-DECISIONS/0227-the-core-holds-nine-rules-each-checked-and-is-built-to-them-in-six-phases.md + - 02-DECISIONS/0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md - 02-DECISIONS/0256-a-machines-hardware-condition-is-read-by-one-module-on-every-machine-and-a-fault-is-said-through-its-health.md --- @@ -209,6 +210,11 @@ The runtime restarts what exits, as now. The node-engine never restarts, recreat being unhealthy. The condition reaches the operator through the conversation (to-be 46), or a healer once a record gives one that act. +One module repairs itself, by its own record: the dnsmasq module's watch restarts the resolver's unit when +it stops answering, at most once every two minutes and three times an hour, and says through its declared +health whether the restart helped ([ADR 0280](../../02-DECISIONS/0280-the-resolver-restarts-itself-when-it-stops-answering-braked-and-said-as-its-health.md)). The node-engine still restarts +nothing; any other restart on health needs a record of its own. + ## 8. A declaration is proved before it is trusted - **`module check`** refuses a `health` field naming an endpoint the module does not declare under @@ -483,7 +489,8 @@ controller. ## What is not decided here -- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231). +- Whether a healer restarts what stays unhealthy (its own record, under ADR 0231). The resolver's own + restart is decided, for that module alone, by ADR 0280. - Health of scheduled work, and of what a module's events do (issue 276). - A slow starter that needs more than five minutes to be ready — a change to the bound, recorded, when one exists.