diff --git a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md index a33c536..2b2fdcd 100644 --- a/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +++ b/03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md @@ -205,8 +205,25 @@ open question 2). A **probe registry** in the controller: each probe is a live invariant of a design, with an id, an interval, a timeout and the condition kind it raises. The controller runs the registry every five -minutes; each probe has thirty seconds. A probe that errors or times out raises `probe-failed` for -itself — an unanswered probe is never a pass. +minutes; each probe has thirty seconds. A probe that errors or times out is said in that run's +verdict at once — an unanswered probe is never a pass — and raises `probe-failed` for itself when the +next run cannot run it either. + +**A finding one look can be wrong about is raised on the second look in a row** +([issue 277](../../04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md)): +a question over the network that went unanswered, a time measured once, a reading a burst can move. +Within a run such a question is asked again before it counts; across runs the finding is raised when +the previous run saw it too, kept while it is open however it is seen, and cleared by the run that no +longer sees it. Held back, it is listed in the verdict as unconfirmed. A definite answer — an address +that is wrong, a stream that is not there — is raised at once. The watchdogs hold a row that cannot +read its facts to the same rule over two ticks. Checked by the controller's tests of the self-check over +runs and of a blind row over ticks. + +**A condition's summary names machines and says the rest in words**; an address, a domain, a path or a +raw error is its evidence, which stays inside the mesh. The condition store holds every summary to the +operator channel's content rule (ADR 0234 §6) and says one that would be withheld in words, keeping it +whole in the evidence; a lint over every condition raised in the controller's test suite fails the +producer that wrote it. | Probe | Asserts | From | |---|---|---| diff --git a/04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md b/04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md new file mode 100644 index 0000000..e5b71bf --- /dev/null +++ b/04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md @@ -0,0 +1,107 @@ +--- +status: located +opened: 2026-10-06 +located-in: [mesh-controller] +fixed-by: novox/mesh-controller PR #91 +amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md +--- + +# 277. One unanswered question was an urgent alert nobody could read + +## Symptom + +Twice on 2026-10-06, about eighty minutes apart, the self-check's D2 (to-be 45 §4: every holder of the +mesh's resolver answers a machine name for IPv4, and NODATA for IPv6) raised its resolver condition, +**urgent**, about the resolver on one machine. Each time, the resolver's machine was briefly loaded: a +push was being applied, and a build was starting some twenty containers. The evidence was one IPv6 +question that timed out — the resolver did not answer within three seconds. Thirty questions asked by +hand right after were all answered at once. Nothing was wrong with the resolver. + +And the operator never read the alert. The condition's summary carried the question's raw error — the +machine's name with the mesh's domain, the resolver's address, and the socket's own words with two +addresses and ports in them. The operator's channel holds every message to a content rule +([ADR 0234](../../02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md) §6: machine +names may appear; domains, addresses, paths and secrets may not), so the message arrived as "this +message carried an IPv4 address, so its words are withheld". An urgent alert that is both false and +unreadable is the worst of both. + +## Cause + +Two faults, each general — D2 is only where they met. + +**A single sample was taken as the invariant failing.** D2 asked each question once, one after +another, and one unanswered question made the resolver wrong. A question over the network to a loaded +machine can go unanswered once without anything being wrong; the design said what a probe asserts, +but not how many looks a finding needs. Auditing every probe and watchdog for the same shape found it +in more places than D2: + +| Where | The single sample | +|---|---| +| D2 | one DNS question per name and family, unanswered once → urgent | +| D3 | one round of the bus's discovery; a holder answering late on a loaded machine → silent | +| D6 | one reading of a consumer's pending count, after a burst → far behind, which healer H4 acts on | +| D8, D13 | one question to each machine's holder; one timeout → the probe fails, or the data is unmeasured | +| D9 | `status` timed once → slow | +| every probe | one error or timeout → `probe-failed` at once | +| every watchdog row | one read of the store or the bus that failed → that row's `probe-failed` at once | + +The watchdogs' own signals were not of this shape: each has a bound measured in intervals (three +missed heartbeats, a tier past three times its measured time), which is already more than one look. + +**A summary carried detail.** Several producers put raw errors, addresses or paths into the one line +the operator reads: D2's socket error; D8's banned addresses and an endpoint's domain; the data probe's +paths, datasets and storage names; the lease's unleased reason. The content rule lives in the +operator's channel and is not trusted to each source — rightly — but nothing in the controller held +its own summaries to it, so a producer could not know it had written a sentence that would be +withheld. + +## Fix + +The fix is novox/mesh-controller PR #91. + +**A finding one look can be wrong about is raised on the second look in a row.** A finding is marked +as such by its source — an unanswered question, a time measured once, a reading a burst can move. It +is raised when the source's previous look saw it too: two runs of the self-check (five minutes apart) +or two ticks of the watchdogs (half a minute). It is kept while it is open, however it is seen, so a +condition is never cleared and raised again by flapping; it clears, as everything does, on the look +that no longer sees it. A finding held back is not a pass: the self-check's verdict lists it as +unconfirmed. A *definite* answer — a resolver answering the wrong address, NXDOMAIN for IPv6, a stream +that is not there — is not marked, and is raised at once. + +Within one run, a question is asked again before it counts as unanswered: D2 asks every question up to +three times, a little apart, all of them at once (so a silent resolver costs a run seconds, not its +whole bound); D3 asks the discovery a second time for what the first did not hear; D8 and D13 ask a +holder again while the probe's bound leaves room. D2's silence, once confirmed, is still urgent: a +resolver silent for five minutes fails every machine whose resolution depends on it. + +Applied to: D2 (silence), D3, D6 (far behind), D9, D13 (unmeasured), the self-check's `probe-failed`, +and every watchdog row's `probe-failed`. + +**A summary says things in machine names and the mesh's words; detail is the evidence.** Every +producer above now keeps the address, the domain, the path and the raw error in the condition's +evidence, which stays inside the mesh, and says the rest in words naming machines. The controller +carries a mirror of the channel's content rule — allowing the mesh's own machine names, as ADR 0234 +says — and the condition store holds every summary to it: one that would be withheld is said in words +there, and kept whole in the evidence, so no future producer can make an alert unreadable again. + +## How it is checked + +- A test over the content rule's table: the shapes the channel refuses are refused, the words + conditions are made of pass, and the mesh's machine names pass. +- **A lint over every condition raised in the controller's test suite**: every observation the + condition store takes in any test — every signals row past its bound, every probe's finding, every + event's condition — and every probe finding a test reads directly is held to the rule; one that would + be withheld fails the suite, naming its source and kind. Proven against the producers as they were: + the data probe's paths fail it. +- Tests of D2 against a stand-in resolver: one that misses a try and answers the next is well; one + that answers nothing is held for the next run and said in machine names, with the address and the + socket's words in the evidence; one that answers wrongly is said at once. +- A test of the self-check over runs: one look raises nothing and is listed as unconfirmed; two in a + row raise; an open condition seen again is kept, not reopened; a pass clears; a probe that cannot + run is a condition only on its second run in a row. The same for a blind watchdog row over two ticks. + +## Design + +To-be 45 §4 said a probe that errors or times out raises `probe-failed`. It now says when: on the +second run in a row, while the verdict says it at once — and states the rule for findings a single +look can be wrong about.