Merge pull request 'Issue 277: one unanswered question was an urgent alert nobody could read' (#147) from issue/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read into main

03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md

@@ -205,8 +205,25 @@ open question 2).
 
 A **probe registry** in the controller: each probe is a live invariant of a design, with an id, an
 interval, a timeout and the condition kind it raises. The controller runs the registry every five
-minutes; each probe has thirty seconds. A probe that errors or times out raises `probe-failed` for
-itself — an unanswered probe is never a pass.
+minutes; each probe has thirty seconds. A probe that errors or times out is said in that run's
+verdict at once — an unanswered probe is never a pass — and raises `probe-failed` for itself when the
+next run cannot run it either.
+
+**A finding one look can be wrong about is raised on the second look in a row**
+([issue 277](../../04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md)):
+a question over the network that went unanswered, a time measured once, a reading a burst can move.
+Within a run such a question is asked again before it counts; across runs the finding is raised when
+the previous run saw it too, kept while it is open however it is seen, and cleared by the run that no
+longer sees it. Held back, it is listed in the verdict as unconfirmed. A definite answer — an address
+that is wrong, a stream that is not there — is raised at once. The watchdogs hold a row that cannot
+read its facts to the same rule over two ticks. Checked by the controller's tests of the self-check over
+runs and of a blind row over ticks.
+
+**A condition's summary names machines and says the rest in words**; an address, a domain, a path or a
+raw error is its evidence, which stays inside the mesh. The condition store holds every summary to the
+operator channel's content rule (ADR 0234 §6) and says one that would be withheld in words, keeping it
+whole in the evidence; a lint over every condition raised in the controller's test suite fails the
+producer that wrote it.
 
 | Probe | Asserts | From |
 |---|---|---|

04-ISSUES/277-one-unanswered-question-was-an-urgent-alert-nobody-could-read/00-report.md

@@ -0,0 +1,107 @@
+---
+status: located
+opened: 2026-10-06
+located-in: [mesh-controller]
+fixed-by: novox/mesh-controller PR #91
+amended-design: 03-DESIGN/01-to-be/45-a-core-that-cannot-fail-silently.md
+---
+
+# 277. One unanswered question was an urgent alert nobody could read
+
+## Symptom
+
+Twice on 2026-10-06, about eighty minutes apart, the self-check's D2 (to-be 45 §4: every holder of the
+mesh's resolver answers a machine name for IPv4, and NODATA for IPv6) raised its resolver condition,
+**urgent**, about the resolver on one machine. Each time, the resolver's machine was briefly loaded: a
+push was being applied, and a build was starting some twenty containers. The evidence was one IPv6
+question that timed out — the resolver did not answer within three seconds. Thirty questions asked by
+hand right after were all answered at once. Nothing was wrong with the resolver.
+
+And the operator never read the alert. The condition's summary carried the question's raw error — the
+machine's name with the mesh's domain, the resolver's address, and the socket's own words with two
+addresses and ports in them. The operator's channel holds every message to a content rule
+([ADR 0234](../../02-DECISIONS/0234-the-mesh-holds-a-conversation-with-its-operator.md) §6: machine
+names may appear; domains, addresses, paths and secrets may not), so the message arrived as "this
+message carried an IPv4 address, so its words are withheld". An urgent alert that is both false and
+unreadable is the worst of both.
+
+## Cause
+
+Two faults, each general — D2 is only where they met.
+
+**A single sample was taken as the invariant failing.** D2 asked each question once, one after
+another, and one unanswered question made the resolver wrong. A question over the network to a loaded
+machine can go unanswered once without anything being wrong; the design said what a probe asserts,
+but not how many looks a finding needs. Auditing every probe and watchdog for the same shape found it
+in more places than D2:
+
+| Where | The single sample |
+|---|---|
+| D2 | one DNS question per name and family, unanswered once → urgent |
+| D3 | one round of the bus's discovery; a holder answering late on a loaded machine → silent |
+| D6 | one reading of a consumer's pending count, after a burst → far behind, which healer H4 acts on |
+| D8, D13 | one question to each machine's holder; one timeout → the probe fails, or the data is unmeasured |
+| D9 | `status` timed once → slow |
+| every probe | one error or timeout → `probe-failed` at once |
+| every watchdog row | one read of the store or the bus that failed → that row's `probe-failed` at once |
+
+The watchdogs' own signals were not of this shape: each has a bound measured in intervals (three
+missed heartbeats, a tier past three times its measured time), which is already more than one look.
+
+**A summary carried detail.** Several producers put raw errors, addresses or paths into the one line
+the operator reads: D2's socket error; D8's banned addresses and an endpoint's domain; the data probe's
+paths, datasets and storage names; the lease's unleased reason. The content rule lives in the
+operator's channel and is not trusted to each source — rightly — but nothing in the controller held
+its own summaries to it, so a producer could not know it had written a sentence that would be
+withheld.
+
+## Fix
+
+The fix is novox/mesh-controller PR #91.
+
+**A finding one look can be wrong about is raised on the second look in a row.** A finding is marked
+as such by its source — an unanswered question, a time measured once, a reading a burst can move. It
+is raised when the source's previous look saw it too: two runs of the self-check (five minutes apart)
+or two ticks of the watchdogs (half a minute). It is kept while it is open, however it is seen, so a
+condition is never cleared and raised again by flapping; it clears, as everything does, on the look
+that no longer sees it. A finding held back is not a pass: the self-check's verdict lists it as
+unconfirmed. A *definite* answer — a resolver answering the wrong address, NXDOMAIN for IPv6, a stream
+that is not there — is not marked, and is raised at once.
+
+Within one run, a question is asked again before it counts as unanswered: D2 asks every question up to
+three times, a little apart, all of them at once (so a silent resolver costs a run seconds, not its
+whole bound); D3 asks the discovery a second time for what the first did not hear; D8 and D13 ask a
+holder again while the probe's bound leaves room. D2's silence, once confirmed, is still urgent: a
+resolver silent for five minutes fails every machine whose resolution depends on it.
+
+Applied to: D2 (silence), D3, D6 (far behind), D9, D13 (unmeasured), the self-check's `probe-failed`,
+and every watchdog row's `probe-failed`.
+
+**A summary says things in machine names and the mesh's words; detail is the evidence.** Every
+producer above now keeps the address, the domain, the path and the raw error in the condition's
+evidence, which stays inside the mesh, and says the rest in words naming machines. The controller
+carries a mirror of the channel's content rule — allowing the mesh's own machine names, as ADR 0234
+says — and the condition store holds every summary to it: one that would be withheld is said in words
+there, and kept whole in the evidence, so no future producer can make an alert unreadable again.
+
+## How it is checked
+
+- A test over the content rule's table: the shapes the channel refuses are refused, the words
+  conditions are made of pass, and the mesh's machine names pass.
+- **A lint over every condition raised in the controller's test suite**: every observation the
+  condition store takes in any test — every signals row past its bound, every probe's finding, every
+  event's condition — and every probe finding a test reads directly is held to the rule; one that would
+  be withheld fails the suite, naming its source and kind. Proven against the producers as they were:
+  the data probe's paths fail it.
+- Tests of D2 against a stand-in resolver: one that misses a try and answers the next is well; one
+  that answers nothing is held for the next run and said in machine names, with the address and the
+  socket's words in the evidence; one that answers wrongly is said at once.
+- A test of the self-check over runs: one look raises nothing and is listed as unconfirmed; two in a
+  row raise; an open condition seen again is kept, not reopened; a pass clears; a probe that cannot
+  run is a condition only on its second run in a row. The same for a blind watchdog row over two ticks.
+
+## Design
+
+To-be 45 §4 said a probe that errors or times out raises `probe-failed`. It now says when: on the
+second run in a row, while the verdict says it at once — and states the rule for findings a single
+look can be wrong about.