Look twice before saying a probe failed, and say conditions in machine names (hq issue 277)

D2 raised a resolver urgent on one query that timed out while its machine was
loaded, and its summary carried the resolver's address and socket text, so the
operator channel withheld the whole alert.

- D2 asks every question up to three times, all at once; a resolver that
  answers nothing is held for the next run and raised urgent when two runs
  in a row find it silent. A wrong answer is still raised at once.
- Findings a single look can be wrong about carry Confirm: raised on the
  second look in a row, kept while open, never cleared-and-reraised. Used by
  D2 silence, D3 (also asks discovery twice), D6 behind, D9, D13 unmeasured,
  probe-failed of the doctor, and blind watchdog rows.
- Probe seat asks (D8, D13) are asked again when the bus brought no answer.
- Summaries name machines and say things in words; addresses, paths,
  domains and raw errors move to the evidence (D2, D5, D8, D9, D13, S12).
- internal/outward mirrors the messenger's content rule, allowing the mesh's
  machine names; the keeper rewords a summary that would be withheld and keeps
  it whole in the evidence; a TestMain lint fails the suite on any raised or
  linted finding that would be withheld.

cmd/mesh-controller/watchdogs.go

@@ -164,6 +164,9 @@ type watchdogs struct {
 	// standing by hears no heartbeat and would call every machine silent.
 	acting func() bool
 
+	// confirm holds back a row blind for one tick: a read that timed out once on a loaded store.
+	confirm confirming
+
 	mu     sync.Mutex
 	ticked time.Time
 	last   *signalFacts
@@ -226,7 +229,9 @@ func (w *watchdogs) see(running context.Context, f *signalFacts) {
 			problems = append(problems, row.Row+": "+err.Error())
 		}
 	}
-	// The rows that could not see, said; the ones that see again, cleared.
+	// The rows that could not see, said once they could not two ticks in a row; the ones that see
+	// again, cleared.
+	blind, _ = w.confirm.pass(running, w.keeper, sourceWatchdogs, blind)
 	if err := w.keeper.Reconcile(running, sourceWatchdogs, blind); err != nil {
 		problems = append(problems, err.Error())
 	}
@@ -259,10 +264,12 @@ func (w *watchdogs) see(running context.Context, f *signalFacts) {
 // sourceWatchdogs is what raises a blind row's condition.
 const sourceWatchdogs = "watchdogs"
 
-// blindRow is a row whose facts could not be gathered, as a condition of its own.
+// blindRow is a row whose facts could not be gathered, as a condition of its own: raised when the next
+// tick cannot gather them either (confirm.go), since one read that did not answer in time is not a
+// watchdog gone blind.
 func blindRow(row signalRow, err error) conditions.Observation {
 	return conditions.Observation{Scope: conditions.ScopeProbe, ID: row.Row, Kind: "probe-failed", Token: "failed",
-		Severity: conditions.Warning,
+		Severity: conditions.Warning, Confirm: true,
 		Summary: fmt.Sprintf("the watchdog of %s (%s) cannot see: what it reads could not be read, so nothing "+
 			"it would raise can be — and nothing it raised before is cleared", row.Row, row.Signal),
 		Said: firstLine(err.Error())}