Say when the mesh is wrong: conditions, watchdogs, the bus's advisories, doctor (hq to-be 45 Phase 1)

Every one of the 48 core failures of research 031 was found by a person
looking; the mesh's answers carried the fact for whoever asked and told
nobody.

- The condition store (to-be 45 §2): mesh-controller_conditions, one key
  per open condition, written by compare-and-set so a person's silence
  and the watchdogs never lose each other's word; every transition kept
  ninety days in mesh-controller_condition-history and said as the
  seat's events condition-raised / condition-changed / condition-cleared
  (the condition at the top level, with event, at, change, why, show),
  offered again while the bus is away. Raised and cleared by observation
  only; a clearing reopened within ten minutes is the same condition with
  its count up, its silence kept. Verbs: conditions, conditions show,
  conditions silence (a hand act, at most a week), conditions history.
- ADR 0224's provider standing is the first kind, provider-failing, held
  by the provider's events; the provider_standing table is no longer read
  or written (left in place: dropping it is the operator's word).
- status leads with the open conditions, urgent first, and says all well
  only with none open; conditions it cannot read are said and not well.
- The signals table compiled in, one watchdog loop over it every 30s: S1
  heartbeat (3 intervals, asleep machines excepted, control node urgent
  after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf,
  S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9
  advisories, S10 self-check silent, S11 node tools silent, S13 stale
  refusals; S12, S14, S15 deferred with their reasons. A row that cannot
  see raises probe-failed and clears nothing. A test generated from the
  table suppresses each signal inside and past its bound.
- The bus's advisories (maximum deliveries, a mesh consumer deleted) and
  the controller's own slow consumer and refused subjects, said in the
  mesh's words.
- doctor: the probe registry D1-D10 (D5 deferred) and DW, every five
  minutes, each in thirty seconds; a probe that cannot run is never a
  pass. D1 validates with mesh-host's own validator. Every run ends with
  the doctor-heartbeat event mesh-watcher listens for.
- The controller is granted its new buckets, events, the two advisories
  and $SRV.INFO; the node tools their tools-alive heartbeat. The streams
  and consumers the controller asserts and the ones D6/D7 expect are one
  derivation.

cmd/mesh-controller/missed_merges.go

@@ -3,6 +3,7 @@ package main
 import (
 	"context"
 	"fmt"
+	"sync"
 	"time"
 
 	"github.com/novox/mesh-controller/internal/inventory"
@@ -59,9 +60,13 @@ func catchingUpOnMerges(ctx context.Context, open *stores, announced merges) {
 			return
 		case <-tick.C:
 		}
+		watchedMerges.begin()
 		err := catchUpOnMerges(ctx, time.Now(), announced, catalogued, f.SourceMoved, func(format string, args ...any) {
 			fmt.Printf(format+"\n", args...)
 		})
+		// What the pass found is what S5 says (novox/hq to-be 45 §3); a pass that could not read
+		// says that instead, and leaves what the last one found standing.
+		watchedMerges.end(time.Now(), err)
 		// A pass that cannot read says so once, not every five minutes, and says when it reads again.
 		why := ""
 		if err != nil {
@@ -114,6 +119,8 @@ func catchUpOnMerges(ctx context.Context, now time.Time, announced merges,
 		for _, e := range moves {
 			names = append(names, e.Manifest.Module)
 		}
+		watchedMerges.found(missedMerge{Owner: a.Owner, Repo: a.Repo, Base: a.Base, Commit: a.Commit,
+			At: a.At, Modules: names})
 		say("%s/%s merged into %s (%.8s), announced %s ago, and the controller never acted on it: the bus "+
 			"did not hand the announcement over (novox/hq issue 266). %s %s behind it; acting on it now",
 			a.Owner, a.Repo, a.Base, a.Commit, now.Sub(a.At).Round(time.Minute), readableList(names),
@@ -129,3 +136,54 @@ func catchUpOnMerges(ctx context.Context, now time.Time, announced merges,
 	}
 	return nil
 }
+
+// missedMerge is one merge the bus announced and never handed over, as a pass found it.
+type missedMerge struct {
+	Owner, Repo, Base, Commit string
+	// At is when the bus took the announcement.
+	At      time.Time
+	Modules []string
+}
+
+// mergeWatch is what the passes found, for S5 (novox/hq to-be 45 §3): **a merge nothing read is
+// said**, urgent, even though the pass acts on it at once — the bus skipping a message is a fault of
+// the transport the mesh's every change rides on, and acting late is the repair, not the absence of
+// the fault. The next pass, finding it acted on, clears it.
+type mergeWatch struct {
+	mu      sync.Mutex
+	passed  time.Time
+	err     error
+	finding []missedMerge
+	missed  []missedMerge
+}
+
+var watchedMerges = &mergeWatch{}
+
+func (w *mergeWatch) begin() {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	w.finding = nil
+}
+
+func (w *mergeWatch) found(m missedMerge) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	w.finding = append(w.finding, m)
+}
+
+func (w *mergeWatch) end(at time.Time, err error) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	w.passed, w.err = at, err
+	if err == nil {
+		w.missed = w.finding
+	}
+}
+
+// last is when the last pass ended, what the last pass that read found, and what the last pass
+// could not read.
+func (w *mergeWatch) last() (time.Time, []missedMerge, error) {
+	w.mu.Lock()
+	defer w.mu.Unlock()
+	return w.passed, append([]missedMerge(nil), w.missed...), w.err
+}