Act under a lease, keep accounts by order, one writer at composition (hq to-be 45 Phase 2)

Two controllers could both act (issue 204), a reconcile's report could
overtake the apply after it and the digest decided (issue 267), and a grant
could make a second writer of a machine's report.

- The lease (internal/lease, ADR 0229): mesh-controller_lease key `holder`,
  15 s age, renewed every 5 s by compare-and-set; the epoch is the revision
  it was taken at. The gate is the clock (stops 3 s before expiry); a refused
  renewal is a loss and the process exits; a holder that stops gives it back.
  serve takes it before asserting the bus. Epochs kept in the store
  (migration 0068 controller_epoch) as a floor: a bucket raised from nothing
  is compacted past it. Unleased (no epoch, S12 urgent) only when nobody
  holds it and the bus will not let it be written. A shell command acts
  under the holder's epoch, or its own lease when none.
- Declarations carry `epoch` inside the signed envelope, only to a machine
  whose latest account carried a report_sequence (mesh-host #35); would-send
  is composed with the epoch last sent. Allot and the send both pass the gate.
- Reports: contract in internal/link/order.go (epoch, sequence,
  report_sequence, older_than, refused_older). Accounts kept by epoch, then
  sequence, then report sequence; older refused, counted; unordered reports
  keep the digest rule. Plans by compare-and-set on a revision, with epoch.
  Conditions and calls carry the epoch and are not written off the lease.
- S12 and S13 (naming the writer by epoch) watched, D5 run; reset of the
  bucket said. Writers table compiled in and enforced in PermissionsFor; the
  controller no longer publishes mesh.control.>. A contract per consumed
  kind, and the empty-on-error lint over the repository.
- mesh-host pinned to its main with the epoch in the validator (D1 validates
  the envelope as sent).

Needs mesh-host's genesis lock with the lease grant (mesh-host PR) for
TestTheInstallersFirstUserListIsWhatTheControllerWouldCompose.

internal/link/watched.go

@@ -5,6 +5,8 @@ import (
 	"encoding/json"
 	"errors"
 	"fmt"
+	"slices"
+	"sort"
 	"strings"
 	"sync"
 	"sync/atomic"
@@ -176,48 +178,130 @@ const StaleRefusal = "is older than what the mesh last said to this node"
 // IsStaleRefusal says a report's refusal is the node-engine refusing a declaration older than it holds.
 func IsStaleRefusal(refused string) bool { return strings.Contains(refused, StaleRefusal) }
 
-// RefusalCount keeps when each machine refused a stale declaration, for S13 (novox/hq to-be 45 §3):
-// more than five from one writer in five minutes is a writer sending what it has moved past.
+// A Refusal is one receiver refusing something older than what it holds (novox/hq to-be 45 §6, ADR 0227
+// rule 2): a machine refusing a declaration, or this controller refusing an account.
+type Refusal struct {
+	// Writer is who sent what was refused, in the mesh's words: WriterEpoch for a declaration, a
+	// machine's node-engine for an account.
+	Writer string
+	// Epoch is the lease epoch the refused declaration claimed, for a controller to be named by; zero
+	// for a declaration that claimed none, and for an account.
+	Epoch int64
+	// Receiver is the machine, or "controller", that refused it.
+	Receiver string
+	At       time.Time
+}
+
+// WriterEpoch is how a refused declaration's writer is said before it is named: by the epoch it
+// claimed, or as a controller that claimed none.
+func WriterEpoch(epoch int64) string {
+	if epoch > 0 {
+		return fmt.Sprintf("the controller of epoch %d", epoch)
+	}
+	return "a controller that claimed no epoch"
+}
+
+// WriterNodeEngine is a machine's node-engine as the writer of an account the controller refused.
+func WriterNodeEngine(node string) string { return "the node-engine on " + node }
+
+// RefusalCount keeps every stale refusal heard, for S13 (novox/hq to-be 45 §3): more than five from
+// one writer in five minutes is a writer sending what it — or the mesh — has moved past.
 type RefusalCount struct {
-	mu  sync.Mutex
-	per map[string][]time.Time
+	mu   sync.Mutex
+	list []Refusal
+	// lifetime is each machine's own count of declarations it refused as older, as its last report
+	// said it (`refused_older`), and named how many of those this process heard as refusal reports
+	// since: what the count rose by beyond them is refusals whose reports never arrived.
+	lifetime map[string]int64
+	named    map[string]int64
 }
 
 // StaleRefusals is this process's count.
-var StaleRefusals = &RefusalCount{per: map[string][]time.Time{}}
+var StaleRefusals = NewRefusalCount()
 
-// Refused records one refusal by a machine.
-func (r *RefusalCount) Refused(node string, at time.Time) {
-	r.mu.Lock()
-	defer r.mu.Unlock()
-	r.per[node] = append(r.per[node], at)
+// NewRefusalCount is an empty count.
+func NewRefusalCount() *RefusalCount {
+	return &RefusalCount{lifetime: map[string]int64{}, named: map[string]int64{}}
 }
 
-// Within is how many refusals each machine made since a moment; older ones are forgotten.
-func (r *RefusalCount) Within(since time.Time) map[string]int {
+// Refused records one refusal.
+func (r *RefusalCount) Refused(f Refusal) {
 	r.mu.Lock()
 	defer r.mu.Unlock()
-	out := map[string]int{}
-	for node, times := range r.per {
-		var kept []time.Time
-		for _, t := range times {
-			if !t.Before(since) {
-				kept = append(kept, t)
-			}
-		}
-		if len(kept) == 0 {
-			delete(r.per, node)
+	r.list = append(r.list, f)
+	if f.Writer != WriterNodeEngine(f.Receiver) {
+		r.named[f.Receiver]++
+	}
+}
+
+// Lifetime reads a machine's own count of declarations it refused as older, from any report: what it
+// rose by since the last, beyond the refusals heard by name, is recorded as refused by a writer the
+// mesh never heard named — the refusal's report was lost, and the count is still a fact.
+func (r *RefusalCount) Lifetime(node string, total int64, at time.Time) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	before, known := r.lifetime[node]
+	r.lifetime[node] = total
+	named := r.named[node]
+	r.named[node] = 0
+	if !known || total <= before {
+		return // the first word since this controller started, or a node-engine that started over
+	}
+	for missing := total - before - named; missing > 0; missing-- {
+		r.list = append(r.list, Refusal{Writer: "a writer whose refused declaration was never reported",
+			Receiver: node, At: at})
+	}
+}
+
+// WriterRefusals is one writer's refusals within a window.
+type WriterRefusals struct {
+	Writer    string
+	Epoch     int64
+	Count     int
+	Receivers []string
+	Last      time.Time
+}
+
+// Within is every writer's refusals since a moment, most first; older ones are forgotten.
+func (r *RefusalCount) Within(since time.Time) []WriterRefusals {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	kept := r.list[:0]
+	by := map[string]*WriterRefusals{}
+	var order []string
+	for _, f := range r.list {
+		if f.At.Before(since) {
 			continue
 		}
-		r.per[node] = kept
-		out[node] = len(kept)
+		kept = append(kept, f)
+		w, ok := by[f.Writer]
+		if !ok {
+			w = &WriterRefusals{Writer: f.Writer, Epoch: f.Epoch}
+			by[f.Writer] = w
+			order = append(order, f.Writer)
+		}
+		w.Count++
+		if !slices.Contains(w.Receivers, f.Receiver) {
+			w.Receivers = append(w.Receivers, f.Receiver)
+		}
+		if f.At.After(w.Last) {
+			w.Last = f.At
+		}
 	}
+	r.list = kept
+	out := make([]WriterRefusals, 0, len(order))
+	for _, writer := range order {
+		w := by[writer]
+		sort.Strings(w.Receivers)
+		out = append(out, *w)
+	}
+	sort.SliceStable(out, func(i, j int) bool { return out[i].Count > out[j].Count })
 	return out
 }
 
 // holding is whether this process holds the controller's consumer of what nodes say: the controller
-// acting, not one standing by for another (issue 213). Until the lease (to-be 45 §6) it is how a
-// watchdog knows it is the one that hears.
+// acting, not one standing by for another (issue 213). Beside the lease (to-be 45 §6), which decides
+// who may act, it is how a watchdog knows this process is the one that hears.
 var holding atomic.Bool
 
 // Holding says this process is the controller acting now.