Say when the mesh is wrong: conditions, watchdogs, the bus's advisories, doctor (hq to-be 45 Phase 1)

Every one of the 48 core failures of research 031 was found by a person
looking; the mesh's answers carried the fact for whoever asked and told
nobody.

- The condition store (to-be 45 §2): mesh-controller_conditions, one key
  per open condition, written by compare-and-set so a person's silence
  and the watchdogs never lose each other's word; every transition kept
  ninety days in mesh-controller_condition-history and said as the
  seat's events condition-raised / condition-changed / condition-cleared
  (the condition at the top level, with event, at, change, why, show),
  offered again while the bus is away. Raised and cleared by observation
  only; a clearing reopened within ten minutes is the same condition with
  its count up, its silence kept. Verbs: conditions, conditions show,
  conditions silence (a hand act, at most a week), conditions history.
- ADR 0224's provider standing is the first kind, provider-failing, held
  by the provider's events; the provider_standing table is no longer read
  or written (left in place: dropping it is the operator's word).
- status leads with the open conditions, urgent first, and says all well
  only with none open; conditions it cannot read are said and not well.
- The signals table compiled in, one watchdog loop over it every 30s: S1
  heartbeat (3 intervals, asleep machines excepted, control node urgent
  after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf,
  S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9
  advisories, S10 self-check silent, S11 node tools silent, S13 stale
  refusals; S12, S14, S15 deferred with their reasons. A row that cannot
  see raises probe-failed and clears nothing. A test generated from the
  table suppresses each signal inside and past its bound.
- The bus's advisories (maximum deliveries, a mesh consumer deleted) and
  the controller's own slow consumer and refused subjects, said in the
  mesh's words.
- doctor: the probe registry D1-D10 (D5 deferred) and DW, every five
  minutes, each in thirty seconds; a probe that cannot run is never a
  pass. D1 validates with mesh-host's own validator. Every run ends with
  the doctor-heartbeat event mesh-watcher listens for.
- The controller is granted its new buckets, events, the two advisories
  and $SRV.INFO; the node tools their tools-alive heartbeat. The streams
  and consumers the controller asserts and the ones D6/D7 expect are one
  derivation.

internal/broker/controller_buckets.go

@@ -19,10 +19,18 @@ import (
 // like the streams, so a bus raised from nothing has them before the first call is served.
 
 // CallsBucket keeps every call of the mesh's own verbs and what came of it; HandActsBucket every act
-// a person did by hand, with why.
+// a person did by hand, with why; ConditionsBucket every condition open now (to-be 45 §2), one key
+// each, and ConditionHistoryBucket every transition of one — raised, changed, silenced, cleared —
+// for ninety days.
+//
+// **The history is a bucket of its own** because its keys expire and an open condition's must not:
+// a bucket has one age for every key, and a condition open longer than the history is kept would
+// otherwise vanish from the store while still true.
 var (
-	CallsBucket    = BucketName(ControllerSeat, "calls")
-	HandActsBucket = BucketName(ControllerSeat, "hand-acts")
+	CallsBucket            = BucketName(ControllerSeat, "calls")
+	HandActsBucket         = BucketName(ControllerSeat, "hand-acts")
+	ConditionsBucket       = BucketName(ControllerSeat, "conditions")
+	ConditionHistoryBucket = BucketName(ControllerSeat, "condition-history")
 )
 
 // The bounds to-be 45 §6 sets for calls: the last thousand, or fourteen days, whichever is fewer.
@@ -37,11 +45,19 @@ const (
 	// HandActsKeptFor is as long as a condition's history (to-be 45 §2): an act by hand is read
 	// back beside what it addressed.
 	HandActsKeptFor = 90 * 24 * time.Hour
+	// ConditionHistoryKeptFor is how long a condition's transitions are kept (to-be 45 §2).
+	ConditionHistoryKeptFor = 90 * 24 * time.Hour
 )
 
 // IsControllerBucket says a bucket is the controller's own, not a module's state nothing declares.
 func IsControllerBucket(bucket string) bool {
-	return bucket == CallsBucket || bucket == HandActsBucket
+	return bucket == CallsBucket || bucket == HandActsBucket || bucket == ConditionsBucket ||
+		bucket == ConditionHistoryBucket
+}
+
+// ControllerBuckets are the controller's own buckets, in the order they are asserted.
+func ControllerBuckets() []string {
+	return []string{CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket}
 }
 
 // ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
@@ -98,5 +114,30 @@ func (j *JetStream) EnsureControllerBuckets() error {
 	}); err != nil {
 		return fmt.Errorf("asserting bucket %s: %w", HandActsBucket, err)
 	}
+	// **No age on the open conditions.** A condition is removed when observation clears it and at no
+	// other moment: one that expired would be a fault the store forgot while it was still true.
+	if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
+		Bucket: ConditionsBucket,
+		Description: "every condition open now, one key each (novox/hq to-be 45 §2): written by the " +
+			"controller alone, raised and cleared by observation, read through `conditions`",
+		History:      1,
+		MaxValueSize: 64 << 10,
+		MaxBytes:     64 << 20,
+		Storage:      jetstream.FileStorage,
+	}); err != nil {
+		return fmt.Errorf("asserting bucket %s: %w", ConditionsBucket, err)
+	}
+	if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
+		Bucket: ConditionHistoryBucket,
+		Description: "every transition of a condition — raised, changed, silenced, cleared — kept ninety " +
+			"days (novox/hq to-be 45 §2): written by the controller alone, read through `conditions history`",
+		History:      1,
+		TTL:          ConditionHistoryKeptFor,
+		MaxValueSize: 64 << 10,
+		MaxBytes:     256 << 20,
+		Storage:      jetstream.FileStorage,
+	}); err != nil {
+		return fmt.Errorf("asserting bucket %s: %w", ConditionHistoryBucket, err)
+	}
 	return nil
 }

internal/broker/controller_buckets_test.go

@@ -2,6 +2,7 @@ package broker
 
 import (
 	"slices"
+	"strings"
 	"testing"
 )
 
@@ -29,3 +30,32 @@ func TestTheControllerMayWriteEveryBucketItWrites(t *testing.T) {
 		t.Error("the controller may write any bucket, a module's state included")
 	}
 }
+
+// **A machine's node tools may say they are there, as that machine and no other** (novox/hq to-be 45
+// S11), and the controller may hear the bus's advisories and ask who answers — read-only, named.
+func TestTheWatchedSignalsMayBeSaidAndHeard(t *testing.T) {
+	tools, err := PermissionsFor(Principal{Kind: KindNodeTools, Node: "anchor", Module: RuntimeModule, PasswordHash: "x"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !slices.Contains(tools.Publish, "mesh.control.anchor.tools-alive") {
+		t.Error("the node tools may not say they are there")
+	}
+	for _, s := range tools.Publish {
+		if strings.Contains(s, "tools-alive") && s != "mesh.control.anchor.tools-alive" {
+			t.Errorf("the node tools may say %s", s)
+		}
+	}
+	controller, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, s := range BusAdvisories {
+		if !slices.Contains(controller.Subscribe, s) {
+			t.Errorf("the controller may not hear %s", s)
+		}
+	}
+	if !slices.Contains(controller.Publish, "$SRV.INFO") || slices.Contains(controller.Subscribe, "$JS.EVENT.>") {
+		t.Error("the controller may not ask who answers, or hears every API call")
+	}
+}

internal/broker/nats.go

@@ -266,6 +266,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
 		sub = append(sub, "mesh.seat."+ControllerSeat+".tool.>")
 		// And says so (novox/hq ADR 0197): it answers discovery for the seat it serves.
 		sub = append(sub, announcing(ControllerSeat)...)
+		// And asks who answers (novox/hq to-be 45 §4, D3): the self-check finds every seat's holder by
+		// the same discovery the console reads. The question only; the answers come to its own inbox.
+		pub = append(pub, "$SRV.INFO")
 
 		// The events it reacts to, and its ack subject on the stream they arrive from
 		// (streams.go). **Each named, not a pattern**: `mesh.mod.*.event.>` would make the
@@ -297,7 +300,14 @@ func PermissionsFor(p Principal) (Permissions, error) {
 		// **And its own buckets** (novox/hq to-be 45 §1): the calls it served and the acts done by
 		// hand, which it alone writes. A put is a publish to the bucket's subject, which `$JS.API.>`
 		// does not cover; each bucket named, not `$KV.>`, which would let it write any module's state.
-		pub = append(pub, "$KV."+CallsBucket+".>", "$KV."+HandActsBucket+".>")
+		for _, bucket := range ControllerBuckets() {
+			pub = append(pub, "$KV."+bucket+".>")
+		}
+		// **And what the bus says about itself, read-only** (novox/hq to-be 45 §3, S9): a durable
+		// consumer that gave up on a message, or one that was deleted. The server already publishes
+		// both in the mesh's own account; the controller says each as a condition in the mesh's words.
+		// Named, not `$JS.EVENT.>`: the other advisories are every API call the mesh makes.
+		sub = append(sub, BusAdvisories...)
 
 	case KindPerson:
 		// Tools, and nothing else. Every subject a person may publish is a tool call; a person
@@ -510,6 +520,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
 		// that varies is the module, so the pattern is the machine's own assignments.
 		sub = append(sub, "mesh.assignment."+p.Node+".*")
 		pub = append(pub, "$JS.API.DIRECT.GET."+AssignmentsStream+".mesh.assignment."+p.Node+".*")
+		// And that it is there (novox/hq to-be 45 §3, S11): its own heartbeat, under its machine's
+		// name and no other's, on core NATS like the host's.
+		pub = append(pub, "mesh.control."+p.Node+".tools-alive")
 		// And every tool on the mesh (ADR 0175, decision 5): any node may call any tool on any
 		// node, as the console already could — the runtime is the console's serving mode.
 		invoked, err := invokedSubjects([]string{"*"})

internal/broker/states_agreement_test.go

@@ -6,6 +6,7 @@ import (
 
 	"github.com/novox/mesh-controller/internal/broker"
 	"github.com/novox/mesh-controller/internal/catalogue"
+	"github.com/novox/mesh-controller/internal/conditions"
 	"github.com/novox/mesh-controller/internal/link"
 )
 
@@ -20,12 +21,15 @@ func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
 		t.Fatalf("the grant is written for the %q seat and the mesh states its facts under %q",
 			broker.ControllerSeat, link.MeshControllerSeat)
 	}
-	for _, event := range []string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore} {
+	// And what is wrong, as it changes, and the self-check's heartbeat (novox/hq to-be 45 §2, §4).
+	states := append([]string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore}, conditions.Events...)
+	states = append(states, conditions.HeartbeatEvent)
+	for _, event := range states {
 		if !slices.Contains(broker.ControllerStates, event) {
 			t.Errorf("the mesh states %q and its account may not publish it", event)
 		}
 	}
-	if len(broker.ControllerStates) != 3 {
+	if len(broker.ControllerStates) != len(states) {
 		t.Errorf("the grant permits %v, which is more than the mesh states", broker.ControllerStates)
 	}
 	// **And the seat says it.** A seat carries the protocol of its role (novox/hq ADR 0129), so the

internal/broker/streams.go

@@ -201,7 +201,23 @@ const ControllerName = "controller"
 // something to say.
 const ControllerSeat = "mesh-controller"
 
-var ControllerStates = []string{"applied", "refused", "built-before"}
+var ControllerStates = []string{"applied", "refused", "built-before",
+	// What is wrong, said as it changes (novox/hq to-be 45 §2): a condition raised, changed in
+	// severity, resolver or silence, and cleared. The operator-channel's holder and any other surface
+	// consume them; the controller tells nobody itself.
+	"condition-raised", "condition-changed", "condition-cleared",
+	// And the self-check's heartbeat, at the end of every run (to-be 45 §4, S10): watched from a
+	// machine that is not the control node, so the controller going quiet is itself said.
+	"doctor-heartbeat"}
+
+// BusAdvisories are what the bus server says about the mesh's own account that the controller
+// reads (novox/hq to-be 45 §3, S9): a durable consumer that handed a message over as often as it
+// may and gave up on it, and one that was deleted. Read-only: an advisory is the server's to
+// publish, and the controller's subscription changes nothing on the bus.
+var BusAdvisories = []string{
+	"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>",
+	"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>",
+}
 
 // ControllerFollows are the events the controller reacts to: the catalogue saying a module's
 // current version moved, and a catalogue that has just started saying it may have missed builds.

internal/broker/testdata/composed.conf

@@ -24,8 +24,8 @@ accounts {
     jetstream: enabled
     users = [
       { user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
-        publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_hand-acts.>", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>"] }
-        subscribe: { allow: ["$JS.API.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
+        publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.refused", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>"] }
+        subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
         allow_responses: { max: 1, ttl: "1m" }
       } }
       { user: "enrol.one", password: "$2a$11$eeeeeeeeeeeeeeeeeeeeee", permissions: {

internal/catalogue/seats.go

@@ -83,7 +83,9 @@ var defaultSeats = append([]Seat{
 	// `assign` and the rest are a role's interface, not a container's, and stay addressable while
 	// the control plane is replaced.
 	{Name: ControllerSeatName, Scope: ScopeMesh, Decision: "novox/hq ADR 0079",
-		Emits:  []string{"applied", "refused", "built-before"},
+		// And what is wrong, as it changes, and the self-check's heartbeat (novox/hq to-be 45 §2, §4).
+		Emits: []string{"applied", "refused", "built-before",
+			"condition-raised", "condition-changed", "condition-cleared", "doctor-heartbeat"},
 		Serves: ControllerVerbs},
 	// The store's first verbs (novox/hq ADR 0159): the smallest set that makes the store askable,
 	// served by whichever module holds the seat with tools of these names.

internal/catalogue/verbs.go

@@ -231,6 +231,33 @@ var ControllerVerbs = []Verb{
 			"kind": "one kind: apply, heartbeat-gap, plan-tier or build; every kind when absent",
 			"days": "how many days back (default 14)",
 		}, nil)},
+	// What is wrong, and the self-check (novox/hq to-be 45 §2, §4).
+	{Name: "conditions", Description: "What is wrong with the mesh now: every open condition, urgent first, " +
+		"then oldest — raised by the watchdogs of the signals table, the self-check's probes and the providers' " +
+		"own words, and cleared when observation says it is resolved, never by hand. Given a key, that one " +
+		"whole with its evidence; with history, every transition lately; with silence, stop one's messages " +
+		"for a while — a hand act, which says why (novox/hq to-be 45 §2).",
+		Input: schema(map[string]string{
+			"key":      "a condition's key: that one whole; with history, only its transitions",
+			"scope":    "only this scope: machine, plan, call, build, merge, provider, seat, bus, core, probe or mesh",
+			"severity": "only urgent, or only warning",
+			"machine":  "only those about this machine",
+			"history":  "\"true\": every raising, change, silence and clearing lately, oldest first",
+			"days":     "with history: how many days back (default 7, at most 90)",
+			"silence":  "a condition's key: send no message for it for a while; it stays open and in status",
+			"for":      "with silence: how long — 30m, 4h, 2d; at most 7d",
+			"why":      "with silence: why — required, and recorded in the hand-act log",
+			"cause":    "with silence: the cause in a word (the condition's kind when absent)",
+		}, nil, "history")},
+	{Name: "doctor", Description: "The self-check (novox/hq to-be 45 §4): the last run's verdict at once — " +
+		"each probe of the design's live invariants passed, failed or could not run, and how long ago. With " +
+		"run, a run now; with probes, the registry; with signals, every row of the signals table and the age " +
+		"of its newest signal. Runs every five minutes on its own; each failure is an open condition.",
+		Input: schema(map[string]string{
+			"run":     "\"true\": run every probe now and answer the verdict",
+			"probes":  "\"true\": the registry — what each probe asserts, and the condition it raises",
+			"signals": "\"true\": the signals table, each row with the age of its newest signal",
+		}, nil, "run", "probes", "signals")},
 	{Name: "build", Description: "Have the build machine build a repository. Answers at once with the build's id: " +
 		"`builds` with that id follows it line by line, and the module is registered when the outcome comes.",
 		Input: schema(map[string]string{

internal/conditions/bus.go

@@ -0,0 +1,159 @@
+package conditions
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"sort"
+	"strconv"
+	"sync/atomic"
+	"time"
+
+	"github.com/nats-io/nats.go"
+	"github.com/nats-io/nats.go/jetstream"
+
+	"github.com/novox/mesh-controller/internal/broker"
+)
+
+// The condition store on the bus (to-be 45 §2, ADR 0201): the controller's two buckets, asserted at
+// its start like its calls and its hand-act log.
+
+// OnTheBus opens the store and its history on a connection.
+func OnTheBus(ctx context.Context, conn *nats.Conn) (Backend, History, error) {
+	api, err := jetstream.New(conn)
+	if err != nil {
+		return nil, nil, err
+	}
+	open, err := api.KeyValue(ctx, broker.ConditionsBucket)
+	if err != nil {
+		return nil, nil, fmt.Errorf("the condition store %s is not on the bus — the controller asserts it at "+
+			"its start, so one older than this has not: %w", broker.ConditionsBucket, err)
+	}
+	history, err := api.KeyValue(ctx, broker.ConditionHistoryBucket)
+	if err != nil {
+		return nil, nil, fmt.Errorf("the condition history %s is not on the bus — the controller asserts it "+
+			"at its start, so one older than this has not: %w", broker.ConditionHistoryBucket, err)
+	}
+	return busStore{open}, &busHistory{api: api, kv: history}, nil
+}
+
+type busStore struct{ kv jetstream.KeyValue }
+
+func (b busStore) Get(ctx context.Context, key string) (Entry, bool, error) {
+	e, err := b.kv.Get(ctx, key)
+	if errors.Is(err, jetstream.ErrKeyNotFound) {
+		return Entry{}, false, nil
+	}
+	if err != nil {
+		return Entry{}, false, err
+	}
+	return Entry{Value: e.Value(), Revision: e.Revision()}, true, nil
+}
+
+func (b busStore) Create(ctx context.Context, key string, value []byte) error {
+	_, err := b.kv.Create(ctx, key, value)
+	if errors.Is(err, jetstream.ErrKeyExists) {
+		return ErrMoved
+	}
+	return err
+}
+
+func (b busStore) Update(ctx context.Context, key string, value []byte, revision uint64) error {
+	_, err := b.kv.Update(ctx, key, value, revision)
+	return moved(err)
+}
+
+func (b busStore) Delete(ctx context.Context, key string, revision uint64) error {
+	return moved(b.kv.Delete(ctx, key, jetstream.LastRevision(revision)))
+}
+
+// moved reads the server's refusal of a compare-and-set as what it is.
+func moved(err error) error {
+	var apiErr *jetstream.APIError
+	if errors.As(err, &apiErr) && apiErr.ErrorCode == jetstream.JSErrCodeStreamWrongLastSequence {
+		return ErrMoved
+	}
+	return err
+}
+
+// All is every key, read through a watch that hands over each current value and then says it has.
+func (b busStore) All(ctx context.Context) (map[string]Entry, error) {
+	w, err := b.kv.WatchAll(ctx, jetstream.IgnoreDeletes())
+	if err != nil {
+		return nil, err
+	}
+	defer func() { _ = w.Stop() }()
+	out := map[string]Entry{}
+	for {
+		select {
+		case <-ctx.Done():
+			return nil, fmt.Errorf("reading the condition store: %w", ctx.Err())
+		case e := <-w.Updates():
+			if e == nil {
+				return out, nil
+			}
+			out[e.Key()] = Entry{Value: e.Value(), Revision: e.Revision()}
+		}
+	}
+}
+
+// busHistory keeps each transition under a key of its time and a sequence, and reads them back
+// from a moment through the stream under the bucket — by time, so a read of the last ten minutes
+// does not read ninety days.
+type busHistory struct {
+	api jetstream.JetStream
+	kv  jetstream.KeyValue
+	seq atomic.Uint64
+}
+
+func (h *busHistory) Append(ctx context.Context, e Event) error {
+	body, err := json.Marshal(e)
+	if err != nil {
+		return err
+	}
+	key := strconv.FormatInt(e.At.UnixNano(), 10) + "-" + strconv.FormatUint(h.seq.Add(1), 10)
+	_, err = h.kv.Put(ctx, key, body)
+	return err
+}
+
+// historyQuiet is how long a read of the history waits for one more transition before it takes the
+// stream as read to its end; it answers at once while it holds something.
+const historyQuiet = 2 * time.Second
+
+func (h *busHistory) Since(ctx context.Context, since time.Time) ([]Event, error) {
+	start := since
+	consumer, err := h.api.OrderedConsumer(ctx, "KV_"+broker.ConditionHistoryBucket, jetstream.OrderedConsumerConfig{
+		DeliverPolicy: jetstream.DeliverByStartTimePolicy, OptStartTime: &start,
+	})
+	if err != nil {
+		return nil, fmt.Errorf("reading the condition history: %w", err)
+	}
+	info, err := consumer.Info(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("reading the condition history: %w", err)
+	}
+	var out []Event
+	pending := info.NumPending
+	for pending > 0 {
+		msg, err := consumer.Next(jetstream.FetchMaxWait(historyQuiet))
+		if err != nil {
+			if ctx.Err() != nil {
+				return nil, ctx.Err()
+			}
+			// Nothing more within the quiet wait: read to its end.
+			break
+		}
+		meta, err := msg.Metadata()
+		if err != nil {
+			break
+		}
+		pending = meta.NumPending
+		var e Event
+		if len(msg.Data()) > 0 && json.Unmarshal(msg.Data(), &e) == nil && e.Key != "" {
+			out = append(out, e)
+		}
+	}
+	sort.SliceStable(out, func(i, j int) bool { return out[i].At.Before(out[j].At) })
+	return out, nil
+}

internal/conditions/bus_test.go

@@ -0,0 +1,117 @@
+package conditions
+
+import (
+	"context"
+	"os"
+	"testing"
+	"time"
+
+	"github.com/nats-io/nats.go/jetstream"
+
+	"github.com/novox/mesh-controller/internal/broker"
+)
+
+// The condition store against a real server: compare-and-set, an unreadable store refused, and the
+// history read back by time are claims about what the bus does.
+
+func busStoreForTest(t *testing.T) (*broker.JetStream, Backend, History) {
+	t.Helper()
+	url := os.Getenv("MESH_TEST_NATS")
+	if url == "" {
+		t.Skip("MESH_TEST_NATS unset")
+	}
+	js, err := broker.Dial(url)
+	if err != nil {
+		t.Fatal(err)
+	}
+	t.Cleanup(js.Close)
+	api, err := jetstream.New(js.Conn())
+	if err != nil {
+		t.Fatal(err)
+	}
+	_ = api.DeleteKeyValue(t.Context(), broker.ConditionsBucket)
+	_ = api.DeleteKeyValue(t.Context(), broker.ConditionHistoryBucket)
+	if err := js.EnsureControllerBuckets(); err != nil {
+		t.Fatal(err)
+	}
+	store, history, err := OnTheBus(t.Context(), js.Conn())
+	if err != nil {
+		t.Fatal(err)
+	}
+	return js, store, history
+}
+
+// **A condition outlives the controller that raised it**, and two writers on the bus cannot lose each
+// other's word: a stale revision is refused as moved.
+func TestNatsTheStoreKeepsConditionsByCompareAndSet(t *testing.T) {
+	_, store, history := busStoreForTest(t)
+	ctx := t.Context()
+	told := &Told{}
+	k := NewKeeper(ctx, Options{Store: store, History: history, Teller: told})
+	if _, err := k.Observe(ctx, silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := k.Observe(ctx, silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	k.Close(context.Background())
+
+	again := NewKeeper(ctx, Options{Store: store, History: history})
+	defer again.Close(context.Background())
+	open, err := again.Open(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(open) != 1 || open[0].Observations != 2 {
+		t.Fatalf("a new keeper read %+v", open)
+	}
+	e, _, err := store.Get(ctx, "machine.ace.silent")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if err := store.Update(ctx, "machine.ace.silent", []byte(`{}`), e.Revision-1); err != ErrMoved {
+		t.Fatalf("a write at a stale revision answered %v", err)
+	}
+	if err := store.Create(ctx, "machine.ace.silent", []byte(`{}`)); err != ErrMoved {
+		t.Fatalf("creating an open condition answered %v", err)
+	}
+	if err := store.Delete(ctx, "machine.ace.silent", e.Revision-1); err != ErrMoved {
+		t.Fatalf("a delete at a stale revision answered %v", err)
+	}
+	if cleared, err := again.Clear(ctx, "machine.ace.silent", "heard"); err != nil || !cleared {
+		t.Fatalf("cleared %v: %v", cleared, err)
+	}
+	// Raised again at once: the store takes a key whose last word was a delete.
+	if _, err := again.Observe(ctx, silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	got, _, _ := again.Get(ctx, "machine.ace.silent")
+	if got.Count != 2 {
+		t.Fatalf("raised again after its clearing as %+v", got)
+	}
+}
+
+// **The history is read back from a moment, oldest first**, through the stream under its bucket.
+func TestNatsTheHistoryIsReadByTime(t *testing.T) {
+	_, store, history := busStoreForTest(t)
+	ctx := t.Context()
+	k := NewKeeper(ctx, Options{Store: store, History: history})
+	if _, err := k.Observe(ctx, silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := k.Clear(ctx, "machine.ace.silent", "heard"); err != nil {
+		t.Fatal(err)
+	}
+	k.Close(context.Background())
+	all, err := history.Since(ctx, time.Now().Add(-time.Hour))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if len(all) != 2 || all[0].Change != ChangeRaised || all[1].Change != ChangeCleared {
+		t.Fatalf("history %+v", all)
+	}
+	none, err := history.Since(ctx, time.Now().Add(time.Hour))
+	if err != nil || len(none) != 0 {
+		t.Fatalf("history from the future: %+v %v", none, err)
+	}
+}

internal/conditions/condition.go

@@ -0,0 +1,232 @@
+// Package conditions is the condition store (novox/hq to-be 45 §2, ADR 0227 rules 5 and 6).
+//
+// **A condition is a durable fact about something the mesh owns that is wrong.** Until this, every
+// one of the forty-eight core failures of research 031 was noticed because a person or an agent
+// looked: the mesh's own answers carried the fact for whoever asked, and told nobody. A condition is
+// raised when an observation says something is wrong past its bound, kept with since-when, evidence
+// and who can resolve it, said on the bus as it changes, and cleared when an observation says it is
+// resolved — never by hand.
+//
+// The controller is the store's only writer (to-be 45 §1). Two of its processes may write at once —
+// the serving controller's watchdogs, and a command a person runs to silence one — so every write
+// is a compare-and-set on the key's revision, and a write that lost the race reads again and redoes
+// itself rather than overwriting what the other said.
+package conditions
+
+import (
+	"fmt"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+)
+
+// Severity is how soon the operator is needed: two levels, no more (to-be 45 §2).
+type Severity string
+
+const (
+	// Urgent needs the operator now.
+	Urgent Severity = "urgent"
+	// Warning needs the operator when they can.
+	Warning Severity = "warning"
+)
+
+// The scopes a condition's key starts with: what kind of thing is wrong.
+const (
+	ScopeMachine  = "machine"
+	ScopePlan     = "plan"
+	ScopeCall     = "call"
+	ScopeBuild    = "build"
+	ScopeMerge    = "merge"
+	ScopeProvider = "provider"
+	ScopeSeat     = "seat"
+	ScopeBus      = "bus"
+	ScopeCore     = "core"
+	ScopeProbe    = "probe"
+	ScopeMesh     = "mesh"
+)
+
+// Scopes is every scope, in the order a person reads them.
+var Scopes = []string{ScopeMachine, ScopePlan, ScopeCall, ScopeBuild, ScopeMerge, ScopeProvider,
+	ScopeSeat, ScopeBus, ScopeCore, ScopeProbe, ScopeMesh}
+
+// Who resolves a condition.
+const (
+	// ResolverSelf clears on observation: the signal returns, the probe passes.
+	ResolverSelf = "self"
+	// ResolverOperator needs a person: a healer's budget spent, or a repair that could only destroy.
+	ResolverOperator = "operator"
+	// ResolverAgent is work handed to an agent (research 017; not raised by anything yet).
+	ResolverAgent = "agent"
+)
+
+// ResolverHealer is the resolver of a condition a registered healer works on (Phase 3).
+func ResolverHealer(name string) string { return "healer:" + name }
+
+// Subject is what the condition is about: its scope, its id within the scope, and the machine it
+// concerns when there is one.
+type Subject struct {
+	Scope   string `json:"scope"`
+	ID      string `json:"id"`
+	Machine string `json:"machine,omitempty"`
+	// Also are the other machines it concerns: a consumer's, for a provider failing it.
+	Also []string `json:"also,omitempty"`
+}
+
+// Evidence is one observation, as it was said.
+type Evidence struct {
+	At   time.Time `json:"at"`
+	Said string    `json:"said"`
+}
+
+// Attempt is one healer's try at a condition (to-be 45 §7; written from Phase 3).
+type Attempt struct {
+	At      time.Time `json:"at"`
+	What    string    `json:"what"`
+	Outcome string    `json:"outcome"`
+}
+
+// Silence is a person saying they know: no messages until it ends (to-be 45 §2). Recorded as a hand
+// act; the condition stays open, and `status` still says it.
+type Silence struct {
+	Until time.Time `json:"until"`
+	By    string    `json:"by"`
+	Why   string    `json:"why"`
+	Since time.Time `json:"since"`
+}
+
+// KeptEvidence is how many observations a condition keeps, newest first.
+const KeptEvidence = 10
+
+// MaxSilence is the longest a condition may be silenced at once: past it, a person says so again.
+const MaxSilence = 7 * 24 * time.Hour
+
+// ReopenWithin is how soon after it cleared a condition raised again is the same one again, with its
+// count increased, rather than news (to-be 45 §2).
+const ReopenWithin = 10 * time.Minute
+
+// Condition is one open condition, as the store keeps it and its events carry it.
+type Condition struct {
+	Key string `json:"key"`
+	// Kind is the condition kind: from the signals table, the probe registry or an event kind.
+	Kind     string   `json:"kind"`
+	Subject  Subject  `json:"subject"`
+	Severity Severity `json:"severity"`
+	// Summary is one line in the mesh's words.
+	Summary string `json:"summary"`
+	// Evidence is the newest observations, at most KeptEvidence, newest first.
+	Evidence []Evidence `json:"evidence"`
+	// Source is the signals-table row, probe or event that raised it: `S1`, `D3`, `provisioner.failing`.
+	Source string `json:"source"`
+	// Raised is when it was first observed this time; LastObserved the newest observation.
+	Raised       time.Time `json:"raised"`
+	LastObserved time.Time `json:"last-observed"`
+	// Observations is how many times it was observed since raised.
+	Observations int `json:"observations"`
+	// Count is how many times it has been raised, a reopening within ReopenWithin counted.
+	Count    int       `json:"count"`
+	Tried    []Attempt `json:"tried,omitempty"`
+	Resolver string    `json:"resolver"`
+	// Silenced is null when no silence is in force: said, not left out, so a reader need not guess.
+	Silenced *Silence `json:"silenced"`
+	// Epoch is the controller lease epoch that last wrote it. Zero until the lease exists (to-be 45
+	// Phase 2): no controller holds an epoch yet, and a number invented here would be one nobody
+	// could compare.
+	Epoch uint64 `json:"epoch"`
+}
+
+// SilencedAt says whether a person's silence is in force at a moment.
+func (c Condition) SilencedAt(now time.Time) bool {
+	return c.Silenced != nil && now.Before(c.Silenced.Until)
+}
+
+// Show is the verb that shows more about a condition, as a message carries it.
+func (c Condition) Show() string { return "mesh-controller.conditions key=" + c.Key }
+
+// Observation is one watchdog, probe or event saying something is wrong now.
+type Observation struct {
+	Scope string
+	// ID is the thing within the scope; several tokens joined by dots where the thing is named by
+	// several (a provider's module, its machine and the consumer).
+	ID string
+	// Token is the last part of the key, short for the kind: `silent` for a machine, `failing` for a
+	// provider. Kind's own word when empty.
+	Token   string
+	Kind    string
+	Machine string
+	// Also are the other machines it concerns.
+	Also     []string
+	Severity Severity
+	Summary  string
+	// Said is this observation's evidence, in the mesh's words; Summary when empty.
+	Said     string
+	Source   string
+	Resolver string
+}
+
+// Key is where the observation's condition is kept: `<scope>.<id>.<kind>`, so the same fault said
+// again is the same condition.
+func (o Observation) Key() string {
+	token := o.Token
+	if token == "" {
+		token = o.Kind
+	}
+	return Key(o.Scope, o.ID, token)
+}
+
+// unsafeKey is anything a key may not hold: the bus takes letters, digits and `-_/=` in a key's
+// tokens, and a `*` or `>` would make one a wildcard.
+var unsafeKey = regexp.MustCompile(`[^A-Za-z0-9_=/-]`)
+
+// Key composes a condition's key from its parts, each token made safe for the bus: a character the
+// bus would refuse becomes `_`, so a key is never refused for the name of the thing it is about.
+func Key(scope, id, token string) string {
+	var parts []string
+	for _, p := range append(append([]string{scope}, strings.Split(id, ".")...), token) {
+		p = unsafeKey.ReplaceAllString(strings.TrimSpace(p), "_")
+		if p == "" {
+			p = "_"
+		}
+		parts = append(parts, p)
+	}
+	return strings.Join(parts, ".")
+}
+
+// check refuses an observation that could not be said: a condition with no kind, no scope the mesh
+// knows, or no severity is one nobody could route.
+func (o Observation) check() error {
+	known := false
+	for _, s := range Scopes {
+		if s == o.Scope {
+			known = true
+		}
+	}
+	switch {
+	case !known:
+		return fmt.Errorf("a condition's scope is one of %s, not %q", strings.Join(Scopes, ", "), o.Scope)
+	case strings.TrimSpace(o.ID) == "":
+		return fmt.Errorf("a %s condition names what it is about", o.Scope)
+	case strings.TrimSpace(o.Kind) == "":
+		return fmt.Errorf("the condition %s has no kind", o.Key())
+	case o.Severity != Urgent && o.Severity != Warning:
+		return fmt.Errorf("the condition %s is urgent or a warning, not %q", o.Key(), o.Severity)
+	case strings.TrimSpace(o.Summary) == "":
+		return fmt.Errorf("the condition %s says nothing", o.Key())
+	case strings.TrimSpace(o.Source) == "":
+		return fmt.Errorf("the condition %s does not say what raised it", o.Key())
+	}
+	return nil
+}
+
+// Order sorts conditions as `status` says them: urgent before warning, then oldest first.
+func Order(list []Condition) {
+	sort.SliceStable(list, func(i, j int) bool {
+		if list[i].Severity != list[j].Severity {
+			return list[i].Severity == Urgent
+		}
+		if !list[i].Raised.Equal(list[j].Raised) {
+			return list[i].Raised.Before(list[j].Raised)
+		}
+		return list[i].Key < list[j].Key
+	})
+}

internal/conditions/events.go

@@ -0,0 +1,73 @@
+package conditions
+
+import "time"
+
+// The events a condition's life emits (to-be 45 §2), as the mesh-controller seat's own: published on
+// `mesh.seat.mesh-controller.event.<name>`, on the events stream, so a consumer that was away catches
+// up. **This is a contract**: the operator-channel's holder is written against these names and the
+// shape of Event, and the controller learns nothing about telling.
+const (
+	// EventRaised: a condition was raised — new, or the same fault again within ReopenWithin of its
+	// clearing (Change says which). A reopened condition is not news: its key is the one the
+	// first message was about.
+	EventRaised = "condition-raised"
+	// EventChanged: its severity, its resolver or its silence changed. Not every observation: a
+	// condition observed again is written, and says nothing.
+	EventChanged = "condition-changed"
+	// EventCleared: an observation says it is resolved. The condition is removed from the store and
+	// the transition kept in its history.
+	EventCleared = "condition-cleared"
+)
+
+// Events is every event a condition's life emits.
+var Events = []string{EventRaised, EventChanged, EventCleared}
+
+// HeartbeatEvent is the self-check's heartbeat (to-be 45 §4, S10), said under the same seat at the end
+// of every run: `{run, at, interval-seconds, counts: {passed, failed, failed-to-run, deferred}, probes,
+// controller, why}`. mesh-watcher, on a machine that is not the control node, listens for it.
+const HeartbeatEvent = "doctor-heartbeat"
+
+// What changed, as an event's Change and a history entry's says it.
+const (
+	ChangeRaised     = "raised"
+	ChangeReopened   = "reopened"
+	ChangeSeverity   = "severity"
+	ChangeResolver   = "resolver"
+	ChangeSilenced   = "silenced"
+	ChangeUnsilenced = "silence-ended"
+	ChangeCleared    = "cleared"
+)
+
+// Event is the body of every condition event, and the shape a history entry keeps: **the condition
+// itself, at the top level** — key, kind, subject, severity, summary, source, raised, last-observed,
+// observations, resolver, silenced (null when not), epoch, and the evidence — with what happened to
+// it beside. One object a consumer reads the same way whichever of the three it is.
+type Event struct {
+	// Condition is the condition after the transition — as it was last held, for a clearing.
+	Condition
+	// Event is the event's own name, so a body read without its subject still says what it is.
+	Event string `json:"event"`
+	// At is when the transition happened.
+	At time.Time `json:"at"`
+	// Change is what happened: raised, reopened, severity, resolver, silenced, silence-ended, cleared.
+	Change string `json:"change"`
+	// Was is the value before, for a severity or resolver change.
+	Was string `json:"was,omitempty"`
+	// Why says why it cleared, or why it was silenced.
+	Why string `json:"why,omitempty"`
+	// Cleared is when it cleared, on a clearing.
+	Cleared *time.Time `json:"cleared,omitempty"`
+	// Show is the verb that shows more.
+	Show string `json:"show"`
+}
+
+// eventFor is the event a change is said under.
+func eventFor(change string) string {
+	switch change {
+	case ChangeRaised, ChangeReopened:
+		return EventRaised
+	case ChangeCleared:
+		return EventCleared
+	}
+	return EventChanged
+}

internal/conditions/memory.go

@@ -0,0 +1,147 @@
+package conditions
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"sort"
+	"sync"
+	"time"
+)
+
+// InMemory is a store and a history held in this process: for tests, and for nothing else — a
+// condition kept here is forgotten by a restart, which is the fault the store exists to remove.
+type InMemory struct {
+	mu       sync.Mutex
+	values   map[string]Entry
+	revision uint64
+	events   []Event
+	// Fail, when set, is what every read and write answers: a store that is away.
+	Fail error
+}
+
+// NewInMemory is an empty store.
+func NewInMemory() *InMemory { return &InMemory{values: map[string]Entry{}} }
+
+func (m *InMemory) Get(_ context.Context, key string) (Entry, bool, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.Fail != nil {
+		return Entry{}, false, m.Fail
+	}
+	e, ok := m.values[key]
+	return e, ok, nil
+}
+
+func (m *InMemory) Create(_ context.Context, key string, value []byte) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.Fail != nil {
+		return m.Fail
+	}
+	if _, ok := m.values[key]; ok {
+		return ErrMoved
+	}
+	m.revision++
+	m.values[key] = Entry{Value: value, Revision: m.revision}
+	return nil
+}
+
+func (m *InMemory) Update(_ context.Context, key string, value []byte, revision uint64) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.Fail != nil {
+		return m.Fail
+	}
+	if e, ok := m.values[key]; !ok || e.Revision != revision {
+		return ErrMoved
+	}
+	m.revision++
+	m.values[key] = Entry{Value: value, Revision: m.revision}
+	return nil
+}
+
+func (m *InMemory) Delete(_ context.Context, key string, revision uint64) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.Fail != nil {
+		return m.Fail
+	}
+	if e, ok := m.values[key]; !ok || e.Revision != revision {
+		return ErrMoved
+	}
+	delete(m.values, key)
+	return nil
+}
+
+func (m *InMemory) All(context.Context) (map[string]Entry, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.Fail != nil {
+		return nil, m.Fail
+	}
+	out := make(map[string]Entry, len(m.values))
+	for k, v := range m.values {
+		out[k] = v
+	}
+	return out, nil
+}
+
+func (m *InMemory) Append(_ context.Context, e Event) error {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.Fail != nil {
+		return m.Fail
+	}
+	m.events = append(m.events, e)
+	return nil
+}
+
+func (m *InMemory) Since(_ context.Context, since time.Time) ([]Event, error) {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	if m.Fail != nil {
+		return nil, m.Fail
+	}
+	var out []Event
+	for _, e := range m.events {
+		if !e.At.Before(since) {
+			out = append(out, e)
+		}
+	}
+	sort.SliceStable(out, func(i, j int) bool { return out[i].At.Before(out[j].At) })
+	return out, nil
+}
+
+// Told is a teller that remembers what it was told, for tests.
+type Told struct {
+	mu     sync.Mutex
+	Events []Event
+	Names  []string
+	Fail   error
+}
+
+func (t *Told) PublishSeatEvent(_ context.Context, seat, event string, body []byte) error {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+	if t.Fail != nil {
+		return t.Fail
+	}
+	if seat != Seat {
+		return errors.New("told under the wrong seat: " + seat)
+	}
+	var e Event
+	if err := json.Unmarshal(body, &e); err != nil {
+		return err
+	}
+	t.Events = append(t.Events, e)
+	t.Names = append(t.Names, event)
+	return nil
+}
+
+// Said is a copy of what was told so far.
+func (t *Told) Said() []Event {
+	t.mu.Lock()
+	defer t.mu.Unlock()
+	return append([]Event(nil), t.Events...)
+}

internal/conditions/store.go

@@ -0,0 +1,502 @@
+package conditions
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+)
+
+// Backend is where the open conditions are kept: one value per key, written by compare-and-set.
+type Backend interface {
+	// Get is one key's value and revision; false when it holds none.
+	Get(ctx context.Context, key string) (Entry, bool, error)
+	// Create writes a key that holds nothing, and fails with ErrMoved when it holds something.
+	Create(ctx context.Context, key string, value []byte) error
+	// Update writes a key at the revision it was read at, and fails with ErrMoved when it moved.
+	Update(ctx context.Context, key string, value []byte, revision uint64) error
+	// Delete removes a key at the revision it was read at, and fails with ErrMoved when it moved.
+	Delete(ctx context.Context, key string, revision uint64) error
+	// All is every key's value. An error is an error: never an empty store (ADR 0227 rule 4).
+	All(ctx context.Context) (map[string]Entry, error)
+}
+
+// Entry is one key's value, at a revision.
+type Entry struct {
+	Value    []byte
+	Revision uint64
+}
+
+// ErrMoved is a compare-and-set that lost: somebody wrote the key since it was read.
+var ErrMoved = errors.New("the condition was written by somebody else since it was read")
+
+// History keeps every transition (to-be 45 §2): appended, read back from a moment.
+type History interface {
+	Append(ctx context.Context, e Event) error
+	// Since is every transition from a moment, oldest first.
+	Since(ctx context.Context, since time.Time) ([]Event, error)
+}
+
+// Teller says a transition on the bus, as the mesh-controller seat's event. The link's bus is one.
+type Teller interface {
+	PublishSeatEvent(ctx context.Context, seat, event string, body []byte) error
+}
+
+// Seat is the role the events are said under (novox/hq ADR 0134): the control plane's.
+const Seat = "mesh-controller"
+
+// Keeper raises, observes, silences and clears conditions, and says each transition.
+type Keeper struct {
+	store   Backend
+	history History
+	teller  Teller
+	now     func() time.Time
+	say     func(format string, args ...any)
+	changed func()
+
+	mu sync.Mutex
+	// cleared is when each recently cleared condition cleared and how often it had been raised, so
+	// one raised again within ReopenWithin is the same one again.
+	cleared map[string]clearing
+
+	// out is the transitions still to be said and kept, in order: said by one goroutine, so a
+	// condition's events arrive in the order they happened, and offered again while the bus is away.
+	out     chan Event
+	drained chan struct{}
+	closing sync.Once
+	// Unsaid counts the transitions given up on, for the self-check to say.
+	unsaid int
+}
+
+type clearing struct {
+	at       time.Time
+	count    int
+	silenced *Silence
+}
+
+// Options are what a Keeper is made with.
+type Options struct {
+	Store   Backend
+	History History
+	// Teller says the transitions; nil says nothing (a test, or a command run with no bus to say on).
+	Teller Teller
+	Now    func() time.Time
+	// Say is where a transition that could not be said or kept is said instead.
+	Say func(format string, args ...any)
+	// Changed is told of every transition, at once — for `status`, which leads with what is open.
+	Changed func()
+}
+
+// TellFor is how long one transition is offered to the bus before it is said lost.
+var TellFor = 10 * time.Minute
+
+// NewKeeper is a keeper over a store. It reads what cleared lately from the history, so a condition
+// that cleared just before this controller started and is raised again now is a reopening.
+func NewKeeper(ctx context.Context, o Options) *Keeper {
+	k := &Keeper{store: o.Store, history: o.History, teller: o.Teller, now: o.Now, say: o.Say, changed: o.Changed,
+		cleared: map[string]clearing{}, out: make(chan Event, 1024), drained: make(chan struct{})}
+	if k.now == nil {
+		k.now = time.Now
+	}
+	if k.say == nil {
+		k.say = func(string, ...any) {}
+	}
+	if k.history != nil {
+		if recent, err := k.history.Since(ctx, k.now().Add(-ReopenWithin)); err == nil {
+			for _, e := range recent {
+				if e.Change == ChangeCleared {
+					k.cleared[e.Key] = clearing{at: e.At, count: e.Condition.Count, silenced: e.Condition.Silenced}
+				}
+			}
+		} else {
+			k.say("what cleared lately could not be read from the condition history, so a condition "+
+				"raised again now is said as new rather than reopened: %v", err)
+		}
+	}
+	go k.telling()
+	return k
+}
+
+// Close says what is still to be said, waiting at most until ctx ends.
+func (k *Keeper) Close(ctx context.Context) {
+	k.closing.Do(func() { close(k.out) })
+	select {
+	case <-k.drained:
+	case <-ctx.Done():
+		k.say("%d condition transition(s) were not yet said when this process ended", len(k.out))
+	}
+}
+
+// Unsaid is how many transitions were given up on since this keeper started.
+func (k *Keeper) Unsaid() int {
+	k.mu.Lock()
+	defer k.mu.Unlock()
+	return k.unsaid
+}
+
+// tries bounds one compare-and-set: two writers rarely race more than once.
+const tries = 8
+
+// Observe records one observation: raises the condition if it is not open, and otherwise adds the
+// evidence. Says a raising, a reopening, and a change of severity or resolver; an observation that
+// changes neither is written and said nowhere.
+func (k *Keeper) Observe(ctx context.Context, o Observation) (Condition, error) {
+	if err := o.check(); err != nil {
+		return Condition{}, err
+	}
+	key := o.Key()
+	for i := 0; i < tries; i++ {
+		now := k.now().UTC()
+		said := o.Said
+		if said == "" {
+			said = o.Summary
+		}
+		entry, found, err := k.store.Get(ctx, key)
+		if err != nil {
+			return Condition{}, fmt.Errorf("reading the condition %s: %w", key, err)
+		}
+		if !found {
+			c := Condition{Key: key, Kind: o.Kind, Subject: Subject{Scope: o.Scope, ID: o.ID, Machine: o.Machine, Also: o.Also},
+				Severity: o.Severity, Summary: o.Summary, Evidence: []Evidence{{At: now, Said: said}},
+				Source: o.Source, Raised: now, LastObserved: now, Observations: 1, Count: 1,
+				Resolver: orSelf(o.Resolver)}
+			change := ChangeRaised
+			k.mu.Lock()
+			if before, ok := k.cleared[key]; ok && now.Sub(before.at) <= ReopenWithin {
+				c.Count, change = before.count+1, ChangeReopened
+				// A silence a person gave the condition before it cleared still holds: they said
+				// they knew, and the same fault again ten minutes later is what they knew about.
+				if before.silenced != nil && now.Before(before.silenced.Until) {
+					c.Silenced = before.silenced
+				}
+			}
+			k.mu.Unlock()
+			body, err := json.Marshal(c)
+			if err != nil {
+				return Condition{}, err
+			}
+			if err := k.store.Create(ctx, key, body); errors.Is(err, ErrMoved) {
+				continue
+			} else if err != nil {
+				return Condition{}, fmt.Errorf("raising the condition %s: %w", key, err)
+			}
+			k.mu.Lock()
+			delete(k.cleared, key)
+			k.mu.Unlock()
+			k.tell(Event{Condition: c, At: now, Change: change})
+			return c, nil
+		}
+		var c Condition
+		if err := json.Unmarshal(entry.Value, &c); err != nil {
+			return Condition{}, fmt.Errorf("the condition %s on the bus cannot be read: %w", key, err)
+		}
+		var changes []Event
+		if o.Severity != c.Severity {
+			changes = append(changes, Event{Change: ChangeSeverity, Was: string(c.Severity)})
+			c.Severity = o.Severity
+		}
+		if r := orSelf(o.Resolver); o.Resolver != "" && r != c.Resolver {
+			changes = append(changes, Event{Change: ChangeResolver, Was: c.Resolver})
+			c.Resolver = r
+		}
+		c.Summary, c.Source, c.LastObserved = o.Summary, o.Source, now
+		if o.Machine != "" {
+			c.Subject.Machine = o.Machine
+		}
+		if len(o.Also) > 0 {
+			c.Subject.Also = o.Also
+		}
+		c.Observations++
+		c.Evidence = append([]Evidence{{At: now, Said: said}}, c.Evidence...)
+		if len(c.Evidence) > KeptEvidence {
+			c.Evidence = c.Evidence[:KeptEvidence]
+		}
+		body, err := json.Marshal(c)
+		if err != nil {
+			return Condition{}, err
+		}
+		if err := k.store.Update(ctx, key, body, entry.Revision); errors.Is(err, ErrMoved) {
+			continue
+		} else if err != nil {
+			return Condition{}, fmt.Errorf("observing the condition %s: %w", key, err)
+		}
+		for _, e := range changes {
+			e.At, e.Condition = now, c
+			k.tell(e)
+		}
+		return c, nil
+	}
+	return Condition{}, fmt.Errorf("the condition %s kept moving under this write; %d tries", key, tries)
+}
+
+// Clear removes a condition an observation says is resolved, and says so. False when none was open.
+func (k *Keeper) Clear(ctx context.Context, key, why string) (bool, error) {
+	for i := 0; i < tries; i++ {
+		entry, found, err := k.store.Get(ctx, key)
+		if err != nil {
+			return false, fmt.Errorf("reading the condition %s: %w", key, err)
+		}
+		if !found {
+			return false, nil
+		}
+		var c Condition
+		if err := json.Unmarshal(entry.Value, &c); err != nil {
+			// Unreadable is not resolved: kept, and said, rather than removed unread.
+			return false, fmt.Errorf("the condition %s on the bus cannot be read, so it is not cleared: %w", key, err)
+		}
+		if err := k.store.Delete(ctx, key, entry.Revision); errors.Is(err, ErrMoved) {
+			continue
+		} else if err != nil {
+			return false, fmt.Errorf("clearing the condition %s: %w", key, err)
+		}
+		now := k.now().UTC()
+		k.mu.Lock()
+		k.cleared[key] = clearing{at: now, count: c.Count, silenced: c.Silenced}
+		k.mu.Unlock()
+		k.tell(Event{Condition: c, At: now, Change: ChangeCleared, Why: why, Cleared: &now})
+		return true, nil
+	}
+	return false, fmt.Errorf("the condition %s kept moving under this clearing; %d tries", key, tries)
+}
+
+// Reconcile is one source's whole observation: every condition it observes is observed, and every
+// condition it raised before and no longer observes is cleared — the observation says it is
+// resolved. A source that could not observe must not call this: an empty observation clears all it
+// raised, which is exactly the fault of saying "none" for "I could not tell" (ADR 0227 rule 4).
+func (k *Keeper) Reconcile(ctx context.Context, source string, observed []Observation) error {
+	all, err := k.Open(ctx)
+	if err != nil {
+		return err
+	}
+	seen := map[string]bool{}
+	var problems []string
+	for _, o := range observed {
+		o.Source = source
+		seen[o.Key()] = true
+		if _, err := k.Observe(ctx, o); err != nil {
+			problems = append(problems, err.Error())
+		}
+	}
+	for _, c := range all {
+		if c.Source != source || seen[c.Key] {
+			continue
+		}
+		if _, err := k.Clear(ctx, c.Key, source+" no longer observes it"); err != nil {
+			problems = append(problems, err.Error())
+		}
+	}
+	if len(problems) > 0 {
+		return errors.New(strings.Join(problems, "; "))
+	}
+	return nil
+}
+
+// Silence stops a condition's messages for a while, with a reason, by somebody (to-be 45 §2). The
+// condition stays open and `status` still says it; recording the act in the hand-act log is the
+// caller's, which knows who acted.
+func (k *Keeper) Silence(ctx context.Context, key string, d time.Duration, by, why string) (Condition, error) {
+	if strings.TrimSpace(why) == "" {
+		return Condition{}, errors.New("a silence says why: --why <text>")
+	}
+	if d <= 0 || d > MaxSilence {
+		return Condition{}, fmt.Errorf("a condition is silenced for a while, at most %s — not %s", MaxSilence, d)
+	}
+	for i := 0; i < tries; i++ {
+		entry, found, err := k.store.Get(ctx, key)
+		if err != nil {
+			return Condition{}, fmt.Errorf("reading the condition %s: %w", key, err)
+		}
+		if !found {
+			return Condition{}, fmt.Errorf("no condition %s is open — `conditions` lists them", key)
+		}
+		var c Condition
+		if err := json.Unmarshal(entry.Value, &c); err != nil {
+			return Condition{}, fmt.Errorf("the condition %s on the bus cannot be read: %w", key, err)
+		}
+		now := k.now().UTC()
+		c.Silenced = &Silence{Until: now.Add(d), By: by, Why: strings.TrimSpace(why), Since: now}
+		body, err := json.Marshal(c)
+		if err != nil {
+			return Condition{}, err
+		}
+		if err := k.store.Update(ctx, key, body, entry.Revision); errors.Is(err, ErrMoved) {
+			continue
+		} else if err != nil {
+			return Condition{}, fmt.Errorf("silencing the condition %s: %w", key, err)
+		}
+		k.tell(Event{Condition: c, At: now, Change: ChangeSilenced, Why: c.Silenced.Why})
+		return c, nil
+	}
+	return Condition{}, fmt.Errorf("the condition %s kept moving under this silence; %d tries", key, tries)
+}
+
+// EndSilences ends every silence that has run out, and says each: the condition is still open, and
+// its messages start again.
+func (k *Keeper) EndSilences(ctx context.Context) error {
+	all, err := k.Open(ctx)
+	if err != nil {
+		return err
+	}
+	now := k.now().UTC()
+	for _, c := range all {
+		if c.Silenced == nil || now.Before(c.Silenced.Until) {
+			continue
+		}
+		for i := 0; i < tries; i++ {
+			entry, found, err := k.store.Get(ctx, c.Key)
+			if err != nil {
+				return err
+			}
+			if !found {
+				break
+			}
+			var held Condition
+			if err := json.Unmarshal(entry.Value, &held); err != nil {
+				return fmt.Errorf("the condition %s on the bus cannot be read: %w", c.Key, err)
+			}
+			if held.Silenced == nil || now.Before(held.Silenced.Until) {
+				break
+			}
+			was := held.Silenced.Why
+			held.Silenced = nil
+			body, err := json.Marshal(held)
+			if err != nil {
+				return err
+			}
+			if err := k.store.Update(ctx, c.Key, body, entry.Revision); errors.Is(err, ErrMoved) {
+				continue
+			} else if err != nil {
+				return err
+			}
+			k.tell(Event{Condition: held, At: now, Change: ChangeUnsilenced, Why: was})
+			break
+		}
+	}
+	return nil
+}
+
+// Open is every open condition, urgent first and then oldest first.
+func (k *Keeper) Open(ctx context.Context) ([]Condition, error) {
+	return Read(ctx, k.store)
+}
+
+// Get is one open condition.
+func (k *Keeper) Get(ctx context.Context, key string) (Condition, bool, error) {
+	return ReadOne(ctx, k.store, key)
+}
+
+// HistorySince is every transition from a moment, oldest first.
+func (k *Keeper) HistorySince(ctx context.Context, since time.Time) ([]Event, error) {
+	if k.history == nil {
+		return nil, errors.New("this keeper has no history to read")
+	}
+	return k.history.Since(ctx, since)
+}
+
+// Read is every open condition in a store, in the order status says them. A value that cannot be
+// read is an error naming its key, never a condition left out (ADR 0227 rule 4).
+func Read(ctx context.Context, store Backend) ([]Condition, error) {
+	all, err := store.All(ctx)
+	if err != nil {
+		return nil, fmt.Errorf("the open conditions cannot be read: %w", err)
+	}
+	out := make([]Condition, 0, len(all))
+	for key, e := range all {
+		var c Condition
+		if err := json.Unmarshal(e.Value, &c); err != nil {
+			return nil, fmt.Errorf("the condition %s cannot be read: %w", key, err)
+		}
+		out = append(out, c)
+	}
+	Order(out)
+	return out, nil
+}
+
+// ReadOne is one open condition from a store.
+func ReadOne(ctx context.Context, store Backend, key string) (Condition, bool, error) {
+	e, found, err := store.Get(ctx, key)
+	if err != nil || !found {
+		return Condition{}, found, err
+	}
+	var c Condition
+	if err := json.Unmarshal(e.Value, &c); err != nil {
+		return Condition{}, false, fmt.Errorf("the condition %s cannot be read: %w", key, err)
+	}
+	return c, true, nil
+}
+
+// tell queues a transition to be kept and said. Never blocks the caller for long: a queue that is
+// full is a bus away for a long time, and the transition is said lost rather than holding a watchdog.
+func (k *Keeper) tell(e Event) {
+	e.Event = eventFor(e.Change)
+	e.Show = e.Condition.Show()
+	if k.changed != nil {
+		k.changed()
+	}
+	defer func() {
+		// A keeper closed while a write was in flight: said, not a panic.
+		if recover() != nil {
+			k.lost(e, errors.New("the keeper was closed"))
+		}
+	}()
+	select {
+	case k.out <- e:
+	default:
+		k.lost(e, errors.New("too many transitions are waiting to be said"))
+	}
+}
+
+func (k *Keeper) lost(e Event, err error) {
+	k.mu.Lock()
+	k.unsaid++
+	k.mu.Unlock()
+	k.say("the condition %s was %s and that could NOT be said or kept: %v", e.Key, e.Change, err)
+}
+
+// telling keeps and says every transition in order, offering each again while the bus is away.
+func (k *Keeper) telling() {
+	defer close(k.drained)
+	for e := range k.out {
+		body, err := json.Marshal(e)
+		if err != nil {
+			k.lost(e, err)
+			continue
+		}
+		deadline := time.Now().Add(TellFor)
+		wait := 200 * time.Millisecond
+		kept, said := k.history == nil, k.teller == nil
+		for {
+			ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
+			if !kept {
+				kept = k.history.Append(ctx, e) == nil
+			}
+			if !said {
+				said = k.teller.PublishSeatEvent(ctx, Seat, e.Event, body) == nil
+			}
+			cancel()
+			if kept && said {
+				break
+			}
+			if time.Now().After(deadline) {
+				what := "said"
+				if !kept {
+					what = "kept in the history"
+				}
+				k.lost(e, fmt.Errorf("not %s within %s", what, TellFor))
+				break
+			}
+			time.Sleep(wait)
+			wait = min(2*wait, 10*time.Second)
+		}
+	}
+}
+
+func orSelf(resolver string) string {
+	if resolver == "" {
+		return ResolverSelf
+	}
+	return resolver
+}

internal/conditions/store_test.go

@@ -0,0 +1,379 @@
+package conditions
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"strings"
+	"testing"
+	"time"
+)
+
+// clock is a time a test moves by hand.
+type clock struct{ at time.Time }
+
+func (c *clock) now() time.Time       { return c.at }
+func (c *clock) pass(d time.Duration) { c.at = c.at.Add(d) }
+func newClock() *clock                { return &clock{at: time.Date(2026, 10, 6, 12, 0, 0, 0, time.UTC)} }
+func keeper(t *testing.T) (*Keeper, *InMemory, *Told, *clock) {
+	t.Helper()
+	store, told, c := NewInMemory(), &Told{}, newClock()
+	k := NewKeeper(t.Context(), Options{Store: store, History: store, Teller: told, Now: c.now,
+		Say: func(f string, a ...any) { t.Logf(f, a...) }})
+	t.Cleanup(func() { k.Close(context.Background()) })
+	return k, store, told, c
+}
+
+// settled waits until the teller has been told n events.
+func settled(t *testing.T, told *Told, n int) []Event {
+	t.Helper()
+	deadline := time.Now().Add(5 * time.Second)
+	for {
+		said := told.Said()
+		if len(said) >= n {
+			return said
+		}
+		if time.Now().After(deadline) {
+			t.Fatalf("told %d event(s), want %d: %+v", len(said), n, said)
+		}
+		time.Sleep(5 * time.Millisecond)
+	}
+}
+
+func silent(node string) Observation {
+	return Observation{Scope: ScopeMachine, ID: node, Kind: "silent", Machine: node, Severity: Warning,
+		Summary: node + " has not been heard from", Source: "S1"}
+}
+
+// **A condition is raised once, observed many times, and said on the bus only when it changes**
+// (to-be 45 §2): an observation that changes nothing is written and said nowhere, or the operator's
+// channel would hear the same fault every thirty seconds.
+func TestAConditionIsSaidWhenItChangesNotWhenItIsSeenAgain(t *testing.T) {
+	k, _, told, c := keeper(t)
+	ctx := t.Context()
+	for i := 0; i < 3; i++ {
+		if _, err := k.Observe(ctx, silent("ace")); err != nil {
+			t.Fatal(err)
+		}
+		c.pass(time.Minute)
+	}
+	urgent := silent("ace")
+	urgent.Severity = Urgent
+	got, err := k.Observe(ctx, urgent)
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Key != "machine.ace.silent" || got.Observations != 4 || got.Count != 1 || got.Severity != Urgent {
+		t.Fatalf("held %+v", got)
+	}
+	if len(got.Evidence) != 4 || !got.Evidence[0].At.Equal(c.at) {
+		t.Fatalf("evidence is not newest first: %+v", got.Evidence)
+	}
+	said := settled(t, told, 2)
+	if said[0].Event != EventRaised || said[0].Change != ChangeRaised || said[1].Event != EventChanged ||
+		said[1].Change != ChangeSeverity || said[1].Was != string(Warning) {
+		t.Fatalf("said %+v", said)
+	}
+	time.Sleep(50 * time.Millisecond)
+	if n := len(told.Said()); n != 2 {
+		t.Fatalf("said %d events for one raising and one change", n)
+	}
+	for i, name := range told.Names {
+		if name != told.Events[i].Event {
+			t.Errorf("event %d published as %s and says it is %s", i, name, told.Events[i].Event)
+		}
+	}
+}
+
+// **Evidence is bounded**: a condition open for a week keeps its newest ten observations, not all.
+func TestEvidenceKeepsTheNewestTen(t *testing.T) {
+	k, _, _, c := keeper(t)
+	var got Condition
+	for i := 0; i < 25; i++ {
+		var err error
+		if got, err = k.Observe(t.Context(), silent("ace")); err != nil {
+			t.Fatal(err)
+		}
+		c.pass(time.Minute)
+	}
+	if len(got.Evidence) != KeptEvidence || got.Observations != 25 {
+		t.Fatalf("kept %d evidence of %d observations", len(got.Evidence), got.Observations)
+	}
+}
+
+// **Cleared and raised again within ten minutes is the same condition again** (to-be 45 §2): its
+// count goes up and it is said as reopened, not as news; a person's silence of it still holds.
+func TestRaisedAgainSoonAfterClearingReopens(t *testing.T) {
+	k, store, told, c := keeper(t)
+	ctx := t.Context()
+	if _, err := k.Observe(ctx, silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := k.Silence(ctx, "machine.ace.silent", time.Hour, "jochen", "the laptop is on the train"); err != nil {
+		t.Fatal(err)
+	}
+	if cleared, err := k.Clear(ctx, "machine.ace.silent", "heard again"); err != nil || !cleared {
+		t.Fatalf("cleared %v: %v", cleared, err)
+	}
+	c.pass(5 * time.Minute)
+	again, err := k.Observe(ctx, silent("ace"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if again.Count != 2 || again.Silenced == nil {
+		t.Fatalf("reopened as %+v", again)
+	}
+	said := settled(t, told, 4)
+	if said[3].Event != EventRaised || said[3].Change != ChangeReopened {
+		t.Fatalf("the reopening was said as %+v", said[3])
+	}
+	// And from a new keeper — the controller restarted between — reading what cleared from history.
+	if _, err := k.Clear(ctx, "machine.ace.silent", "heard again"); err != nil {
+		t.Fatal(err)
+	}
+	settled(t, told, 5)
+	k.Close(context.Background())
+	next := NewKeeper(ctx, Options{Store: store, History: store, Now: c.now})
+	defer next.Close(context.Background())
+	c.pass(time.Minute)
+	third, err := next.Observe(ctx, silent("ace"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if third.Count != 3 {
+		t.Fatalf("a controller restarted between cleared and raised said it as new: %+v", third)
+	}
+	// Past the window it is news.
+	if _, err := next.Clear(ctx, "machine.ace.silent", "heard again"); err != nil {
+		t.Fatal(err)
+	}
+	c.pass(ReopenWithin + time.Minute)
+	fourth, err := next.Observe(ctx, silent("ace"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if fourth.Count != 1 || fourth.Silenced != nil {
+		t.Fatalf("raised past the window as %+v", fourth)
+	}
+}
+
+// **A source's whole observation clears what it no longer observes, and only its own.** A watchdog
+// that stops seeing a fault says it is resolved; it does not clear what another raised.
+func TestReconcileClearsOnlyTheSourcesOwn(t *testing.T) {
+	k, _, told, _ := keeper(t)
+	ctx := t.Context()
+	other := Observation{Scope: ScopeProbe, ID: "D3", Kind: "probe-failed", Token: "failed", Severity: Warning,
+		Summary: "D3 did not answer", Source: "doctor"}
+	if _, err := k.Observe(ctx, other); err != nil {
+		t.Fatal(err)
+	}
+	if err := k.Reconcile(ctx, "S1", []Observation{silent("ace"), silent("g14")}); err != nil {
+		t.Fatal(err)
+	}
+	if err := k.Reconcile(ctx, "S1", []Observation{silent("g14")}); err != nil {
+		t.Fatal(err)
+	}
+	open, err := k.Open(ctx)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var keys []string
+	for _, c := range open {
+		keys = append(keys, c.Key)
+	}
+	if strings.Join(keys, ",") != "machine.g14.silent,probe.D3.failed" {
+		t.Fatalf("open after the second observation: %v", keys)
+	}
+	said := settled(t, told, 4)
+	last := said[3]
+	if last.Event != EventCleared || last.Key != "machine.ace.silent" || last.Why == "" {
+		t.Fatalf("the clearing was said as %+v", last)
+	}
+}
+
+// **A store that cannot be read is never an empty one** (ADR 0227 rule 4): reconciling against it
+// clears nothing and says why.
+func TestAnUnreadableStoreClearsNothing(t *testing.T) {
+	k, store, _, _ := keeper(t)
+	ctx := t.Context()
+	if _, err := k.Observe(ctx, silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	store.Fail = errors.New("the bus is away")
+	if err := k.Reconcile(ctx, "S1", nil); err == nil {
+		t.Fatal("reconciled against a store it could not read")
+	}
+	if _, err := k.Open(ctx); err == nil {
+		t.Fatal("an unreadable store answered as read")
+	}
+	store.Fail = nil
+	store.values["machine.g14.silent"] = Entry{Value: []byte("{not a condition"), Revision: 99}
+	if _, err := k.Open(ctx); err == nil || !strings.Contains(err.Error(), "machine.g14.silent") {
+		t.Fatalf("an unreadable condition was left out rather than said: %v", err)
+	}
+	if cleared, err := k.Clear(ctx, "machine.g14.silent", "x"); err == nil || cleared {
+		t.Fatal("an unreadable condition was cleared unread")
+	}
+}
+
+// **A silence is bounded, says why, and ends on its own** (to-be 45 §2): the condition stays open
+// through it, and its messages start again when it ends.
+func TestASilenceIsBoundedAndEnds(t *testing.T) {
+	k, _, told, c := keeper(t)
+	ctx := t.Context()
+	if _, err := k.Observe(ctx, silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	if _, err := k.Silence(ctx, "machine.ace.silent", 8*24*time.Hour, "jochen", "away"); err == nil {
+		t.Fatal("silenced for longer than a week")
+	}
+	if _, err := k.Silence(ctx, "machine.ace.silent", time.Hour, "jochen", " "); err == nil {
+		t.Fatal("silenced without a reason")
+	}
+	if _, err := k.Silence(ctx, "machine.nothing.silent", time.Hour, "jochen", "x"); err == nil {
+		t.Fatal("silenced a condition that is not open")
+	}
+	held, err := k.Silence(ctx, "machine.ace.silent", time.Hour, "jochen", "on the train")
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !held.SilencedAt(c.at) || held.Silenced.By != "jochen" {
+		t.Fatalf("silenced as %+v", held.Silenced)
+	}
+	c.pass(30 * time.Minute)
+	if err := k.EndSilences(ctx); err != nil {
+		t.Fatal(err)
+	}
+	c.pass(31 * time.Minute)
+	if err := k.EndSilences(ctx); err != nil {
+		t.Fatal(err)
+	}
+	got, _, _ := k.Get(ctx, "machine.ace.silent")
+	if got.Silenced != nil {
+		t.Fatalf("a silence past its end still held: %+v", got.Silenced)
+	}
+	said := settled(t, told, 3)
+	if said[1].Change != ChangeSilenced || said[2].Change != ChangeUnsilenced || said[2].Event != EventChanged {
+		t.Fatalf("said %+v", said)
+	}
+}
+
+// **Two writers never lose each other's word.** The serving controller observes while a person's
+// command silences: the write that lost the compare-and-set reads again and redoes itself.
+func TestAWriteThatLostTheRaceRedoesItself(t *testing.T) {
+	k, store, _, _ := keeper(t)
+	ctx := t.Context()
+	if _, err := k.Observe(ctx, silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	racing := &racingStore{InMemory: store, before: func() {
+		// Another process silences between this keeper's read and its write.
+		other := NewKeeper(ctx, Options{Store: store})
+		defer other.Close(context.Background())
+		if _, err := other.Silence(ctx, "machine.ace.silent", time.Hour, "jochen", "known"); err != nil {
+			t.Error(err)
+		}
+	}}
+	k.store = racing
+	got, err := k.Observe(ctx, silent("ace"))
+	if err != nil {
+		t.Fatal(err)
+	}
+	if got.Silenced == nil || got.Observations != 2 {
+		t.Fatalf("the observation overwrote the silence: %+v", got)
+	}
+}
+
+// racingStore lets another writer in once, between a read and the write after it.
+type racingStore struct {
+	*InMemory
+	before func()
+	done   bool
+}
+
+func (r *racingStore) Update(ctx context.Context, key string, value []byte, revision uint64) error {
+	if !r.done {
+		r.done = true
+		r.before()
+	}
+	return r.InMemory.Update(ctx, key, value, revision)
+}
+
+// **An observation that could not be routed is refused**, naming what it lacks.
+func TestAnObservationSaysWhatItIs(t *testing.T) {
+	k, _, _, _ := keeper(t)
+	for _, o := range []Observation{
+		{Scope: "elsewhere", ID: "x", Kind: "k", Severity: Warning, Summary: "s", Source: "S1"},
+		{Scope: ScopeMachine, ID: "x", Kind: "k", Severity: "loud", Summary: "s", Source: "S1"},
+		{Scope: ScopeMachine, ID: "x", Kind: "k", Severity: Warning, Source: "S1"},
+		{Scope: ScopeMachine, ID: "x", Kind: "k", Severity: Warning, Summary: "s"},
+	} {
+		if _, err := k.Observe(t.Context(), o); err == nil {
+			t.Errorf("observed %+v", o)
+		}
+	}
+}
+
+// **A key holds nothing the bus would refuse or read as a wildcard**, whatever the thing is called.
+func TestAKeyIsSafeForTheBus(t *testing.T) {
+	if got := Key(ScopeProvider, "keycloak.novox.my app*", "failing"); got != "provider.keycloak.novox.my_app_.failing" {
+		t.Fatalf("key %q", got)
+	}
+	if got := Key(ScopeBus, "EVENTS.>", "consumer-lost"); got != "bus.EVENTS._.consumer-lost" {
+		t.Fatalf("key %q", got)
+	}
+}
+
+// **The event's shape is a contract** (to-be 45 §2): the operator-channel's holder is written against
+// these field names. A rename here is a channel that reads nothing, so they are held still.
+func TestTheEventShapeIsTheContract(t *testing.T) {
+	k, _, told, _ := keeper(t)
+	if _, err := k.Observe(t.Context(), silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	said := settled(t, told, 1)
+	body, err := json.Marshal(said[0])
+	if err != nil {
+		t.Fatal(err)
+	}
+	var shape map[string]any
+	if err := json.Unmarshal(body, &shape); err != nil {
+		t.Fatal(err)
+	}
+	// The condition at the top level, kebab-case, beside what happened to it.
+	for _, field := range []string{"event", "at", "change", "show", "key", "kind", "subject", "severity",
+		"summary", "evidence", "source", "raised", "last-observed", "observations", "count", "resolver",
+		"silenced", "epoch"} {
+		if _, ok := shape[field]; !ok {
+			t.Errorf("the event carries no %q: %s", field, body)
+		}
+	}
+	if shape["silenced"] != nil {
+		t.Errorf("an unsilenced condition says silenced %v, not null", shape["silenced"])
+	}
+	subject, _ := shape["subject"].(map[string]any)
+	if subject["scope"] != "machine" || subject["id"] != "ace" || subject["machine"] != "ace" {
+		t.Errorf("subject %v", shape["subject"])
+	}
+	if said[0].Show != "mesh-controller.conditions key=machine.ace.silent" {
+		t.Errorf("show is %q", said[0].Show)
+	}
+}
+
+// **A transition the bus will not take is offered again**, and said lost only after TellFor.
+func TestATransitionIsOfferedAgainWhileTheBusIsAway(t *testing.T) {
+	store, told, c := NewInMemory(), &Told{Fail: errors.New("no responders")}, newClock()
+	k := NewKeeper(t.Context(), Options{Store: store, History: store, Teller: told, Now: c.now})
+	defer k.Close(context.Background())
+	if _, err := k.Observe(t.Context(), silent("ace")); err != nil {
+		t.Fatal(err)
+	}
+	time.Sleep(300 * time.Millisecond)
+	told.mu.Lock()
+	told.Fail = nil
+	told.mu.Unlock()
+	said := settled(t, told, 1)
+	if said[0].Key != "machine.ace.silent" || k.Unsaid() != 0 {
+		t.Fatalf("said %+v, unsaid %d", said, k.Unsaid())
+	}
+}

internal/inventory/plans.go

@@ -28,6 +28,10 @@ type Plan struct {
 	Tiers   [][]string             `json:"tiers"`
 	Modules map[string]*PlanModule `json:"modules"`
 	Note    string                 `json:"note,omitempty"`
+	// TierEntered is when the plan entered the tier it is at (novox/hq to-be 45 Phase 0), read and
+	// never written from here: a save measures the tier it leaves and stamps the next. What the
+	// watchdog of a plan's progress (S3) reads.
+	TierEntered time.Time `json:"tier_entered,omitempty"`
 }
 
 // PlanModule is one module's state within a plan.
@@ -129,7 +133,8 @@ func (i *Inventory) PlanByID(ctx context.Context, id string) (Plan, error) {
 
 func (i *Inventory) plans(ctx context.Context, tail string) ([]Plan, error) {
 	rows, err := i.store.Pool().Query(ctx,
-		`select id, repository, commit_hash, created, updated, state, tier, tiers, modules, note, branch
+		`select id, repository, commit_hash, created, updated, state, tier, tiers, modules, note, branch,
+		        coalesce(tier_entered, created)
 		   from release_plan `+tail)
 	if err != nil {
 		return nil, err
@@ -140,7 +145,7 @@ func (i *Inventory) plans(ctx context.Context, tail string) ([]Plan, error) {
 		var p Plan
 		var tiers, modules []byte
 		if err := rows.Scan(&p.ID, &p.Repository, &p.Commit, &p.Created, &p.Updated, &p.State,
-			&p.Tier, &tiers, &modules, &p.Note, &p.Branch); err != nil {
+			&p.Tier, &tiers, &modules, &p.Note, &p.Branch, &p.TierEntered); err != nil {
 			return nil, err
 		}
 		if err := json.Unmarshal(tiers, &p.Tiers); err != nil {

internal/inventory/standing.go

@@ -1,76 +0,0 @@
-package inventory
-
-import (
-	"context"
-	"time"
-)
-
-// ProviderStanding is a consumer a provider says it keeps failing (novox/hq ADR 0224).
-type ProviderStanding struct {
-	// Module is the provider's module, and ProviderNode the machine it runs on.
-	Module       string `json:"module"`
-	ProviderNode string `json:"provider-node"`
-	// Provision is the interface it provides, e.g. `oidc-client`.
-	Provision string `json:"provision"`
-	// Consumer is the identity the mesh derived for the consumer, ConsumerNode its machine.
-	Consumer     string `json:"consumer"`
-	ConsumerNode string `json:"consumer-node"`
-	// Class is what kind of failure: credentials-rejected, unreachable, secret-unreadable, refused.
-	Class string `json:"class"`
-	Error string `json:"error"`
-	// Since is when the unbroken run of failures began; Attempts how many it has been.
-	Since    time.Time `json:"since"`
-	Attempts int       `json:"attempts"`
-	// SaidAt is when the controller last heard it. A provider says it again every quarter of an hour
-	// while it lasts, so an old one is a provider that stopped saying anything.
-	SaidAt time.Time `json:"said-at"`
-}
-
-// SayAgainWithin is how long a failing standing stays current without being said again: twice the
-// quarter of an hour a provider repeats it at. Older, and status says the provider has gone quiet.
-const SayAgainWithin = 30 * time.Minute
-
-// Quiet says the provider has not repeated this standing for longer than it would while it lasts.
-func (s ProviderStanding) Quiet(now time.Time) bool { return now.Sub(s.SaidAt) > SayAgainWithin }
-
-// KeepStanding records a provider's newest word: failing keeps it, recovered removes it, and says
-// whether a recovery removed anything.
-func (i *Inventory) KeepStanding(ctx context.Context, failing bool, s ProviderStanding) (bool, error) {
-	if !failing {
-		tag, err := i.store.Pool().Exec(ctx,
-			`delete from provider_standing where module = $1 and provider_node = $2 and consumer = $3`,
-			s.Module, s.ProviderNode, s.Consumer)
-		return err == nil && tag.RowsAffected() > 0, err
-	}
-	_, err := i.store.Pool().Exec(ctx, `
-		insert into provider_standing
-			(module, provider_node, consumer, consumer_node, provision, class, error, since, attempts, said_at)
-		values ($1, $2, $3, $4, $5, $6, $7, $8, $9, now())
-		on conflict (module, provider_node, consumer) do update set
-			consumer_node = excluded.consumer_node, provision = excluded.provision,
-			class = excluded.class, error = excluded.error, since = excluded.since,
-			attempts = excluded.attempts, said_at = excluded.said_at`,
-		s.Module, s.ProviderNode, s.Consumer, s.ConsumerNode, s.Provision, s.Class, s.Error, s.Since, s.Attempts)
-	return false, err
-}
-
-// FailingProviders is every consumer a provider last said it keeps failing, oldest run first.
-func (i *Inventory) FailingProviders(ctx context.Context) ([]ProviderStanding, error) {
-	rows, err := i.store.Pool().Query(ctx, `
-		select module, provider_node, provision, consumer, consumer_node, class, error, since, attempts, said_at
-		from provider_standing order by since, module, consumer`)
-	if err != nil {
-		return nil, err
-	}
-	defer rows.Close()
-	var out []ProviderStanding
-	for rows.Next() {
-		var s ProviderStanding
-		if err := rows.Scan(&s.Module, &s.ProviderNode, &s.Provision, &s.Consumer, &s.ConsumerNode,
-			&s.Class, &s.Error, &s.Since, &s.Attempts, &s.SaidAt); err != nil {
-			return nil, err
-		}
-		out = append(out, s)
-	}
-	return out, rows.Err()
-}

internal/inventory/standing_test.go

@@ -1,130 +0,0 @@
-package inventory
-
-import (
-	"slices"
-	"testing"
-	"time"
-
-	"github.com/novox/mesh-controller/internal/broker"
-	"github.com/novox/mesh-controller/internal/catalogue"
-)
-
-// A provider's standing (novox/hq ADR 0224), from the grant that lets it say so to the row status
-// reads.
-
-// The broker spells the events itself because it cannot import the catalogue; the two agree.
-func TestTheBrokerAndTheCatalogueNameTheSameStandingEvents(t *testing.T) {
-	if broker.ProvisionerFailing != catalogue.ProvisionerFailing ||
-		broker.ProvisionerRecovered != catalogue.ProvisionerRecovered {
-		t.Fatal("the broker and the catalogue disagree about what a provider's standing is called")
-	}
-}
-
-// **Every provider may say it, whatever its manifest lists**: a provider whose manifest forgot the
-// events would have its announcement refused by the bus, and fail its consumers as silently as on
-// 2026-10-05 (issue 179). A module that receives no contributions provides nothing and is given
-// nothing.
-func TestEveryProviderIsGrantedItsStandingAndNothingElseIs(t *testing.T) {
-	provider := catalogue.Manifest{Module: "keycloak", Version: "1",
-		Emits: []string{"client.created"}, Receives: map[string]string{"oidc-client": "/x/mesh.json"}}
-	consumer := catalogue.Manifest{Module: "grafana", Version: "1", Emits: []string{"dashboard.saved"}}
-
-	d := declaredFor(provider, nil)
-	for _, e := range []string{"client.created", catalogue.ProvisionerFailing, catalogue.ProvisionerRecovered} {
-		if !slices.Contains(d.Emits, e) {
-			t.Fatalf("a provider is not granted %s: %v", e, d.Emits)
-		}
-	}
-	perms, err := broker.PermissionsFor(broker.Principal{Kind: broker.KindModule, Node: "anchor",
-		Module: "keycloak", Emits: d.Emits})
-	if err != nil {
-		t.Fatal(err)
-	}
-	if !slices.Contains(perms.Publish, "mesh.mod.keycloak.event.provisioner.failing") {
-		t.Fatalf("the bus would refuse a provider's standing: %v", perms.Publish)
-	}
-
-	if got := declaredFor(consumer, nil).Emits; slices.Contains(got, catalogue.ProvisionerFailing) {
-		t.Fatalf("a module that provides nothing was granted a provider's standing: %v", got)
-	}
-	// Declared by hand as well: said once.
-	provider.Emits = append(provider.Emits, catalogue.ProvisionerFailing)
-	n := 0
-	for _, e := range provider.EmitsAll() {
-		if e == catalogue.ProvisionerFailing {
-			n++
-		}
-	}
-	if n != 1 {
-		t.Fatalf("%v", provider.EmitsAll())
-	}
-}
-
-// And the controller may hear it from every provider, and only those two events.
-func TestTheControllerHearsEveryProvidersStanding(t *testing.T) {
-	perms, err := broker.PermissionsFor(broker.Principal{Kind: broker.KindController})
-	if err != nil {
-		t.Fatal(err)
-	}
-	for _, want := range []string{"mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered"} {
-		if !slices.Contains(perms.Subscribe, want) {
-			t.Fatalf("the controller may not hear %s: %v", want, perms.Subscribe)
-		}
-	}
-	if slices.Contains(perms.Subscribe, "mesh.mod.*.event.>") {
-		t.Fatal("the controller hears every event in the mesh")
-	}
-}
-
-func TestAFailingStandingIsKeptUntilItRecovers(t *testing.T) {
-	inv := ForTest(t)
-	ctx := t.Context()
-	since := time.Date(2026, 10, 5, 0, 49, 0, 0, time.UTC)
-	s := ProviderStanding{Module: "keycloak", ProviderNode: "anchor", Provision: "oidc-client",
-		Consumer: "mesh_home_grafana", ConsumerNode: "home-server", Class: "credentials-rejected",
-		Error: "401 invalid_grant", Since: since, Attempts: 60}
-	if _, err := inv.KeepStanding(ctx, true, s); err != nil {
-		t.Fatal(err)
-	}
-	// Said again: one row, the newest word.
-	s.Attempts = 31000
-	if _, err := inv.KeepStanding(ctx, true, s); err != nil {
-		t.Fatal(err)
-	}
-	got, err := inv.FailingProviders(ctx)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if len(got) != 1 || got[0].Attempts != 31000 || !got[0].Since.Equal(since) || got[0].ConsumerNode != "home-server" ||
-		got[0].Class != "credentials-rejected" || got[0].SaidAt.IsZero() {
-		t.Fatalf("%+v", got)
-	}
-	if got[0].Quiet(time.Now()) {
-		t.Fatal("a standing just said reads as quiet")
-	}
-	if !got[0].Quiet(time.Now().Add(SayAgainWithin + time.Minute)) {
-		t.Fatal("a standing not said again for longer than a provider repeats it does not read as quiet")
-	}
-
-	// The same consumer from another machine's provider is its own row.
-	other := s
-	other.ProviderNode = "laptop"
-	if _, err := inv.KeepStanding(ctx, true, other); err != nil {
-		t.Fatal(err)
-	}
-	cleared, err := inv.KeepStanding(ctx, false, s)
-	if err != nil || !cleared {
-		t.Fatalf("recovered cleared nothing: %v %v", cleared, err)
-	}
-	cleared, err = inv.KeepStanding(ctx, false, s)
-	if err != nil || cleared {
-		t.Fatalf("a recovery for nothing kept said it cleared something: %v %v", cleared, err)
-	}
-	got, err = inv.FailingProviders(ctx)
-	if err != nil {
-		t.Fatal(err)
-	}
-	if len(got) != 1 || got[0].ProviderNode != "laptop" {
-		t.Fatalf("%+v", got)
-	}
-}

internal/link/advisories.go

@@ -0,0 +1,227 @@
+package link
+
+import (
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+	"sync"
+	"time"
+
+	"github.com/nats-io/nats.go"
+
+	"github.com/novox/mesh-controller/internal/broker"
+)
+
+// What the bus says about itself, in the mesh's words (novox/hq to-be 45 §3, S9).
+//
+// **The server already says it; nothing listened.** A durable consumer that hands a message over as
+// often as it may gives up on it and says so on `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES`; one
+// deleted says so on `…CONSUMER.DELETED`. The controller's own connection is told when it falls behind
+// (a slow consumer: the client library dropped messages — issue 184, the controller deaf for 24
+// minutes) and when the bus refuses it a subject (a permissions violation — issues 183, 217, 265,
+// days each as a line in a client library's output). Each is recorded here, named in the mesh's words —
+// which consumer of whose, which subject — for the watchdog to say as a condition, as the refused
+// reply of issue 265 is said today.
+//
+// What the bus says about **other** principals' connections — a module's slow consumer, a module
+// refused a subject — the server publishes only to a system account, which the mesh's bus does not
+// have; that half is not heard yet (to-be 45 S9, recorded as deferred).
+
+// The advisory kinds, as conditions name them.
+const (
+	AdvisorySlowConsumer  = "slow-consumer"
+	AdvisoryMaxDeliveries = "max-deliveries"
+	AdvisoryRefused       = "refused"
+	AdvisoryConsumerLost  = "consumer-lost"
+)
+
+// Advisory is one thing the bus said, kept as its newest word and how often it was said.
+type Advisory struct {
+	Kind string
+	// ID names what it is about, for the condition's key: `<stream>.<consumer>`, or `controller`.
+	ID string
+	// Stream and Consumer are the consumer it is about, when it is about one.
+	Stream, Consumer string
+	// Said is the newest saying, in the mesh's words.
+	Said        string
+	First, Last time.Time
+	Count       int
+}
+
+// AdvisoryLog keeps what the bus said lately.
+type AdvisoryLog struct {
+	mu   sync.Mutex
+	seen map[string]*Advisory
+}
+
+// Advisories is this process's log.
+var Advisories = &AdvisoryLog{seen: map[string]*Advisory{}}
+
+// Heard records one advisory.
+func (l *AdvisoryLog) Heard(a Advisory, at time.Time) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	key := a.Kind + "/" + a.ID
+	if had, ok := l.seen[key]; ok {
+		had.Last, had.Said, had.Count = at, a.Said, had.Count+1
+		return
+	}
+	a.First, a.Last, a.Count = at, at, 1
+	l.seen[key] = &a
+}
+
+// Since is every advisory said at or after a moment, and forgets the older ones.
+func (l *AdvisoryLog) Since(since time.Time) []Advisory {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	var out []Advisory
+	for key, a := range l.seen {
+		if a.Last.Before(since) {
+			delete(l.seen, key)
+			continue
+		}
+		out = append(out, *a)
+	}
+	return out
+}
+
+// jsAdvisory is the part of a JetStream advisory the mesh reads.
+type jsAdvisory struct {
+	Type       string `json:"type"`
+	Stream     string `json:"stream"`
+	Consumer   string `json:"consumer"`
+	StreamSeq  uint64 `json:"stream_seq"`
+	Deliveries uint64 `json:"deliveries"`
+	Action     string `json:"action"`
+}
+
+// ReadAdvisory is one JetStream advisory as the mesh says it; false for one it does not watch.
+func ReadAdvisory(subject string, body []byte) (Advisory, bool) {
+	var a jsAdvisory
+	if err := json.Unmarshal(body, &a); err != nil || a.Stream == "" || a.Consumer == "" {
+		return Advisory{}, false
+	}
+	who := ConsumerInWords(a.Stream, a.Consumer)
+	id := a.Stream + "." + a.Consumer
+	switch {
+	case strings.HasPrefix(subject, "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES."):
+		return Advisory{Kind: AdvisoryMaxDeliveries, ID: id, Stream: a.Stream, Consumer: a.Consumer,
+			Said: fmt.Sprintf("%s handed message %d over %d times and gave up on it: it will not be delivered "+
+				"again, and what it asked for was not done", who, a.StreamSeq, a.Deliveries)}, true
+	case strings.HasPrefix(subject, "$JS.EVENT.ADVISORY.CONSUMER.DELETED."):
+		if !MeshNamed(a.Stream, a.Consumer) {
+			// A reader's own consumer, gone when it finished — every watch of a bucket and every
+			// read-back of a stream makes one, many a minute: not something the mesh defines.
+			return Advisory{}, false
+		}
+		return Advisory{Kind: AdvisoryConsumerLost, ID: id, Stream: a.Stream, Consumer: a.Consumer,
+			Said: fmt.Sprintf("%s was deleted from the bus", who)}, true
+	}
+	return Advisory{}, false
+}
+
+// MeshNamed says a consumer is one of the durable consumers the mesh defines, by its name's shape:
+// the controller's own, a machine's declaration consumer, a module's (`<node>_<module>`), a seat's
+// worker. Every other consumer is a reader's own — an ordered consumer, a bucket's watcher — named at
+// random by the client library, deleted when the read is done, and not the mesh's to say anything of.
+func MeshNamed(stream, name string) bool {
+	switch {
+	case strings.HasPrefix(stream, "KV_") || stream == broker.AssignmentsStream:
+		return false // the mesh defines no durable consumer on a bucket's stream, or the memberships'
+	case stream == "NODES":
+		return true
+	case strings.HasPrefix(stream, "SEAT_"):
+		return strings.HasSuffix(name, "_worker")
+	case name == broker.ControllerName:
+		return true
+	case stream == broker.EventsStream:
+		return strings.Contains(name, "_")
+	}
+	return false
+}
+
+// ConsumerInWords is a durable consumer as the mesh says it: whose, and for what.
+func ConsumerInWords(stream, name string) string {
+	switch {
+	case name == broker.ControllerName:
+		return "the controller's consumer on " + stream
+	case stream == "NODES":
+		return "how " + name + " hears what it should be (its declaration consumer)"
+	case strings.HasPrefix(stream, "SEAT_") && strings.HasSuffix(name, "_worker"):
+		seat := strings.ToLower(strings.ReplaceAll(strings.TrimPrefix(stream, "SEAT_"), "_", "-"))
+		return "the worker every holder of " + seat + " takes its asks from"
+	case stream == broker.EventsStream:
+		if node, module, ok := strings.Cut(name, "_"); ok {
+			return "how " + module + " on " + node + " hears what it consumes"
+		}
+	}
+	return "the consumer " + name + " on " + stream
+}
+
+// HearAdvisories subscribes what the bus says about the mesh's account, and listens for what it
+// tells this connection, until the returned function is called. Read-only: nothing is published.
+func (s *Server) HearAdvisories(logf func(string, ...any)) (func(), error) {
+	if s.js == nil {
+		return nil, errors.New("this control plane is not on the bus, so it cannot hear what the bus says")
+	}
+	conn := s.js.Conn()
+	var subs []*nats.Subscription
+	stop := func() {
+		for _, sub := range subs {
+			_ = sub.Unsubscribe()
+		}
+	}
+	for _, subject := range broker.BusAdvisories {
+		sub, err := conn.Subscribe(subject, func(m *nats.Msg) {
+			if a, ok := ReadAdvisory(m.Subject, m.Data); ok {
+				Advisories.Heard(a, time.Now())
+				logf("the bus says: %s", a.Said)
+			}
+		})
+		if err != nil {
+			stop()
+			return nil, fmt.Errorf("listening to what the bus says (%s): %w", subject, err)
+		}
+		subs = append(subs, sub)
+	}
+	WatchConnection(conn, logf)
+	return stop, nil
+}
+
+// WatchConnection records what the bus tells this connection about itself: it fell behind, or a
+// subject was refused it. Chained before whatever handler the connection had, which still runs. A
+// refused answer to a call is the call log's to say (issue 265), and is not said twice.
+func WatchConnection(conn *nats.Conn, logf func(string, ...any)) {
+	before := conn.ErrorHandler()
+	conn.SetErrorHandler(func(c *nats.Conn, sub *nats.Subscription, err error) {
+		if a, ok := connectionAdvisory(sub, err); ok {
+			Advisories.Heard(a, time.Now())
+			logf("the bus says: %s", a.Said)
+		}
+		if before != nil {
+			before(c, sub, err)
+		}
+	})
+}
+
+// connectionAdvisory is an error the bus handed the controller's connection, as an advisory.
+func connectionAdvisory(sub *nats.Subscription, err error) (Advisory, bool) {
+	switch {
+	case errors.Is(err, nats.ErrSlowConsumer):
+		subject := "a subscription"
+		if sub != nil {
+			subject = sub.Subject
+		}
+		return Advisory{Kind: AdvisorySlowConsumer, ID: "controller",
+			Said: fmt.Sprintf("the controller fell behind on %s and the client dropped messages it was sent", subject)}, true
+	case errors.Is(err, nats.ErrPermissionViolation):
+		if m := refusedPublish.FindStringSubmatch(err.Error()); m != nil && strings.HasPrefix(m[1], "_INBOX.") &&
+			!strings.HasPrefix(m[1], "_INBOX.enrol.") {
+			return Advisory{}, false // a refused answer to a call, which the call log says against its call
+		}
+		return Advisory{Kind: AdvisoryRefused, ID: "controller",
+			Said: "the bus refused the controller: " + err.Error()}, true
+	}
+	return Advisory{}, false
+}

internal/link/advisories_test.go

@@ -0,0 +1,33 @@
+package link
+
+import "testing"
+
+// **A reader's own consumer gone is not a consumer lost**: every bucket watch and stream read-back
+// makes and deletes one, many a minute, and the bus says so each time (found running the controller
+// against a real bus: five a tick).
+func TestOnlyTheMeshsOwnConsumersAreSaidLost(t *testing.T) {
+	deleted := func(stream, consumer string) bool {
+		_, ok := ReadAdvisory("$JS.EVENT.ADVISORY.CONSUMER.DELETED."+stream+"."+consumer,
+			[]byte(`{"type":"io.nats.jetstream.advisory.v1.consumer_action","stream":"`+stream+`","consumer":"`+consumer+`","action":"delete"}`))
+		return ok
+	}
+	for _, c := range [][2]string{{"EVENTS", "controller"}, {"CONTROL", "controller"}, {"NODES", "anchor"},
+		{"EVENTS", "anchor_shop"}, {"SEAT_NODE_BUILD_AGENT", "SEAT_NODE_BUILD_AGENT_worker"}} {
+		if !deleted(c[0], c[1]) {
+			t.Errorf("%s on %s deleted is not said", c[1], c[0])
+		}
+	}
+	for _, c := range [][2]string{{"KV_mesh-controller_conditions", "381UWW5Y"}, {"EVENTS", "E7vVYoe6"},
+		{"SEAT_NODE_BUILD_AGENT", "Rcnt6jla"}, {"ASSIGNMENTS", "x"}} {
+		if deleted(c[0], c[1]) {
+			t.Errorf("a reader's own consumer %s on %s is said lost", c[1], c[0])
+		}
+	}
+	a, ok := ReadAdvisory("$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.EVENTS.anchor_shop",
+		[]byte(`{"stream":"EVENTS","consumer":"anchor_shop","stream_seq":7,"deliveries":5}`))
+	if !ok || a.Kind != AdvisoryMaxDeliveries || a.ID != "EVENTS.anchor_shop" ||
+		a.Said != "how shop on anchor hears what it consumes handed message 7 over 5 times and gave up on it: "+
+			"it will not be delivered again, and what it asked for was not done" {
+		t.Fatalf("%+v", a)
+	}
+}

internal/link/bus.go

@@ -71,6 +71,10 @@ const (
 	// next heartbeat, and a stream of them is the mesh's least valuable message competing for
 	// retention with its most valuable (design 25 §3).
 	AliveSubjects = "mesh.control.*.alive"
+
+	// ToolsAliveSubjects is every machine's node tools saying they are there (novox/hq to-be 45 §3,
+	// S11): core NATS like the host's, for the same reason.
+	ToolsAliveSubjects = "mesh.control.*.tools-alive"
 )
 
 // ReportSubject is where one node says what it did. On the CONTROL stream, because it is the
@@ -80,6 +84,9 @@ func ReportSubject(node string) string { return "mesh.control." + node + ".repor
 // AliveSubject is one node's heartbeat.
 func AliveSubject(node string) string { return "mesh.control." + node + ".alive" }
 
+// ToolsAliveSubject is one machine's node tools' heartbeat.
+func ToolsAliveSubject(node string) string { return "mesh.control." + node + ".tools-alive" }
+
 // EventSubject is where a module's event lands. Derived from the emitter, never taken from the
 // caller: a source that could differ from the subject is an envelope that can lie about its
 // origin, and on NATS the account's permissions make the subject the authority (design 29 §2).

internal/link/calls.go

@@ -271,6 +271,23 @@ func (l *CallLog) finish(c *Call, answer []byte, failed, answeredAlready bool) {
 // kept durably, every other the bus holds — a call a controller before this one served included.
 // Memory wins for a call in both, being the newer word on it. A bus that cannot be read is said in
 // the error beside what memory holds, never answered as no calls.
+// Running is every call this process is serving that has not finished, oldest first, without
+// answers: what the watchdog of a call's bound (novox/hq to-be 45 S7) reads. This process's own,
+// because a call another controller left running is said abandoned when this one starts.
+func (l *CallLog) Running() []Call {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	var out []Call
+	for _, c := range l.calls {
+		if c.State == CallRunning {
+			running := *c
+			running.Answer = nil
+			out = append(out, running)
+		}
+	}
+	return out
+}
+
 func (l *CallLog) Recent() ([]Call, error) {
 	l.mu.Lock()
 	out := make([]Call, 0, len(l.calls))

internal/link/protocol.go

@@ -138,6 +138,20 @@ type Signed struct {
 // is current.
 type Alive struct {
 	Node string `json:"node"`
+	// IntervalSeconds is how often the node says it is there (novox/hq to-be 45 §3, S1): the
+	// watchdog's bound is three of them. Zero from a host older than that, which is read as the
+	// interval hosts have always used.
+	IntervalSeconds int `json:"interval_seconds,omitempty"`
+}
+
+// ToolsAlive is a machine's node tools saying they are there (novox/hq to-be 45 §3, S11): the
+// runtime every module's tools and every held seat's verbs are served by. Its own word, apart from the
+// host's, because a host heard and a runtime gone is a machine nobody can ask anything.
+type ToolsAlive struct {
+	Node            string `json:"node"`
+	IntervalSeconds int    `json:"interval_seconds,omitempty"`
+	// Version is the runtime's build, as it says it.
+	Version string `json:"version,omitempty"`
 }
 
 // Report is what a node states after applying. It states; the owning context writes.

internal/link/receive.go

@@ -25,11 +25,13 @@ import (
 // on the wire: the wire is the transport's business, and a kind that travelled would be a third
 // name for the same thing.
 const (
-	KindEnrolment   = "enrolment"
-	KindReport      = "report"
-	KindHeartbeat   = "heartbeat"
-	KindBuilt       = "built"
-	KindModuleMoved = "module-moved"
+	KindEnrolment = "enrolment"
+	KindReport    = "report"
+	KindHeartbeat = "heartbeat"
+	// KindToolsHeartbeat is a machine's node tools saying they are there (novox/hq to-be 45 S11).
+	KindToolsHeartbeat = "tools-heartbeat"
+	KindBuilt          = "built"
+	KindModuleMoved    = "module-moved"
 	// KindSourceMoved is the forge announcing a merge: a source moved, and what it produces is
 	// built without anybody telling the mesh (novox/hq 04-ISSUES/131).
 	KindSourceMoved = "source-moved"

internal/link/receive_nats.go

@@ -89,6 +89,10 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
 		return nil // stopped while standing by
 	}
 	defer func() { _ = said.Unsubscribe() }()
+	// This controller is the one acting now: its watchdogs and self-check may say what they see
+	// (novox/hq to-be 45 §3). One standing by hears nothing, and would call every machine silent.
+	holding.Store(true)
+	defer holding.Store(false)
 
 	// Heartbeats, on core NATS and off any stream (design 25 §3). Their own subscription because
 	// they are their own guarantee: a lost one is the next one.
@@ -98,6 +102,12 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
 		return fmt.Errorf("subscribing to heartbeats: %w", err)
 	}
 	defer func() { _ = alive.Unsubscribe() }()
+	// And each machine's node tools, on the same channel: as cheap, and lost the same way.
+	toolsAlive, err := conn.ChanSubscribe(ToolsAliveSubjects, beats)
+	if err != nil {
+		return fmt.Errorf("subscribing to the node tools' heartbeats: %w", err)
+	}
+	defer func() { _ = toolsAlive.Unsubscribe() }()
 
 	// The events the controller follows, when something is listening for them.
 	var events chan *nats.Msg
@@ -159,6 +169,8 @@ func (n *natsInbound) deliver(ctx context.Context, act func(context.Context, Con
 	}
 	m := &natsControl{kind: kind, msg: msg, on: n}
 	if streamed {
+		// The event loop took one: what S4 watches (novox/hq to-be 45 §3).
+		Loop.Took(time.Now())
 		// A message with no metadata is not from a stream, whatever it was delivered on, and the
 		// window has nothing to hold it by. Said by leaving the sequence at zero.
 		if meta, err := msg.Metadata(); err == nil {
@@ -225,6 +237,8 @@ func kindOfSubject(subject string) (string, bool) {
 			return KindReport, true
 		case "alive":
 			return KindHeartbeat, true
+		case "tools-alive":
+			return KindToolsHeartbeat, true
 		}
 	}
 	switch subject {

internal/link/serve.go

@@ -170,6 +170,8 @@ func (s *Server) act(ctx context.Context, m Control) {
 		s.reported(ctx, m)
 	case KindHeartbeat:
 		s.heartbeat(m)
+	case KindToolsHeartbeat:
+		s.toolsHeartbeat(m)
 	case KindBuilt:
 		s.wasBuilt(ctx, m)
 	case KindModuleMoved:
@@ -268,6 +270,8 @@ func (s *Server) heartbeat(m Control) {
 		_ = m.Drop()
 		return
 	}
+	// The interval it says, for the watchdog's bound (novox/hq to-be 45 S1).
+	HostBeats.Heard(alive.Node, time.Now(), time.Duration(alive.IntervalSeconds)*time.Second, "")
 	if s.listener != nil {
 		if _, err := s.listener.Heard(context.Background(), Report{Node: alive.Node}); err != nil {
 			s.log.Printf("could not record that %s is here: %v", alive.Node, err)
@@ -276,6 +280,22 @@ func (s *Server) heartbeat(m Control) {
 	_ = m.Took()
 }
 
+// toolsHeartbeat records that a machine's node tools were heard from (novox/hq to-be 45 S11), and
+// nothing else: in this process's memory, for the watchdog — the next one is a minute away.
+func (s *Server) toolsHeartbeat(m Control) {
+	var alive ToolsAlive
+	if err := json.Unmarshal(m.Body(), &alive); err != nil || alive.Node == "" {
+		_ = m.Drop()
+		return
+	}
+	// The machine is the one in the subject the bus let the runtime publish on, never the body's.
+	if node, ok := nodeOfToolsAlive(m.Subject()); ok {
+		alive.Node = node
+	}
+	ToolsBeats.Heard(alive.Node, time.Now(), time.Duration(alive.IntervalSeconds)*time.Second, alive.Version)
+	_ = m.Took()
+}
+
 // reported records what a node says it did.
 //
 // A node states; nothing here writes anything the node claimed about itself beyond that it was

internal/link/watched.go

@@ -0,0 +1,227 @@
+package link
+
+import (
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"strings"
+	"sync"
+	"sync/atomic"
+	"time"
+
+	"github.com/nats-io/nats.go"
+
+	"github.com/novox/mesh-controller/internal/broker"
+)
+
+// What the serving controller hears that its watchdogs read (novox/hq to-be 45 §3).
+//
+// **In memory, on purpose.** A heartbeat is the least valuable message the mesh sends — the next one
+// is a minute away — and what a watchdog needs of it is the newest moment and the interval it said.
+// A machine's last word is kept in the store as well (`last_seen`), which is what S1 reads; these are
+// what the store does not keep: the interval, the node tools' word, and when the event loop last took
+// a message.
+
+// Beat is one emitter's newest heartbeat.
+type Beat struct {
+	At time.Time
+	// Every is the interval it said; zero when it said none.
+	Every   time.Duration
+	Version string
+}
+
+// Beats is the newest heartbeat of each machine, for one kind of emitter.
+type Beats struct {
+	mu      sync.Mutex
+	started time.Time
+	heard   map[string]Beat
+}
+
+// NewBeats is an empty record, started now.
+func NewBeats() *Beats { return &Beats{started: time.Now(), heard: map[string]Beat{}} }
+
+// HostBeats are the node-engines' heartbeats (S1); ToolsBeats the node tools' (S11).
+var (
+	HostBeats  = NewBeats()
+	ToolsBeats = NewBeats()
+)
+
+// Heard records one heartbeat.
+func (b *Beats) Heard(node string, at time.Time, every time.Duration, version string) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	b.heard[node] = Beat{At: at, Every: every, Version: version}
+}
+
+// Of is one machine's newest heartbeat; false when none was heard since this process started.
+func (b *Beats) Of(node string) (Beat, bool) {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	beat, ok := b.heard[node]
+	return beat, ok
+}
+
+// Started is when this record began: a machine not heard since is silent since then at the most.
+func (b *Beats) Started() time.Time {
+	b.mu.Lock()
+	defer b.mu.Unlock()
+	return b.started
+}
+
+// nodeOfToolsAlive is the machine a node tools' heartbeat names in its subject.
+func nodeOfToolsAlive(subject string) (string, bool) {
+	rest, ok := strings.CutPrefix(subject, "mesh.control.")
+	if !ok {
+		return "", false
+	}
+	node, kind, ok := strings.Cut(rest, ".")
+	if !ok || kind != "tools-alive" || node == "" {
+		return "", false
+	}
+	return node, true
+}
+
+// LoopActivity is when the controller's event loop last took a message from a stream (S4).
+type LoopActivity struct {
+	mu   sync.Mutex
+	took time.Time
+}
+
+// Loop is this process's event loop.
+var Loop = &LoopActivity{}
+
+// Took records that the loop was handed a message.
+func (l *LoopActivity) Took(at time.Time) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	l.took = at
+}
+
+// Last is when the loop last took one; zero when it has not since this process started.
+func (l *LoopActivity) Last() time.Time {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	return l.took
+}
+
+// PowerState is what a machine last said about its power (novox/hq ADR 0211): `sleeping` or
+// `shutting-down` until it says `booted` or `woke`.
+type PowerState struct {
+	State string
+	At    time.Time
+}
+
+// PowerModule is the module whose events say a machine's power (ADR 0211), and PowerEvents its
+// events that say whether the machine is about to be away or is back.
+const PowerModule = "power"
+
+var PowerEvents = map[string]bool{"sleeping": true, "shutting-down": true, "booted": true, "woke": true}
+
+// PowerStates is each machine's newest word about its power since a moment, read back from the
+// events stream on a consumer of its own that acknowledges nothing (as AnnouncedMerges reads). The
+// machine is the one the runtime stamped on the event (`x-node`); an event without one names none.
+func (s *Server) PowerStates(ctx context.Context, since time.Time) (map[string]PowerState, error) {
+	if s.js == nil {
+		return nil, errors.New("this control plane is not on the bus, so it cannot read what machines said of their power")
+	}
+	sub, err := s.js.Context().SubscribeSync(EventSubject(PowerModule, ">"), nats.OrderedConsumer(), nats.StartTime(since))
+	if err != nil {
+		return nil, fmt.Errorf("reading what machines said of their power: %w", err)
+	}
+	defer func() { _ = sub.Unsubscribe() }()
+	out := map[string]PowerState{}
+	for {
+		wait, cancel := context.WithTimeout(ctx, readQuiet)
+		msg, err := sub.NextMsgWithContext(wait)
+		cancel()
+		if err != nil {
+			if ctx.Err() != nil {
+				return nil, ctx.Err()
+			}
+			break
+		}
+		meta, err := msg.Metadata()
+		if err != nil {
+			continue
+		}
+		state := strings.TrimPrefix(msg.Subject, "mesh.mod."+PowerModule+".event.")
+		node := msg.Header.Get("x-node")
+		if PowerEvents[state] && node != "" {
+			at := meta.Timestamp
+			var body struct {
+				At time.Time `json:"at"`
+			}
+			if json.Unmarshal(msg.Data, &body) == nil && !body.At.IsZero() {
+				at = body.At
+			}
+			if before, ok := out[node]; !ok || !at.Before(before.At) {
+				out[node] = PowerState{State: state, At: at}
+			}
+		}
+		if meta.NumPending == 0 {
+			break
+		}
+	}
+	return out, nil
+}
+
+// Away says whether a power state is a machine that said it would be away.
+func (p PowerState) Away() bool { return p.State == "sleeping" || p.State == "shutting-down" }
+
+// StaleRefusal is the node-engine's words for a declaration it refused because it is older than the
+// one it holds (mesh-host `refuseOlder`, novox/hq issue 107): the receiver's refusal S13 counts.
+const StaleRefusal = "is older than what the mesh last said to this node"
+
+// IsStaleRefusal says a report's refusal is the node-engine refusing a declaration older than it holds.
+func IsStaleRefusal(refused string) bool { return strings.Contains(refused, StaleRefusal) }
+
+// RefusalCount keeps when each machine refused a stale declaration, for S13 (novox/hq to-be 45 §3):
+// more than five from one writer in five minutes is a writer sending what it has moved past.
+type RefusalCount struct {
+	mu  sync.Mutex
+	per map[string][]time.Time
+}
+
+// StaleRefusals is this process's count.
+var StaleRefusals = &RefusalCount{per: map[string][]time.Time{}}
+
+// Refused records one refusal by a machine.
+func (r *RefusalCount) Refused(node string, at time.Time) {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	r.per[node] = append(r.per[node], at)
+}
+
+// Within is how many refusals each machine made since a moment; older ones are forgotten.
+func (r *RefusalCount) Within(since time.Time) map[string]int {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	out := map[string]int{}
+	for node, times := range r.per {
+		var kept []time.Time
+		for _, t := range times {
+			if !t.Before(since) {
+				kept = append(kept, t)
+			}
+		}
+		if len(kept) == 0 {
+			delete(r.per, node)
+			continue
+		}
+		r.per[node] = kept
+		out[node] = len(kept)
+	}
+	return out
+}
+
+// holding is whether this process holds the controller's consumer of what nodes say: the controller
+// acting, not one standing by for another (issue 213). Until the lease (to-be 45 §6) it is how a
+// watchdog knows it is the one that hears.
+var holding atomic.Bool
+
+// Holding says this process is the controller acting now.
+func Holding() bool { return holding.Load() }
+
+// JetStream is the connection the server consumes on.
+func (s *Server) JetStream() *broker.JetStream { return s.js }