Say when the mesh is wrong: conditions, watchdogs, the bus's advisories, doctor (hq to-be 45 Phase 1)

Every one of the 48 core failures of research 031 was found by a person
looking; the mesh's answers carried the fact for whoever asked and told
nobody.

- The condition store (to-be 45 §2): mesh-controller_conditions, one key
  per open condition, written by compare-and-set so a person's silence
  and the watchdogs never lose each other's word; every transition kept
  ninety days in mesh-controller_condition-history and said as the
  seat's events condition-raised / condition-changed / condition-cleared
  (the condition at the top level, with event, at, change, why, show),
  offered again while the bus is away. Raised and cleared by observation
  only; a clearing reopened within ten minutes is the same condition with
  its count up, its silence kept. Verbs: conditions, conditions show,
  conditions silence (a hand act, at most a week), conditions history.
- ADR 0224's provider standing is the first kind, provider-failing, held
  by the provider's events; the provider_standing table is no longer read
  or written (left in place: dropping it is the operator's word).
- status leads with the open conditions, urgent first, and says all well
  only with none open; conditions it cannot read are said and not well.
- The signals table compiled in, one watchdog loop over it every 30s: S1
  heartbeat (3 intervals, asleep machines excepted, control node urgent
  after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf,
  S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9
  advisories, S10 self-check silent, S11 node tools silent, S13 stale
  refusals; S12, S14, S15 deferred with their reasons. A row that cannot
  see raises probe-failed and clears nothing. A test generated from the
  table suppresses each signal inside and past its bound.
- The bus's advisories (maximum deliveries, a mesh consumer deleted) and
  the controller's own slow consumer and refused subjects, said in the
  mesh's words.
- doctor: the probe registry D1-D10 (D5 deferred) and DW, every five
  minutes, each in thirty seconds; a probe that cannot run is never a
  pass. D1 validates with mesh-host's own validator. Every run ends with
  the doctor-heartbeat event mesh-watcher listens for.
- The controller is granted its new buckets, events, the two advisories
  and $SRV.INFO; the node tools their tools-alive heartbeat. The streams
  and consumers the controller asserts and the ones D6/D7 expect are one
  derivation.

internal/broker/controller_buckets.go

@@ -19,10 +19,18 @@ import (
 // like the streams, so a bus raised from nothing has them before the first call is served.
 
 // CallsBucket keeps every call of the mesh's own verbs and what came of it; HandActsBucket every act
-// a person did by hand, with why.
+// a person did by hand, with why; ConditionsBucket every condition open now (to-be 45 §2), one key
+// each, and ConditionHistoryBucket every transition of one — raised, changed, silenced, cleared —
+// for ninety days.
+//
+// **The history is a bucket of its own** because its keys expire and an open condition's must not:
+// a bucket has one age for every key, and a condition open longer than the history is kept would
+// otherwise vanish from the store while still true.
 var (
-	CallsBucket    = BucketName(ControllerSeat, "calls")
-	HandActsBucket = BucketName(ControllerSeat, "hand-acts")
+	CallsBucket            = BucketName(ControllerSeat, "calls")
+	HandActsBucket         = BucketName(ControllerSeat, "hand-acts")
+	ConditionsBucket       = BucketName(ControllerSeat, "conditions")
+	ConditionHistoryBucket = BucketName(ControllerSeat, "condition-history")
 )
 
 // The bounds to-be 45 §6 sets for calls: the last thousand, or fourteen days, whichever is fewer.
@@ -37,11 +45,19 @@ const (
 	// HandActsKeptFor is as long as a condition's history (to-be 45 §2): an act by hand is read
 	// back beside what it addressed.
 	HandActsKeptFor = 90 * 24 * time.Hour
+	// ConditionHistoryKeptFor is how long a condition's transitions are kept (to-be 45 §2).
+	ConditionHistoryKeptFor = 90 * 24 * time.Hour
 )
 
 // IsControllerBucket says a bucket is the controller's own, not a module's state nothing declares.
 func IsControllerBucket(bucket string) bool {
-	return bucket == CallsBucket || bucket == HandActsBucket
+	return bucket == CallsBucket || bucket == HandActsBucket || bucket == ConditionsBucket ||
+		bucket == ConditionHistoryBucket
+}
+
+// ControllerBuckets are the controller's own buckets, in the order they are asserted.
+func ControllerBuckets() []string {
+	return []string{CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket}
 }
 
 // ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
@@ -98,5 +114,30 @@ func (j *JetStream) EnsureControllerBuckets() error {
 	}); err != nil {
 		return fmt.Errorf("asserting bucket %s: %w", HandActsBucket, err)
 	}
+	// **No age on the open conditions.** A condition is removed when observation clears it and at no
+	// other moment: one that expired would be a fault the store forgot while it was still true.
+	if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
+		Bucket: ConditionsBucket,
+		Description: "every condition open now, one key each (novox/hq to-be 45 §2): written by the " +
+			"controller alone, raised and cleared by observation, read through `conditions`",
+		History:      1,
+		MaxValueSize: 64 << 10,
+		MaxBytes:     64 << 20,
+		Storage:      jetstream.FileStorage,
+	}); err != nil {
+		return fmt.Errorf("asserting bucket %s: %w", ConditionsBucket, err)
+	}
+	if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
+		Bucket: ConditionHistoryBucket,
+		Description: "every transition of a condition — raised, changed, silenced, cleared — kept ninety " +
+			"days (novox/hq to-be 45 §2): written by the controller alone, read through `conditions history`",
+		History:      1,
+		TTL:          ConditionHistoryKeptFor,
+		MaxValueSize: 64 << 10,
+		MaxBytes:     256 << 20,
+		Storage:      jetstream.FileStorage,
+	}); err != nil {
+		return fmt.Errorf("asserting bucket %s: %w", ConditionHistoryBucket, err)
+	}
 	return nil
 }

internal/broker/controller_buckets_test.go

@@ -2,6 +2,7 @@ package broker
 
 import (
 	"slices"
+	"strings"
 	"testing"
 )
 
@@ -29,3 +30,32 @@ func TestTheControllerMayWriteEveryBucketItWrites(t *testing.T) {
 		t.Error("the controller may write any bucket, a module's state included")
 	}
 }
+
+// **A machine's node tools may say they are there, as that machine and no other** (novox/hq to-be 45
+// S11), and the controller may hear the bus's advisories and ask who answers — read-only, named.
+func TestTheWatchedSignalsMayBeSaidAndHeard(t *testing.T) {
+	tools, err := PermissionsFor(Principal{Kind: KindNodeTools, Node: "anchor", Module: RuntimeModule, PasswordHash: "x"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	if !slices.Contains(tools.Publish, "mesh.control.anchor.tools-alive") {
+		t.Error("the node tools may not say they are there")
+	}
+	for _, s := range tools.Publish {
+		if strings.Contains(s, "tools-alive") && s != "mesh.control.anchor.tools-alive" {
+			t.Errorf("the node tools may say %s", s)
+		}
+	}
+	controller, err := PermissionsFor(Principal{Kind: KindController, PasswordHash: "x"})
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, s := range BusAdvisories {
+		if !slices.Contains(controller.Subscribe, s) {
+			t.Errorf("the controller may not hear %s", s)
+		}
+	}
+	if !slices.Contains(controller.Publish, "$SRV.INFO") || slices.Contains(controller.Subscribe, "$JS.EVENT.>") {
+		t.Error("the controller may not ask who answers, or hears every API call")
+	}
+}

internal/broker/nats.go

@@ -266,6 +266,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
 		sub = append(sub, "mesh.seat."+ControllerSeat+".tool.>")
 		// And says so (novox/hq ADR 0197): it answers discovery for the seat it serves.
 		sub = append(sub, announcing(ControllerSeat)...)
+		// And asks who answers (novox/hq to-be 45 §4, D3): the self-check finds every seat's holder by
+		// the same discovery the console reads. The question only; the answers come to its own inbox.
+		pub = append(pub, "$SRV.INFO")
 
 		// The events it reacts to, and its ack subject on the stream they arrive from
 		// (streams.go). **Each named, not a pattern**: `mesh.mod.*.event.>` would make the
@@ -297,7 +300,14 @@ func PermissionsFor(p Principal) (Permissions, error) {
 		// **And its own buckets** (novox/hq to-be 45 §1): the calls it served and the acts done by
 		// hand, which it alone writes. A put is a publish to the bucket's subject, which `$JS.API.>`
 		// does not cover; each bucket named, not `$KV.>`, which would let it write any module's state.
-		pub = append(pub, "$KV."+CallsBucket+".>", "$KV."+HandActsBucket+".>")
+		for _, bucket := range ControllerBuckets() {
+			pub = append(pub, "$KV."+bucket+".>")
+		}
+		// **And what the bus says about itself, read-only** (novox/hq to-be 45 §3, S9): a durable
+		// consumer that gave up on a message, or one that was deleted. The server already publishes
+		// both in the mesh's own account; the controller says each as a condition in the mesh's words.
+		// Named, not `$JS.EVENT.>`: the other advisories are every API call the mesh makes.
+		sub = append(sub, BusAdvisories...)
 
 	case KindPerson:
 		// Tools, and nothing else. Every subject a person may publish is a tool call; a person
@@ -510,6 +520,9 @@ func PermissionsFor(p Principal) (Permissions, error) {
 		// that varies is the module, so the pattern is the machine's own assignments.
 		sub = append(sub, "mesh.assignment."+p.Node+".*")
 		pub = append(pub, "$JS.API.DIRECT.GET."+AssignmentsStream+".mesh.assignment."+p.Node+".*")
+		// And that it is there (novox/hq to-be 45 §3, S11): its own heartbeat, under its machine's
+		// name and no other's, on core NATS like the host's.
+		pub = append(pub, "mesh.control."+p.Node+".tools-alive")
 		// And every tool on the mesh (ADR 0175, decision 5): any node may call any tool on any
 		// node, as the console already could — the runtime is the console's serving mode.
 		invoked, err := invokedSubjects([]string{"*"})

internal/broker/states_agreement_test.go

@@ -6,6 +6,7 @@ import (
 
 	"github.com/novox/mesh-controller/internal/broker"
 	"github.com/novox/mesh-controller/internal/catalogue"
+	"github.com/novox/mesh-controller/internal/conditions"
 	"github.com/novox/mesh-controller/internal/link"
 )
 
@@ -20,12 +21,15 @@ func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
 		t.Fatalf("the grant is written for the %q seat and the mesh states its facts under %q",
 			broker.ControllerSeat, link.MeshControllerSeat)
 	}
-	for _, event := range []string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore} {
+	// And what is wrong, as it changes, and the self-check's heartbeat (novox/hq to-be 45 §2, §4).
+	states := append([]string{link.KeyApplied, link.KeyRefused, link.KeyBuiltBefore}, conditions.Events...)
+	states = append(states, conditions.HeartbeatEvent)
+	for _, event := range states {
 		if !slices.Contains(broker.ControllerStates, event) {
 			t.Errorf("the mesh states %q and its account may not publish it", event)
 		}
 	}
-	if len(broker.ControllerStates) != 3 {
+	if len(broker.ControllerStates) != len(states) {
 		t.Errorf("the grant permits %v, which is more than the mesh states", broker.ControllerStates)
 	}
 	// **And the seat says it.** A seat carries the protocol of its role (novox/hq ADR 0129), so the

internal/broker/streams.go

@@ -201,7 +201,23 @@ const ControllerName = "controller"
 // something to say.
 const ControllerSeat = "mesh-controller"
 
-var ControllerStates = []string{"applied", "refused", "built-before"}
+var ControllerStates = []string{"applied", "refused", "built-before",
+	// What is wrong, said as it changes (novox/hq to-be 45 §2): a condition raised, changed in
+	// severity, resolver or silence, and cleared. The operator-channel's holder and any other surface
+	// consume them; the controller tells nobody itself.
+	"condition-raised", "condition-changed", "condition-cleared",
+	// And the self-check's heartbeat, at the end of every run (to-be 45 §4, S10): watched from a
+	// machine that is not the control node, so the controller going quiet is itself said.
+	"doctor-heartbeat"}
+
+// BusAdvisories are what the bus server says about the mesh's own account that the controller
+// reads (novox/hq to-be 45 §3, S9): a durable consumer that handed a message over as often as it
+// may and gave up on it, and one that was deleted. Read-only: an advisory is the server's to
+// publish, and the controller's subscription changes nothing on the bus.
+var BusAdvisories = []string{
+	"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>",
+	"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>",
+}
 
 // ControllerFollows are the events the controller reacts to: the catalogue saying a module's
 // current version moved, and a catalogue that has just started saying it may have missed builds.

internal/broker/testdata/composed.conf

@@ -24,8 +24,8 @@ accounts {
     jetstream: enabled
     users = [
       { user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
-        publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_hand-acts.>", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.refused", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>"] }
-        subscribe: { allow: ["$JS.API.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
+        publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.control.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.refused", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>"] }
+        subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.gitea.event.pull.merged", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
         allow_responses: { max: 1, ttl: "1m" }
       } }
       { user: "enrol.one", password: "$2a$11$eeeeeeeeeeeeeeeeeeeeee", permissions: {