Keep what each machine says of what it runs, raise it, and gate on it (hq ADR 0240, to-be 48 Phase A)

The gate judged a module by what the mesh saw from outside, so a container that
crash-looped after it applied passed it. Each machine's node-engine now states
the health of every long-running resource it runs; the controller keeps the
newest statement per machine, raises module.<module>.<machine>.unhealthy on the
second statement in a row, clears it on the first that does not say it, and the
gate passes a module only when every long-running resource of it is stated
healthy since the send. An engine that states nothing is judged as before.
This commit is contained in:
jochen
2026-10-07 02:28:16 +02:00
parent 5d3e52219b
commit 1cc6a2d759
22 changed files with 997 additions and 5 deletions
+49
View File
@@ -262,6 +262,13 @@ type Report struct {
// `witness` and `not-reversible` are sent only to a machine whose report carries it.
Witness int `json:"witness,omitempty"`
// Health is the machine's word on every long-running resource it runs for a module (novox/hq ADR
// 0240, to-be 48 §4; mesh-host's internal/liveness): its state, since when, its failing streak and
// the restarts its node-engine counted. **Absent from an engine older than the judging**, which is
// read as "not known" — never as healthy, never as a reason to raise anything; present with no
// resources from a machine that runs nothing long-lived.
Health *Health `json:"health,omitempty"`
// Rekey is a node taking a found tunnel's key as its overlay key after enrolment (novox/hq
// ADR 0105). A report carrying one is not an account of the machine: it moves the node's
// overlay key and tunnel and nothing else.
@@ -404,3 +411,45 @@ func EnrolProof(secret string, public []byte, overlay, sealing, serving string)
return []byte("novox-mesh-enrol\x00" + secret + "\x00" + base64.StdEncoding.EncodeToString(public) +
"\x00" + overlay + "\x00" + sealing + "\x00" + serving)
}
// LivenessContract is the version of the health statement this controller reads (ADR 0240 Phase A).
const LivenessContract = 1
// Health is one statement of a machine's long-running resources (to-be 48 §4): in every report, as the
// event HealthSubject between reports on each change, and again every minute while one is not healthy.
// The node-engine's own (mesh-host internal/link Health); a test on each side holds the field names.
type Health struct {
Contract int `json:"contract"`
// At is when the engine looked, on the machine's clock: the order of its statements.
At time.Time `json:"at"`
Resources []ResourceHealth `json:"resources"`
}
// The states a resource is said in (ADR 0240 §4).
const (
StateHealthy = "healthy"
StateUnhealthy = "unhealthy"
StateStarting = "starting"
StateHeld = "held"
StateUnknown = "unknown"
)
// ResourceHealth is one long-running resource's state.
type ResourceHealth struct {
Module string `json:"module"`
Resource string `json:"resource"`
Kind string `json:"kind"`
Target string `json:"target"`
State string `json:"state"`
Reason string `json:"reason,omitempty"`
Since time.Time `json:"since"`
Streak int `json:"streak,omitempty"`
Restarts int `json:"restarts,omitempty"`
}
// HealthSaid is the health event's body: the machine and its statement. The machine is read from the
// subject the bus let it publish on, never from here.
type HealthSaid struct {
Node string `json:"node"`
Health Health `json:"health"`
}