Keep what each machine says of what it runs, raise it, and gate on it (hq ADR 0240, to-be 48 Phase A)
The gate judged a module by what the mesh saw from outside, so a container that crash-looped after it applied passed it. Each machine's node-engine now states the health of every long-running resource it runs; the controller keeps the newest statement per machine, raises module.<module>.<machine>.unhealthy on the second statement in a row, clears it on the first that does not say it, and the gate passes a module only when every long-running resource of it is stated healthy since the send. An engine that states nothing is judged as before.
This commit is contained in:
@@ -262,6 +262,13 @@ type Report struct {
|
||||
// `witness` and `not-reversible` are sent only to a machine whose report carries it.
|
||||
Witness int `json:"witness,omitempty"`
|
||||
|
||||
// Health is the machine's word on every long-running resource it runs for a module (novox/hq ADR
|
||||
// 0240, to-be 48 §4; mesh-host's internal/liveness): its state, since when, its failing streak and
|
||||
// the restarts its node-engine counted. **Absent from an engine older than the judging**, which is
|
||||
// read as "not known" — never as healthy, never as a reason to raise anything; present with no
|
||||
// resources from a machine that runs nothing long-lived.
|
||||
Health *Health `json:"health,omitempty"`
|
||||
|
||||
// Rekey is a node taking a found tunnel's key as its overlay key after enrolment (novox/hq
|
||||
// ADR 0105). A report carrying one is not an account of the machine: it moves the node's
|
||||
// overlay key and tunnel and nothing else.
|
||||
@@ -404,3 +411,45 @@ func EnrolProof(secret string, public []byte, overlay, sealing, serving string)
|
||||
return []byte("novox-mesh-enrol\x00" + secret + "\x00" + base64.StdEncoding.EncodeToString(public) +
|
||||
"\x00" + overlay + "\x00" + sealing + "\x00" + serving)
|
||||
}
|
||||
|
||||
// LivenessContract is the version of the health statement this controller reads (ADR 0240 Phase A).
|
||||
const LivenessContract = 1
|
||||
|
||||
// Health is one statement of a machine's long-running resources (to-be 48 §4): in every report, as the
|
||||
// event HealthSubject between reports on each change, and again every minute while one is not healthy.
|
||||
// The node-engine's own (mesh-host internal/link Health); a test on each side holds the field names.
|
||||
type Health struct {
|
||||
Contract int `json:"contract"`
|
||||
// At is when the engine looked, on the machine's clock: the order of its statements.
|
||||
At time.Time `json:"at"`
|
||||
Resources []ResourceHealth `json:"resources"`
|
||||
}
|
||||
|
||||
// The states a resource is said in (ADR 0240 §4).
|
||||
const (
|
||||
StateHealthy = "healthy"
|
||||
StateUnhealthy = "unhealthy"
|
||||
StateStarting = "starting"
|
||||
StateHeld = "held"
|
||||
StateUnknown = "unknown"
|
||||
)
|
||||
|
||||
// ResourceHealth is one long-running resource's state.
|
||||
type ResourceHealth struct {
|
||||
Module string `json:"module"`
|
||||
Resource string `json:"resource"`
|
||||
Kind string `json:"kind"`
|
||||
Target string `json:"target"`
|
||||
State string `json:"state"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Since time.Time `json:"since"`
|
||||
Streak int `json:"streak,omitempty"`
|
||||
Restarts int `json:"restarts,omitempty"`
|
||||
}
|
||||
|
||||
// HealthSaid is the health event's body: the machine and its statement. The machine is read from the
|
||||
// subject the bus let it publish on, never from here.
|
||||
type HealthSaid struct {
|
||||
Node string `json:"node"`
|
||||
Health Health `json:"health"`
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user