jochen
1cc6a2d759
The gate judged a module by what the mesh saw from outside, so a container that crash-looped after it applied passed it. Each machine's node-engine now states the health of every long-running resource it runs; the controller keeps the newest statement per machine, raises module.<module>.<machine>.unhealthy on the second statement in a row, clears it on the first that does not say it, and the gate passes a module only when every long-running resource of it is stated healthy since the send. An engine that states nothing is judged as before.
57 lines
2.2 KiB
Go
57 lines
2.2 KiB
Go
package link
|
|
|
|
import (
|
|
"context"
|
|
"encoding/json"
|
|
"strings"
|
|
)
|
|
|
|
// A machine's health statement (novox/hq ADR 0240, to-be 48 §4).
|
|
//
|
|
// **The node-engine owns every verdict; the controller keeps the last word per machine.** A statement
|
|
// arrives in every report and as an event between reports — on each change, and again every minute while
|
|
// a resource is not healthy. What the controller does with it — keep it, refuse an older one, raise
|
|
// `module.<module>.<machine>.unhealthy` on the second statement that says so — is the Healths it is given.
|
|
|
|
// Healths keeps what machines state of their long-running resources.
|
|
type Healths interface {
|
|
// Stated keeps one machine's statement, from a report or the event. An older statement than the one
|
|
// kept is refused there, by its time on the machine.
|
|
Stated(ctx context.Context, node string, h Health) error
|
|
}
|
|
|
|
// Hears says where the machines' health statements are kept. Their subscription is the heartbeats':
|
|
// core, and always made, so nothing is asked of the bus here.
|
|
func (s *Server) Hears(h Healths) { s.healths = h }
|
|
|
|
// healthSaid acts on one health event. Core NATS, so there is nothing to hold: a statement that could not
|
|
// be kept is said in the log, and the next one — a minute away while anything is not healthy — is kept.
|
|
func (s *Server) healthSaid(ctx context.Context, m Control) {
|
|
defer func() { _ = m.Took() }()
|
|
var said HealthSaid
|
|
if err := json.Unmarshal(m.Body(), &said); err != nil || said.Health.Contract == 0 {
|
|
return
|
|
}
|
|
// The machine is the one in the subject the bus let it publish on, never the body's.
|
|
node, ok := nodeOfHealth(m.Subject())
|
|
if !ok || s.healths == nil {
|
|
return
|
|
}
|
|
if err := s.healths.Stated(ctx, node, said.Health); err != nil {
|
|
s.log.Printf("could not keep what %s says of its resources' health: %v", node, err)
|
|
}
|
|
}
|
|
|
|
// nodeOfHealth is the machine a health statement names in its subject.
|
|
func nodeOfHealth(subject string) (string, bool) {
|
|
rest, ok := strings.CutPrefix(subject, "mesh.control.")
|
|
if !ok {
|
|
return "", false
|
|
}
|
|
node, kind, ok := strings.Cut(rest, ".")
|
|
if !ok || kind != "health" || node == "" || strings.Contains(node, ".") {
|
|
return "", false
|
|
}
|
|
return node, true
|
|
}
|