jochen
bb1607e424
Every one of the 48 core failures of research 031 was found by a person looking; the mesh's answers carried the fact for whoever asked and told nobody. - The condition store (to-be 45 §2): mesh-controller_conditions, one key per open condition, written by compare-and-set so a person's silence and the watchdogs never lose each other's word; every transition kept ninety days in mesh-controller_condition-history and said as the seat's events condition-raised / condition-changed / condition-cleared (the condition at the top level, with event, at, change, why, show), offered again while the bus is away. Raised and cleared by observation only; a clearing reopened within ten minutes is the same condition with its count up, its silence kept. Verbs: conditions, conditions show, conditions silence (a hand act, at most a week), conditions history. - ADR 0224's provider standing is the first kind, provider-failing, held by the provider's events; the provider_standing table is no longer read or written (left in place: dropping it is the operator's word). - status leads with the open conditions, urgent first, and says all well only with none open; conditions it cannot read are said and not well. - The signals table compiled in, one watchdog loop over it every 30s: S1 heartbeat (3 intervals, asleep machines excepted, control node urgent after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf, S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9 advisories, S10 self-check silent, S11 node tools silent, S13 stale refusals; S12, S14, S15 deferred with their reasons. A row that cannot see raises probe-failed and clears nothing. A test generated from the table suppresses each signal inside and past its bound. - The bus's advisories (maximum deliveries, a mesh consumer deleted) and the controller's own slow consumer and refused subjects, said in the mesh's words. - doctor: the probe registry D1-D10 (D5 deferred) and DW, every five minutes, each in thirty seconds; a probe that cannot run is never a pass. D1 validates with mesh-host's own validator. Every run ends with the doctor-heartbeat event mesh-watcher listens for. - The controller is granted its new buckets, events, the two advisories and $SRV.INFO; the node tools their tools-alive heartbeat. The streams and consumers the controller asserts and the ones D6/D7 expect are one derivation.
74 lines
3.1 KiB
Go
74 lines
3.1 KiB
Go
package conditions
|
|
|
|
import "time"
|
|
|
|
// The events a condition's life emits (to-be 45 §2), as the mesh-controller seat's own: published on
|
|
// `mesh.seat.mesh-controller.event.<name>`, on the events stream, so a consumer that was away catches
|
|
// up. **This is a contract**: the operator-channel's holder is written against these names and the
|
|
// shape of Event, and the controller learns nothing about telling.
|
|
const (
|
|
// EventRaised: a condition was raised — new, or the same fault again within ReopenWithin of its
|
|
// clearing (Change says which). A reopened condition is not news: its key is the one the
|
|
// first message was about.
|
|
EventRaised = "condition-raised"
|
|
// EventChanged: its severity, its resolver or its silence changed. Not every observation: a
|
|
// condition observed again is written, and says nothing.
|
|
EventChanged = "condition-changed"
|
|
// EventCleared: an observation says it is resolved. The condition is removed from the store and
|
|
// the transition kept in its history.
|
|
EventCleared = "condition-cleared"
|
|
)
|
|
|
|
// Events is every event a condition's life emits.
|
|
var Events = []string{EventRaised, EventChanged, EventCleared}
|
|
|
|
// HeartbeatEvent is the self-check's heartbeat (to-be 45 §4, S10), said under the same seat at the end
|
|
// of every run: `{run, at, interval-seconds, counts: {passed, failed, failed-to-run, deferred}, probes,
|
|
// controller, why}`. mesh-watcher, on a machine that is not the control node, listens for it.
|
|
const HeartbeatEvent = "doctor-heartbeat"
|
|
|
|
// What changed, as an event's Change and a history entry's says it.
|
|
const (
|
|
ChangeRaised = "raised"
|
|
ChangeReopened = "reopened"
|
|
ChangeSeverity = "severity"
|
|
ChangeResolver = "resolver"
|
|
ChangeSilenced = "silenced"
|
|
ChangeUnsilenced = "silence-ended"
|
|
ChangeCleared = "cleared"
|
|
)
|
|
|
|
// Event is the body of every condition event, and the shape a history entry keeps: **the condition
|
|
// itself, at the top level** — key, kind, subject, severity, summary, source, raised, last-observed,
|
|
// observations, resolver, silenced (null when not), epoch, and the evidence — with what happened to
|
|
// it beside. One object a consumer reads the same way whichever of the three it is.
|
|
type Event struct {
|
|
// Condition is the condition after the transition — as it was last held, for a clearing.
|
|
Condition
|
|
// Event is the event's own name, so a body read without its subject still says what it is.
|
|
Event string `json:"event"`
|
|
// At is when the transition happened.
|
|
At time.Time `json:"at"`
|
|
// Change is what happened: raised, reopened, severity, resolver, silenced, silence-ended, cleared.
|
|
Change string `json:"change"`
|
|
// Was is the value before, for a severity or resolver change.
|
|
Was string `json:"was,omitempty"`
|
|
// Why says why it cleared, or why it was silenced.
|
|
Why string `json:"why,omitempty"`
|
|
// Cleared is when it cleared, on a clearing.
|
|
Cleared *time.Time `json:"cleared,omitempty"`
|
|
// Show is the verb that shows more.
|
|
Show string `json:"show"`
|
|
}
|
|
|
|
// eventFor is the event a change is said under.
|
|
func eventFor(change string) string {
|
|
switch change {
|
|
case ChangeRaised, ChangeReopened:
|
|
return EventRaised
|
|
case ChangeCleared:
|
|
return EventCleared
|
|
}
|
|
return EventChanged
|
|
}
|