Say when the mesh is wrong: conditions, watchdogs, the bus's advisories, doctor (hq to-be 45 Phase 1)
Every one of the 48 core failures of research 031 was found by a person looking; the mesh's answers carried the fact for whoever asked and told nobody. - The condition store (to-be 45 §2): mesh-controller_conditions, one key per open condition, written by compare-and-set so a person's silence and the watchdogs never lose each other's word; every transition kept ninety days in mesh-controller_condition-history and said as the seat's events condition-raised / condition-changed / condition-cleared (the condition at the top level, with event, at, change, why, show), offered again while the bus is away. Raised and cleared by observation only; a clearing reopened within ten minutes is the same condition with its count up, its silence kept. Verbs: conditions, conditions show, conditions silence (a hand act, at most a week), conditions history. - ADR 0224's provider standing is the first kind, provider-failing, held by the provider's events; the provider_standing table is no longer read or written (left in place: dropping it is the operator's word). - status leads with the open conditions, urgent first, and says all well only with none open; conditions it cannot read are said and not well. - The signals table compiled in, one watchdog loop over it every 30s: S1 heartbeat (3 intervals, asleep machines excepted, control node urgent after 30 min), S2 report after a send, S3 plan tier, S4 event loop deaf, S5 merge not acted, S6 ask lost, S7 call hung, S8 provider silent, S9 advisories, S10 self-check silent, S11 node tools silent, S13 stale refusals; S12, S14, S15 deferred with their reasons. A row that cannot see raises probe-failed and clears nothing. A test generated from the table suppresses each signal inside and past its bound. - The bus's advisories (maximum deliveries, a mesh consumer deleted) and the controller's own slow consumer and refused subjects, said in the mesh's words. - doctor: the probe registry D1-D10 (D5 deferred) and DW, every five minutes, each in thirty seconds; a probe that cannot run is never a pass. D1 validates with mesh-host's own validator. Every run ends with the doctor-heartbeat event mesh-watcher listens for. - The controller is granted its new buckets, events, the two advisories and $SRV.INFO; the node tools their tools-alive heartbeat. The streams and consumers the controller asserts and the ones D6/D7 expect are one derivation.
This commit is contained in:
@@ -83,7 +83,9 @@ var defaultSeats = append([]Seat{
|
||||
// `assign` and the rest are a role's interface, not a container's, and stay addressable while
|
||||
// the control plane is replaced.
|
||||
{Name: ControllerSeatName, Scope: ScopeMesh, Decision: "novox/hq ADR 0079",
|
||||
Emits: []string{"applied", "refused", "built-before"},
|
||||
// And what is wrong, as it changes, and the self-check's heartbeat (novox/hq to-be 45 §2, §4).
|
||||
Emits: []string{"applied", "refused", "built-before",
|
||||
"condition-raised", "condition-changed", "condition-cleared", "doctor-heartbeat"},
|
||||
Serves: ControllerVerbs},
|
||||
// The store's first verbs (novox/hq ADR 0159): the smallest set that makes the store askable,
|
||||
// served by whichever module holds the seat with tools of these names.
|
||||
|
||||
@@ -231,6 +231,33 @@ var ControllerVerbs = []Verb{
|
||||
"kind": "one kind: apply, heartbeat-gap, plan-tier or build; every kind when absent",
|
||||
"days": "how many days back (default 14)",
|
||||
}, nil)},
|
||||
// What is wrong, and the self-check (novox/hq to-be 45 §2, §4).
|
||||
{Name: "conditions", Description: "What is wrong with the mesh now: every open condition, urgent first, " +
|
||||
"then oldest — raised by the watchdogs of the signals table, the self-check's probes and the providers' " +
|
||||
"own words, and cleared when observation says it is resolved, never by hand. Given a key, that one " +
|
||||
"whole with its evidence; with history, every transition lately; with silence, stop one's messages " +
|
||||
"for a while — a hand act, which says why (novox/hq to-be 45 §2).",
|
||||
Input: schema(map[string]string{
|
||||
"key": "a condition's key: that one whole; with history, only its transitions",
|
||||
"scope": "only this scope: machine, plan, call, build, merge, provider, seat, bus, core, probe or mesh",
|
||||
"severity": "only urgent, or only warning",
|
||||
"machine": "only those about this machine",
|
||||
"history": "\"true\": every raising, change, silence and clearing lately, oldest first",
|
||||
"days": "with history: how many days back (default 7, at most 90)",
|
||||
"silence": "a condition's key: send no message for it for a while; it stays open and in status",
|
||||
"for": "with silence: how long — 30m, 4h, 2d; at most 7d",
|
||||
"why": "with silence: why — required, and recorded in the hand-act log",
|
||||
"cause": "with silence: the cause in a word (the condition's kind when absent)",
|
||||
}, nil, "history")},
|
||||
{Name: "doctor", Description: "The self-check (novox/hq to-be 45 §4): the last run's verdict at once — " +
|
||||
"each probe of the design's live invariants passed, failed or could not run, and how long ago. With " +
|
||||
"run, a run now; with probes, the registry; with signals, every row of the signals table and the age " +
|
||||
"of its newest signal. Runs every five minutes on its own; each failure is an open condition.",
|
||||
Input: schema(map[string]string{
|
||||
"run": "\"true\": run every probe now and answer the verdict",
|
||||
"probes": "\"true\": the registry — what each probe asserts, and the condition it raises",
|
||||
"signals": "\"true\": the signals table, each row with the age of its newest signal",
|
||||
}, nil, "run", "probes", "signals")},
|
||||
{Name: "build", Description: "Have the build machine build a repository. Answers at once with the build's id: " +
|
||||
"`builds` with that id follows it line by line, and the module is registered when the outcome comes.",
|
||||
Input: schema(map[string]string{
|
||||
|
||||
Reference in New Issue
Block a user