Research 031 counted the repairs people made by hand: a push to unstick a plan waiting on a report, a controller restarted to make an object again, a plan closed, a consumer re-made from now. Each was the ordinary path taken again by someone who noticed. The healer registry makes each a registered response to one condition kind, with a budget, a settle and its event: - H1 sent-not-reported: ask the machine's node-engine to report again (mesh.node.<n>.ask.report); if it does not report what it was sent, send it again, never moving a build a policy or a plan holds back - H2 stalled: close a plan whose wait is superseded or finished - H3 holder-silent / consumer-lost: the send's own assertion of the bus's objects (issue 208's note) - H4 consumer-behind: consumer-reset, only for a consumer the stream table marks resettable (the controller's own events consumer) - H5 is the identity provider's own repair (ADR 0224 §5), registered only Success is the observation clearing the condition, never the healer; a spent budget hands the condition to the operator, urgent, with what was tried, and no healer touches it again. Every act is begun in the store before it is made (migration 0070), kept in the condition's tried as "healer Hn" and said as the seat event healer-acted; a heal is never a hand act. More than twelve acts in an hour stop every healer until an hour after the last, said urgently. Only the lease holder heals. S15 is live: a cause repaired by hand twice in a fortnight raises healer-wanted, naming the healer that was not enough where one exists. D6's far-behind finding has its own kind, consumer-behind. Nodes are granted the question; the controller's grant gains healer-acted (genesis lock in mesh-host). `healers` lists the registry, the acts and the brake; status counts the week's heals.
300 lines
14 KiB
Go
300 lines
14 KiB
Go
package main
|
|
|
|
import (
|
|
"encoding/json"
|
|
"fmt"
|
|
"github.com/novox/mesh-controller/internal/catalogue"
|
|
"github.com/novox/mesh-controller/internal/conditions"
|
|
"github.com/novox/mesh-controller/internal/inventory"
|
|
"sort"
|
|
"time"
|
|
)
|
|
|
|
// The same answers, in a shape something other than a person can read.
|
|
//
|
|
// A board reads through interfaces and holds nothing (novox/hq 03-DESIGN/01-to-be/11-a-board.md).
|
|
// Everything it needs is already answered by these commands — as text, for people, which is not
|
|
// something a page can read. So each of them can say it again as JSON.
|
|
//
|
|
// **`--json` rather than a serving API**, because nothing needs one yet: whatever serves a board
|
|
// runs the command, and the constraint in the design holds either way — the board never touches a
|
|
// context's store. An API is the larger thing and should wait until something is asking for it.
|
|
//
|
|
// **These shapes are hard to change once anything is built against them.** So they stay close to
|
|
// what the domain already calls things, and carry no summary field that would have to be kept
|
|
// true. Nothing here is derived that a reader could not derive.
|
|
|
|
// meshStatus is what `status --json` says: the three questions, in the order they are asked.
|
|
type meshStatus struct {
|
|
// Wrong is every machine whose last declaration was refused or partly failed.
|
|
Wrong []machineDoing `json:"wrong"`
|
|
// Quiet is every machine not heard from lately. Not the same as wrong: new, switched off and
|
|
// unreachable are not "tried and could not".
|
|
Quiet []machineQuiet `json:"quiet"`
|
|
// Behind is every module built from something older than its source has.
|
|
Behind []moduleBehind `json:"behind"`
|
|
// Waiting is every machine not running what the mesh would send it. The same question as
|
|
// Behind one level down: that says the catalogue is old, this says a machine is — and only
|
|
// this one has somebody's change waiting inside it.
|
|
Waiting []machineWaiting `json:"waiting"`
|
|
// Reported is every machine's last word beside when it was last sent a declaration. A
|
|
// machine whose report is newer than its send has acted on the current declaration; one
|
|
// whose is older is still working — and Waiting cannot tell those apart, because the sent
|
|
// digest is recorded at send, not at apply.
|
|
Reported []machineReported `json:"reported"`
|
|
// Plans is what the last merges produced and where each stands (novox/hq ADR 0162): the
|
|
// open ones first, each saying its tier, what it waits for, and whether it has waited too long.
|
|
Plans []planStatus `json:"plans"`
|
|
// Unresolved is every machine that cannot be worked out at all, with what the mesh said when
|
|
// it tried. **A machine here is in none of the lists above**: nothing was computed for it, so
|
|
// there is nothing to compare it against and nothing it can be behind — which is why a
|
|
// document without this field described a wholly blocked mesh as a well one.
|
|
//
|
|
// Per machine, and data. One node failing must never take the document away from a reader
|
|
// asking about the others.
|
|
Unresolved []machineUnresolved `json:"unresolved"`
|
|
// Network is why the private network could not be computed, when it could not; absent when it
|
|
// could. Almost always a consequence of Unresolved: a node that does not resolve is not on the
|
|
// network, and a mesh whose hub is that node has no hub.
|
|
Network string `json:"network,omitempty"`
|
|
// Machines is how many the mesh knows about, so a reader can tell "none wrong" from
|
|
// "none at all".
|
|
Machines int `json:"machines"`
|
|
// Adopted is every node still adopted (novox/hq ADR 0100); absent when none is.
|
|
Adopted []string `json:"adopted,omitempty"`
|
|
// Untaken is every module assigned to a machine that is holding what it found rather than
|
|
// running what the module declares, because nothing took it (novox/hq 04-ISSUES/125). Absent
|
|
// when nothing is held.
|
|
//
|
|
// **A document without this said an outage was a well mesh.** Read from what each machine
|
|
// reported, so it is the machine's account and not the mesh's take-time listing.
|
|
Untaken []machineUntaken `json:"untaken,omitempty"`
|
|
// Filtered is every converged machine that is not filtered by the mesh alone (novox/hq ADR
|
|
// 0168), one entry per rule set the mesh did not write — the found firewall in force again,
|
|
// or a chain nobody speaks for. Absent when every converged machine is filtered by the mesh
|
|
// alone. A document without this called a machine well while a predecessor's chain refused
|
|
// what the mesh declared open.
|
|
Filtered []machineFiltered `json:"filtered,omitempty"`
|
|
// Unheld is every module on a machine whose resources are applied through a seat nothing on
|
|
// that machine holds, with the modules that could hold it (novox/hq ADR 0207). Absent when every
|
|
// dependency is met. Reported, not refused, until the switch.
|
|
Unheld []catalogue.Unheld `json:"unheld,omitempty"`
|
|
// HandActsThisWeek is how many acts were done by hand in the last seven days (novox/hq to-be 45
|
|
// §7): every one is a repair a healer could have made. Absent where the log is not on hand;
|
|
// HandActsUnread says why when it could not be read, rather than reading as none.
|
|
HandActsThisWeek *int `json:"handActsThisWeek,omitempty"`
|
|
HandActsUnread string `json:"handActsUnread,omitempty"`
|
|
// HealsThisWeek is what the healers did in the last seven days (novox/hq to-be 45 §7); HealsUnread
|
|
// why it could not be read.
|
|
HealsThisWeek *healsCount `json:"healsThisWeek,omitempty"`
|
|
HealsUnread string `json:"healsUnread,omitempty"`
|
|
// Conditions is every open condition, urgent first and then oldest first (novox/hq to-be 45 §2):
|
|
// what is wrong, as the watchdogs, the self-check and the providers say it. Always present — an
|
|
// empty list is "none open" — unless they could not be read, which ConditionsUnread says.
|
|
Conditions []conditions.Condition `json:"conditions"`
|
|
ConditionsUnread string `json:"conditionsUnread,omitempty"`
|
|
// Failing is every consumer a provider says it keeps failing (novox/hq ADR 0224): the open
|
|
// conditions of that kind, carried here as well because ADR 0224 names this field. Absent when no
|
|
// provider says so. A document without it called the mesh well while the identity provider
|
|
// refused every consumer for a day (04-ISSUES/179).
|
|
Failing []conditions.Condition `json:"failing,omitempty"`
|
|
// Overflowing is every module whose identity overflows the bound of a provision it requires, and
|
|
// so is left out of its provider's grants (novox/hq ADR 0225). Absent when every identity fits.
|
|
Overflowing []catalogue.Overflow `json:"overflowing,omitempty"`
|
|
}
|
|
|
|
// machineFiltered is one rule set on a converged machine that the mesh did not write and that
|
|
// refuses traffic: where it is, whose the host reads it as, and what it refuses.
|
|
type machineFiltered struct {
|
|
Node string `json:"node"`
|
|
Where string `json:"where"`
|
|
Owner string `json:"owner"`
|
|
Refuses string `json:"refuses"`
|
|
}
|
|
|
|
// machineUntaken is one module a machine is holding rather than running, and how many resources of
|
|
// it are held.
|
|
type machineUntaken struct {
|
|
Node string `json:"node"`
|
|
Module string `json:"module"`
|
|
// Held is how many of the module's resources the machine is keeping as it found them. Zero is
|
|
// impossible here: a module with nothing held is not in this list.
|
|
Held int `json:"held"`
|
|
}
|
|
|
|
type machineUnresolved struct {
|
|
Node string `json:"node"`
|
|
// Problem is the mesh's own words, whole — newlines and all. It lists every requirement that
|
|
// could not be met, and a first line alone would name one of them and hide the rest.
|
|
Problem string `json:"problem"`
|
|
}
|
|
|
|
type machineDoing struct {
|
|
Node string `json:"node"`
|
|
// Outcome is refused or failed. Kept distinct all the way out: they are fixed in different
|
|
// places, and one word for both sends half the readers to the wrong one.
|
|
Outcome string `json:"outcome"`
|
|
Refused string `json:"refused,omitempty"`
|
|
Failed []struct {
|
|
ID string `json:"id"`
|
|
Error string `json:"error"`
|
|
} `json:"failed,omitempty"`
|
|
Applied int `json:"applied"`
|
|
At time.Time `json:"at"`
|
|
// Since is when this same failure was first reported and Times how many reports in a row
|
|
// have said it; Stuck is the mesh's word for "enough of them" (novox/hq 04-ISSUES/065).
|
|
Since *time.Time `json:"since,omitempty"`
|
|
Times int `json:"times"`
|
|
Stuck bool `json:"stuck"`
|
|
}
|
|
|
|
type machineReported struct {
|
|
Node string `json:"node"`
|
|
Outcome string `json:"outcome"`
|
|
At *time.Time `json:"at,omitempty"`
|
|
Sent *time.Time `json:"sent,omitempty"`
|
|
// Current is whether the last report names the declaration last sent. Not derivable from
|
|
// the timestamps beside it: an apply begun under the previous declaration reports after the
|
|
// next send, newer and still about the old words.
|
|
Current bool `json:"current"`
|
|
}
|
|
|
|
type machineWaiting struct {
|
|
Node string `json:"node"`
|
|
// Never is true when nothing has ever been sent to it. Not out of date: nobody has ever asked
|
|
// this machine to be anything, and the two read differently to whoever is looking.
|
|
Never bool `json:"never"`
|
|
Sent *time.Time `json:"sent,omitempty"`
|
|
}
|
|
|
|
type machineQuiet struct {
|
|
Node string `json:"node"`
|
|
// LastSeen is absent when the machine has never spoken, which is a different thing from
|
|
// having been quiet for a while.
|
|
LastSeen *time.Time `json:"lastSeen,omitempty"`
|
|
}
|
|
|
|
type moduleBehind struct {
|
|
Module string `json:"module"`
|
|
BuiltFrom string `json:"builtFrom"`
|
|
Head string `json:"head"`
|
|
On []string `json:"on"`
|
|
}
|
|
|
|
// statusAsJSON answers the same questions as the text form, from the same reading.
|
|
//
|
|
// **It takes the whole reading rather than a growing argument list**, which is what let a new
|
|
// answer be added to the text form and forgotten here — the two are one function's output in two
|
|
// shapes, and they must not be able to differ about what was asked.
|
|
//
|
|
// It never fails on account of the mesh. Every per-machine problem in here is a field, so one
|
|
// machine that cannot be worked out cannot stop a caller reading about the others: a
|
|
// machine-readable interface that stops being machine-readable exactly when something is wrong is
|
|
// one nobody can build an alarm on.
|
|
func statusAsJSON(asked answers) ([]byte, error) {
|
|
wrong, nodes, quiet := asked.wrong, asked.nodes, asked.quiet
|
|
behind, sources := asked.behind, asked.sources
|
|
waiting, reported := asked.waiting, asked.reported
|
|
|
|
out := meshStatus{Machines: len(nodes), Wrong: []machineDoing{},
|
|
Quiet: []machineQuiet{}, Behind: []moduleBehind{}, Waiting: []machineWaiting{},
|
|
Reported: []machineReported{}, Unresolved: []machineUnresolved{},
|
|
Network: asked.network, Adopted: adoptedNodes(nodes), Plans: planStatuses(asked.plans, time.Now(), asked.paused)}
|
|
// In a stated order, so two readings of an unchanged mesh are the same document.
|
|
untakenNodes := make([]string, 0, len(asked.untaken))
|
|
for name := range asked.untaken {
|
|
untakenNodes = append(untakenNodes, name)
|
|
}
|
|
sort.Strings(untakenNodes)
|
|
for _, name := range untakenNodes {
|
|
modules := make([]string, 0, len(asked.untaken[name]))
|
|
for m := range asked.untaken[name] {
|
|
modules = append(modules, m)
|
|
}
|
|
sort.Strings(modules)
|
|
for _, m := range modules {
|
|
out.Untaken = append(out.Untaken,
|
|
machineUntaken{Node: name, Module: m, Held: asked.untaken[name][m]})
|
|
}
|
|
}
|
|
filteredNodes := make([]string, 0, len(asked.filtered))
|
|
for name := range asked.filtered {
|
|
filteredNodes = append(filteredNodes, name)
|
|
}
|
|
sort.Strings(filteredNodes)
|
|
for _, name := range filteredNodes {
|
|
f := asked.filtered[name]
|
|
if fw := f.FoundFirewall; fw != nil && fw.Active {
|
|
out.Filtered = append(out.Filtered, machineFiltered{Node: name, Where: "the found firewall",
|
|
Owner: inventory.FilterFoundFirewall, Refuses: fw.Kind + " is in force again"})
|
|
}
|
|
for _, x := range f.Others() {
|
|
out.Filtered = append(out.Filtered, machineFiltered{Node: name, Where: x.Where, Owner: x.Owner, Refuses: x.Refuses})
|
|
}
|
|
}
|
|
out.Unheld = asked.unheld
|
|
out.HandActsThisWeek, out.HandActsUnread = asked.handActs, asked.handActsUnread
|
|
out.HealsThisWeek, out.HealsUnread = asked.heals, asked.healsUnread
|
|
out.Conditions, out.ConditionsUnread = asked.conditions, asked.conditionsUnread
|
|
if out.Conditions == nil {
|
|
out.Conditions = []conditions.Condition{}
|
|
}
|
|
out.Failing = providerStandings(asked.conditions)
|
|
out.Overflowing = asked.overflowing
|
|
for name := range asked.refused {
|
|
out.Unresolved = append(out.Unresolved, machineUnresolved{
|
|
Node: name, Problem: asked.refused[name]})
|
|
}
|
|
sort.Slice(out.Unresolved, func(i, j int) bool {
|
|
return out.Unresolved[i].Node < out.Unresolved[j].Node
|
|
})
|
|
for _, r := range reported {
|
|
out.Reported = append(out.Reported, machineReported{
|
|
Node: r.Node, Outcome: r.Outcome, At: r.At, Sent: r.Sent, Current: r.Current})
|
|
}
|
|
|
|
for _, m := range waiting {
|
|
out.Waiting = append(out.Waiting, machineWaiting{
|
|
Node: m.Node, Never: m.Never, Sent: m.SentAt})
|
|
}
|
|
|
|
for _, d := range wrong {
|
|
row := machineDoing{
|
|
Node: d.Node, Outcome: d.Outcome, Refused: d.Refused, Applied: d.Applied, At: d.At,
|
|
Since: d.Since, Times: d.Times, Stuck: d.Stuck(),
|
|
}
|
|
for _, f := range d.Failed {
|
|
row.Failed = append(row.Failed, struct {
|
|
ID string `json:"id"`
|
|
Error string `json:"error"`
|
|
}{ID: f.ID, Error: f.Error})
|
|
}
|
|
out.Wrong = append(out.Wrong, row)
|
|
}
|
|
for _, n := range quiet {
|
|
row := machineQuiet{Node: n.Name}
|
|
if !n.LastSeen.IsZero() {
|
|
seen := n.LastSeen
|
|
row.LastSeen = &seen
|
|
}
|
|
out.Quiet = append(out.Quiet, row)
|
|
}
|
|
for module, on := range behind {
|
|
from := sources[module]
|
|
out.Behind = append(out.Behind, moduleBehind{
|
|
Module: module, BuiltFrom: from.BuiltFrom, Head: from.Head, On: on,
|
|
})
|
|
}
|
|
return json.MarshalIndent(out, "", " ")
|
|
}
|
|
|
|
// say prints a value as JSON, for the commands that can answer either way.
|
|
func say(value any) error {
|
|
body, err := json.MarshalIndent(value, "", " ")
|
|
if err != nil {
|
|
return err
|
|
}
|
|
fmt.Println(string(body))
|
|
return nil
|
|
}
|