Compare commits

...
7 Commits
Author SHA1 Message Date
jochen ea92af2a7d Say a plan's refused step once, not on every tick, and call a walk a walk (hq issue 296, ADR 0244)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery covered: a later merge that contains it was delivered: novox/mesh-controller@bf11a8baacda (merged as 96c34c52 into main, walk plan-17914006…
A refused step was logged on every 30-second tick with the same words; it is said, and kept,
when it is new. The refusal named the retired "release plan".
2026-10-07 20:47:01 +02:00
jochen 875a4e5758 Count a plan's time from its tier, and save it only when a step changed it (hq issue 296)
plans said a build queued for minutes had been building for a few seconds: the line counted from
the last save, and every advance saved the plan whether or not it moved, bumping its revision and
saying plan-moved on the bus. The line, status and LATE now count from when the plan entered its
tier with the stalled condition's bound, and an advance that changes nothing writes nothing.
2026-10-07 20:42:13 +02:00
mesh-admin 85d664438f Merge pull request 'Raise a machine's network from what its engine says, once (hq ADR 0241)' (#113) from feat/machine-network-health into main 2026-10-07 18:16:10 +00:00
mesh-admin f6aea4bfbb Merge pull request 'Promise the delivery seat's checks verb, optional until its holder serves it (hq ADR 0239)' (#117) from feat/delivery-checks-verb into main 2026-10-07 17:58:59 +00:00
jochen 1fdff00794 Promise the delivery seat's checks verb, optional until its holder serves it (hq ADR 0239)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/delivery-checks-verb closed: a member was closed before the group merged
What the mesh's checks said of a pull request had no verb: the verdict was read from this
controller's journal. mesh-delivery answers it as checks. A verb added to a mesh seat whose holder
lives in another repository deadlocked: this controller would refuse the holder that does not serve
it, the one before it the holder that does, and every catalogue check between the two would fail on
mesh-delivery. So a verb can be marked optional — served or not, the holder holds — until every
holder serves it.
2026-10-07 19:36:57 +02:00
jochen b8bbf9c79f Hold the network statement's field names on the controller's side (hq ADR 0241)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer delivery to the same trunk took over its walk
mesh/delivery-group group feat/machine-network-health closed: a member was closed before the group merged
2026-10-07 18:53:21 +02:00
jochen 8bcf787258 Raise a machine's network from what its engine says, once (hq ADR 0241)
A VPN client rewrote the laptop's resolver file and nothing said so. The
engine now states its machine's networking; the controller keeps it with
the machine's health (migration 0077) and raises the rewrite as its own
finding naming the writer, the machine's own faults as machine.<m>.network,
and what several machines cannot reach once, there. The gate waits on a
rewrite it did not make rather than putting back a good build.
2026-10-07 18:52:16 +02:00
21 changed files with 1305 additions and 50 deletions
+2
View File
@@ -720,6 +720,8 @@ type answers struct {
plans []inventory.Plan
// paused is whether the build seat takes work, which a plan waiting on it says (ADR 0219).
paused pauseView
// tierBounds is how long each repository's plan tier may take before it is late (novox/hq issue 296).
tierBounds tierBounds
// refused is why a machine cannot be worked out at all, by name. A different thing from every
// other answer here: those are about a machine that was told something, and this is about one
// that cannot be told anything — it never reaches waiting, because nothing was computed for it
+11
View File
@@ -283,6 +283,8 @@ type machineWord struct {
facts gateFacts
on map[string]string
whole string
// waiting is what holds the machine on something shown to be another's (ADR 0241): the judging waits.
waiting string
}
// kindCoreBehind is D10's kind: a machine runs core components older than the mesh holds, or has not
@@ -346,6 +348,13 @@ func aboutTheMachine(machine string, moved []string, since time.Time, f gateFact
}
case coreBehind:
// Not what the send moved: said nowhere against it.
case c.Kind == kindNetworkRewritten || c.Kind == kindNetworkUnreachable && c.Subject.ID != machine:
// **Shown to be somebody else's** (ADR 0241): another program rewrote the resolver file the send
// did not move, or the machine cannot reach another that is down. Nothing the send did; the
// judging waits for it rather than putting back a build at the bound.
if w.waiting == "" {
w.waiting = machine + "'s network: " + said
}
default:
if w.whole == "" {
w.whole = machine + " as a whole: " + said
@@ -433,6 +442,8 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
h, said = healthNotYet, on
} else if w.whole != "" {
h, said = healthNotYet, w.whole
} else if w.waiting != "" {
h, said = healthWaiting, w.waiting
}
}
if h != healthGood && !slices.Contains(failing, j.module) {
+359
View File
@@ -0,0 +1,359 @@
package main
import (
"context"
"fmt"
"slices"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A machine says how its network is (novox/hq ADR 0241, which extends ADR 0240 from what a module runs
// to the machine it runs on).
//
// **Every machine's node-engine judges its own networking** — the resolver file the uplink holder
// declared, the names through every resolver it lists, the tunnel's handshake with the hub, the bus, the
// default route — on the two-look rule, and states it beside its resources. The controller keeps the
// newest statement per machine and raises from all of them together:
//
// - **an outside writer of the resolver file is its own finding**, `machine.<m>.<owner>.rewritten`:
// the file the uplink holder declares was rewritten by another program, named where the engine could
// name it. The names failing through what that program wrote are that finding's consequence, said in
// it — never a second condition;
// - **what is the machine's own** — its route, its tunnel, its resolvers answering wrong, a resolver
// that is no mesh machine — is `machine.<m>.network`;
// - **what points at another machine is said once, there** (the provider hold of ADR 0240 rule 5, for
// the network): a failure toward the hub or toward a mesh resolver is held under that machine when it
// is down on the record — silent, or its own network unhealthy — or when a second machine finds the
// same; then `machine.<x>.unreachable` names every machine that cannot reach it, and none of them
// raises anything of its own for it. One machine alone failing toward a healthy one is its own.
//
// Each is a warning; urgent on the control node, or when the bus cannot be reached, or for the hub.
// Cleared on the first statement that no longer says it. An engine older than this judging says nothing
// of its network, and nothing is raised for it.
// The conditions a machine's network raises.
const (
kindMachineNetwork = "machine-network"
kindNetworkRewritten = "network-rewritten"
kindNetworkUnreachable = "network-unreachable"
sourceNetwork = "network"
)
// networkKinds are the kinds this judging owns: every open one it no longer says, it clears.
var networkKinds = []string{kindMachineNetwork, kindNetworkRewritten, kindNetworkUnreachable}
// networkFacts is what one judging of every machine's network reads.
type networkFacts struct {
healths map[string]inventory.NodeHealth
// byAddress is each machine's address on the private network; hub the hub's name; control the
// control node's.
byAddress map[string]string
hub string
control string
// silent is every machine whose silence is an open condition.
silent map[string]bool
}
// judgeNetworks raises and clears every machine's network conditions from every machine's newest
// statement, after one machine's statement was kept.
func judgeNetworks(ctx context.Context, inv *inventory.Inventory, k *conditions.Keeper, now time.Time) error {
healths, err := inv.Healths(ctx)
if err != nil {
return err
}
overlays, err := inv.Overlays(ctx)
if err != nil {
return err
}
open, err := k.Open(ctx)
if err != nil {
return err
}
f := networkFacts{healths: healths, byAddress: map[string]string{}, control: controlHost(ctx, inv),
silent: map[string]bool{}}
for _, o := range overlays {
if o.Address != "" {
f.byAddress[o.Address] = o.Name
}
if o.Hub {
f.hub = o.Name
}
}
for _, c := range open {
if c.Subject.Scope == conditions.ScopeMachine && c.Kind == "silent" {
f.silent[c.Subject.ID] = true
}
}
var problems []string
said := map[string]bool{}
for _, o := range networkObservations(f) {
said[o.Key()] = true
if _, err := k.Observe(ctx, o); err != nil {
problems = append(problems, err.Error())
}
}
for _, c := range open {
if !slices.Contains(networkKinds, c.Kind) || said[c.Key] {
continue
}
why := "no machine says it any more"
if c.Subject.Machine != "" {
why = c.Subject.Machine + "'s network no longer says it"
}
if _, err := k.Clear(ctx, c.Key, why); err != nil {
problems = append(problems, err.Error())
}
}
if len(problems) > 0 {
return fmt.Errorf("%s", strings.Join(problems, "; "))
}
return nil
}
// pointed is one machine's failing part that points at another machine.
type pointed struct {
from string
part inventory.NetworkPart
}
// networkObservations is every network condition the statements say now. Pure.
func networkObservations(f networkFacts) []conditions.Observation {
machines := make([]string, 0, len(f.healths))
for m := range f.healths {
machines = append(machines, m)
}
sort.Strings(machines)
unhealthy := func(m string) []inventory.NetworkPart {
h := f.healths[m]
if h.Network == nil {
return nil
}
var out []inventory.NetworkPart
for _, p := range h.Network.Parts {
if p.State == link.StateUnhealthy {
out = append(out, p)
}
}
return out
}
// The machines a part points at, other than its own: the hub, and each mesh resolver by its address.
// An address that is no mesh machine's — the resolver a VPN client wrote in — is the machine's own.
targets := func(m string, p inventory.NetworkPart) ([]string, bool) {
if len(p.Toward) == 0 {
return nil, false
}
var out []string
for _, t := range p.Toward {
x := f.byAddress[t]
if t == link.TowardHub {
x = f.hub
}
if x == "" || x == m {
return nil, false
}
if !slices.Contains(out, x) {
out = append(out, x)
}
}
return out, true
}
// First pass: what points at whom.
pointing := map[string][]pointed{}
for _, m := range machines {
for _, p := range unhealthy(m) {
if xs, ok := targets(m, p); ok {
for _, x := range xs {
pointing[x] = append(pointing[x], pointed{from: m, part: p})
}
}
}
}
from := func(x string) []string {
var out []string
for _, pt := range pointing[x] {
if !slices.Contains(out, pt.from) {
out = append(out, pt.from)
}
}
sort.Strings(out)
return out
}
// A machine is down on the record when its silence is open or its own network is unhealthy, or when
// two machines find it unreachable: then what points at it is held there.
down := func(x string) bool {
return f.silent[x] || len(unhealthy(x)) > 0 || len(from(x)) >= 2
}
var out []conditions.Observation
saysOwn := map[string]bool{}
for _, m := range machines {
parts := unhealthy(m)
if len(parts) == 0 {
continue
}
var own []inventory.NetworkPart
var rewritten *inventory.NetworkPart
for i, p := range parts {
if p.Part == link.PartResolvConf {
rewritten = &parts[i]
continue
}
if xs, ok := targets(m, p); ok && allDown(xs, down) {
continue // held at the machines it points at
}
own = append(own, p)
}
if rewritten != nil {
out = append(out, rewrittenObservation(m, *rewritten, parts, f))
// The names failing through what another program wrote are that finding's, said in it.
kept := own[:0]
for _, p := range own {
if p.Part != link.PartNames {
kept = append(kept, p)
}
}
own = kept
}
if len(own) > 0 {
out = append(out, machineNetworkObservation(m, own, f, from(m)))
saysOwn[m] = true
}
}
// Said once, at the machine everybody points at — unless its own network condition already says it
// (listed there), or its silence does.
targetsSorted := make([]string, 0, len(pointing))
for x := range pointing {
targetsSorted = append(targetsSorted, x)
}
sort.Strings(targetsSorted)
for _, x := range targetsSorted {
if !down(x) || saysOwn[x] || f.silent[x] {
continue
}
out = append(out, unreachableObservation(x, pointing[x], from(x), f))
}
return out
}
func allDown(xs []string, down func(string) bool) bool {
for _, x := range xs {
if !down(x) {
return false
}
}
return len(xs) > 0
}
// rewrittenObservation is the resolver file rewritten by another program: its own finding, naming the
// writer where the engine could, and what it costs the machine.
func rewrittenObservation(m string, p inventory.NetworkPart, all []inventory.NetworkPart, f networkFacts) conditions.Observation {
writer := ""
if p.Writer != "" {
writer = " (" + p.Writer + ")"
}
cost := "the names through it are not yet judged"
said := []string{p.Part + ": " + p.Said}
for _, q := range all {
if q.Part == link.PartNames {
cost = strings.TrimSuffix(q.Reason, ".")
said = append(said, q.Part+": "+q.Said)
}
}
if cost == "the names through it are not yet judged" {
cost = "the names still resolve through what it wrote"
}
id := m
if p.Owner != "" {
// Named by the module whose file it is: a send that moved that module is what the gate
// holds it on (issue 281's rule — what names a moved module is that module's).
id = m + "." + p.Owner
}
severity := conditions.Warning
if m == f.control {
severity = conditions.Urgent
}
summary := fmt.Sprintf("the resolver file on %s was rewritten by another program%s — %s until the "+
"node-engine writes it back at its next reconcile, or that program gives it back", m, writer, cost)
if p.Owner != "" {
summary = fmt.Sprintf("the resolver file %s writes on %s was rewritten by another program%s — %s until "+
"the node-engine writes it back at its next reconcile, or that program gives it back", p.Owner, m, writer, cost)
}
return conditions.Observation{Scope: conditions.ScopeMachine, ID: id, Token: "rewritten", Kind: kindNetworkRewritten,
Machine: m, Severity: severity, Source: sourceNetwork, Summary: summary,
Said: fmt.Sprintf("since %s: %s", p.Since.UTC().Format("2006-01-02 15:04:05 MST"), strings.Join(said, " | "))}
}
// machineNetworkObservation is what is wrong with a machine's own networking.
func machineNetworkObservation(m string, parts []inventory.NetworkPart, f networkFacts, waiting []string) conditions.Observation {
var words, said []string
severity := conditions.Warning
for _, p := range parts {
words = append(words, p.Reason)
said = append(said, fmt.Sprintf("%s since %s: %s", p.Part, p.Since.UTC().Format("2006-01-02 15:04:05 MST"), p.Said))
if p.Part == link.PartBus {
severity = conditions.Urgent
}
}
if m == f.control || m == f.hub {
severity = conditions.Urgent
}
summary := fmt.Sprintf("%s's network is not healthy: %s", m, strings.Join(words, "; "))
if len(waiting) > 0 {
severity = conditions.Urgent
summary += fmt.Sprintf("; %s cannot reach it", strings.Join(waiting, ", "))
}
return conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Token: "network", Kind: kindMachineNetwork,
Machine: m, Severity: severity, Source: sourceNetwork, Summary: summary, Said: strings.Join(said, " | ")}
}
// unreachableObservation is one machine others cannot reach, said once there.
func unreachableObservation(x string, pts []pointed, from []string, f networkFacts) conditions.Observation {
var what []string
var said []string
for _, pt := range pts {
w := map[string]string{link.PartTunnel: "the tunnel to it", link.PartBus: "the bus on it",
link.PartNames: "its resolver"}[pt.part.Part]
if w == "" {
w = pt.part.Part
}
if !slices.Contains(what, w) {
what = append(what, w)
}
said = append(said, fmt.Sprintf("%s: %s: %s", pt.from, pt.part.Part, pt.part.Said))
}
severity := conditions.Warning
if x == f.hub || x == f.control || slices.Contains(what, "the bus on it") {
severity = conditions.Urgent
}
return conditions.Observation{Scope: conditions.ScopeMachine, ID: x, Token: "unreachable", Kind: kindNetworkUnreachable,
Machine: x, Also: from, Severity: severity, Source: sourceNetwork,
Summary: fmt.Sprintf("%s cannot be reached from %s: %s", x, strings.Join(from, ", "), strings.Join(what, ", ")),
Said: strings.Join(said, " | ")}
}
// networkLines is what `node show` says of a machine's networking.
func networkLines(h inventory.NodeHealth, had bool, now time.Time) []string {
if !had || h.Network == nil {
return []string{" its node-engine does not say how its network is — it is older than that judging (ADR 0241)"}
}
out := []string{fmt.Sprintf(" its network: %s since %s", h.Network.State, h.Network.Since.Local().Format("2006-01-02 15:04"))}
for _, p := range h.Network.Parts {
line := fmt.Sprintf(" %-10s %s", p.State, p.Part)
if p.Reason != "" {
line += " — " + p.Reason
}
if p.Writer != "" {
line += " (" + p.Writer + ")"
}
out = append(out, line)
}
return out
}
+306
View File
@@ -0,0 +1,306 @@
package main
import (
"encoding/json"
"os"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A machine says how its network is (novox/hq ADR 0241, "how it is checked"): the resolver file rewritten
// by another program is one finding, naming the writer, with the names it costs said in it; what is the
// machine's own is `machine.<m>.network`; what points at another machine that is down is said once, there;
// one machine alone failing toward a healthy one is its own; the control node, the hub and the bus are
// urgent; an engine that says nothing of its network raises nothing; and the gate waits on what is shown to
// be another's.
func aNetwork(state string, parts ...inventory.NetworkPart) *inventory.NetworkHealth {
for i := range parts {
if parts[i].State == "" {
parts[i].State = link.StateUnhealthy
}
parts[i].Since = h0
}
return &inventory.NetworkHealth{State: state, Since: h0, Parts: parts}
}
func netFacts(healths map[string]*inventory.NetworkHealth) networkFacts {
f := networkFacts{healths: map[string]inventory.NodeHealth{}, hub: "anchor", control: "anchor",
byAddress: map[string]string{"10.77.0.1": "anchor", "10.77.0.2": "laptop", "10.77.0.3": "spare"},
silent: map[string]bool{}}
for m, n := range healths {
f.healths[m] = inventory.NodeHealth{Node: m, Network: n}
}
return f
}
var (
rewrittenByVPN = inventory.NetworkPart{Part: link.PartResolvConf, Reason: "the resolver file was rewritten by another program",
Said: "/etc/resolv.conf lists 172.16.5.5 where 10.77.0.1 is declared", Writer: "FortiClient", Owner: "networkmanager"}
namesThroughVPN = inventory.NetworkPart{Part: link.PartNames, Reason: "mesh names do not resolve",
Said: "172.16.5.5 — anchor.internal (IPv4): says no such name", Toward: []string{"172.16.5.5"}}
tunnelDown = inventory.NetworkPart{Part: link.PartTunnel, Reason: "the tunnel to the hub has not handshaken for over five minutes",
Said: "mesh0's newest handshake with the hub was 9m0s ago", Toward: []string{link.TowardHub}}
noRoute = inventory.NetworkPart{Part: link.PartRoute, Reason: "the machine has no default route", Said: "no default route"}
)
func keysOf(obs []conditions.Observation) []string {
var keys []string
for _, o := range obs {
keys = append(keys, o.Key())
}
return keys
}
func TestAResolverFileRewrittenIsOneFindingNamingItsWriterAndWhatItCosts(t *testing.T) {
obs := networkObservations(netFacts(map[string]*inventory.NetworkHealth{
"anchor": aNetwork(link.StateHealthy),
"laptop": aNetwork(link.StateUnhealthy, rewrittenByVPN, namesThroughVPN),
}))
if keys := keysOf(obs); len(keys) != 1 || keys[0] != "machine.laptop.networkmanager.rewritten" {
t.Fatalf("want one finding, the rewrite, got %v", keys)
}
o := obs[0]
for _, want := range []string{"laptop", "rewritten by another program (FortiClient)", "mesh names do not resolve",
"next reconcile"} {
if !strings.Contains(o.Summary, want) {
t.Errorf("the summary does not say %q: %s", want, o.Summary)
}
}
if strings.Contains(o.Summary, "172.16.") || strings.Contains(o.Summary, "/etc/") {
t.Errorf("an address or a path reached the summary: %s", o.Summary)
}
if !strings.Contains(o.Said, "172.16.5.5") || o.Severity != conditions.Warning || o.Machine != "laptop" {
t.Errorf("the evidence or the severity is wrong: %+v", o)
}
}
func TestOneMachineFailingTowardAHealthyHubIsItsOwn(t *testing.T) {
obs := networkObservations(netFacts(map[string]*inventory.NetworkHealth{
"anchor": aNetwork(link.StateHealthy),
"laptop": aNetwork(link.StateUnhealthy, tunnelDown),
"spare": aNetwork(link.StateHealthy),
}))
if keys := keysOf(obs); len(keys) != 1 || keys[0] != "machine.laptop.network" {
t.Fatalf("want the laptop's own, got %v", keys)
}
if obs[0].Severity != conditions.Warning {
t.Fatalf("a laptop's own tunnel is a warning, got %s", obs[0].Severity)
}
}
func TestTwoMachinesThatCannotReachTheHubAreSaidOnceAtTheHub(t *testing.T) {
obs := networkObservations(netFacts(map[string]*inventory.NetworkHealth{
"anchor": aNetwork(link.StateHealthy),
"laptop": aNetwork(link.StateUnhealthy, tunnelDown),
"spare": aNetwork(link.StateUnhealthy, tunnelDown),
}))
if keys := keysOf(obs); len(keys) != 1 || keys[0] != "machine.anchor.unreachable" {
t.Fatalf("want one condition at the hub, got %v", keys)
}
o := obs[0]
if o.Severity != conditions.Urgent || !strings.Contains(o.Summary, "laptop, spare") || len(o.Also) != 2 {
t.Fatalf("the hub's condition is %+v", o)
}
}
func TestWhatPointsAtASilentHubIsHeldUnderItsSilence(t *testing.T) {
f := netFacts(map[string]*inventory.NetworkHealth{"laptop": aNetwork(link.StateUnhealthy, tunnelDown)})
f.silent["anchor"] = true
if obs := networkObservations(f); len(obs) != 0 {
t.Fatalf("the hub's silence says it; got %v", keysOf(obs))
}
}
func TestAHubWhoseOwnNetworkIsUnhealthyListsWhoCannotReachIt(t *testing.T) {
obs := networkObservations(netFacts(map[string]*inventory.NetworkHealth{
"anchor": aNetwork(link.StateUnhealthy, noRoute),
"laptop": aNetwork(link.StateUnhealthy, tunnelDown),
}))
if keys := keysOf(obs); len(keys) != 1 || keys[0] != "machine.anchor.network" {
t.Fatalf("want the hub's own, holding the laptop's, got %v", keys)
}
if o := obs[0]; o.Severity != conditions.Urgent || !strings.Contains(o.Summary, "laptop cannot reach it") {
t.Fatalf("the hub's condition is %+v", o)
}
}
func TestOneMachineFailingAHealthyMeshResolverIsItsOwnAndTwoAreTheResolvers(t *testing.T) {
silentResolver := inventory.NetworkPart{Part: link.PartNames, Reason: "1 of its 2 resolvers do not answer as the mesh's do",
Said: "10.77.0.3 — no answer within 1s", Toward: []string{"10.77.0.3"}}
obs := networkObservations(netFacts(map[string]*inventory.NetworkHealth{
"anchor": aNetwork(link.StateHealthy), "spare": aNetwork(link.StateHealthy),
"laptop": aNetwork(link.StateUnhealthy, silentResolver),
}))
if keys := keysOf(obs); len(keys) != 1 || keys[0] != "machine.laptop.network" {
t.Fatalf("one machine alone: want its own, got %v", keys)
}
obs = networkObservations(netFacts(map[string]*inventory.NetworkHealth{
"anchor": aNetwork(link.StateUnhealthy, silentResolver), "spare": aNetwork(link.StateHealthy),
"laptop": aNetwork(link.StateUnhealthy, silentResolver),
}))
if keys := keysOf(obs); len(keys) != 1 || keys[0] != "machine.spare.unreachable" {
t.Fatalf("two machines: want it said once at the resolver's machine, got %v", keys)
}
if !strings.Contains(obs[0].Summary, "its resolver") {
t.Fatalf("the resolver's machine is said %s", obs[0].Summary)
}
}
func TestTheControlNodeAndTheBusAreUrgent(t *testing.T) {
f := netFacts(map[string]*inventory.NetworkHealth{"anchor": aNetwork(link.StateUnhealthy, rewrittenByVPN)})
if obs := networkObservations(f); len(obs) != 1 || obs[0].Severity != conditions.Urgent {
t.Fatalf("the control node's rewritten file: %+v", obs)
}
bus := inventory.NetworkPart{Part: link.PartBus, Reason: "the bus cannot be reached", Said: "no link"}
f = netFacts(map[string]*inventory.NetworkHealth{"laptop": aNetwork(link.StateUnhealthy, bus, noRoute)})
if obs := networkObservations(f); len(obs) != 1 || obs[0].Severity != conditions.Urgent {
t.Fatalf("the bus unreachable from the laptop: %+v", obs)
}
}
func TestAnEngineThatSaysNothingOfItsNetworkRaisesNothing(t *testing.T) {
f := netFacts(map[string]*inventory.NetworkHealth{"laptop": nil, "anchor": aNetwork(link.StateHealthy)})
if obs := networkObservations(f); len(obs) != 0 {
t.Fatalf("got %v", keysOf(obs))
}
}
// Through the store and the keeper: the statement kept, the rewrite raised from it, cleared when the file
// is written back, and node show saying it.
func TestARewrittenResolverFileIsRaisedFromTheStatementAndClearedWhenWrittenBack(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv, k := open.inventory, conditionsFrom
say := func(at time.Time, n *link.NetworkHealth) {
t.Helper()
if err := stateHealth(ctx, inv, k, "laptop", link.Health{Contract: link.ReadinessContract, At: at, Network: n}, at); err != nil {
t.Fatal(err)
}
}
openKeys := func() []string {
t.Helper()
list, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
var keys []string
for _, c := range list {
keys = append(keys, c.Key)
}
return keys
}
healthy := &link.NetworkHealth{State: link.StateHealthy, Since: h0, Parts: []link.NetworkPart{
{Part: link.PartResolvConf, State: link.StateHealthy, Since: h0}}}
rewritten := &link.NetworkHealth{State: link.StateUnhealthy, Since: h0.Add(time.Minute), Parts: []link.NetworkPart{
{Part: link.PartResolvConf, State: link.StateUnhealthy, Reason: rewrittenByVPN.Reason, Said: rewrittenByVPN.Said,
Writer: "FortiClient", Owner: "networkmanager", Since: h0.Add(time.Minute)},
{Part: link.PartNames, State: link.StateUnhealthy, Reason: namesThroughVPN.Reason, Said: namesThroughVPN.Said,
Toward: namesThroughVPN.Toward, Since: h0.Add(time.Minute)}}}
say(h0, healthy)
if keys := openKeys(); len(keys) != 0 {
t.Fatalf("a healthy network raised %v", keys)
}
say(h0.Add(time.Minute), rewritten)
if keys := openKeys(); len(keys) != 1 || keys[0] != "machine.laptop.networkmanager.rewritten" {
t.Fatalf("the rewrite raised %v", keys)
}
kept, had, err := inv.HealthOf(ctx, "laptop")
if err != nil || !had || kept.Network == nil || kept.Network.Parts[0].Writer != "FortiClient" {
t.Fatalf("the statement's network was not kept: %+v %v", kept.Network, err)
}
if lines := strings.Join(networkLines(kept, had, h0), "\n"); !strings.Contains(lines, "FortiClient") ||
!strings.Contains(lines, "unhealthy resolv-conf") {
t.Fatalf("node show says:\n%s", lines)
}
say(h0.Add(2*time.Minute), healthy)
if keys := openKeys(); len(keys) != 0 {
t.Fatalf("written back, still open: %v", keys)
}
// An engine older than the judging says no network: nothing raised, and node show says it is not known.
say(h0.Add(3*time.Minute), nil)
kept, had, _ = inv.HealthOf(ctx, "laptop")
if keys := openKeys(); len(keys) != 0 || kept.Network != nil {
t.Fatalf("an older engine: %v %+v", keys, kept.Network)
}
if lines := strings.Join(networkLines(kept, had, h0), "\n"); !strings.Contains(lines, "older than that judging") {
t.Fatalf("node show says:\n%s", lines)
}
}
// The gate: a resolver file another program rewrote waits the judging rather than failing it at the
// bound; the same, when the send moved the module whose file it is, is that module's; a machine's own
// network fault holds the machine as a whole, as before.
func TestTheGateWaitsOnARewriteItDidNotMakeAndHoldsTheOwnerOnOneItMoved(t *testing.T) {
since := h0
rewrite := conditions.Condition{Key: "machine.laptop.networkmanager.rewritten", Kind: kindNetworkRewritten,
Subject: conditions.Subject{Scope: conditions.ScopeMachine, ID: "laptop.networkmanager", Machine: "laptop"},
Summary: "the resolver file was rewritten", Raised: since.Add(time.Minute), Source: sourceNetwork}
f := gateFacts{judged: true, open: []conditions.Condition{rewrite}}
if w := aboutTheMachine("laptop", []string{"letta"}, since, f); w.waiting == "" || w.whole != "" || len(w.on) != 0 {
t.Fatalf("a rewrite the send did not make: %+v", w)
}
if w := aboutTheMachine("laptop", []string{"networkmanager", "letta"}, since, f); w.on["networkmanager"] == "" ||
w.on["letta"] != "" || w.waiting != "" {
t.Fatalf("a rewrite of the file a moved module owns: %+v", w)
}
own := conditions.Condition{Key: "machine.laptop.network", Kind: kindMachineNetwork,
Subject: conditions.Subject{Scope: conditions.ScopeMachine, ID: "laptop", Machine: "laptop"},
Summary: "laptop's network is not healthy", Raised: since.Add(time.Minute), Source: sourceNetwork}
if w := aboutTheMachine("laptop", []string{"letta"}, since, gateFacts{judged: true,
open: []conditions.Condition{own}}); w.whole == "" || w.waiting != "" {
t.Fatalf("the machine's own network: %+v", w)
}
unreachable := conditions.Condition{Key: "machine.anchor.unreachable", Kind: kindNetworkUnreachable,
Subject: conditions.Subject{Scope: conditions.ScopeMachine, ID: "anchor", Machine: "anchor", Also: []string{"laptop", "spare"}},
Summary: "anchor cannot be reached", Raised: since.Add(time.Minute), Source: sourceNetwork}
if w := aboutTheMachine("laptop", []string{"letta"}, since, gateFacts{judged: true,
open: []conditions.Condition{unreachable}}); w.waiting == "" || w.whole != "" {
t.Fatalf("a machine that cannot reach the hub: %+v", w)
}
}
// TestTheDrillsStatementsRaiseAndClearTheRewrite replays the drill of ADR 0241 (mesh-host
// internal/network TestDrill…): what a node-engine said in a throwaway container while its resolver file
// was declared, rewritten as a VPN client rewrites it, and written back — recorded, with the mesh's names
// and addresses replaced by this test mesh's. Healthy raises nothing; the rewrite, on its second look, is
// raised as one finding naming the writer; written back, it clears.
func TestTheDrillsStatementsRaiseAndClearTheRewrite(t *testing.T) {
raw, err := os.ReadFile("testdata/network-drill.json")
if err != nil {
t.Fatal(err)
}
var said []link.Health
if err := json.Unmarshal(raw, &said); err != nil {
t.Fatal(err)
}
open := aMesh(t)
ctx := t.Context()
k := conditionsFrom
var raisedAt []int
for i, h := range said {
if err := stateHealth(ctx, open.inventory, k, "laptop", h, h.At); err != nil {
t.Fatal(err)
}
list, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
for _, c := range list {
if c.Key != "machine.laptop.networkmanager.rewritten" || !strings.Contains(c.Summary, "(FortiClient)") {
t.Fatalf("statement %d raised %s: %s", i, c.Key, c.Summary)
}
raisedAt = append(raisedAt, i)
}
}
// Five statements: healthy, healthy, one failing look (still healthy), unhealthy, written back.
if len(raisedAt) != 1 || raisedAt[0] != 3 {
t.Fatalf("the rewrite was open after statements %v; want after the fourth alone", raisedAt)
}
}
+19 -2
View File
@@ -83,8 +83,16 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
for module := range unhealthy {
streaks[module] = prev.Streaks[module] + 1
}
var network *inventory.NetworkHealth
if h.Network != nil {
network = &inventory.NetworkHealth{State: h.Network.State, Since: h.Network.Since, Parts: []inventory.NetworkPart{}}
for _, p := range h.Network.Parts {
network.Parts = append(network.Parts, inventory.NetworkPart{Part: p.Part, State: p.State, Reason: p.Reason,
Said: p.Said, Writer: p.Writer, Owner: p.Owner, Toward: p.Toward, Since: p.Since, Streak: p.Streak})
}
}
stored, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: node, Contract: h.Contract, SaidAt: h.At,
HeardAt: now, Resources: resources, Streaks: streaks})
HeardAt: now, Resources: resources, Streaks: streaks, Network: network})
if err != nil || !stored {
if err == nil {
healthRefused.Add(1)
@@ -94,7 +102,16 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
if k == nil {
return nil
}
return judgeModuleHealth(ctx, inv, k, node, unhealthy, streaks, now)
err = judgeModuleHealth(ctx, inv, k, node, unhealthy, streaks, now)
// And every machine's network, from every machine's newest statement (ADR 0241): a statement about one
// machine can hold another's finding, or release it.
if nerr := judgeNetworks(ctx, inv, k, now); nerr != nil {
if err == nil {
return nerr
}
return fmt.Errorf("%w; %v", err, nerr)
}
return err
}
// judgeModuleHealth raises a module's condition on a machine on the second statement in a row that says a
+3
View File
@@ -546,6 +546,9 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
for _, line := range healthLines(h, had, time.Now()) {
fmt.Println(line)
}
for _, line := range networkLines(h, had, time.Now()) {
fmt.Println(line)
}
}
held, err := inv.Profile(ctx, name)
+121
View File
@@ -0,0 +1,121 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// novox/hq issue 296: a plan saved a few seconds ago that has stood in its tier for forty minutes says
// forty minutes, and LATE — the line counted from the last save, which every advance made.
func TestAPlansLineCountsFromItsTierNotFromItsLastSave(t *testing.T) {
now := time.Date(2026, 10, 7, 20, 30, 0, 0, time.UTC)
asked := now.Add(-40 * time.Minute)
p := inventory.Plan{ID: "plan-t", Repository: "novox/mesh-catalog", Commit: "3ad22356", State: inventory.PlanBuilding,
Created: now.Add(-40 * time.Minute), Updated: now.Add(-3 * time.Second), TierEntered: now.Add(-40 * time.Minute),
Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &asked}}}
line := planLine(p, now)
if !strings.Contains(line, "building for 40m0s") || !strings.Contains(line, "LATE") {
t.Fatalf("a plan in its tier for 40m, saved 3s ago, reads %q", line)
}
st := planStatuses([]inventory.Plan{p}, now, pauseView{}, nil)[0]
if !st.Late || !st.Since.Equal(p.TierEntered) {
t.Errorf("status --json says %+v", st)
}
if _, late := openPlans([]inventory.Plan{p}, now, pauseView{}, nil); late != 1 {
t.Errorf("status counted %d late", late)
}
// LATE is the stalled condition said on the line: the same start and the same bound, measured or not.
bounds := boundsOf([]inventory.Duration{{Subject: p.Repository, Took: 20 * time.Minute}})
if got := bounds.of(p.Repository); got != time.Hour {
t.Fatalf("a repository whose tiers took 20m is given %s", got)
}
if got := bounds.of("novox/other"); got != tierAtLeast {
t.Fatalf("a repository with nothing measured is given %s", got)
}
for _, in := range []time.Duration{40 * time.Minute, 70 * time.Minute} {
q := p
q.TierEntered = now.Add(-in)
bound := bounds.of(q.Repository)
late := strings.Contains(planLineWith(q, now, pauseView{}, bound), "LATE")
stalled := len(watchPlans(&signalFacts{now: now, plans: []planFacts{{id: q.ID, repository: q.Repository,
commit: q.Commit, tiers: 1, entered: inTierSince(q), bound: bound}}})) == 1
if late != stalled || late != (in > bound) {
t.Errorf("in its tier %s with a bound of %s: LATE %v, stalled %v", in, bound, late, stalled)
}
}
// A plan read without its tier's stamp counts from its last save, the only time it has.
old := p
old.TierEntered = time.Time{}
if line := planLine(old, now); !strings.Contains(line, "for 3s") || strings.Contains(line, "LATE") {
t.Errorf("a plan without its tier's stamp reads %q", line)
}
// A walk that waited for its delivery's word counts its wait from when it was opened, and its tier
// from the word.
waiting := p
waiting.Delivery = &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}
waiting.Modules = map[string]*inventory.PlanModule{}
if line := planLine(waiting, now); !strings.Contains(line, "for 40m0s") || strings.Contains(line, "LATE") {
t.Errorf("a walk waiting 40m reads %q", line)
}
if st := planStatuses([]inventory.Plan{waiting}, now, pauseView{}, nil)[0]; st.Late {
t.Errorf("a waiting walk is called late: %+v", st)
}
let := now.Add(-5 * time.Minute)
gone := p
gone.Delivery = &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat, Go: &let}
if line := planLine(gone, now); !strings.Contains(line, "building for 5m0s") || strings.Contains(line, "LATE") {
t.Errorf("a walk let go 5m ago, opened 40m ago, reads %q", line)
}
}
// novox/hq issue 296: a plan whose tier stands still is not written again by every advance — its revision,
// its last save and its `plan-moved` stay where its last change left them.
func TestAPlanStandingStillIsNotSavedAgain(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := asksRecorded(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "app", Version: "1"},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/app", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "c1aaaaaaaa",
Paths: []string{"modules/app/index.ts"}, ModuleDirs: []string{"modules/app"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
recent, err := inv.RecentPlans(ctx, 1)
if err != nil || len(recent) != 1 || len(*asked) != 1 {
t.Fatalf("no plan building: %v %v, asked %v", recent, err, *asked)
}
before := recent[0]
if before.State != inventory.PlanBuilding || before.Modules["app"] == nil || before.Modules["app"].State != "asked" {
t.Fatalf("the plan is not waiting on its build: %+v", before)
}
for range 3 {
advanceHeld(ctx, open)
}
after, err := inv.PlanByID(ctx, before.ID)
if err != nil {
t.Fatal(err)
}
if after.Revision != before.Revision || !after.Updated.Equal(before.Updated) {
t.Fatalf("a plan that did not move was saved again: revision %d → %d, saved %s → %s",
before.Revision, after.Revision, before.Updated, after.Updated)
}
if len(*asked) != 1 {
t.Fatalf("advancing asked again: %v", *asked)
}
}
+6 -6
View File
@@ -276,21 +276,21 @@ func TestAPlanWaitingOnAPausedSeatSaysSoAndIsNotLate(t *testing.T) {
Modules: map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &asked}}}
all := pauseOf([]string{"g14", "ace"}, map[string]link.HolderState{"ace": {Paused: true}, "g14": {Paused: true}})
line := planLineWith(p, now, all)
line := planLineWith(p, now, all, tierAtLeast)
if !strings.Contains(line, "waiting: the build seat is paused on ace, g14 (asked 12m0s ago)") || strings.Contains(line, "LATE") {
t.Errorf("a plan on a paused seat reads %q", line)
}
if st := planStatuses([]inventory.Plan{p}, now, all)[0]; st.Late || !strings.Contains(st.Waiting, "paused on ace, g14") {
if st := planStatuses([]inventory.Plan{p}, now, all, nil)[0]; st.Late || !strings.Contains(st.Waiting, "paused on ace, g14") {
t.Errorf("status --json says %+v", st)
}
if _, late := openPlans([]inventory.Plan{p}, all); late != 0 {
if _, late := openPlans([]inventory.Plan{p}, now, all, nil); late != 0 {
t.Errorf("a plan waiting on a paused seat counted late")
}
longAgo := now.Add(-25 * time.Hour)
forgotten := p
forgotten.Modules = map[string]*inventory.PlanModule{"a": {State: "asked", AskedAt: &longAgo}}
if line := planLineWith(forgotten, now, all); !strings.Contains(line, "PAUSED OVER A DAY") || strings.Contains(line, "LATE") {
if line := planLineWith(forgotten, now, all, tierAtLeast); !strings.Contains(line, "PAUSED OVER A DAY") || strings.Contains(line, "LATE") {
t.Errorf("a plan paused over a day reads %q", line)
}
@@ -298,10 +298,10 @@ func TestAPlanWaitingOnAPausedSeatSaysSoAndIsNotLate(t *testing.T) {
if some.All || !reflect.DeepEqual(some.Nodes, []string{"ace"}) {
t.Fatalf("%+v", some)
}
if line := planLineWith(p, now, some); !strings.Contains(line, "LATE") {
if line := planLineWith(p, now, some, tierAtLeast); !strings.Contains(line, "LATE") {
t.Errorf("a seat paused on one holder of two made the plan not late: %q", line)
}
if st := planStatuses([]inventory.Plan{p}, now, some)[0]; !st.Late {
if st := planStatuses([]inventory.Plan{p}, now, some, nil)[0]; !st.Late {
t.Errorf("status --json: %+v", st)
}
// A plan rolling out, or with nothing asked, is not waiting on the seat.
+1 -1
View File
@@ -199,7 +199,7 @@ func statusAsJSON(asked answers) ([]byte, error) {
out := meshStatus{Machines: len(nodes), Wrong: []machineDoing{},
Quiet: []machineQuiet{}, Behind: []moduleBehind{}, Waiting: []machineWaiting{},
Reported: []machineReported{}, Unresolved: []machineUnresolved{},
Network: asked.network, Adopted: adoptedNodes(nodes), Plans: planStatuses(asked.plans, time.Now(), asked.paused)}
Network: asked.network, Adopted: adoptedNodes(nodes), Plans: planStatuses(asked.plans, time.Now(), asked.paused, asked.tierBounds)}
// In a stated order, so two readings of an unchanged mesh are the same document.
untakenNodes := make([]string, 0, len(asked.untaken))
for name := range asked.untaken {
+1 -1
View File
@@ -246,7 +246,7 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
}
}
if len(refused) > 0 {
return nil, fmt.Errorf("%w: %s — a release plan sends them, one machine at a time, each judged (novox/hq ADR "+
return nil, fmt.Errorf("%w: %s — a walk sends them, one machine at a time, each judged (novox/hq ADR "+
"0236); `upgrade backlog` lists them", errUngated, strings.Join(refused, "; "))
}
return kept, nil
+73 -20
View File
@@ -2,6 +2,7 @@ package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
@@ -506,23 +507,33 @@ func advanceHeld(ctx context.Context, open *stores) {
for i := range plans {
p := &plans[i]
for p.Open() {
// **Saved only when the step changed it** (novox/hq issue 296). Every step was saved, and
// the steps run on every tick and after every build outcome on the mesh: a plan whose tier
// stood still for half an hour was written every few seconds, its revision moved, `plan-moved`
// said on the bus, and its last save read as when it began building.
before := planSnapshot(*p)
moved, err := advanceOnce(ctx, open, p, edges, rollsOut)
if err != nil {
fmt.Printf("%s: %v\n", p.ID, err)
// Kept in the plan, so `plans` says why it has not moved rather than the log alone;
// the state is left as it was and the step is tried again on the next tick.
// the state is left as it was and the step is tried again on the next tick. Said and
// kept when it is new: the same refusal on every tick is one fact, not one per tick.
p.Note = "tier " + fmt.Sprint(p.Tier) + ": " + err.Error() + " — tried again"
if err := inv.SavePlan(ctx, p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
if planSnapshot(*p) != before {
fmt.Printf("%s: %v\n", p.ID, err)
if err := inv.SavePlan(ctx, p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
}
}
break
}
if p.State == inventory.PlanFailed {
sayUnsent(p, rollsOut)
}
if err := inv.SavePlan(ctx, p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
break
if planSnapshot(*p) != before {
if err := inv.SavePlan(ctx, p); err != nil {
fmt.Printf("%s: cannot keep the plan: %v\n", p.ID, err)
break
}
}
if !moved {
break
@@ -531,6 +542,18 @@ func advanceHeld(ctx context.Context, open *stores) {
}
}
// planSnapshot is a plan as a save would write it, to compare before and after a step: what the store
// stamps itself (when it was saved, its revision and epoch, when its tier was entered) is left out.
func planSnapshot(p inventory.Plan) string {
p.Updated, p.Revision, p.Epoch, p.TierEntered = time.Time{}, 0, 0, time.Time{}
b, err := json.Marshal(p)
if err != nil {
// Unreadable is never the same: the plan is saved, as every step was before.
return fmt.Sprintf("unmarshallable %p %v", &p, err)
}
return string(b)
}
// advanceOnce takes one step of one plan and says whether anything changed.
func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
edges []inventory.Edge, rollsOut func(string) bool) (bool, error) {
@@ -1070,12 +1093,37 @@ func planTicker(ctx context.Context, open *stores) {
}
}
// planLine is one plan as `status` says it.
func planLine(p inventory.Plan, now time.Time) string { return planLineWith(p, now, pauseView{}) }
// planLine is one plan as `status` says it, given the least bound of a tier.
func planLine(p inventory.Plan, now time.Time) string {
return planLineWith(p, now, pauseView{}, tierAtLeast)
}
// inTierSince is when a plan began waiting on what it waits on now, which `plans`, `status` and the
// watchdog of a plan's progress (S3) all count from (novox/hq issue 296): a walk waiting for its
// delivery's word from when it was opened, as S16 counts; any other plan from when it entered its tier,
// or, when the delivery let it go after that, from the go. Never from the plan's last save: a plan is
// saved for many reasons while its tier stands still, and a clock reset by each save said a build queued
// for minutes had been building for a few seconds. A plan read without the stamp (one from before it
// was kept) counts from its last save, the only time it has.
func inTierSince(p inventory.Plan) time.Time {
if p.Waiting() {
return p.Created
}
since := p.TierEntered
if since.IsZero() {
since = p.Updated
}
if d := p.Delivery; d != nil && d.Go != nil && d.Go.After(since) {
since = *d.Go
}
return since
}
// planLineWith is planLine knowing whether the build seat is paused (novox/hq ADR 0219): a plan
// waiting on builds nobody will take until a person resumes the seat says so, and is not late.
func planLineWith(p inventory.Plan, now time.Time, pause pauseView) string {
// waiting on builds nobody will take until a person resumes the seat says so, and is not late. bound is
// the plan's tier bound, the one its stalled condition is raised at (tierBounds): LATE is that condition
// said on the line (novox/hq issue 296).
func planLineWith(p inventory.Plan, now time.Time, pause pauseView, bound time.Duration) string {
where := fmt.Sprintf("tier %d of %d", min(p.Tier+1, len(p.Tiers)), len(p.Tiers))
switch p.State {
case inventory.PlanDone:
@@ -1085,7 +1133,7 @@ func planLineWith(p inventory.Plan, now time.Time, pause pauseView) string {
case inventory.PlanSuperseded:
return fmt.Sprintf("%s %s %s", p.Repository, short(p.Commit), p.Note)
}
since := now.Sub(p.Updated).Round(time.Second)
since := now.Sub(inTierSince(p)).Round(time.Second)
if p.Waiting() {
// Waiting for its delivery's word is no lateness of the walk's (novox/hq ADR 0239).
return fmt.Sprintf("%s %s %s, %s, for %s", p.Repository, short(p.Commit), where, waitingNote(p), since)
@@ -1094,7 +1142,7 @@ func planLineWith(p inventory.Plan, now time.Time, pause pauseView) string {
return fmt.Sprintf("%s %s %s, %s", p.Repository, short(p.Commit), where, waiting)
}
late := ""
if since > planWaitBound {
if since > bound {
late = " — LATE"
}
what := "building"
@@ -1167,17 +1215,19 @@ type planStatus struct {
Late bool `json:"late"`
}
func planStatuses(plans []inventory.Plan, now time.Time, pause pauseView) []planStatus {
func planStatuses(plans []inventory.Plan, now time.Time, pause pauseView, bounds tierBounds) []planStatus {
out := make([]planStatus, 0, len(plans))
for _, p := range plans {
ps := planStatus{ID: p.ID, Repository: p.Repository, Commit: p.Commit, State: p.State,
Tier: p.Tier, Tiers: len(p.Tiers), Since: p.Updated}
if p.Open() {
ps.Since = inTierSince(p)
ps.Waiting = p.Note
if ps.Waiting == "" {
ps.Waiting = "builds of tier " + fmt.Sprint(p.Tier)
}
ps.Late = now.Sub(p.Updated) > planWaitBound
// A walk waiting for its delivery's word is no tier late (ADR 0239): S16 is its watchdog.
ps.Late = !p.Waiting() && now.Sub(ps.Since) > bounds.of(p.Repository)
// Paused is a person's decision, not lateness (novox/hq ADR 0219).
if waiting, paused := pausedWaiting(p, pause, now); paused {
ps.Waiting, ps.Late = waiting, false
@@ -1189,16 +1239,16 @@ func planStatuses(plans []inventory.Plan, now time.Time, pause pauseView) []plan
}
// openPlans is the open plans among the recent ones, and how many have waited past the bound.
func openPlans(plans []inventory.Plan, pause pauseView) ([]inventory.Plan, int) {
func openPlans(plans []inventory.Plan, now time.Time, pause pauseView, bounds tierBounds) ([]inventory.Plan, int) {
var open []inventory.Plan
late := 0
for _, p := range plans {
if p.Open() {
open = append(open, p)
if _, paused := pausedWaiting(p, pause, time.Now()); paused {
if _, paused := pausedWaiting(p, pause, now); paused || p.Waiting() {
continue
}
if time.Since(p.Updated) > planWaitBound {
if now.Sub(inTierSince(p)) > bounds.of(p.Repository) {
late++
}
}
@@ -1240,7 +1290,9 @@ func plansCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
fmt.Printf("%s — %s\n", p.ID, planLineWith(p, now, buildSeatPause(ctx, inv, []inventory.Plan{p})))
bounds := readTierBounds(ctx, inv, now)
fmt.Printf("%s — %s\n", p.ID, planLineWith(p, now, buildSeatPause(ctx, inv, []inventory.Plan{p}),
bounds.of(p.Repository)))
if r := p.Release; r != nil {
// A release plan's walk (ADR 0236): machines done, the one judged, those to come.
fmt.Printf(" machines in order: %s; done: %s; skipped: %s\n", strings.Join(r.Order, ", "),
@@ -1369,8 +1421,9 @@ func plansCommand(ctx context.Context, args []string) error {
return nil
}
pause := buildSeatPause(ctx, inv, plans)
bounds := readTierBounds(ctx, inv, now)
for _, p := range plans {
fmt.Printf("%-28s %s\n", p.ID, planLineWith(p, now, pause))
fmt.Printf("%-28s %s\n", p.ID, planLineWith(p, now, pause, bounds.of(p.Repository)))
}
return nil
}
+5 -3
View File
@@ -143,14 +143,14 @@ func printStatus(asked answers) error {
len(quiet), strings.Join(said, "\n "))
}
if open, late := openPlans(asked.plans, asked.paused); len(open) > 0 {
if open, late := openPlans(asked.plans, time.Now(), asked.paused, asked.tierBounds); len(open) > 0 {
fmt.Printf("%d plan(s) open", len(open))
if late > 0 {
fmt.Printf(", %d waiting past %s", late, planWaitBound)
fmt.Printf(", %d in a tier past its bound", late)
}
fmt.Println(":")
for _, p := range open {
fmt.Printf(" %s\n", planLineWith(p, time.Now(), asked.paused))
fmt.Printf(" %s\n", planLineWith(p, time.Now(), asked.paused, asked.tierBounds.of(p.Repository)))
}
fmt.Println()
}
@@ -472,6 +472,8 @@ func theThreeQuestions(ctx context.Context, open *stores) (answers, error) {
}
// Whether the build seat takes work, for a plan waiting on it (novox/hq ADR 0219).
out.paused = buildSeatPause(ctx, inv, out.plans)
// And how long a tier may take, the bound a plan is late at (novox/hq issue 296).
out.tierBounds = readTierBounds(ctx, inv, time.Now())
// And how many repairs were done by hand this week (novox/hq to-be 45 §7) — where there is a bus
// to read the log from; a process with none has no log to count.
if _, onBus := broker.BusAddress(); onBus == nil {
+170
View File
@@ -0,0 +1,170 @@
[
{
"contract": 2,
"at": "2026-10-07T16:48:58.691388404Z",
"resources": [],
"network": {
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z",
"parts": [
{
"part": "resolv-conf",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "names",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "bus",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "route",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
}
]
}
},
{
"contract": 2,
"at": "2026-10-07T16:49:28.691388404Z",
"resources": [],
"network": {
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z",
"parts": [
{
"part": "resolv-conf",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "names",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "bus",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "route",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
}
]
}
},
{
"contract": 2,
"at": "2026-10-07T16:49:58.691388404Z",
"resources": [],
"network": {
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z",
"parts": [
{
"part": "resolv-conf",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z",
"streak": 1
},
{
"part": "names",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z",
"streak": 1
},
{
"part": "bus",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "route",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
}
]
}
},
{
"contract": 2,
"at": "2026-10-07T16:50:28.691388404Z",
"resources": [],
"network": {
"state": "unhealthy",
"since": "2026-10-07T16:50:28.691388404Z",
"parts": [
{
"part": "resolv-conf",
"state": "unhealthy",
"reason": "the resolver file was rewritten by another program",
"said": "/etc/resolv.conf differs from what networkmanager declares: it lists 192.0.2.53 where 10.77.0.2, 10.77.0.1 is declared; changed 2026-10-07T16:48:28Z; its own header names FortiClient",
"writer": "FortiClient",
"owner": "networkmanager",
"since": "2026-10-07T16:50:28.691388404Z",
"streak": 2
},
{
"part": "names",
"state": "unhealthy",
"reason": "neither mesh names nor public names resolve",
"said": "within 1s: 192.0.2.53 — anchor.internal (IPv4): no answer within 1s; anchor.internal (IPv6): no answer within 1s; example.com (IPv4): no answer within 1s",
"toward": [
"192.0.2.53"
],
"since": "2026-10-07T16:50:28.691388404Z",
"streak": 2
},
{
"part": "bus",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "route",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
}
]
}
},
{
"contract": 2,
"at": "2026-10-07T16:50:58.691388404Z",
"resources": [],
"network": {
"state": "healthy",
"since": "2026-10-07T16:50:58.691388404Z",
"parts": [
{
"part": "resolv-conf",
"state": "healthy",
"since": "2026-10-07T16:50:58.691388404Z"
},
{
"part": "names",
"state": "healthy",
"since": "2026-10-07T16:50:58.691388404Z"
},
{
"part": "bus",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
},
{
"part": "route",
"state": "healthy",
"since": "2026-10-07T16:48:58.691388404Z"
}
]
}
}
]
+50 -8
View File
@@ -439,13 +439,9 @@ func gatherPlans(ctx context.Context, inv *inventory.Inventory, now time.Time) (
if len(plans) == 0 {
return nil, nil, nil
}
tiers, err := inv.Durations(ctx, inventory.DurationPlanTier, now.Add(-14*24*time.Hour))
bounds, err := measuredTierBounds(ctx, inv, now)
if err != nil {
return nil, nil, fmt.Errorf("the measured plan tiers cannot be read: %w", err)
}
measured := map[string][]time.Duration{}
for _, d := range tiers {
measured[d.Subject] = append(measured[d.Subject], d.Took)
return nil, nil, err
}
pause := buildSeatPause(ctx, inv, plans)
var out []planFacts
@@ -459,13 +455,59 @@ func gatherPlans(ctx context.Context, inv *inventory.Inventory, now time.Time) (
continue
}
_, paused := pausedWaiting(p, pause, now)
bound := bounds.of(p.Repository)
out = append(out, planFacts{id: p.ID, repository: p.Repository, commit: p.Commit, tier: p.Tier,
tiers: len(p.Tiers), entered: p.TierEntered, bound: max(tierAtLeast, 3*p90(measured[p.Repository])),
waiting: planLineWith(p, now, pause), paused: paused})
tiers: len(p.Tiers), entered: inTierSince(p), bound: bound,
waiting: planLineWith(p, now, pause, bound), paused: paused})
}
return out, waits, nil
}
// tierBounds is how long a plan's tier may take before it is stalled (S3) and `plans` and `status` call
// it LATE, by repository: three times the ninetieth percentile of the tiers measured for it, and never
// less than tierAtLeast. One bound for the watchdog and the lines a person reads, so the two cannot
// disagree about one plan (novox/hq issue 296). A repository nothing was measured for, or a nil
// tierBounds, is given tierAtLeast.
type tierBounds map[string]time.Duration
func (b tierBounds) of(repository string) time.Duration {
if d, ok := b[repository]; ok {
return d
}
return tierAtLeast
}
// measuredTierBounds is the tier bounds from the last fourteen days of measured plan tiers.
func measuredTierBounds(ctx context.Context, inv *inventory.Inventory, now time.Time) (tierBounds, error) {
tiers, err := inv.Durations(ctx, inventory.DurationPlanTier, now.Add(-14*24*time.Hour))
if err != nil {
return nil, fmt.Errorf("the measured plan tiers cannot be read: %w", err)
}
return boundsOf(tiers), nil
}
// readTierBounds is measuredTierBounds for a line a person reads: what cannot be read is said, and
// every plan is then given the least bound, as the watchdog gives a repository nothing was measured for.
func readTierBounds(ctx context.Context, inv *inventory.Inventory, now time.Time) tierBounds {
bounds, err := measuredTierBounds(ctx, inv, now)
if err != nil {
fmt.Fprintf(os.Stderr, "%v; a plan is called late past %s\n", err, tierAtLeast)
}
return bounds
}
func boundsOf(tiers []inventory.Duration) tierBounds {
measured := map[string][]time.Duration{}
for _, d := range tiers {
measured[d.Subject] = append(measured[d.Subject], d.Took)
}
out := tierBounds{}
for repository, took := range measured {
out[repository] = max(tierAtLeast, 3*p90(took))
}
return out
}
// p90 is the ninetieth percentile of measurements; zero for none.
func p90(took []time.Duration) time.Duration {
if len(took) == 0 {
+11 -1
View File
@@ -9,7 +9,7 @@ package catalogue
// DeliverySeat is the seat mesh-delivery holds.
const DeliverySeat = "mesh-delivery"
// deliveryVerbs are the delivery seat's tools: five that read, and the acts of a person and of healer H2.
// deliveryVerbs are the delivery seat's tools: six that read, and the acts of a person and of healer H2.
func deliveryVerbs() []Verb {
return []Verb{
{Name: "deliveries", Description: "Every delivery not final, and those that ended in the last day, one line " +
@@ -32,6 +32,16 @@ func deliveryVerbs() []Verb {
Input: schema(map[string]string{"repository": "owner/repository",
"paths": "the files it changes, comma-separated, from the repository's root",
"base": "the branch it merges into (default main)"}, []string{"repository", "paths"})},
{Name: "checks", Description: "What the mesh's checks said of a pull request's head or of one commit: each " +
"of the commit's mesh statuses (mesh/merge-gate, mesh/repo-check, mesh/delivery, …) with its state, " +
"description and when it was set; the merge check's full verdict as the controller said it — each " +
"layer's summary, the machine that ran it, when, its build id and its report; and whether the branch's " +
"protection would let it merge, every required status being success.",
Input: schema(map[string]string{"repository": "owner/repository",
"number": "a pull request's number: its head is read",
"commit": "a commit's sha, or the start of one, instead of a pull request"}, []string{"repository"}),
// Added after the seat's first holder shipped: optional until mesh-delivery serves it everywhere.
Optional: true},
{Name: "table", Description: "The state table every delivery runs by: each transition with its guard, " +
"each state's bound and what healer H2 may do once it has passed; and the machine steps' table.",
Input: schema(map[string]string{}, nil)},
+60
View File
@@ -0,0 +1,60 @@
package catalogue
import (
"reflect"
"strings"
"testing"
)
// The delivery seat answers "what did the mesh's checks say of this pull request?" as a verb of its own,
// `checks` (novox/hq ADR 0239): read by repository and a pull request's number or a commit.
func TestTheDeliverySeatPromisesChecks(t *testing.T) {
seat, ok := SeatNamed(DeliverySeat)
if !ok {
t.Fatal("the delivery seat is not in the set")
}
var checks *Verb
for i := range seat.Serves {
if seat.Serves[i].Name == "checks" {
checks = &seat.Serves[i]
}
}
if checks == nil {
t.Fatalf("the delivery seat promises %v and not checks", VerbNames(seat.Serves))
}
props, _ := checks.Input["properties"].(map[string]any)
for _, arg := range []string{"repository", "number", "commit"} {
if _, has := props[arg]; !has {
t.Errorf("checks takes no %q", arg)
}
}
if req, _ := checks.Input["required"].([]string); !reflect.DeepEqual(req, []string{"repository"}) {
t.Errorf("checks requires %v, wanted only the repository", req)
}
// Added after the seat's holder shipped, it is optional: the holder serving the ten verbs before it still
// holds the seat, and one serving all eleven does too — so the controller and the catalogue can move in
// either order, and no check of the catalogue fails on a module nobody touched between the two.
if !checks.Optional {
t.Fatal("checks is a condition of holding before its holder serves it")
}
var before []string
for _, v := range VerbNames(seat.Serves) {
if v != "checks" {
before = append(before, v)
}
}
m := Manifest{Module: "mesh-delivery", Claims: []Claim{{Name: DeliverySeat, Scope: ScopeMesh, Serves: before}}}
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder without checks yet: %v", err)
}
m.Claims[0].Serves = VerbNames(seat.Serves)
if err := CanHold(m, seat); err != nil {
t.Fatalf("a holder serving every verb: %v", err)
}
// Every other verb is still a condition of holding.
m.Claims[0].Serves = append([]string{"checks"}, before[1:]...)
if err := CanHold(m, seat); err == nil || !strings.Contains(err.Error(), before[0]) {
t.Fatalf("a holder without %s: %v", before[0], err)
}
}
+9 -1
View File
@@ -22,6 +22,14 @@ type Verb struct {
Description string `json:"description,omitempty"`
Input map[string]any `json:"input,omitempty"`
Output map[string]any `json:"output,omitempty"`
// Optional marks a verb added to a mesh seat whose holder lives in another repository (design 33 §7,
// additive within a version): a holder that serves it is accepted, and one that does not yet still
// holds the seat. Without it the addition would be a deadlock — this controller refusing the holder that
// does not serve the verb, the controller before it refusing the holder that does — and every check of
// the catalogue between the two would fail on a module nobody touched. Once every holder serves it,
// the mark is removed and the verb is a condition of holding like the rest. Never stored or said: the
// seat set's protocol is the compiled one.
Optional bool `json:"-"`
}
func (v *Verb) UnmarshalJSON(raw []byte) error {
@@ -442,7 +450,7 @@ func unservedVerbs(tools []string, promised []Verb) []string {
}
var missing []string
for _, v := range promised {
if !has[v.Name] {
if !has[v.Name] && !v.Optional {
missing = append(missing, v.Name)
}
}
+42 -7
View File
@@ -40,6 +40,29 @@ type NodeHealth struct {
Resources []ResourceHealth
// Streaks is, per module, how many statements in a row said a resource of it was unhealthy.
Streaks map[string]int
// Network is the machine's own networking as its engine said it (novox/hq ADR 0241); nil from an
// engine older than that judging.
Network *NetworkHealth
}
// NetworkHealth is a machine's networking as its engine said it (ADR 0241).
type NetworkHealth struct {
State string `json:"state"`
Since time.Time `json:"since"`
Parts []NetworkPart `json:"parts"`
}
// NetworkPart is one part of it: resolv-conf, names, tunnel, bus or route.
type NetworkPart struct {
Part string `json:"part"`
State string `json:"state"`
Reason string `json:"reason,omitempty"`
Said string `json:"said,omitempty"`
Writer string `json:"writer,omitempty"`
Owner string `json:"owner,omitempty"`
Toward []string `json:"toward,omitempty"`
Since time.Time `json:"since"`
Streak int `json:"streak,omitempty"`
}
// HealthOf is a machine's newest statement; false when its node-engine has never stated one.
@@ -60,7 +83,7 @@ func (i *Inventory) Healths(ctx context.Context) (map[string]NodeHealth, error)
func (i *Inventory) healths(ctx context.Context, only string) (map[string]NodeHealth, error) {
rows, err := i.store.Pool().Query(ctx,
`select n.name, h.contract, h.said_at, h.heard_at, h.resources, h.streaks
`select n.name, h.contract, h.said_at, h.heard_at, h.resources, h.streaks, h.network
from node_health h join node n on n.id = h.node
where $1 = '' or n.name = $1`, only)
if err != nil {
@@ -70,8 +93,8 @@ func (i *Inventory) healths(ctx context.Context, only string) (map[string]NodeHe
out := map[string]NodeHealth{}
for rows.Next() {
var h NodeHealth
var resources, streaks []byte
if err := rows.Scan(&h.Node, &h.Contract, &h.SaidAt, &h.HeardAt, &resources, &streaks); err != nil {
var resources, streaks, network []byte
if err := rows.Scan(&h.Node, &h.Contract, &h.SaidAt, &h.HeardAt, &resources, &streaks, &network); err != nil {
return nil, err
}
if err := json.Unmarshal(resources, &h.Resources); err != nil {
@@ -80,6 +103,11 @@ func (i *Inventory) healths(ctx context.Context, only string) (map[string]NodeHe
if err := json.Unmarshal(streaks, &h.Streaks); err != nil {
return nil, fmt.Errorf("%s's health cannot be read: %w", h.Node, err)
}
if len(network) > 0 {
if err := json.Unmarshal(network, &h.Network); err != nil {
return nil, fmt.Errorf("%s's network health cannot be read: %w", h.Node, err)
}
}
out[h.Node] = h
}
return out, rows.Err()
@@ -102,18 +130,25 @@ func (i *Inventory) RecordHealth(ctx context.Context, h NodeHealth) (bool, error
if err != nil {
return false, err
}
var network []byte
if h.Network != nil {
if network, err = json.Marshal(h.Network); err != nil {
return false, err
}
}
heard := h.HeardAt
if heard.IsZero() {
heard = time.Now()
}
var node string
err = i.store.Pool().QueryRow(ctx,
`insert into node_health (node, contract, said_at, heard_at, resources, streaks)
select id, $2, $3, $4, $5, $6 from node where name = $1
`insert into node_health (node, contract, said_at, heard_at, resources, streaks, network)
select id, $2, $3, $4, $5, $6, $7 from node where name = $1
on conflict (node) do update set contract = excluded.contract, said_at = excluded.said_at,
heard_at = excluded.heard_at, resources = excluded.resources, streaks = excluded.streaks
heard_at = excluded.heard_at, resources = excluded.resources, streaks = excluded.streaks,
network = excluded.network
where node_health.said_at <= excluded.said_at
returning node`, h.Node, h.Contract, h.SaidAt, heard, resources, streaks).Scan(&node)
returning node`, h.Node, h.Contract, h.SaidAt, heard, resources, streaks, network).Scan(&node)
if errors.Is(err, pgx.ErrNoRows) {
if _, nerr := i.NodeByName(ctx, h.Node); nerr != nil {
return false, nerr
@@ -0,0 +1,10 @@
-- A machine says how its network is (novox/hq ADR 0241, which extends ADR 0240 from what a module runs to
-- the machine it runs on).
--
-- Beside the state of every long-running resource, each machine's node-engine states its own networking:
-- the resolver file the uplink holder declared and who rewrote it, the names through every resolver it
-- lists, the tunnel's handshake with the hub, the bus and the default route — the worst of them, since
-- when, and each part. Kept with the machine's newest statement, replaced with it, so `node show`, the
-- gate and a controller started again read the same word. Null for a machine whose node-engine is older
-- than this judging: its network is not known — never healthy, never a reason to raise anything.
alter table node_health add column network jsonb;
+39
View File
@@ -428,6 +428,45 @@ type Health struct {
// At is when the engine looked, on the machine's clock: the order of its statements.
At time.Time `json:"at"`
Resources []ResourceHealth `json:"resources"`
// Network is the machine's own networking, judged by its engine (novox/hq ADR 0241); nil from an engine
// older than that judging, which is "not known", never healthy.
Network *NetworkHealth `json:"network,omitempty"`
}
// NetworkHealth is a machine's networking in one statement (ADR 0241): the worst of its parts, since
// when, and each part. The node-engine's own (mesh-host internal/link NetworkHealth).
type NetworkHealth struct {
State string `json:"state"`
Since time.Time `json:"since"`
Parts []NetworkPart `json:"parts"`
}
// The parts of a machine's networking its engine judges (ADR 0241).
const (
PartResolvConf = "resolv-conf"
PartNames = "names"
PartTunnel = "tunnel"
PartBus = "bus"
PartRoute = "route"
// TowardHub is what a part failing toward the hub names in Toward.
TowardHub = "hub"
)
// NetworkPart is one part, as the engine judged it on its second look.
type NetworkPart struct {
Part string `json:"part"`
State string `json:"state"`
// Reason is in words with no address, path or domain; Said is the detail, kept as evidence.
Reason string `json:"reason,omitempty"`
Said string `json:"said,omitempty"`
// Writer is the program that rewrote the resolver file, when the engine could name it; Owner the
// module whose file it is.
Writer string `json:"writer,omitempty"`
Owner string `json:"owner,omitempty"`
// Toward is what a failure points at: "hub", or each resolver's address that failed.
Toward []string `json:"toward,omitempty"`
Since time.Time `json:"since"`
Streak int `json:"streak,omitempty"`
}
// The states a resource is said in (ADR 0240 §4).
+7
View File
@@ -33,6 +33,13 @@ func TestTheWireFormatIsExactlyTheseFieldNames(t *testing.T) {
[]string{"module", "resource", "kind", "target", "state", "reason", "since", "streak", "restarts",
"check", "needs"}},
{HealthSaid{Node: "n"}, []string{"node", "health"}},
// novox/hq ADR 0241: the machine's own networking, beside its resources.
{Health{Contract: ReadinessContract, Resources: []ResourceHealth{}, Network: &NetworkHealth{State: "unhealthy",
Parts: []NetworkPart{}}}, []string{"contract", "at", "resources", "network"}},
{NetworkHealth{State: "unhealthy", Parts: []NetworkPart{}}, []string{"state", "since", "parts"}},
{NetworkPart{Part: "resolv-conf", State: "unhealthy", Reason: "r", Said: "s", Writer: "w", Owner: "o",
Toward: []string{"hub"}, Streak: 2}, []string{"part", "state", "reason", "said", "writer", "owner", "toward",
"since", "streak"}},
} {
raw, err := json.Marshal(c.value)
if err != nil {