Author SHA1 Message Date
jochen 4291fee68e Hold a consumer's findings under its unhealthy provider, and say them once there (hq ADR 0240, to-be 48 Phase C)
mesh/merge-gate the mesh is mapping this head onto its module graph
mesh/delivery checking
With twelve consumers of the database provision, one provider down would be
twelve conditions for one fault and twelve gates failed for something none of
them did. A consumer's check names the provision it exercises; while the
provider composed for it — its recorded binding, or the machine its credential
comes from — is unhealthy on the record, what that check finds raises nothing
of its own: the provider's condition lists it as waiting and is urgent, and
the consumer's gate waits, past its bound too, rather than putting a build
back. Liveness findings and checks naming no provision stay the consumer's own,
and once the provider is healthy a consumer still failing is raised at once.
2026-10-07 16:17:52 +02:00
jochen b98fd0f396 Read how a module says each resource is ready, and send it to engines that read it (hq ADR 0240, to-be 48 Phase B)
mesh/merge-gate the mesh is mapping this head onto its module graph
mesh/delivery checking
A module could say nothing about what ready means for what it runs, so a web
application with its port open and its requests hanging passed everything for
eleven hours (issue 145). A long-running resource now carries `health` — the
image's own check adopted by name, http, tcp, exec, unit or a module's own tool,
with its timing — refused near its author when it names a port or an address,
an endpoint the module does not declare, a tool it does not serve, a tool check
alone, or a timing outside the record's bounds. It is composed with the endpoint
as the port this machine published it on, and sent only to a node-engine whose
statement says it reads it: an older one would refuse the whole declaration.
The engine is granted its own machine's instance of each health tool. `module
check` warns of every long-running resource without `health`, counts them for
the catalogue, and refuses them from 2026-11-18. A check's findings stay out of
a condition's summary. The node-engine's validator is vendored at its Phase B
commit, so what is composed is judged by the words the engine takes.
2026-10-07 16:17:50 +02:00
mesh-admin 863ebd4277 Merge pull request 'A drill is recorded through its own verb, and S15 never counts it (hq issue 292)' (#109) from fix/a-drill-is-no-repair into main 2026-10-07 12:05:40 +00:00
jochen 2799e95035 Record a drill through its own verb, so S15 never counts a deliberate test as a repair
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
Two ADR 0240 drills recorded with hand-act record --cause drill raised
mesh.hand-acts.drill.healer-wanted. hand-act drill (seat verb drill) records
them as a person's decision; hand-act record now refuses the cause, so the
two recorded before it clear on the next tick and a repair cannot pass for a
drill by the word it gives.
2026-10-07 13:55:31 +02:00
mesh-admin 582f4a082a Merge pull request 'Run a verb from the controller's own running image; refuse what it cannot run as a handover (hq issue 289)' (#108) from fix/a-verb-survives-the-handover into main 2026-10-07 01:05:42 +00:00
jochen 6c7af5c63f Run a verb from the controller's own running image, and refuse what it cannot run as a handover (hq issue 289)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
The witness moves a running build aside into a directory the controller's user
cannot enter, then deletes it; a verb exec'd from os.Executable() in that window
failed with permission denied. /proc/self/exe stays valid while the process lives.
A verb that still cannot start, or arrives while the controller stops, is refused
with link.ErrHandingOver and marked retry: handing-over.
2026-10-07 02:45:09 +02:00
mesh-admin 9d15f3a39e Merge pull request 'Keep what each machine says of what it runs, raise it, and gate on it (hq ADR 0240, to-be 48 Phase A)' (#107) from feat/a-module-says-how-it-is-healthy into main 2026-10-07 00:45:02 +00:00
jochen 725fcd977e Hold a dotted module on its own health condition at the gate
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
mesh/delivery-group group feat/a-module-says-how-it-is-healthy delivered: every member is delivered
2026-10-07 02:28:16 +02:00
jochen 1cc6a2d759 Keep what each machine says of what it runs, raise it, and gate on it (hq ADR 0240, to-be 48 Phase A)
The gate judged a module by what the mesh saw from outside, so a container that
crash-looped after it applied passed it. Each machine's node-engine now states
the health of every long-running resource it runs; the controller keeps the
newest statement per machine, raises module.<module>.<machine>.unhealthy on the
second statement in a row, clears it on the first that does not say it, and the
gate passes a module only when every long-running resource of it is stated
healthy since the send. An engine that states nothing is judged as before.
2026-10-07 02:28:16 +02:00
mesh-admin 5d3e52219b Merge pull request 'The seat calls a gate that raised no machine the mesh composes an error, whatever judged it (hq issue 285)' (#106) from fix/seat-refuses-a-gate-that-raised-nothing into main 2026-10-07 00:28:00 +00:00
mesh-admin 099c176fa9 Merge pull request 'Facts: name a repository owner/repository, without the forge's address (hq issue 288)' (#105) from fix/facts-name-repositories-without-the-forge into main 2026-10-07 00:27:50 +00:00
jochen ec3769a8a6 Check again: the first check was redelivered mid-run and collided with its own store
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery held for a person: merged, and the controller opened no walk for it within 10m0s — nothing it holds follows that branch, or the merge was…
2026-10-07 02:17:48 +02:00
jochen 8b2abd08cd Remove what an earlier delivery of a check left before raising its store again
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
An ask redelivered after the build agent stopped mid-check (the rollout it was checking updated it)
found its own throwaway store under its name, and the check said it could not run (mesh-controller#105,
build-1791331512096605198).
2026-10-07 02:17:46 +02:00
jochen 858b4672dd The seat calls a gate that raised no machine the mesh composes an error, whatever judged it
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A check asked by the controller before its own build of the issue 285 fix was registered was judged by
the controller the mesh ran then, which passed 0 of 4 composing. The judge is the running controller by
design, so the rule is read where the verdict is taken too: from the machines the verdict lists.
2026-10-07 02:06:49 +02:00
jochen 3d7ccc8aeb Name a repository in the facts as owner/repository, without the forge's address
mesh/merge-gate error: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; the check could not run: a throwaway postgr…
mesh/repo-check error: the check could not run: a throwaway postgres:17-alpine could not be raised: docker run --label mesh.build=build-1791331512096605198…
mesh/delivery superseded: a newer head of the same pull request
The snapshot promises no address, and every module's repository, reads and sources carried the URL the
mesh clones from. A check matches repositories by owner and name, so nothing it reads is lost (novox/hq
issue 288).
2026-10-07 02:04:35 +02:00
mesh-admin b39eaa485a Merge pull request 'The gate raises the mesh as it is, and a baseline that does not compose is an error; check-here runs a check as the seat does (hq issues 282, 283)' (#104) from fix/gate-baseline-composes into main 2026-10-06 23:59:43 +00:00
60 changed files with 3263 additions and 45 deletions
+46
View File
@@ -8,6 +8,7 @@ import (
"path/filepath"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
@@ -107,6 +108,28 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
names = append(names, name)
}
sort.Strings(names)
// **Every long-running resource says how it is ready** (novox/hq ADR 0240 rule 8): warned until the
// date, refused from it. The count is the catalogue's: its merge check keeps the number and lets a
// change lower it, never raise it.
undeclared := 0
required := !checkNow().Before(catalogue.HealthRequiredFrom)
for _, name := range names {
missing := catalogue.Undeclared(shelf[name])
undeclared += len(missing)
if len(missing) == 0 {
continue
}
if required {
for _, id := range missing {
fmt.Fprintf(out, "%s: %s stays up and does not say how it is ready: a long-running resource declares "+
"health since %s (novox/hq ADR 0240 rule 8)\n", name, id, catalogue.HealthRequiredFrom.Format("2006-01-02"))
}
failed += len(missing)
faulted[name] = true
}
}
for _, name := range names {
m := shelf[name]
if faulted[name] {
@@ -138,8 +161,24 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
fmt.Fprintf(out, ", keeps %s", strings.Join(kept, ", "))
}
// How what it runs is ready (ADR 0240): each declared check, and what is judged by liveness alone.
var checks []string
for _, r := range m.Resources {
if h, has, _ := catalogue.ReadHealth(r); has {
checks = append(checks, fmt.Sprintf("%v by %s", r["id"], catalogue.HealthWords(h)))
}
}
if len(checks) > 0 {
fmt.Fprintf(out, ", ready: %s", strings.Join(checks, "; "))
}
if missing := catalogue.Undeclared(m); len(missing) > 0 {
fmt.Fprintf(out, "; WARNING: %s stay(s) up and say(s) not how it is ready — judged by liveness alone, "+
"refused from %s (ADR 0240 rule 8)", strings.Join(missing, ", "), catalogue.HealthRequiredFrom.Format("2006-01-02"))
}
fmt.Fprintln(out)
}
// The count the catalogue keeps (ADR 0240 rule 8), in a line its merge check reads.
fmt.Fprintf(out, "%s %d\n", UndeclaredHealthLine, undeclared)
if failed > 0 {
return fmt.Errorf("%d problem(s) in %d manifest(s)", failed, len(paths))
}
@@ -150,6 +189,13 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
return nil
}
// UndeclaredHealthLine starts the line `module check` says the count of long-running resources without
// `health` in, over the manifests given: the catalogue's merge check compares it with the number it keeps.
const UndeclaredHealthLine = "long-running resources without health:"
// checkNow is the clock `module check` judges the date by; a test sets it.
var checkNow = time.Now
func joinInvokes(invokes []string) string {
if len(invokes) == 1 && invokes[0] == "*" {
return "every tool"
+2 -2
View File
@@ -17,7 +17,7 @@ import (
)
// `check-here` is a pull request's merge check run on the machine at hand **exactly as the build seat
// runs it** (novox/hq issue 286): the same code (builder.Check), the same ask the controller would make of
// runs it** (novox/hq issue 283): the same code (builder.Check), the same ask the controller would make of
// the seat — the gate's modules and judge by the planner's own answer over the facts snapshot, the
// repositories beside it at the refs the snapshot says the seat clones them at — in the toolchain image
// the mesh holds, as the user the build seat runs as, against a throwaway store and bus of the versions
@@ -126,7 +126,7 @@ func checkHereCommand(ctx context.Context, args []string) error {
}
if len(toolchains) == 0 {
return errors.New("the facts snapshot names no toolchain: it was taken by a controller from before " +
"issue 286, and the seat's toolchain cannot be known here")
"issue 283, and the seat's toolchain cannot be known here")
}
beside := map[string]builder.Beside{}
for d, ref := range f.Beside {
+4 -3
View File
@@ -406,11 +406,11 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
if err != nil {
return snapshot.Facts{}, err
}
mod := snapshot.Module{Name: e.Manifest.Module, Repository: e.Source.Repository, Path: e.Source.Path,
mod := snapshot.Module{Name: e.Manifest.Module, Repository: snapshot.RepositoryName(e.Source.Repository), Path: e.Source.Path,
Commit: e.Source.BuiltFrom, Provided: e.Provided, RollOut: current[e.Manifest.Module].RollOut,
Manifest: raw}
for _, r := range read[e.Manifest.Module] {
mod.Reads = append(mod.Reads, r.Repository)
mod.Reads = append(mod.Reads, snapshot.RepositoryName(r.Repository))
}
f.Modules = append(f.Modules, mod)
if e.Provided || e.Source.Repository == "" {
@@ -426,7 +426,8 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
if commit == "" {
commit = s.BuiltFrom
}
f.Sources = append(f.Sources, snapshot.Source{Repository: repository, Commit: commit, Modules: count[repository]})
f.Sources = append(f.Sources, snapshot.Source{Repository: snapshot.RepositoryName(repository), Commit: commit,
Modules: count[repository]})
}
for _, e := range edges {
f.Edges = append(f.Edges, snapshot.Edge{From: e.From, To: e.To, Kind: e.Kind})
+36 -1
View File
@@ -77,6 +77,10 @@ type health int
const (
healthGood health = iota
// healthWaiting is not a pass and not a fault: what the module's checks find waits on an unhealthy
// provider (ADR 0240 rule 5), so the judging waits — past the bound too — rather than putting back a
// build for something it did not do.
healthWaiting
healthNotYet
healthBroken
)
@@ -105,6 +109,12 @@ type gateFacts struct {
// holder is who holds the controller lease, for judging the controller.
holder *lease.Holder
holderErr error
// health is each machine's newest health statement (ADR 0240); a machine absent never stated one.
// healthErr is why they could not be read.
health map[string]inventory.NodeHealth
healthErr error
// heldOn is, per "<module>@<machine>", the provider its findings are held under (ADR 0240 rule 5).
heldOn map[string]string
}
// gatherGateFacts reads what a judging needs, from the store, the bus and this controller's memory. A
@@ -127,6 +137,9 @@ var gatherGateFacts = func(ctx context.Context, open *stores, component string)
for _, n := range nodes {
f.engines[n.Name] = n.HostVersion
}
// What each machine says of its long-running resources (ADR 0240): unreadable is said, never read as
// healthy.
f.health, f.healthErr = inv.Healths(ctx)
if d := doctorFrom; d != nil {
if d.keeper != nil {
f.judged = true
@@ -140,6 +153,17 @@ var gatherGateFacts = func(ctx context.Context, open *stores, component string)
} else {
f.servedErr = errors.New("this process does not serve the mesh, so it cannot ask the bus who serves what")
}
// Whose findings wait on an unhealthy provider (ADR 0240 rule 5): their gates wait, not fail.
if f.healthErr == nil && f.openErr == nil {
if hold, err := readHolding(ctx, inv, f.open); err == nil {
f.heldOn = map[string]string{}
for machine := range f.health {
for module, p := range hold.heldModules(machine) {
f.heldOn[module+"@"+machine] = p.Module + " on " + p.Node
}
}
}
}
if theLease != nil {
h, found, err := theLease.holder(ctx)
switch {
@@ -193,7 +217,9 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
if !onIt {
continue
}
if c.Subject.Scope == conditions.ScopeMachine || slices.Contains(strings.Split(c.Subject.ID, "."), module) {
// A module's own health condition names it whole, its name's dots and all (ADR 0240).
ownHealth := c.Subject.Scope == conditions.ScopeModule && c.Subject.ID == module+"."+machine
if c.Subject.Scope == conditions.ScopeMachine || ownHealth || slices.Contains(strings.Split(c.Subject.ID, "."), module) {
return healthNotYet, fmt.Sprintf("raised since it was sent: %s — %s", c.Key, c.Summary)
}
}
@@ -241,6 +267,11 @@ func judgeHealth(module, component string, m catalogue.Manifest, machine string,
return healthNotYet, fmt.Sprintf("the node tools on %s do not serve %s's tools", machine, module)
}
}
// **And what it runs is stated healthy** (ADR 0240 §4): every long-running resource of it on that
// machine, in a statement heard since the send. A resource still starting makes the judging wait.
if h, why := moduleHealthWord(module, machine, since, f); h != healthGood {
return h, why
}
}
return healthGood, ""
}
@@ -421,6 +452,10 @@ func judgeMoves(ctx context.Context, open *stores, g *inventory.PlanGate, pairs
// What broke is put back; what was only not yet healthy beside it is too — they moved together.
g.Failing = failing
decide(g, inventory.GateFailed, why, now)
case worst == healthWaiting:
// Waiting on a provider that is unhealthy: not a pass, and not a failure at the bound either —
// the provider's own condition says what is wrong (ADR 0240 rule 5).
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
case worst == healthNotYet:
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, why, failing
if now.Sub(*g.Since) > gateBound {
+42
View File
@@ -346,6 +346,48 @@ func TestTheHealthDefinitions(t *testing.T) {
!strings.Contains(why, "first run") {
t.Errorf("a controller not ready is %v (%s)", h, why)
}
// What the module runs (novox/hq ADR 0240 §4): a judging passes only when every long-running
// resource of it on that machine is stated healthy since the send; starting and unhealthy wait.
f = applied("anchor")
stated := func(state, reason string, heard time.Time) {
f.health = map[string]inventory.NodeHealth{"anchor": {Node: "anchor", Contract: 1, SaidAt: heard, HeardAt: heard,
Resources: []inventory.ResourceHealth{
{Module: "app", Resource: "app.server", Kind: "container", Target: "app-server", State: state, Reason: reason},
{Module: "other", Resource: "other.server", Kind: "container", State: link.StateUnhealthy, Reason: "down"}}}}
}
plain := catalogue.Manifest{Module: "app"}
if h, why := judgeHealth("app", "", plain, "anchor", since, f); h != healthGood {
t.Errorf("a machine whose engine states no health is judged as before, not %v (%s)", h, why)
}
stated(link.StateStarting, "", now)
if h, why := judgeHealth("app", "", plain, "anchor", since, f); h != healthNotYet || !strings.Contains(why, "starting") {
t.Errorf("a resource still starting is not yet a pass: %v (%s)", h, why)
}
stated(link.StateUnhealthy, "restarting", now)
if h, why := judgeHealth("app", "", plain, "anchor", since, f); h != healthNotYet || !strings.Contains(why, "restarting") {
t.Errorf("a resource unhealthy fails the judging: %v (%s)", h, why)
}
stated(link.StateHealthy, "", since.Add(-time.Minute))
if h, why := judgeHealth("app", "", plain, "anchor", since, f); h != healthNotYet {
t.Errorf("a statement from before the send says nothing of the new build: %v (%s)", h, why)
}
stated(link.StateHealthy, "", now)
if h, why := judgeHealth("app", "", plain, "anchor", since, f); h != healthGood {
t.Errorf("every resource of it healthy since the send — another module's state is not its — is %v (%s)", h, why)
}
f.healthErr = errors.New("the store is away")
if h, _ := judgeHealth("app", "", plain, "anchor", since, f); h != healthNotYet {
t.Errorf("health that cannot be read is never read as healthy: %v", h)
}
// And the condition it raises after the send holds it, as every condition about it does.
f.healthErr = nil
f.judged = true
f.open = []conditions.Condition{{Key: "module.app.anchor.unhealthy", Kind: kindModuleUnhealthy, Subject: conditions.Subject{
Scope: conditions.ScopeModule, ID: "app.anchor", Machine: "anchor"}, Raised: now, Summary: "app on anchor is not healthy"}}
if h, why := judgeHealth("app", "", plain, "anchor", since, f); h != healthNotYet || !strings.Contains(why, "module.app.anchor.unhealthy") {
t.Errorf("a module held on its own unhealthy condition is %v (%s)", h, why)
}
// A machine that refused what it was sent: broken.
f = applied("anchor")
r := f.reports["anchor"]
+54 -3
View File
@@ -41,6 +41,11 @@ type handActVerb struct {
// causeLeakedInLogs is the cause a rotation after a value was printed into a log gives.
const causeLeakedInLogs = "leaked-in-logs"
// causeDrill is the cause of every act `hand-act drill` records, and the one cause `hand-act record`
// refuses: a drill has its own verb, so whether an act was a drill is said by the verb a person chose,
// never by a word typed into a repair's cause (novox/hq issue 292).
const causeDrill = "drill"
// handActVerbs is every verb that writes the hand-act log (novox/hq to-be 45 §7). **S15 reads it**:
// an act recorded by a verb whose entry names a decision is the mesh working as decided, never a
// repair, and does not count toward `healer-wanted` — whatever cause it gives. A verb not listed, or
@@ -57,8 +62,14 @@ var handActVerbs = []handActVerb{
{Verb: "broker consumer-reset"},
// Silencing the same condition twice says the condition, or what it watches, wants mending.
{Verb: "conditions silence"},
// An act done outside the mesh: the mesh cannot tell a repair from a decision there, so it counts.
{Verb: "hand-act record"},
// An act done outside the mesh: the mesh cannot tell a repair from a decision there, so it counts —
// except a drill recorded through it before `hand-act drill` existed (2026-10-07). It refuses the
// cause since, so no act recorded through it now carries it.
{Verb: "hand-act record", Decision: "a drill recorded before `hand-act drill` existed: a person's " +
"deliberate test, never a repair", DecidedFor: []string{causeDrill}},
// A drill: something broken on purpose to see the mesh raise and clear it. A person's test, never a
// repair, however often it is run.
{Verb: "hand-act drill", Decision: "a drill is a person's deliberate test of the mesh, never a repair"},
// A person's decisions by design.
{Verb: "retire approve", Decision: "nothing is retired past its bound without a person (ADR 0230)"},
{Verb: "retire reject", Decision: "keeping a consumer active is a person's word (ADR 0230)"},
@@ -174,6 +185,10 @@ func handActCommand(ctx context.Context, args []string) error {
return errors.New("hand-act record says the cause too: --cause <word>, the word a second " +
"act for the same reason will use — it is how a repair done twice is found")
}
if strings.EqualFold(strings.TrimSpace(*f.cause), causeDrill) {
return errors.New("a drill is not recorded as a repair: hand-act drill <what was done> --why <text> " +
"records it as the person's deliberate test it is, which no healer is wanted for. Nothing was recorded")
}
act := link.HandAct{Verb: "hand-act record", Args: []string{what}, Why: strings.TrimSpace(*f.why),
Cause: strings.TrimSpace(*f.cause), Condition: strings.TrimSpace(*f.condition)}
return onTheBus(func(conn *nats.Conn) error {
@@ -186,8 +201,12 @@ func handActCommand(ctx context.Context, args []string) error {
return nil
})
}
if len(args) > 0 && args[0] == "drill" {
return handActDrill(ctx, args[1:])
}
if len(args) > 0 && args[0] != "list" && !strings.HasPrefix(args[0], "-") {
return errors.New("hand-act record <what> --why <text> --cause <word> | hand-acts [--days N] [--json]")
return errors.New("hand-act record <what> --why <text> --cause <word> | hand-act drill <what> --why <text> " +
"| hand-acts [--days N] [--json]")
}
if len(args) > 0 && args[0] == "list" {
args = args[1:]
@@ -239,6 +258,38 @@ func handActCommand(ctx context.Context, args []string) error {
})
}
// handActDrill is `hand-act drill`: an act done on purpose to test the mesh — a module stopped, a
// process killed — recorded so the conditions it raises are read as the drill they are. Its cause is
// always causeDrill and S15 never counts it (handActVerbs).
func handActDrill(ctx context.Context, args []string) error {
set := flag.NewFlagSet("hand-act drill", flag.ContinueOnError)
why := set.String("why", "", "what the drill tests — recorded in the hand-act log (novox/hq to-be 45 §7)")
condition := set.String("condition", "", "the key of the condition the drill is meant to raise, if any")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
what := strings.TrimSpace(strings.Join(positionals, " "))
if what == "" {
return errors.New("hand-act drill <what was done on purpose> --why <what it tests> [--condition <key>]")
}
if strings.TrimSpace(*why) == "" {
return errors.New("a drill says what it tests: --why <text> (recorded in the hand-act log, novox/hq " +
"to-be 45 §7). Nothing was recorded")
}
act := link.HandAct{Verb: "hand-act drill", Args: []string{what}, Why: strings.TrimSpace(*why),
Cause: causeDrill, Condition: strings.TrimSpace(*condition)}
return onTheBus(func(conn *nats.Conn) error {
written, err := link.RecordHandAct(ctx, conn, act)
if err != nil {
return fmt.Errorf("the drill could not be recorded: %w", err)
}
fmt.Printf("recorded as %s: %s drilled %q, because %q — a drill, which no healer is wanted for\n",
written.ID, written.By, what, written.Why)
return nil
})
}
// repairs are the acts that are not a person's decision by design: what S15 counts.
func repairs(acts []link.HandAct) []link.HandAct {
out := make([]link.HandAct, 0, len(acts))
+29
View File
@@ -92,3 +92,32 @@ func TestDurationsAreSummarisedPerSubject(t *testing.T) {
t.Fatalf("a minute between words suggests %q", got[1].Suggests)
}
}
// **A drill has its own verb** — `hand-act drill`, the seat's `drill` — and `hand-act record` refuses
// the cause, so a repair cannot pass for a drill by the word it gives.
func TestADrillIsRecordedThroughItsOwnVerb(t *testing.T) {
err := handActCommand(context.Background(), []string{"record", "stopped searxng", "--why", "a test", "--cause", "drill"})
if err == nil || !strings.Contains(err.Error(), "hand-act drill") {
t.Errorf("hand-act record --cause drill was not sent to the drill verb: %v", err)
}
if err := handActCommand(context.Background(), []string{"drill", "stopped searxng"}); err == nil ||
!strings.Contains(err.Error(), "--why") {
t.Errorf("a drill without what it tests: %v", err)
}
if err := handActCommand(context.Background(), []string{"drill", "--why", "a test"}); err == nil ||
!strings.Contains(err.Error(), "hand-act drill <what") {
t.Errorf("a drill without what was done: %v", err)
}
argv, err := argvFor("drill", map[string]any{"what": "stopped searxng", "why": "ADR 0240 phase A",
"condition": "machine.ace.module.searxng.unhealthy"})
if want := "hand-act drill stopped searxng --why ADR 0240 phase A --condition machine.ace.module.searxng.unhealthy"; err != nil ||
strings.Join(argv, " ") != want {
t.Errorf("the seat's drill: %v %v, want %q", argv, err, want)
}
if _, err := argvFor("drill", map[string]any{"what": "stopped searxng"}); err == nil {
t.Error("the seat's drill without why was not refused")
}
if repairingCommand([]string{"hand-act", "drill", "x"}) != "hand-act drill" {
t.Error("a drill through `command` is not held to why")
}
}
+47
View File
@@ -0,0 +1,47 @@
package main
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
)
// Every catalogue module that runs something long-lived declares how it is ready (novox/hq ADR 0240 rule
// 8): `module check` warns and counts the undeclared before the date, and refuses them from it.
func TestModuleCheckCountsTheUndeclaredAndRefusesThemFromTheDate(t *testing.T) {
dir := t.TempDir()
digest := "@sha256:" + strings.Repeat("a", 64)
path := filepath.Join(dir, "module.json")
os.WriteFile(path, []byte(`{"module":"web","listens":[{"name":"web","port":80,"from":"mesh"}],"resources":[
{"id":"server","type":"container","name":"web","image":"registry.example/web`+digest+`","ports":["80"],
"health":{"kind":"http","endpoint":"web"}},
{"id":"worker","type":"container","name":"web-worker","image":"registry.example/web`+digest+`"},
{"id":"seed","type":"container","name":"web-seed","image":"registry.example/web`+digest+`","run-once":true}]}`), 0o600)
defer func() { checkNow = time.Now }()
checkNow = func() time.Time { return catalogue.HealthRequiredFrom.Add(-time.Hour) }
var out bytes.Buffer
if err := moduleCheck([]string{path}, &out); err != nil {
t.Fatalf("refused before the date: %v\n%s", err, out.String())
}
for _, want := range []string{"ready: server by http / on web every 30s", "WARNING: worker stay(s) up",
catalogue.HealthRequiredFrom.Format("2006-01-02"), UndeclaredHealthLine + " 1"} {
if !strings.Contains(out.String(), want) {
t.Errorf("the check does not say %q:\n%s", want, out.String())
}
}
checkNow = func() time.Time { return catalogue.HealthRequiredFrom }
out.Reset()
if err := moduleCheck([]string{path}, &out); err == nil {
t.Fatalf("a long-running resource without health passed after the date:\n%s", out.String())
}
if !strings.Contains(out.String(), "web: worker stays up and does not say how it is ready") {
t.Errorf("the refusal does not name the resource:\n%s", out.String())
}
}
+1 -1
View File
@@ -123,7 +123,7 @@ func run() error {
// The merge gate: every machine of the snapshot composed with a change (novox/hq to-be 45 §9).
case "merge-gate":
return mergeGateCommand(ctx, args[1:])
// A pull request's merge check run here exactly as the build seat runs it (novox/hq issue 286).
// A pull request's merge check run here exactly as the build seat runs it (novox/hq issue 283).
case "check-here":
return checkHereCommand(ctx, args[1:])
case "upgrade":
+3 -3
View File
@@ -108,7 +108,7 @@ type mergeVerdict struct {
Facts time.Time `json:"facts"`
Failures []string `json:"failures,omitempty"`
// Errors are what kept the gate from judging: a machine the mesh composes that the gate could not
// raise as it is. Any one makes the verdict an error — never a pass (novox/hq issue 285).
// raise as it is. Any one makes the verdict an error — never a pass (novox/hq issue 282).
Errors []string `json:"errors,omitempty"`
Warnings []string `json:"warnings,omitempty"`
Notes []string `json:"notes,omitempty"`
@@ -222,7 +222,7 @@ func mergeGateCommand(ctx context.Context, args []string) error {
}
// errMergeGateCouldNotJudge is the gate's own error: the mesh as it is could not be raised, so nothing
// the change does to it can be seen. Never a pass (novox/hq issue 285).
// the change does to it can be seen. Never a pass (novox/hq issue 282).
var errMergeGateCouldNotJudge = errors.New("the merge gate could not judge the change")
// errMergeGateFailed is the gate's own failure: the verdict says why, so the error says nothing more.
@@ -351,7 +351,7 @@ func judgeChange(ctx context.Context, in mergeCheckInput) (mergeVerdict, error)
v.Failures = append(v.Failures, fmt.Sprintf("%s: nothing could be sent to it with this change — %s",
gm.Described, firstOr(newOnly(gm.Change.Problems, gm.Base.Problems), gm.Change.Problems)))
case !gm.Base.Composes && m.Declaration.Composes:
// **The mesh composes it and the gate could not raise it as it is** (novox/hq issue 285): a fact
// **The mesh composes it and the gate could not raise it as it is** (novox/hq issue 282): a fact
// the snapshot does not carry, or one the gate does not raise. Then the change is judged against a
// machine that is not the mesh's — broken against broken, which passes whatever the change does —
// so the gate cannot judge, and says so: an error, never a pass.
+5 -5
View File
@@ -323,11 +323,11 @@ func busMesh(t *testing.T) snapshot.Facts {
return f
}
// **Issue 285**: every machine running a module on the bus failed to compose in the gate's store, with
// **Issue 282**: every machine running a module on the bus failed to compose in the gate's store, with
// the change and without — the store held the module's credential and no account for it, which
// composition refuses (issue 203) — and the gate passed every change, "0 of 4 compose". The account the
// mesh issued is raised with its credential, so the machine composes in the gate as on the mesh.
func TestIssue285AModuleOnTheBusComposesInTheGate(t *testing.T) {
func TestIssue282AModuleOnTheBusComposesInTheGate(t *testing.T) {
f := busMesh(t)
v := gateJudged(t, f, "")
if v.Verdict != "pass" {
@@ -343,10 +343,10 @@ func TestIssue285AModuleOnTheBusComposesInTheGate(t *testing.T) {
}
}
// **Issue 285**: a machine the mesh composes that the gate cannot raise as it is leaves the change judged
// **Issue 282**: a machine the mesh composes that the gate cannot raise as it is leaves the change judged
// against a machine that is not the mesh's — broken against broken, which passes whatever the change does.
// That is an error, never a pass.
func TestIssue285AMachineTheGateCannotRaiseIsAnErrorNeverAPass(t *testing.T) {
func TestIssue282AMachineTheGateCannotRaiseIsAnErrorNeverAPass(t *testing.T) {
f := busMesh(t)
for i := range f.Machines {
// A fact the snapshot does not carry: the module's credential, gone from the laptop's.
@@ -386,7 +386,7 @@ func TestAWithheldPathIsStoodInForByAPath(t *testing.T) {
}
}
// **Issue 286**: a check run by hand clones beside a change what the seat clones — one rule, read by the
// **Issue 283**: a check run by hand clones beside a change what the seat clones — one rule, read by the
// controller's ask and by the facts — and finds the repository it checks from its origin.
func TestACheckByHandClonesWhatTheSeatClones(t *testing.T) {
for dir, refs := range map[string]map[string]string{
+325
View File
@@ -0,0 +1,325 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"sync/atomic"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A module says how it is healthy, and the node-engine judges it (novox/hq ADR 0240, to-be 48 §4 and §5,
// Phase A).
//
// **The node-engine owns every verdict; the controller keeps the last word and raises the condition.** A
// machine states, in every report and as an event between reports, the state of every long-running
// resource it runs for a module. The controller keeps the newest statement per machine (node_health), and
// raises `module.<module>.<machine>.unhealthy` when two statements in a row say a resource of the module
// is unhealthy — one statement is listed as unconfirmed, as the self-check does a finding one look can be
// wrong about (to-be 45 §4, issue 277) — and clears it on the first that does not. The release gate reads
// the stated health: a judging passes a module only when every long-running resource of it on that
// machine is stated healthy, so a resource still starting is not yet a pass.
//
// **An engine older than the judging states nothing**, and its machine's health is not known: never
// healthy, never a reason to raise anything, and the gate judges it as it did before.
// The condition a module's health raises.
const (
kindModuleUnhealthy = "module-unhealthy"
// sourceHealth is what raised it: the machine's own statement.
sourceHealth = "health"
// moduleUnhealthyUrgentAfter is how long it stands before it is urgent (to-be 48 §4).
moduleUnhealthyUrgentAfter = 4 * time.Hour
// moduleUnhealthyAfter is how many statements in a row raise it.
moduleUnhealthyAfter = 2
)
// healthRefused counts the statements refused as older than the one kept, for the log and a test.
var healthRefused atomic.Int64
// moduleHealth keeps what the machines state, for the link (link.Healths).
type moduleHealth struct {
inv *inventory.Inventory
keeper func() *conditions.Keeper
}
func (m moduleHealth) Stated(ctx context.Context, node string, h link.Health) error {
return stateHealth(ctx, m.inv, m.keeper(), node, h, time.Now())
}
// stateHealth keeps one machine's statement and raises or clears its modules' conditions from it. An
// older statement than the one kept is refused, by when the engine looked.
func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Keeper, node string, h link.Health,
now time.Time) error {
if h.Contract == 0 {
return nil
}
prev, had, err := inv.HealthOf(ctx, node)
if err != nil {
return err
}
if had && h.At.Before(prev.SaidAt) {
healthRefused.Add(1)
return nil
}
unhealthy := map[string][]inventory.ResourceHealth{}
resources := make([]inventory.ResourceHealth, 0, len(h.Resources))
for _, r := range h.Resources {
kept := inventory.ResourceHealth{Module: r.Module, Resource: r.Resource, Kind: r.Kind, Target: r.Target,
State: r.State, Reason: r.Reason, Since: r.Since, Streak: r.Streak, Restarts: r.Restarts,
Check: r.Check, Needs: r.Needs}
resources = append(resources, kept)
if r.State == link.StateUnhealthy && r.Module != "" {
unhealthy[r.Module] = append(unhealthy[r.Module], kept)
}
}
streaks := map[string]int{}
for module := range unhealthy {
streaks[module] = prev.Streaks[module] + 1
}
stored, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: node, Contract: h.Contract, SaidAt: h.At,
HeardAt: now, Resources: resources, Streaks: streaks})
if err != nil || !stored {
if err == nil {
healthRefused.Add(1)
}
return err
}
if k == nil {
return nil
}
return judgeModuleHealth(ctx, inv, k, node, unhealthy, streaks, now)
}
// judgeModuleHealth raises a module's condition on a machine on the second statement in a row that says a
// resource of it is unhealthy — or on the first while it is already open — and clears every one this
// statement no longer says. **A consumer whose findings wait on an unhealthy provider is held** (to-be 48
// §6): raised as nothing of its own, listed at the provider's condition, which is urgent while anyone
// waits on it.
func judgeModuleHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Keeper, node string,
unhealthy map[string][]inventory.ResourceHealth, streaks map[string]int, now time.Time) error {
open, err := k.Open(ctx)
if err != nil {
return err
}
standing := map[string]conditions.Condition{}
for _, c := range open {
if c.Kind == kindModuleUnhealthy && c.Subject.Machine == node {
standing[c.Key] = c
}
}
var hold *holding
if inv != nil {
if hold, err = readHolding(ctx, inv, open); err != nil {
return err
}
}
var problems []string
modules := make([]string, 0, len(unhealthy))
for m := range unhealthy {
modules = append(modules, m)
}
sort.Strings(modules)
seen := map[string]bool{}
heldOn := map[string]string{}
providers := map[catalogue.Chosen]bool{}
for _, m := range modules {
o := moduleUnhealthyObservation(m, node, unhealthy[m])
if hold != nil {
if p, held := hold.heldUnder(node, m, unhealthy[m]); held {
// Held under the provider's condition: nothing of its own, and the provider's says it waits.
heldOn[o.Key()] = p.Module + " on " + p.Node
providers[p] = true
continue
}
if waiters := hold.waitersOn(catalogue.Chosen{Node: node, Module: m}); len(waiters) > 0 {
o.Severity = conditions.Urgent
o.Said += "; " + waitingWords(waiters)
o.Summary += fmt.Sprintf("; %d consumer(s) wait on it", len(waiters))
}
}
seen[o.Key()] = true
c, isOpen := standing[o.Key()]
if streaks[m] < moduleUnhealthyAfter && !isOpen {
continue // unconfirmed: one statement can be wrong; `node show` lists it
}
if isOpen && now.Sub(c.Raised) >= moduleUnhealthyUrgentAfter {
o.Severity = conditions.Urgent
}
if _, err := k.Observe(ctx, o); err != nil {
problems = append(problems, err.Error())
}
}
for key, c := range standing {
if seen[key] {
continue
}
module := strings.TrimSuffix(c.Subject.ID, "."+node)
why := fmt.Sprintf("%s says no resource of %s is unhealthy", node, module)
if on, held := heldOn[key]; held {
why = fmt.Sprintf("what %s finds on %s waits on %s, which is unhealthy: held under its condition", module, node, on)
}
if _, err := k.Clear(ctx, key, why); err != nil {
problems = append(problems, err.Error())
}
}
// And each provider a consumer here now waits on, when its own condition is open: said again with who
// waits on it, so the wait is listed at the provider whichever machine's statement arrived first.
for p := range providers {
if err := sayWaiters(ctx, k, hold, p, now); err != nil {
problems = append(problems, err.Error())
}
}
if len(problems) > 0 {
return fmt.Errorf("%s", strings.Join(problems, "; "))
}
return nil
}
// sayWaiters observes a provider's open condition again, with who waits on it, from its machine's newest
// statement. Nothing when its condition is not open: it is raised by its own statements, on its own looks.
func sayWaiters(ctx context.Context, k *conditions.Keeper, hold *holding, p catalogue.Chosen, now time.Time) error {
var raisedAt *conditions.Condition
for i, c := range hold.open {
if c.Key == moduleUnhealthyKey(p.Module, p.Node) {
raisedAt = &hold.open[i]
}
}
if raisedAt == nil {
return nil
}
var rs []inventory.ResourceHealth
for _, r := range hold.healths[p.Node].Resources {
if r.Module == p.Module && r.State == link.StateUnhealthy {
rs = append(rs, r)
}
}
if len(rs) == 0 {
return nil
}
o := moduleUnhealthyObservation(p.Module, p.Node, rs)
if waiters := hold.waitersOn(p); len(waiters) > 0 {
o.Severity = conditions.Urgent
o.Said += "; " + waitingWords(waiters)
o.Summary += fmt.Sprintf("; %d consumer(s) wait on it", len(waiters))
}
_, err := k.Observe(ctx, o)
return err
}
// moduleUnhealthyObservation is a module unhealthy on a machine, in words: the summary names the module,
// the machine and what is wrong with each resource; the detail — targets, streaks, since — is evidence.
func moduleUnhealthyObservation(module, node string, rs []inventory.ResourceHealth) conditions.Observation {
var words, said []string
for _, r := range rs {
words = append(words, fmt.Sprintf("its %s %s %s", r.Kind, r.Resource, reasonWords(r)))
said = append(said, fmt.Sprintf("%s (%s %s): %s, %d look(s) in a row, %d restart(s) counted, since %s",
r.Resource, r.Kind, r.Target, orNotSaid(r.Reason), r.Streak, r.Restarts,
r.Since.UTC().Format("2006-01-02 15:04:05 MST")))
}
return conditions.Observation{Scope: conditions.ScopeModule, ID: module + "." + node, Token: "unhealthy",
Kind: kindModuleUnhealthy, Machine: node, Severity: conditions.Warning, Source: sourceHealth,
Summary: fmt.Sprintf("%s on %s is not healthy: %s", module, node, strings.Join(words, "; ")),
Said: strings.Join(said, "; ")}
}
// reasonWords is why a resource is unhealthy, as a person reads it.
func reasonWords(r inventory.ResourceHealth) string {
switch r.Reason {
case "restarting":
return fmt.Sprintf("keeps restarting (%d restart(s) counted)", r.Restarts)
case "down":
return "is not running"
case "":
return "is unhealthy"
}
// What a declared check found says an endpoint, a path or an address: evidence, never the summary the
// operator's channel carries (ADR 0234 §6). The summary names the check.
if r.Check != "" {
return "fails its " + r.Check + " check"
}
return "is unhealthy: " + r.Reason
}
func orNotSaid(s string) string {
if s == "" {
return "no reason said"
}
return s
}
// moduleHealthWord is the gate's reading of a module's stated health on a machine (ADR 0240 §4, ADR 0236
// §2 as amended): good when every long-running resource of it is stated healthy in a statement heard since
// the send; not yet otherwise, saying which. A machine that never stated health is judged as before.
func moduleHealthWord(module, machine string, since time.Time, f gateFacts) (health, string) {
if f.healthErr != nil {
return healthNotYet, "what " + machine + " says of its resources' health cannot be read: " + firstLine(f.healthErr.Error())
}
h, states := f.health[machine]
if !states {
return healthGood, ""
}
if h.HeardAt.Before(since) {
return healthNotYet, fmt.Sprintf("%s has not said how what %s runs is since it was sent", machine, module)
}
for _, r := range h.Resources {
if r.Module != module {
continue
}
switch r.State {
case link.StateHealthy:
case link.StateStarting:
return healthNotYet, fmt.Sprintf("its %s %s on %s is still starting", r.Kind, r.Resource, machine)
case link.StateUnhealthy:
if on, held := f.heldOn[module+"@"+machine]; held {
return healthWaiting, fmt.Sprintf("its %s %s on %s waits on %s, which is unhealthy", r.Kind,
r.Resource, machine, on)
}
return healthNotYet, fmt.Sprintf("its %s %s on %s %s", r.Kind, r.Resource, machine, reasonWords(r))
default:
return healthNotYet, fmt.Sprintf("its %s %s on %s is %s%s", r.Kind, r.Resource, machine, r.State,
reasonAfter(r.Reason))
}
}
return healthGood, ""
}
func reasonAfter(s string) string {
if s == "" {
return ""
}
return ": " + s
}
// healthLines is what `node show` says of a machine's long-running resources: each with its state and
// since when, an unhealthy one said once marked unconfirmed.
func healthLines(h inventory.NodeHealth, had bool, now time.Time) []string {
if !had {
return []string{" its node-engine does not say how what it runs is — it is older than the judging (ADR 0240)"}
}
if len(h.Resources) == 0 {
return []string{fmt.Sprintf(" it runs nothing long-lived for a module (said %s ago)", roughly(now.Sub(h.HeardAt)))}
}
out := []string{fmt.Sprintf(" what it runs, as it said %s ago:", roughly(now.Sub(h.HeardAt)))}
for _, r := range h.Resources {
line := fmt.Sprintf(" %-10s %-34s %s %s, since %s", r.State, r.Resource, r.Kind, r.Target,
r.Since.Local().Format("2006-01-02 15:04"))
if r.Reason != "" {
line += " — " + r.Reason
}
if r.Restarts > 0 {
line += fmt.Sprintf(", %d restart(s) counted", r.Restarts)
}
if r.State == link.StateUnhealthy && h.Streaks[r.Module] < moduleUnhealthyAfter {
line += " (unconfirmed: said once)"
}
out = append(out, line)
}
return out
}
+196
View File
@@ -0,0 +1,196 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A module's stated health, as the controller keeps it and raises from it (novox/hq ADR 0240, "how it is
// checked", rule 4): one unhealthy statement raises nothing and is listed unconfirmed; two raise; a
// healthy one clears; a condition from before the send clears at the new build's start; an older
// statement is refused; and an engine that states nothing raises nothing.
var h0 = time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)
func aStatement(at time.Time, states ...string) link.Health {
h := link.Health{Contract: link.LivenessContract, At: at}
for i, s := range states {
r := link.ResourceHealth{Module: "letta", Resource: "letta.server", Kind: "container", Target: "letta-server",
State: s, Since: at}
if i > 0 {
r.Module, r.Resource, r.Target = "mqtt", "mqtt.broker", "mosquitto.service"
}
if s == link.StateUnhealthy {
r.Reason, r.Restarts, r.Streak = "restarting", 4, 2
}
h.Resources = append(h.Resources, r)
}
return h
}
func TestTwoUnhealthyStatementsRaiseTheModulesConditionAndAHealthyOneClearsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
k := conditionsFrom
const key = "module.letta.anchor.unhealthy"
openKeys := func() []string {
t.Helper()
list, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
var keys []string
for _, c := range list {
keys = append(keys, c.Key)
}
return keys
}
if err := stateHealth(ctx, inv, k, "anchor", aStatement(h0, link.StateHealthy, link.StateHealthy), h0); err != nil {
t.Fatal(err)
}
// One statement: nothing raised, and `node show` lists it as unconfirmed.
if err := stateHealth(ctx, inv, k, "anchor", aStatement(h0.Add(time.Minute), link.StateUnhealthy, link.StateHealthy),
h0.Add(time.Minute)); err != nil {
t.Fatal(err)
}
if keys := openKeys(); len(keys) != 0 {
t.Fatalf("one statement raised %v", keys)
}
kept, had, err := inv.HealthOf(ctx, "anchor")
if err != nil || !had {
t.Fatalf("the statement was not kept: %v %v", had, err)
}
if lines := strings.Join(healthLines(kept, had, h0.Add(time.Minute)), "\n"); !strings.Contains(lines, "unconfirmed") ||
!strings.Contains(lines, "letta.server") {
t.Fatalf("node show does not list the first statement as unconfirmed:\n%s", lines)
}
// The second in a row raises it — the module's own, never the other module's on the machine.
if err := stateHealth(ctx, inv, k, "anchor", aStatement(h0.Add(2*time.Minute), link.StateUnhealthy, link.StateHealthy),
h0.Add(2*time.Minute)); err != nil {
t.Fatal(err)
}
if keys := openKeys(); len(keys) != 1 || keys[0] != key {
t.Fatalf("two statements raised %v, not %s", keys, key)
}
c, _, _ := k.Get(ctx, key)
if c.Severity != conditions.Warning || c.Resolver != conditions.ResolverSelf || c.Subject.Machine != "anchor" ||
!strings.Contains(c.Summary, "letta on anchor") || !strings.Contains(c.Summary, "keeps restarting") ||
!strings.Contains(c.Evidence[0].Said, "letta-server") {
t.Fatalf("the condition does not say it in words with its evidence: %+v", c)
}
// Standing four hours, it is urgent.
if err := stateHealth(ctx, inv, k, "anchor", aStatement(h0.Add(5*time.Hour), link.StateUnhealthy, link.StateHealthy),
time.Now().Add(5*time.Hour)); err != nil {
t.Fatal(err)
}
if c, _, _ := k.Get(ctx, key); c.Severity != conditions.Urgent {
t.Fatalf("unhealthy for four hours is still %s", c.Severity)
}
// A new build's start — every start begins in `starting` — clears it: what follows is the new build's.
if err := stateHealth(ctx, inv, k, "anchor", aStatement(h0.Add(6*time.Hour), link.StateStarting, link.StateHealthy),
h0.Add(6*time.Hour)); err != nil {
t.Fatal(err)
}
if keys := openKeys(); len(keys) != 0 {
t.Fatalf("the first statement that says no resource is unhealthy did not clear it: %v", keys)
}
// And the streak starts again: one unhealthy statement after it raises nothing.
if err := stateHealth(ctx, inv, k, "anchor", aStatement(h0.Add(7*time.Hour), link.StateUnhealthy), h0.Add(7*time.Hour)); err != nil {
t.Fatal(err)
}
if keys := openKeys(); len(keys) != 0 {
t.Fatalf("one statement after a clearing raised %v", keys)
}
}
func TestAnOlderHealthStatementIsRefused(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
before := healthRefused.Load()
if err := stateHealth(ctx, inv, nil, "anchor", aStatement(h0.Add(time.Minute), link.StateHealthy), h0); err != nil {
t.Fatal(err)
}
// An event said before the report that overtook it arrives late.
if err := stateHealth(ctx, inv, nil, "anchor", aStatement(h0, link.StateUnhealthy), h0); err != nil {
t.Fatal(err)
}
kept, _, err := inv.HealthOf(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if !kept.SaidAt.Equal(h0.Add(time.Minute)) || kept.Resources[0].State != link.StateHealthy {
t.Fatalf("the older statement replaced the newer: %+v", kept)
}
if healthRefused.Load() != before+1 {
t.Fatalf("the refusal was not counted")
}
}
// **An engine older than the judging states nothing**: its reports raise nothing, keep nothing, and the
// gate judges its machine as before — never healthy for having said nothing, never unhealthy.
func TestAReportWithNoHealthRaisesAndKeepsNothing(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
l := nudgingListener{Enrolment: link.Enrolment{Inventory: open.inventory}}
if _, err := l.Heard(ctx, link.Report{Node: "anchor", Declared: "d1", Applied: []string{"letta.server"}}); err != nil {
t.Fatal(err)
}
if _, had, err := open.inventory.HealthOf(ctx, "anchor"); err != nil || had {
t.Fatalf("health kept for an engine that said none: %v %v", had, err)
}
if lines := healthLines(inventory.NodeHealth{}, false, time.Now()); !strings.Contains(lines[0], "older than the judging") {
t.Fatalf("node show: %v", lines)
}
// And one that does, through the report, is kept.
h := aStatement(time.Now().UTC(), link.StateHealthy)
if _, err := l.Heard(ctx, link.Report{Node: "anchor", Declared: "d1", Health: &h}); err != nil {
t.Fatal(err)
}
if kept, had, err := open.inventory.HealthOf(ctx, "anchor"); err != nil || !had || len(kept.Resources) != 1 {
t.Fatalf("the report's health was not kept: %+v %v %v", kept, had, err)
}
}
// A declared `health` is sent only to an engine whose own statement says it reads it (novox/hq ADR 0240
// Phase B): an older engine is strict and would refuse the whole declaration for the field.
func TestHealthIsSentOnlyToAnEngineThatSaysItReadsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
reads := func() bool {
t.Helper()
got, err := engineReadsHealth(ctx, inv, "anchor")
if err != nil {
t.Fatal(err)
}
return got
}
if reads() {
t.Fatal("an engine that never stated anything is sent health")
}
if err := stateHealth(ctx, inv, nil, "anchor", aStatement(h0, link.StateHealthy), h0); err != nil {
t.Fatal(err)
}
if reads() {
t.Fatal("an engine judging liveness alone is sent health")
}
later := aStatement(h0.Add(time.Minute), link.StateHealthy)
later.Contract = link.ReadinessContract
if err := stateHealth(ctx, inv, nil, "anchor", later, h0.Add(time.Minute)); err != nil {
t.Fatal(err)
}
if !reads() {
t.Fatal("an engine that reads health is not sent it")
}
}
+11
View File
@@ -537,6 +537,17 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
}
}
// What it runs for its modules, and how each is (novox/hq ADR 0240): "is it working" answered without
// a terminal on the machine.
if h, had, err := inv.HealthOf(ctx, name); err != nil {
fmt.Printf("\n what it says of what it runs could NOT be read: %v\n", err)
} else {
fmt.Println()
for _, line := range healthLines(h, had, time.Now()) {
fmt.Println(line)
}
}
held, err := inv.Profile(ctx, name)
if err != nil {
return err
+17
View File
@@ -17,6 +17,7 @@ import (
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
)
@@ -860,7 +861,12 @@ func renderingFor(ctx context.Context, open *stores, node string,
if artifactStore != "" {
reach = map[string]string{"mesh-artifact-store": artifactStore}
}
readsHealth, err := engineReadsHealth(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{
ReadsHealth: readsHealth,
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
@@ -872,6 +878,17 @@ func renderingFor(ctx context.Context, open *stores, node string,
}, record, nil
}
// engineReadsHealth says whether a machine's node-engine reads a declared `health` (novox/hq ADR 0240
// Phase B), by its own newest statement: one older, or one that never stated anything, is not sent the
// field, because it parses strictly and would refuse the whole declaration for it.
func engineReadsHealth(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.ReadinessContract, nil
}
// zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR
// 0199): the zone settled from that node's settings, the node's private address, the port the
// answering listen is published on there.
+192
View File
@@ -0,0 +1,192 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A provider down is said once, at the provider (novox/hq ADR 0240 rule 5, to-be 48 §6, Phase C).
//
// With twelve consumers of the database provision and thirty-six of a route, one provider down would be
// twelve conditions for one fault and twelve gates failed for something none of them did. So a consumer's
// check names, in `needs`, the provision it exercises; while **that provision's provider for this
// consumer** — the one the controller composed the consumer against: its recorded binding (ADR 0232), or
// the provider its credential for the provision is from — is unhealthy on the record, what the check finds
// is held under the provider's condition: listed there as waiting on it, raised as nothing of its own, and
// the consumer's gate waits rather than fails. The provider's condition is urgent while consumers wait.
//
// **Only what the check finds is held.** A consumer that is down or restarting is its own, whatever its
// provider does; so is anything a check that names no provision finds, and anything found while the
// provider is healthy. A machine-level fault is never pinned on a module (issue 281), and this does not
// change that.
// holding is what one reading of the record needs to say who waits on whom: every machine's newest
// statement, the open conditions, and the catalogue — read once, asked many times.
type holding struct {
ctx context.Context
inv *inventory.Inventory
healths map[string]inventory.NodeHealth
open []conditions.Condition
shelf map[string]catalogue.Manifest
// providers memoises providerFor by machine, consumer and provision.
providers map[string]providerLookup
}
type providerLookup struct {
chosen catalogue.Chosen
ok bool
}
// readHolding reads what the hold is judged from.
func readHolding(ctx context.Context, inv *inventory.Inventory, open []conditions.Condition) (*holding, error) {
healths, err := inv.Healths(ctx)
if err != nil {
return nil, err
}
return &holding{ctx: ctx, inv: inv, healths: healths, open: open, providers: map[string]providerLookup{}}, nil
}
// heldFinding says a resource's state is a finding of its declared check that names a provision: what
// may be held. Down and restarting are liveness, the resource's own.
func heldFinding(r inventory.ResourceHealth) bool {
return r.State == link.StateUnhealthy && r.Check != "" && r.Needs != "" &&
r.Reason != "down" && r.Reason != "restarting"
}
// providerFor is the provider composed for a consumer's provision: its recorded binding, else the
// machine its credential for the provision comes from and the module there that provides it.
func (h *holding) providerFor(machine, consumer, provision string) (catalogue.Chosen, bool) {
key := machine + "\x00" + consumer + "\x00" + provision
if p, known := h.providers[key]; known {
return p.chosen, p.ok
}
chosen, ok := h.lookUpProvider(machine, consumer, provision)
h.providers[key] = providerLookup{chosen, ok}
return chosen, ok
}
func (h *holding) lookUpProvider(machine, consumer, provision string) (catalogue.Chosen, bool) {
if bound, err := h.inv.BindingsFor(h.ctx, machine); err == nil {
if c, ok := bound[consumer][provision]; ok && c.Node != "" && c.Module != "" {
return c, true
}
}
secrets, err := h.inv.SecretsOf(h.ctx, machine, consumer)
if err != nil {
return catalogue.Chosen{}, false
}
for _, s := range secrets {
if s.Name != provision || s.Provider == "" {
continue
}
if h.shelf == nil {
if h.shelf, err = h.inv.Catalogue(h.ctx); err != nil {
return catalogue.Chosen{}, false
}
}
assigned, err := h.inv.Assigned(h.ctx, s.Provider)
if err != nil {
return catalogue.Chosen{}, false
}
for _, module := range assigned {
for _, offer := range h.shelf[module].Offers() {
if offer == provision {
return catalogue.Chosen{Node: s.Provider, Module: module}, true
}
}
}
}
return catalogue.Chosen{}, false
}
// unhealthy says a provider is unhealthy on the record: its condition is open, or its machine's newest
// statement says a resource of it is unhealthy.
func (h *holding) unhealthy(p catalogue.Chosen) bool {
key := moduleUnhealthyKey(p.Module, p.Node)
for _, c := range h.open {
if c.Key == key {
return true
}
}
for _, r := range h.healths[p.Node].Resources {
if r.Module == p.Module && r.State == link.StateUnhealthy {
return true
}
}
return false
}
// heldUnder is the provider a consumer's unhealthy resources wait on: when every one of them is a finding
// of a check naming a provision whose provider for this consumer is unhealthy on the record. False when any
// is the consumer's own.
func (h *holding) heldUnder(machine, module string, rs []inventory.ResourceHealth) (catalogue.Chosen, bool) {
var on catalogue.Chosen
for _, r := range rs {
if r.State != link.StateUnhealthy {
continue
}
if !heldFinding(r) {
return catalogue.Chosen{}, false
}
p, ok := h.providerFor(machine, module, r.Needs)
if !ok || (p.Node == machine && p.Module == module) || !h.unhealthy(p) {
return catalogue.Chosen{}, false
}
on = p
}
return on, on.Module != ""
}
// waitersOn is every consumer held under a provider, as "<module> on <machine>", sorted.
func (h *holding) waitersOn(p catalogue.Chosen) []string {
var out []string
for machine, nh := range h.healths {
byModule := map[string][]inventory.ResourceHealth{}
for _, r := range nh.Resources {
if r.Module != "" && r.State == link.StateUnhealthy {
byModule[r.Module] = append(byModule[r.Module], r)
}
}
for module, rs := range byModule {
if on, held := h.heldUnder(machine, module, rs); held && on == p {
out = append(out, module+" on "+machine)
}
}
}
sort.Strings(out)
return out
}
// heldModules is, for one machine's statement, each module whose finding is held, with the provider.
func (h *holding) heldModules(machine string) map[string]catalogue.Chosen {
out := map[string]catalogue.Chosen{}
byModule := map[string][]inventory.ResourceHealth{}
for _, r := range h.healths[machine].Resources {
if r.Module != "" && r.State == link.StateUnhealthy {
byModule[r.Module] = append(byModule[r.Module], r)
}
}
for module, rs := range byModule {
if on, held := h.heldUnder(machine, module, rs); held {
out[module] = on
}
}
return out
}
// moduleUnhealthyKey is a module's health condition's key on a machine.
func moduleUnhealthyKey(module, machine string) string {
return conditions.ScopeModule + "." + module + "." + machine + ".unhealthy"
}
// waitingWords is the provider's evidence that consumers wait on it.
func waitingWords(waiters []string) string {
return fmt.Sprintf("waiting on it — %s", strings.Join(waiters, ", "))
}
+172
View File
@@ -0,0 +1,172 @@
package main
import (
"context"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A provider down is said once, at the provider (novox/hq ADR 0240 rule 5, "how it is checked"): one
// unhealthy database provider and three consumers failing their checks that need it — one condition, at the
// provider, urgent, the consumers listed as waiting; their gates wait, not fail; once the provider is
// healthy, a consumer still failing is its own. A consumer failing while its provider is healthy is raised
// on its own from the start.
func TestOneProviderDownAndThreeConsumersFailingAreOneCondition(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
k := conditionsFrom
register(t, open, catalogue.Manifest{Module: "db", Version: "1",
Provides: []catalogue.Offer{{Name: "postgres-database", Scope: catalogue.ScopeMesh}}})
consumers := []string{"shop", "wiki", "crm"}
for _, c := range consumers {
register(t, open, catalogue.Manifest{Module: c, Version: "1", Requires: []string{"postgres-database"}})
}
if _, err := inv.Assign(ctx, "anchor", "db"); err != nil {
t.Fatal(err)
}
machineOf := map[string]string{"shop": "laptop", "wiki": "laptop", "crm": "anchor"}
for _, c := range consumers {
if _, err := inv.Assign(ctx, machineOf[c], c); err != nil {
t.Fatal(err)
}
if err := inv.RecordBindings(ctx, machineOf[c], []inventory.Binding{{Machine: machineOf[c], Consumer: c,
Provision: "postgres-database", Provider: catalogue.Chosen{Node: "anchor", Module: "db"}}}); err != nil {
t.Fatal(err)
}
}
at := h0
say := func(machine string, rs ...link.ResourceHealth) {
t.Helper()
at = at.Add(time.Second)
for i := range rs {
rs[i].Since = at
}
if err := stateHealth(ctx, inv, k, machine, link.Health{Contract: link.ReadinessContract, At: at, Resources: rs}, at); err != nil {
t.Fatal(err)
}
}
dbDown := link.ResourceHealth{Module: "db", Resource: "db.server", Kind: "container", Target: "db", State: link.StateUnhealthy,
Reason: "its command: the database refuses connections", Check: "exec"}
dbUp := dbDown
dbUp.State, dbUp.Reason = link.StateHealthy, ""
failing := func(module string) link.ResourceHealth {
return link.ResourceHealth{Module: module, Resource: module + ".web", Kind: "container", Target: module,
State: link.StateUnhealthy, Reason: "http /health on web: answered 500", Check: "http", Needs: "postgres-database"}
}
fine := func(module string) link.ResourceHealth {
r := failing(module)
r.State, r.Reason = link.StateHealthy, ""
return r
}
openKeys := func() []conditions.Condition {
t.Helper()
list, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
return list
}
// Two looks of the provider down and its consumers failing, in either order on each machine.
for look := 0; look < 2; look++ {
say("laptop", failing("shop"), failing("wiki"))
say("anchor", dbDown, failing("crm"))
}
raised := openKeys()
if len(raised) != 1 || raised[0].Key != "module.db.anchor.unhealthy" {
var keys []string
for _, c := range raised {
keys = append(keys, c.Key)
}
t.Fatalf("one provider down and three consumers failing raised %v; want the provider's alone", keys)
}
c := raised[0]
if c.Severity != conditions.Urgent {
t.Errorf("consumers wait on the provider and its condition is %s", c.Severity)
}
said := c.Evidence[0].Said
for _, w := range []string{"shop on laptop", "wiki on laptop", "crm on anchor"} {
if !strings.Contains(said, w) {
t.Errorf("the provider's condition does not list %s as waiting on it: %s", w, said)
}
}
// Their gates wait, not fail: the judging is neither a pass nor a fault, past the bound too.
f, err := gatherGateFacts(ctx, open, "")
if err != nil {
t.Fatal(err)
}
f.open, f.openErr, f.judged = raised, nil, false
for _, m := range consumers {
h, why := moduleHealthWord(m, machineOf[m], h0, f)
if h != healthWaiting || !strings.Contains(why, "waits on db on anchor") {
t.Errorf("%s's gate: %v %q; want it waiting on its provider", m, h, why)
}
}
// And a whole judging past the bound puts nothing back: it waits.
long := h0.Add(-time.Hour)
for _, m := range []string{"anchor", "laptop"} {
f.reports[m] = inventory.Reported{Node: m, Outcome: inventory.OutcomeApplied, At: &f.now, Current: true}
}
gatherWas := gatherGateFacts
defer func() { gatherGateFacts = gatherWas }()
gatherGateFacts = func(context.Context, *stores, string) (gateFacts, error) { return f, nil }
g := &inventory.PlanGate{Machines: []string{"laptop"}, Since: &long}
verdict, err := judgeMoves(ctx, open, g, []judged{{module: "shop", node: "laptop"}}, time.Now())
if err != nil || verdict != "" || !strings.Contains(g.Last, "waits on db on anchor") {
t.Fatalf("a held consumer's gate past its bound: verdict %q (%v), last %q; want it waiting", verdict, err, g.Last)
}
// The provider healthy again: a consumer still failing is now its own, at once (its streak stood).
say("anchor", dbUp, fine("crm"))
say("laptop", failing("shop"), fine("wiki"))
var keys []string
for _, c := range openKeys() {
keys = append(keys, c.Key)
}
if !slices.Equal(keys, []string{"module.shop.laptop.unhealthy"}) {
t.Fatalf("after the provider recovered: %v; want shop's own and nothing else", keys)
}
}
// A consumer failing while its provider is healthy is raised on its own.
func TestAConsumerFailingBesideAHealthyProviderIsItsOwn(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: "db", Version: "1",
Provides: []catalogue.Offer{{Name: "postgres-database", Scope: catalogue.ScopeMesh}}})
register(t, open, catalogue.Manifest{Module: "shop", Version: "1", Requires: []string{"postgres-database"}})
for _, a := range [][2]string{{"anchor", "db"}, {"laptop", "shop"}} {
if _, err := inv.Assign(ctx, a[0], a[1]); err != nil {
t.Fatal(err)
}
}
if err := inv.RecordBindings(ctx, "laptop", []inventory.Binding{{Machine: "laptop", Consumer: "shop",
Provision: "postgres-database", Provider: catalogue.Chosen{Node: "anchor", Module: "db"}}}); err != nil {
t.Fatal(err)
}
healthy := link.ResourceHealth{Module: "db", Resource: "db.server", Kind: "container", Target: "db", State: link.StateHealthy, Check: "exec"}
shop := link.ResourceHealth{Module: "shop", Resource: "shop.web", Kind: "container", Target: "shop", State: link.StateUnhealthy,
Reason: "http /health on web: answered 500", Check: "http", Needs: "postgres-database"}
for i := 1; i <= 2; i++ {
at := h0.Add(time.Duration(i) * time.Minute)
_ = stateHealth(ctx, inv, conditionsFrom, "anchor", link.Health{Contract: 2, At: at, Resources: []link.ResourceHealth{healthy}}, at)
if err := stateHealth(ctx, inv, conditionsFrom, "laptop", link.Health{Contract: 2, At: at, Resources: []link.ResourceHealth{shop}}, at); err != nil {
t.Fatal(err)
}
}
list, _ := conditionsFrom.Open(ctx)
if len(list) != 1 || list[0].Key != "module.shop.laptop.unhealthy" {
t.Fatalf("a consumer failing beside a healthy provider raised %v", list)
}
}
+7
View File
@@ -128,6 +128,10 @@ func serve(ctx context.Context) (err error) {
stopActing()
case <-ctx.Done():
}
// Stopping, whichever way: a verb arriving from here is refused as a handover, so its caller
// asks the controller after this one rather than have a command started and killed with this
// process (novox/hq issue 289).
handingOver.Store(true)
}()
defer func() {
select {
@@ -182,6 +186,9 @@ func serve(ctx context.Context) (err error) {
if err := server.Watches(standings{keeper: func() *conditions.Keeper { return conditionsFrom }}); err != nil {
return err
}
// And what each machine says of what it runs between its reports, kept and raised from (novox/hq ADR
// 0240): the gate, `node show` and the conditions read it.
server.Hears(moduleHealth{inv: inv, keeper: func() *conditions.Keeper { return conditionsFrom }})
// And what they say about consumers the mesh stopped asking for: one waiting for a person is an
// urgent condition, and an act asked of a provider some other way is recorded by hand (ADR 0230).
if err := server.KeepsRetirements(retirements{keeper: func() *conditions.Keeper { return conditionsFrom },
+230
View File
@@ -1,9 +1,18 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
"slices"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The replays of the controller's incidents (novox/hq to-be 45 §9, M9): each a scripted replay of what
@@ -111,3 +120,224 @@ func TestReplay273AConsumerBesideItsStoreStaysBoundToIt(t *testing.T) {
t.Fatalf("the resolver was bound to %q; its seat is held on anchor (issue 258)", network)
}
}
// **R-crashloop — a module that crash-loops after it applied fails its gate on its first machine** (novox/hq
// ADR 0240; research 032 §6). On the home server the agent server restarted about a hundred times while the
// mesh read it applied, its tools served and no condition raised — the gate judged a module by what the
// mesh saw from outside, and nothing looked at what it ran. The outcome asserted: a build whose container
// exits at start never passes its gate on the first machine, is put back there at the bound, and never
// reaches the second.
//
// The machine is heard as a node-engine says it: its report of the apply, then the same account said again
// later, each carrying what the engine states of what it runs — `starting` as the apply ends, then the
// crash loop. What it states of the crash loop is MESH_REPLAY_STATEMENT when the lab's replay ran the
// engine against a real container (mesh-lab replays, R-crashloop), and otherwise what the engine said of
// one, kept below. Heard as bytes, so an older controller reads them as it reads any report — this file is
// written only with what the controller had before the judging, for the prover to lay over that commit.
func TestReplayCrashLoopFailsItsGateOnTheFirstMachine(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
crashLoop := []byte(`{"contract":1,"at":"2026-10-07T00:00:00Z","resources":[{"module":"app","resource":"app.server",` +
`"kind":"container","target":"app-server","state":"unhealthy","reason":"restarting","since":"2026-10-07T00:00:00Z",` +
`"streak":5,"restarts":2}]}`)
if path := os.Getenv("MESH_REPLAY_STATEMENT"); path != "" {
raw, err := os.ReadFile(path)
if err != nil {
t.Fatalf("the engine's statement of the crash loop: %v", err)
}
crashLoop = raw
}
// `null` is an engine that states nothing — older than the judging — and its reports carry no health.
var stated map[string]any
if err := json.Unmarshal(crashLoop, &stated); err != nil {
t.Fatal(err)
}
for _, b := range []inventory.Build{
{ID: "build-1", Module: "app", Commit: "c1", Repository: "novox/mesh-catalog", Path: "modules/app",
Asked: time.Now().Add(-2 * time.Hour), At: time.Now().Add(-2 * time.Hour)},
{ID: "build-2", Module: "app", Commit: "c2", Repository: "novox/mesh-catalog", Path: "modules/app",
Asked: time.Now().Add(-time.Minute), At: time.Now().Add(-time.Minute)},
} {
manifest, _ := json.Marshal(catalogue.Manifest{Module: "app", Version: b.Commit})
b.Manifest = manifest
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
registerAt := func(commit string, asked time.Time) {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "app", Version: commit},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/app", BuiltFrom: commit,
Head: commit, Asked: asked}); err != nil {
t.Fatal(err)
}
}
registerAt("c1", time.Now().Add(-2*time.Hour))
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.Assign(ctx, n, "app"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, n), "d-"+n+"-c1", map[string]string{"app": "c1"}); err != nil {
t.Fatal(err)
}
}
registerAt("c2", time.Now().Add(-time.Minute))
// The machine: what it is sent is applied, and its report says what runs is starting.
listener := nudgingListener{Enrolment: link.Enrolment{Inventory: inv}}
sequence := int64(0)
say := func(node, digest, state string) {
t.Helper()
sequence++
health := map[string]any{}
for k, v := range stated {
health[k] = v
}
health["at"] = time.Now().UTC().Format(time.RFC3339Nano)
if state != "" && stated != nil {
resources := []any{}
for _, r := range stated["resources"].([]any) {
kept := map[string]any{}
for k, v := range r.(map[string]any) {
kept[k] = v
}
kept["state"], kept["reason"] = state, ""
resources = append(resources, kept)
}
health["resources"] = resources
}
said := map[string]any{"node": node, "applied": []string{"app.server"}, "declared": digest,
"report_sequence": sequence}
if stated != nil {
said["health"] = health
}
body, _ := json.Marshal(said)
var report link.Report
if err := json.Unmarshal(body, &report); err != nil {
t.Fatal(err)
}
if _, err := listener.Heard(ctx, report); err != nil {
t.Fatal(err)
}
}
var sent [][]string
last := map[string]string{}
n := 0
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
sent = append(sent, append([]string(nil), names...))
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
for _, node := range names {
n++
digest := fmt.Sprintf("d-%s-%d", node, n)
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), digest, map[string]string{"app": current["app"].Commit}); err != nil {
return nil, err
}
last[node] = digest
say(node, digest, "starting")
}
return names, nil
}
t.Cleanup(func() { sendRollout = wasSend })
wasSettle, wasEvery, wasBound := gateSettle, gateEvery, gateBound
gateSettle, gateEvery = 0, 0
t.Cleanup(func() { gateSettle, gateEvery, gateBound = wasSettle, wasEvery, wasBound })
built := time.Now().UTC()
plan := inventory.Plan{ID: "plan-crashloop", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c2",
Created: built, State: inventory.PlanBuilding, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {State: "built", BuiltAt: &built, Commit: "c2", Build: "build-2"}}}
if err := inv.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
advancePlans(ctx, open) // the first machine is sent the new build, and starts it
if len(sent) == 0 || !slices.Equal(sent[0], []string{"anchor"}) {
t.Fatalf("sent %v, not the first machine first", sent)
}
advancePlans(ctx, open) // judged while it starts
// Its container exits at start, and the runtime restarts it: the machine says so, again and again.
for i := 0; i < 6 && len(sent) == 1; i++ {
say("anchor", last["anchor"], "")
advancePlans(ctx, open)
}
gateBound = -time.Second // and the bound passes
advancePlans(ctx, open)
p, err := inv.PlanByID(ctx, "plan-crashloop")
if err != nil {
t.Fatal(err)
}
gate := p.Modules["app"].Gate
for _, names := range sent {
if slices.Contains(names, "laptop") {
t.Fatalf("the crash loop passed its gate on anchor and was sent to laptop: sent %v, the gate %+v", sent, gate)
}
}
if gate == nil || gate.Verdict != inventory.GateFailed || p.State != inventory.PlanFailed {
t.Fatalf("a crash-looping build did not fail its gate on the first machine: the plan is %s (%s), its gate %+v",
p.State, p.Note, gate)
}
if current, _ := inv.CurrentBuilds(ctx); current["app"].Commit != "c1" {
t.Fatalf("the module is registered at %s, not put back to c1", current["app"].Commit)
}
}
// **R145 — a web application that accepts TCP and answers nothing is raised within two looks** (novox/hq
// ADR 0240 rule 4 and Phase B, issue 145). For eleven hours a web application's port was open and its
// program ran while every request hung, and the mesh said its machine was healthy; a person found it.
// Liveness cannot see it and a TCP check cannot either: the port is open. The module's declared HTTP check
// can. The engine's half (mesh-host internal/liveness TestReplaySilentWebAppIsSaidUnhealthy) states what it
// found looking at such a program; here the controller hears that statement on two looks in a row and
// raises the module's condition — the second, never the first. `null` is an engine older than the
// readiness check: it states the program alive, and nothing is raised.
func TestReplaySilentWebAppIsRaisedWithinTwoLooks(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
silent := []byte(`{"contract":2,"at":"2026-10-07T00:00:00Z","resources":[{"module":"app","resource":"app.server",` +
`"kind":"container","target":"app-server","state":"unhealthy","reason":"http / on web: no answer within 5s",` +
`"since":"2026-10-07T00:00:00Z","streak":3,"check":"http"}]}`)
if path := os.Getenv("MESH_REPLAY_STATEMENT"); path != "" {
raw, err := os.ReadFile(path)
if err != nil {
t.Fatalf("the engine's statement of the silent web application: %v", err)
}
silent = raw
}
var h link.Health
if err := json.Unmarshal(silent, &h); err != nil {
t.Fatal(err)
}
if h.Contract == 0 {
t.Fatal("the engine states nothing of the silent web application: it is older than the judging")
}
open1 := func() []conditions.Condition {
t.Helper()
list, err := conditionsFrom.Open(ctx)
if err != nil {
t.Fatal(err)
}
return list
}
for look := 1; look <= 2; look++ {
at := time.Now().Add(time.Duration(look) * time.Second)
h.At = at
if err := stateHealth(ctx, open.inventory, conditionsFrom, "anchor", h, at); err != nil {
t.Fatal(err)
}
raised := open1()
switch {
case look == 1 && len(raised) != 0:
t.Fatalf("one look raised %v", raised[0].Key)
case look == 2 && (len(raised) != 1 || raised[0].Key != "module.app.anchor.unhealthy"):
t.Fatalf("two looks in a row did not raise the module's condition: %v", raised)
case look == 2:
t.Logf("raised on the second look: %s", raised[0].Summary)
}
}
}
+27 -4
View File
@@ -456,6 +456,15 @@ func (a *verbArguments) commandLine() ([]string, error) {
argv = append(argv, "--condition", c)
}
return argv, nil
case "drill":
if err := need("what", "why"); err != nil {
return nil, err
}
argv := []string{"hand-act", "drill", str("what"), "--why", str("why")}
if c := str("condition"); c != "" {
argv = append(argv, "--condition", c)
}
return argv, nil
case "hand-acts":
argv := []string{"hand-acts", "--json"}
if d := str("days"); d != "" {
@@ -723,6 +732,8 @@ func repairingCommand(argv []string) string {
return "plans " + argv[1]
case argv[0] == "broker" && len(argv) > 1 && argv[1] == "consumer-reset":
return "broker consumer-reset"
case argv[0] == "hand-act" && len(argv) > 1 && argv[1] == "drill":
return "hand-act drill"
case argv[0] == "hand-act":
return "hand-act record"
case argv[0] == "conditions" && len(argv) > 1 && argv[1] == "silence":
@@ -740,12 +751,18 @@ func isWhyFlag(word string) bool {
}
// runVerb runs this binary with the given command line and gathers what it said.
//
// **This binary is the image this process runs, never the file at the path it started from**
// (novox/hq issue 289): the node-engine's witness moves a running build aside when it places the next
// one, into a directory only it may enter, and deletes it once the next is proved — while this process
// may still be serving. A verb run from the path then failed with "permission denied" for the seconds
// the old controller still answered. selfCommand runs what this process is running, wherever its file
// went; a verb it still cannot start is refused as a handover, which the caller asks again.
func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
self, err := os.Executable()
if err != nil {
return verbAnswer{}, err
if handingOver.Load() {
return verbAnswer{}, fmt.Errorf("%w: this controller is stopping and runs no new command", link.ErrHandingOver)
}
cmd := exec.CommandContext(ctx, self, argv...)
cmd := selfCommand(ctx, argv)
// The same environment: the stores' credentials, the bus, the broker — everything a command run
// from a shell in this container would have, because it is that.
cmd.Env = os.Environ()
@@ -773,6 +790,12 @@ func runVerb(ctx context.Context, argv []string) (verbAnswer, error) {
var exit *exec.ExitError
if runErr != nil && !errors.As(runErr, &exit) {
// Not the command refusing — the command not running at all, which is this process's fault.
if errors.Is(runErr, os.ErrPermission) || errors.Is(runErr, os.ErrNotExist) {
// Its own image unreachable: a build replaced under a process that has not yet stopped.
// Refused as a handover, so the caller asks the controller that follows.
return answer, fmt.Errorf("%w: could not run %s from this controller's own build: %v",
link.ErrHandingOver, strings.Join(argv, " "), runErr)
}
return answer, fmt.Errorf("could not run %s: %w", strings.Join(argv, " "), runErr)
}
return answer, nil
+46
View File
@@ -0,0 +1,46 @@
package main
import (
"context"
"os"
"os/exec"
"runtime"
"sync/atomic"
)
// startedFrom is the path this process's executable had when it started: what a verb's command line
// is named in a process listing, and what is run where the image cannot be named otherwise.
var startedFrom, _ = os.Executable()
// ownImage is how a process names the executable it is running, as long as it runs, wherever the file
// has gone since. On Linux the kernel keeps it: /proc/self/exe is the running image itself, not a path,
// so it is valid after the file is renamed into a directory this process may not enter, or deleted —
// which is what the node-engine's witness does to a build it replaces (novox/hq issue 289). Read in the
// child, it names the child's image, which until the exec is this process's.
//
// os.Executable reads the same link and returns the path it points at *now*: correct at start and
// wrong the moment the file moves, which is the fault. A variable so a test can name another.
var ownImage = func() string {
if runtime.GOOS == "linux" {
if _, err := os.Stat("/proc/self/exe"); err == nil {
return "/proc/self/exe"
}
}
return startedFrom
}
// selfCommand is this binary run with a command line: the image this process runs, named in a process
// listing as the path it started from.
func selfCommand(ctx context.Context, argv []string) *exec.Cmd {
cmd := exec.CommandContext(ctx, ownImage(), argv...)
if startedFrom != "" {
cmd.Args[0] = startedFrom
}
return cmd
}
// handingOver is set when the serving controller begins to stop — a signal from its supervisor, a lease
// lost. From then a verb that would run a command is refused as a handover rather than started and
// killed with this process: the caller asks again, and the controller after this one answers
// (novox/hq issue 289).
var handingOver atomic.Bool
+177
View File
@@ -0,0 +1,177 @@
package main
import (
"bufio"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"runtime"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// The roles a copy of this test binary plays in TestAVerbRunsWhileItsBuildIsMovedOrDeleted.
const selfExecRole = "MESH_CONTROLLER_SELFEXEC_ROLE"
// TestSelfExecServer is a controller standing in, when run as one: it waits until its build has been
// moved, then runs a verb as the seat runs one, and prints what came of it as JSON.
func TestSelfExecServer(t *testing.T) {
if os.Getenv(selfExecRole) != "server" {
t.Skip("run by TestAVerbRunsWhileItsBuildIsMovedOrDeleted")
}
line, _ := bufio.NewReader(os.Stdin).ReadString('\n')
if strings.TrimSpace(line) != "go" {
t.Fatalf("told %q", line)
}
if err := os.Setenv(selfExecRole, "verb"); err != nil {
t.Fatal(err)
}
answer, err := runVerb(t.Context(), []string{"-test.run", "^TestSelfExecVerb$", "-test.v"})
said := map[string]any{"ok": answer.OK, "output": answer.Output}
if err != nil {
said["error"] = err.Error()
}
body, _ := json.Marshal(said)
fmt.Println("ANSWER " + string(body))
}
// TestSelfExecVerb is the verb, when run as one.
func TestSelfExecVerb(t *testing.T) {
if os.Getenv(selfExecRole) != "verb" {
t.Skip("run by TestSelfExecServer")
}
fmt.Println("the verb ran")
}
// **A verb runs while the build it was started from is moved where its user may not go, or deleted**
// (novox/hq issue 289). The node-engine's witness moves a running controller's build into a directory
// only it may enter as it places the next, and deletes it once the next is proved; the controller
// still serving in between ran its verbs from the path and answered "permission denied".
//
// A copy of this test binary is the controller: started from one place, moved into a directory closed
// to everyone (or deleted), and only then asked to run a verb.
func TestAVerbRunsWhileItsBuildIsMovedOrDeleted(t *testing.T) {
if runtime.GOOS != "linux" {
t.Skip("the image is named through /proc on Linux only")
}
self, err := os.Executable()
if err != nil {
t.Fatal(err)
}
for _, how := range []string{"moved into a closed directory", "deleted"} {
t.Run(how, func(t *testing.T) {
root := t.TempDir()
placed := filepath.Join(root, "mesh-controller", "mesh-controller")
if err := os.MkdirAll(filepath.Dir(placed), 0o755); err != nil {
t.Fatal(err)
}
copyFile(t, self, placed)
server := exec.Command(placed, "-test.run", "^TestSelfExecServer$", "-test.v")
server.Env = append(os.Environ(), selfExecRole+"=server")
stdin, err := server.StdinPipe()
if err != nil {
t.Fatal(err)
}
stdout, err := server.StdoutPipe()
if err != nil {
t.Fatal(err)
}
server.Stderr = os.Stderr
if err := server.Start(); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = server.Process.Kill(); _ = server.Wait() })
// What the witness does to a running build, once the process runs.
switch how {
case "deleted":
if err := os.RemoveAll(filepath.Dir(placed)); err != nil {
t.Fatal(err)
}
default:
kept := filepath.Join(root, ".witness", "mesh-controller", "previous")
if err := os.MkdirAll(filepath.Dir(kept), 0o700); err != nil {
t.Fatal(err)
}
if err := os.Rename(filepath.Dir(placed), kept); err != nil {
t.Fatal(err)
}
if err := os.Chmod(filepath.Join(root, ".witness"), 0); err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = os.Chmod(filepath.Join(root, ".witness"), 0o700) })
}
if _, err := io.WriteString(stdin, "go\n"); err != nil {
t.Fatal(err)
}
out, _ := io.ReadAll(stdout)
_ = server.Wait()
var said struct {
OK bool `json:"ok"`
Output string `json:"output"`
Error string `json:"error"`
}
found := false
for _, line := range strings.Split(string(out), "\n") {
if rest, ok := strings.CutPrefix(line, "ANSWER "); ok {
found = json.Unmarshal([]byte(rest), &said) == nil
}
}
if !found {
t.Fatalf("the controller standing in answered nothing:\n%s", out)
}
if said.Error != "" || !said.OK || !strings.Contains(said.Output, "the verb ran") {
t.Fatalf("the verb did not run from the controller's own image after its build was %s: %+v", how, said)
}
})
}
}
// A verb the controller cannot start from its own build is refused as a handover — marked so its
// caller asks again — never a permission error; and so is one arriving once the controller is stopping.
func TestAVerbThatCannotRunIsRefusedAsAHandover(t *testing.T) {
closed := filepath.Join(t.TempDir(), "closed")
if err := os.MkdirAll(closed, 0o700); err != nil {
t.Fatal(err)
}
was := ownImage
ownImage = func() string { return filepath.Join(closed, "gone", "mesh-controller") }
t.Cleanup(func() { ownImage = was })
_, err := runVerb(t.Context(), []string{"status"})
if !errors.Is(err, link.ErrHandingOver) {
t.Fatalf("a build that is not there was answered %v, not a handover", err)
}
ownImage = was
handingOver.Store(true)
t.Cleanup(func() { handingOver.Store(false) })
if _, err := runVerb(t.Context(), []string{"-test.run", "^$"}); !errors.Is(err, link.ErrHandingOver) {
t.Fatalf("a controller stopping ran a verb: %v", err)
}
}
func copyFile(t *testing.T, from, to string) {
t.Helper()
in, err := os.Open(from)
if err != nil {
t.Fatal(err)
}
defer in.Close()
out, err := os.OpenFile(to, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0o755)
if err != nil {
t.Fatal(err)
}
if _, err := io.Copy(out, in); err != nil {
t.Fatal(err)
}
if err := out.Close(); err != nil {
t.Fatal(err)
}
}
+44
View File
@@ -550,3 +550,47 @@ func TestAWalkWaitingFourHoursIsUrgentAndNamesTheWayOn(t *testing.T) {
t.Fatalf("it does not say how on: %q", got[0].Summary)
}
}
// **A drill is a person's deliberate test, never a repair** — the log of 2026-10-07: two drills of ADR
// 0240 recorded through `hand-act record --cause drill` before `hand-act drill` existed raised
// `mesh.hand-acts.drill.healer-wanted`. A drill through its own verb never counts, the two recorded
// before it clear on the next tick, and the cause word alone on any other verb still counts — whether
// an act is a drill is said by the verb chosen, not by a word typed into a repair's cause.
func TestADrillIsNoRepairAndItsHealerWantedClears(t *testing.T) {
now := time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)
f := calm(now)
f.handActs = []link.HandAct{actOf(now.Add(-26*time.Hour), "hand-act drill", causeDrill),
actOf(now.Add(-time.Hour), "hand-act drill", causeDrill), actOf(now.Add(-time.Minute), "hand-act drill", causeDrill)}
if got := watchHandActs(f); len(got) != 0 {
t.Errorf("drills repeated asked for a healer: %+v", got)
}
for _, verb := range []string{"push", "conditions silence", "plans close", "a verb nobody listed"} {
f := calm(now)
f.handActs = []link.HandAct{actOf(now.Add(-26*time.Hour), verb, causeDrill), actOf(now.Add(-time.Hour), verb, causeDrill)}
if got := watchHandActs(f); len(got) != 1 {
t.Errorf("%s with the cause %q passed for a drill: %+v", verb, causeDrill, got)
}
}
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
defer k.Close(context.Background())
before := []conditions.Observation{{Scope: conditions.ScopeMesh, ID: "hand-acts." + causeDrill,
Token: "healer-wanted", Kind: "healer-wanted", Severity: conditions.Warning,
Summary: "\"drill\" was repaired by hand 2 times in 14 days"}}
if err := k.Reconcile(t.Context(), "S15", before); err != nil {
t.Fatal(err)
}
if open, _ := k.Open(t.Context()); len(open) != 1 {
t.Fatalf("the condition of the build before was not open: %+v", open)
}
f = calm(now)
f.handActs = []link.HandAct{actOf(now.Add(-11*time.Hour), "hand-act record", causeDrill),
actOf(now.Add(-30*time.Minute), "hand-act record", causeDrill)}
w := &watchdogs{keeper: k, started: now.Add(-time.Hour)}
w.see(t.Context(), f)
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("a healer-wanted for two drills stayed open: %+v", open)
}
}
+10
View File
@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"fmt"
"os"
"sync"
"time"
@@ -204,6 +205,15 @@ func (l nudgingListener) Heard(ctx context.Context, report link.Report) (bool, e
if news {
l.summary.nudge()
}
// What it says of its long-running resources (novox/hq ADR 0240): kept, and its modules' conditions
// raised or cleared from it. Only from an account of the machine the store took.
if err == nil && report.Health != nil && report.Superseded == "" && report.Rekey == nil && report.Node != "" &&
l.Enrolment.Inventory != nil {
if herr := stateHealth(ctx, l.Enrolment.Inventory, conditionsFrom, report.Node, *report.Health, now); herr != nil {
fmt.Fprintf(os.Stderr, "mesh-controller: could not keep what %s says of its resources' health: %v\n",
report.Node, herr)
}
}
if err == nil && l.open != nil && startedWell(report) {
// Off the report's path: replacing a given value sends the machine, and a report waits for
// nothing it caused (novox/hq ADR 0228).
+1 -1
View File
@@ -35,4 +35,4 @@ require (
// committed. Every build (the build agent's `go build`, the Dockerfile) compiles from vendor/ and
// fetches nothing; go refuses to build when vendor/ and this file disagree, so a pin moved without
// `go mod vendor` fails loudly, at once, everywhere.
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261006095519-3e80b7ae325e
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261007120832-bdd44154ccac
+2
View File
@@ -1,5 +1,7 @@
git.novox.be/novox/mesh-host v0.0.0-20261006095519-3e80b7ae325e h1:g9h4QRaAMg5yaJLwqtb0FoOs23DVGUYpW6qvnQ3oY5A=
git.novox.be/novox/mesh-host v0.0.0-20261006095519-3e80b7ae325e/go.mod h1:VlilMCRZ5yyNXg7SNigNBLr0Gt32jrGw5KSNq5JAVYs=
git.novox.be/novox/mesh-host v0.0.0-20261007120832-bdd44154ccac h1:KvnKtJ2rWeIE/t4GweK+JL0OjKSNxsrVP3/nMdpii8o=
git.novox.be/novox/mesh-host v0.0.0-20261007120832-bdd44154ccac/go.mod h1:VlilMCRZ5yyNXg7SNigNBLr0Gt32jrGw5KSNq5JAVYs=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+25
View File
@@ -122,6 +122,10 @@ type Principal struct {
// key, and nothing else of the bucket, so it can judge a new controller build and put the previous
// one back.
WitnessesController bool
// Checks are the tools a machine principal's node-engine asks as a declared health check, each
// `<module>.<tool>` (novox/hq ADR 0240, to-be 48 §3): asked of the instance on its own machine and
// nowhere else.
Checks []string
// PasswordHash is the bcrypt hash the mesh minted. The plaintext is sealed to the principal
// and never appears here: this file is written to a node's disk and read by a server, and a
@@ -1021,5 +1025,26 @@ func WitnessSubjects(p Principal) []string {
if p.WitnessesController {
out = append(out, lease.LeaseReadSubject(LeaseBucket))
}
return append(out, CheckSubjects(p)...)
}
// CheckSubjects are the tools a machine's node-engine asks as declared health checks (novox/hq ADR 0240,
// to-be 48 §3): each `<module>.<tool>` on this machine's instance — `mesh.mod.<module>.tool.<tool>.<node>`
// — and never the plain subject, which any machine's instance may answer. Sorted and once each.
func CheckSubjects(p Principal) []string {
seen := map[string]bool{}
var out []string
for _, c := range p.Checks {
module, tool, ok := strings.Cut(c, ".")
if !ok || !safeSubject.MatchString(module) || !safeSubject.MatchString(tool) {
continue
}
subject := "mesh.mod." + module + ".tool." + tool + "." + p.Node
if !seen[subject] {
seen[subject] = true
out = append(out, subject)
}
}
sort.Strings(out)
return out
}
+6 -1
View File
@@ -44,6 +44,9 @@ type Declared struct {
// SnapshotsTheBus says the module holds mesh-broker — it is the bus — and so is the one module
// granted the snapshot API, to copy the bus's streams for the night's backup (novox/hq ADR 0235).
SnapshotsTheBus bool
// Checks are the module's own tools its health asks, each `<module>.<tool>` (novox/hq ADR 0240, to-be
// 48 §3): the machine's node-engine asks them of its own node tools, and is granted that and no more.
Checks []string
}
// Records is what composing a user list needs to know about the mesh, and nothing more.
@@ -73,12 +76,14 @@ func Users(r Records) ([]Principal, error) {
for _, node := range sortedCopy(r.Nodes) {
witness := false
var checks []string
for _, d := range r.Assigned[node] {
if d.Module == controllerModule {
witness = true
}
checks = append(checks, d.Checks...)
}
out = append(out, Principal{Kind: KindNode, Node: node, WitnessesController: witness})
out = append(out, Principal{Kind: KindNode, Node: node, WitnessesController: witness, Checks: checks})
// **Where the runtime is assigned, the machine gets one runtime principal in place of the
// runtime module's own** (novox/hq ADR 0175, to-be 38). It carries every module on the
// node: its serving grants are the union of theirs. Every other module keeps its own
+28
View File
@@ -35,3 +35,31 @@ func TestTheWitnessIsGrantedWhatItReadsAndNoMore(t *testing.T) {
}
}
}
// A module's health that asks one of its own tools is asked by the machine's node-engine, of the instance
// on its own machine and nowhere else (novox/hq ADR 0240, to-be 48 §3).
func TestTheEngineIsGrantedTheToolsItsModulesHealthAsks(t *testing.T) {
users, err := Users(Records{Nodes: []string{"control", "edge"},
Assigned: map[string][]Declared{"edge": {{Module: "keycloak", Checks: []string{"keycloak.keycloak_admin_health"}}}}})
if err != nil {
t.Fatal(err)
}
for _, u := range users {
if u.Kind != KindNode {
continue
}
perms, err := PermissionsFor(u)
if err != nil {
t.Fatal(err)
}
mine := "mesh.mod.keycloak.tool.keycloak_admin_health.edge"
if got := slices.Contains(perms.Publish, mine); got != (u.Node == "edge") {
t.Errorf("%s may ask keycloak's health tool on edge: %v", u.Node, got)
}
for _, p := range perms.Publish {
if p == "mesh.mod.keycloak.tool.keycloak_admin_health" || p == "mesh.mod.keycloak.tool.>" {
t.Errorf("%s may ask the tool of any machine: %s", u.Node, p)
}
}
}
}
+3 -1
View File
@@ -86,7 +86,9 @@ var WritersTable = []WriterRow{
Others: "read", Subjects: []string{"mesh.node.*.declare"}, Writes: isController},
{State: "a machine's applied state and its report", Writer: "the node-engine's apply queue",
KeptIn: "the machine; the report on the bus", Others: "the reconcile and a delivery enqueue, never apply",
Subjects: []string{"mesh.control.*.report"}, Writes: ownMachine},
// And its health statement between reports (novox/hq ADR 0240): the same writer stating the same
// machine, inside the grant it already had (`mesh.control.<its own>.>`).
Subjects: []string{"mesh.control.*.report", "mesh.control.*.health"}, Writes: ownMachine},
{State: "the controller lease", Writer: "the controller instance holding it", KeptIn: "key-value " + LeaseBucket,
Others: "a candidate waits", Subjects: kvOf(LeaseBucket), Writes: isController},
{State: "plans and their tiers", Writer: "controller (lease holder), compare-and-set on the plan's revision",
+69 -7
View File
@@ -298,6 +298,13 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
say("check", "the facts of %s (%s): %d machine(s), the bus at %s, the store at %s", f.Taken.Format(time.RFC3339),
short(strings.TrimPrefix(digest, "sha256:")), len(f.Machines), f.Versions.Bus, f.Versions.Store)
// **What an earlier delivery of this same ask left is removed first** (novox/hq issue 285): an ask is
// redelivered when the holder that took it stopped mid-check — the build agent itself updated by the
// rollout it is checking — and its containers, named by the ask's id, are still there. Raising the
// store again under that name was refused, and the check said it could not run.
if n, err := RemoveContainersOf(ctx, run, spec.ID); err == nil && n > 0 {
say("check", "removed %d throwaway container(s) an earlier delivery of this check left", n)
}
// The throwaway store and bus, of the versions the mesh runs, removed whatever happens.
defer func() {
removing, done := context.WithTimeout(context.Background(), time.Minute)
@@ -519,21 +526,24 @@ func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFil
case "fail":
return "fail", said.Summary
case "error":
// The gate could not raise the mesh as it is (novox/hq issue 285): said in its own words.
// The gate could not raise the mesh as it is (novox/hq issue 282): said in its own words.
return "error", said.Summary
}
}
// The gate could not judge: not the change's fault, and never a pass.
return "error", "the merge gate could not judge the change: " + lastLine(out.String())
}
var said struct {
Verdict string `json:"verdict"`
Summary string `json:"summary"`
}
raw, err := os.ReadFile(verdictFile)
if err != nil || json.Unmarshal(raw, &said) != nil || said.Verdict == "" {
if err != nil {
return "error", "the merge gate said no verdict"
}
said, ok := readGateVerdict(raw)
if !ok {
return "error", "the merge gate said no verdict"
}
if verdict, summary, raised := gateRaisedTheMesh(said); !raised {
return verdict, summary
}
// 3. **The replays of what the mesh runs** (to-be 45 §9, M9): mesh-lab's, from its main — reviewed code,
// so given the container runtime the resolver replay raises containers with — against the bus of the
@@ -683,6 +693,58 @@ func newProblems(change, base string) []string {
return out
}
// gateVerdict is what of the gate's verdict the seat reads: the verdict, and every machine — whether it
// composes on the mesh and whether it composed in the gate's store without the change.
type gateVerdict struct {
Verdict string `json:"verdict"`
Summary string `json:"summary"`
Machines []struct {
Described string `json:"described"`
Live bool `json:"live-composes"`
Base struct {
Composes bool `json:"composes"`
} `json:"base"`
} `json:"machines"`
}
// readGateVerdict reads the verdict the gate wrote, from its first line that opens a JSON document: a
// judge from before the verdict was alone on its output printed what composition said ahead of it.
func readGateVerdict(raw []byte) (gateVerdict, bool) {
var v gateVerdict
text := string(raw)
if i := strings.Index(text, "\n{"); i >= 0 && !strings.HasPrefix(strings.TrimSpace(text), "{") {
text = text[i+1:]
}
if json.Unmarshal([]byte(text), &v) != nil || v.Verdict == "" {
return gateVerdict{}, false
}
return v, true
}
// gateRaisedTheMesh is the seat's own reading of a verdict that passes (novox/hq issue 285): **a machine
// the mesh composes that did not compose in the gate's store without the change makes the gate an error,
// never a pass** — the change was judged against a machine that is not the mesh's, broken against broken.
// The judge says so itself since issue 285, but the judge is the controller the mesh runs, and one from
// before it passed such a verdict; read here too, the rule holds whatever judged. It answers false, with
// the verdict to report, when the gate did not raise the mesh.
func gateRaisedTheMesh(v gateVerdict) (string, string, bool) {
if v.Verdict != "pass" && v.Verdict != "warning" {
return "", "", true
}
var unraised []string
for _, m := range v.Machines {
if m.Live && !m.Base.Composes {
unraised = append(unraised, m.Described)
}
}
if len(unraised) == 0 {
return "", "", true
}
return "error", fmt.Sprintf("the mesh as it is could not be raised, so the change cannot be judged against it: "+
"%d of %d machines compose on the mesh and not in the gate (the first: %s) — novox/hq issue 285",
len(unraised), len(v.Machines), unraised[0]), false
}
// gitShow is a file as a ref has it.
func gitShow(ctx context.Context, dir, ref, file string) ([]byte, error) {
cmd := exec.CommandContext(ctx, "git", "show", ref+":"+filepath.ToSlash(file))
@@ -835,7 +897,7 @@ func (t *tail) String() string {
// whatFailed is the line of a failed script's output that says what failed, for the status a pull request
// shows: the first failing test, the first failing package, the files not formatted — a bare "FAIL" or a
// file's name said nothing a reader could act on (novox/hq issue 286) — and the last line otherwise.
// file's name said nothing a reader could act on (novox/hq issue 283) — and the last line otherwise.
func whatFailed(s string) string {
lines := strings.Split(strings.TrimSpace(s), "\n")
for i, line := range lines {
+52 -1
View File
@@ -415,7 +415,7 @@ func TestABuildSaysWhetherItsCommitIsOnTheTrunk(t *testing.T) {
}
}
// **Issue 286**: a failed merge-check.sh was said by its last line — a bare "FAIL", or the name of a file
// **Issue 283**: a failed merge-check.sh was said by its last line — a bare "FAIL", or the name of a file
// gofmt listed — which named nothing a reader could act on. The status says what failed.
func TestAFailedScriptIsSaidByWhatFailed(t *testing.T) {
for out, want := range map[string]string{
@@ -429,3 +429,54 @@ func TestAFailedScriptIsSaidByWhatFailed(t *testing.T) {
}
}
}
// **Issue 285**: a judge from before the rule passed "every machine composes with the change as it did
// without (0 of 4 compose)". The seat reads the machines itself: a machine the mesh composes that did not
// compose in the gate's store makes the gate an error, whatever judged it.
func TestTheSeatCallsAGateThatRaisedNoMachineAnError(t *testing.T) {
old := "bus users without a credential: controller\n" + `{"verdict":"pass","summary":"every machine composes with the change as it did without (0 of 2 compose)",
"machines":[{"described":"the hub","live-composes":true,"base":{"composes":false}},
{"described":"a machine","live-composes":true,"base":{"composes":false}}]}`
v, ok := readGateVerdict([]byte(old))
if !ok {
t.Fatal("a verdict after a line of composition's was not read")
}
verdict, summary, raised := gateRaisedTheMesh(v)
if raised || verdict != "error" || !strings.Contains(summary, "2 of 2 machines") || !strings.Contains(summary, "the hub") {
t.Errorf("0 of 2 composing is %q %q", verdict, summary)
}
good := `{"verdict":"pass","summary":"(2 of 2 compose)","machines":[{"described":"the hub","live-composes":true,"base":{"composes":true}},
{"described":"a laptop","live-composes":false,"base":{"composes":false}}]}`
v, _ = readGateVerdict([]byte(good))
if _, _, raised := gateRaisedTheMesh(v); !raised {
t.Error("a machine that does not compose on the mesh either was read as the gate's failure")
}
failed := `{"verdict":"fail","summary":"x","machines":[{"described":"the hub","live-composes":true,"base":{"composes":false}}]}`
v, _ = readGateVerdict([]byte(failed))
if _, _, raised := gateRaisedTheMesh(v); !raised {
t.Error("a failing verdict is the change's, and stands")
}
}
// **Issue 285**: an ask redelivered after its holder stopped mid-check found its own throwaway store still
// there under its name, and the check said it could not run. What an earlier delivery left goes first.
func TestARedeliveredCheckRemovesWhatItsEarlierDeliveryLeft(t *testing.T) {
registry := checkEnvironment(t)
repo, head := aCheckedRepository(t, map[string]string{CheckScript: "echo checked\n"})
id := fmt.Sprintf("check-again-%d", time.Now().UnixNano())
if out, err := exec.Command("docker", "run", "-d", "--rm", "--label", BuildLabel+"="+id, "--name", id+"-store",
"postgres:17-alpine", "sleep", "300").CombinedOutput(); err != nil {
t.Skipf("no container for the earlier delivery: %v %s", err, out)
}
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: repo, Ref: head, Owner: "novox",
Repo: "hq", Number: 7, Toolchain: goToolchain}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatalf("a redelivered check could not run: %v", err)
}
if v.Repo == nil || v.Repo.Verdict != "pass" {
t.Errorf("the repository's check answered %+v", v.Repo)
}
if left := labelled(id); len(left) > 0 {
t.Errorf("the check left %d container(s) behind", len(left))
}
}
+9
View File
@@ -235,6 +235,12 @@ type Rendering struct {
// mounts and environment. A node setting fixed at installation; empty means the default,
// /var/lib — see dir_into.go.
DataRoot string
// ReadsHealth says this machine's node-engine reads a resource's `health` (novox/hq ADR 0240 Phase B:
// its statement's contract is link.ReadinessContract or later). An older engine parses strictly and
// refuses a field it does not know, whole — so to it the field is not sent, and what it runs is
// judged by liveness alone.
ReadsHealth bool
}
// machinePort is where a module's port lives on this machine, or the port itself when the mesh has
@@ -957,6 +963,9 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
return nil, err
}
publishedOn(copied, m.Module, with)
// How it is ready, in the node-engine's words: its endpoint as the port this machine
// published it on — or not sent at all to an engine older than the field (ADR 0240).
healthInto(copied, m, with)
// The account's environment and every module's shell code, where this module holds the
// seat that places them (novox/hq ADR 0203, ADR 0204). Gathered from every module on
// the node, as the jails are, and **last of every placeholder pass**: shell code is a
+460
View File
@@ -0,0 +1,460 @@
package catalogue
import (
"fmt"
"sort"
"strconv"
"strings"
"time"
)
// A module says how each long-running resource is ready (novox/hq ADR 0240 rule 2, to-be 48 §2 and §8,
// Phase B).
//
// **On the resource, beside its other fields**: one kind and its timing. The node-engine judges every
// long-running resource alive with no declaration (Phase A); this is how a module says what *ready* means
// for one — the image's own check adopted by name, an HTTP request to a declared endpoint, a TCP connect,
// a command in the container, the unit's own readiness, or one of the module's own tools.
//
// **An endpoint is named, never a port or an address**: the check follows the machine's port for that
// endpoint as the endpoint does, so a port this machine gave elsewhere moves the check with it. The
// controller composes the name into the port the machine published it on, and sends the field only to a
// node-engine that reads it (link.ReadinessContract): an older, strict engine refuses a field it does not
// know, whole.
//
// **The bounds are the record's**: an interval not under ten seconds, a timeout under the interval, at
// least two failing looks in a row, and a grace plus the failing looks within five minutes — so a
// resource broken from its start is said within the gate's ten. Refused here, near the author, and again
// by the node-engine, far away, in the same words.
// HealthField is the resource field.
const HealthField = "health"
// The kinds of check (to-be 48 §2).
const (
HealthRuntime = "runtime"
HealthHTTP = "http"
HealthTCP = "tcp"
HealthExec = "exec"
HealthUnit = "unit"
HealthTool = "tool"
)
// The bounds and the defaults (ADR 0240 rule 2).
const (
HealthIntervalDefault = 30 * time.Second
HealthIntervalFloor = 10 * time.Second
HealthTimeoutDefault = 5 * time.Second
HealthLooksDefault = 3
HealthLooksFloor = 2
HealthGraceDefault = 60 * time.Second
// HealthWithin is the most a grace and the failing looks may take together: a resource broken
// from its start is said within the gate's ten minutes with room for its judgings.
HealthWithin = 5 * time.Minute
)
// HealthRequiredFrom is when `module check` refuses a long-running resource without `health` (ADR 0240
// rule 8): six weeks after liveness was first judged live (2026-10-07), unless the catalogue's count of
// undeclared resources reached zero first — which its own counter enforces by never letting it rise.
var HealthRequiredFrom = time.Date(2026, 11, 18, 0, 0, 0, 0, time.UTC)
// Health is one resource's declaration, read.
type Health struct {
Kind string
// Endpoint is the `listens` name an http or tcp check looks at.
Endpoint string
// Path, Status, Body and Scheme are an http check's: the path asked, the status expected (zero: any
// status under 400), a text the answer must hold, and http or https.
Path string
Status int
Body string
Scheme string
// Command is an exec check's command, run by the container's shell.
Command string
// Tool is a tool check's tool, one of the module's own.
Tool string
// The timing, with the defaults applied.
Interval, Timeout, Grace time.Duration
Looks int
// Needs is the provision the check exercises (to-be 48 §6): while its provider for this consumer is
// unhealthy, what this check finds is held under the provider's condition.
Needs string
}
// healthKeys are the keys a `health` field may carry; anything else is refused by name.
var healthKeys = map[string]bool{"kind": true, "endpoint": true, "path": true, "status": true, "body": true,
"scheme": true, "command": true, "tool": true, "interval": true, "timeout": true, "looks": true,
"grace": true, "needs": true}
// healthAddressKeys are what a check may not be aimed by: a port or an address does not follow the
// machine's port for the endpoint, and a manifest is the same on every machine.
var healthAddressKeys = map[string]bool{"port": true, "address": true, "host": true, "url": true, "ip": true}
// LongRunning says whether a manifest resource stays up: a container that is no step and on no schedule,
// a service stated running, a process that is no step and on no schedule (ADR 0240 rule 1).
func LongRunning(r map[string]any) bool {
switch fmt.Sprint(r["type"]) {
case "container", "process":
if once, _ := r["run-once"].(bool); once {
return false
}
return r["schedule"] == nil
case "service":
return fmt.Sprint(r["state"]) == "running"
}
return false
}
// ReadHealth reads a resource's `health` field, defaults applied. False when it carries none.
func ReadHealth(r map[string]any) (Health, bool, []string) {
raw, present := r[HealthField]
if !present {
return Health{}, false, nil
}
id := fmt.Sprint(r["id"])
fields, ok := raw.(map[string]any)
if !ok {
return Health{}, true, []string{fmt.Sprintf("%s: health is %T; it is an object with a kind and its timing", id, raw)}
}
var problems []string
keys := make([]string, 0, len(fields))
for k := range fields {
keys = append(keys, k)
}
sort.Strings(keys)
for _, k := range keys {
switch {
case healthAddressKeys[k]:
problems = append(problems, fmt.Sprintf("%s: its health names a %s; a check names an endpoint the module "+
"declares under listens, by its name, so it follows the port this machine gives it (ADR 0240 rule 2)", id, k))
case !healthKeys[k]:
problems = append(problems, fmt.Sprintf("%s: its health says %q, which a health check does not have", id, k))
}
}
text := func(key string) string {
v, present := fields[key]
if !present {
return ""
}
s, ok := v.(string)
if !ok {
problems = append(problems, fmt.Sprintf("%s: its health's %s is %T; it is text", id, key, v))
}
return strings.TrimSpace(s)
}
duration := func(key string, fallback time.Duration) time.Duration {
s := text(key)
if s == "" {
return fallback
}
d, err := time.ParseDuration(s)
if err != nil || d < 0 {
problems = append(problems, fmt.Sprintf("%s: its health's %s is %q; it is a duration such as \"30s\"", id, key, s))
return fallback
}
return d
}
number := func(key string, fallback int) int {
v, present := fields[key]
if !present {
return fallback
}
f, ok := v.(float64)
if !ok || f != float64(int(f)) {
problems = append(problems, fmt.Sprintf("%s: its health's %s is %v; it is a whole number", id, key, v))
return fallback
}
return int(f)
}
h := Health{Kind: text("kind"), Endpoint: text("endpoint"), Path: text("path"), Body: text("body"),
Scheme: text("scheme"), Command: text("command"), Tool: text("tool"), Needs: text("needs"),
Interval: duration("interval", HealthIntervalDefault), Timeout: duration("timeout", HealthTimeoutDefault),
Grace: duration("grace", HealthGraceDefault), Looks: number("looks", HealthLooksDefault),
Status: number("status", 0)}
return h, true, problems
}
// healthProblems is everything wrong with a manifest's `health` fields, in the manifest's words (to-be
// 48 §8): a field on something that does not stay up, a kind its resource cannot have, an endpoint the
// module does not declare, a port or an address, a timing outside its bounds, a tool the module does not
// serve, and a tool check with no check of another kind beside it on the module.
func healthProblems(m Manifest) []string {
var problems []string
endpoints := map[string]Listening{}
for _, l := range m.Listens {
if n := strings.TrimSpace(l.Name); n != "" {
endpoints[n] = l
}
}
tools := map[string]bool{}
for _, t := range m.Tools {
tools[t] = true
}
wants := map[string]bool{}
for _, w := range m.Wants() {
wants[w] = true
}
var toolChecks []string
otherKinds := 0
for _, r := range m.Resources {
h, has, read := ReadHealth(r)
if !has {
continue
}
id, typ := fmt.Sprint(r["id"]), fmt.Sprint(r["type"])
where := m.Module + ": " + id
for _, p := range read {
problems = append(problems, m.Module+": "+p)
}
if !LongRunning(r) {
problems = append(problems, fmt.Sprintf("%s declares health and does not stay up: a step, anything on a "+
"schedule and a service not stated running are judged by their step and their schedule (ADR 0240 rule 1)", where))
continue
}
switch h.Kind {
case HealthRuntime, HealthExec:
if typ != "container" {
problems = append(problems, fmt.Sprintf("%s is a %s and its health is %q, which only a container has: "+
"the runtime runs it inside the container", where, typ, h.Kind))
}
case HealthUnit:
if typ == "container" {
problems = append(problems, fmt.Sprintf("%s is a container and its health is %q, which is a service's "+
"or a process's own readiness", where, h.Kind))
}
case HealthHTTP, HealthTCP, HealthTool:
case "":
problems = append(problems, fmt.Sprintf("%s declares health with no kind: %s", where, healthKindsWords()))
default:
problems = append(problems, fmt.Sprintf("%s declares health of kind %q: %s", where, h.Kind, healthKindsWords()))
}
switch h.Kind {
case HealthHTTP, HealthTCP:
l, declared := endpoints[h.Endpoint]
switch {
case h.Endpoint == "":
problems = append(problems, fmt.Sprintf("%s's %s check names no endpoint: it names one the module "+
"declares under listens, by its name", where, h.Kind))
case !declared:
problems = append(problems, fmt.Sprintf("%s's %s check names the endpoint %q, which %s does not declare "+
"under listens (%s)", where, h.Kind, h.Endpoint, m.Module, namedEndpointsWords(endpoints)))
case l.At() != "tcp":
problems = append(problems, fmt.Sprintf("%s's %s check names %q, which is %s: a check connects over tcp",
where, h.Kind, h.Endpoint, l.At()))
}
default:
if h.Endpoint != "" {
problems = append(problems, fmt.Sprintf("%s's %s check names an endpoint, which only an http or tcp check "+
"looks at", where, h.Kind))
}
}
if h.Kind != HealthHTTP && (h.Path != "" || h.Status != 0 || h.Body != "" || h.Scheme != "") {
problems = append(problems, fmt.Sprintf("%s's %s check says a path, a status, a body or a scheme, which "+
"only an http check has", where, h.Kind))
}
if h.Kind == HealthHTTP {
if h.Path != "" && !strings.HasPrefix(h.Path, "/") {
problems = append(problems, fmt.Sprintf("%s's http check asks %q; a path starts with /", where, h.Path))
}
if h.Status != 0 && (h.Status < 100 || h.Status > 599) {
problems = append(problems, fmt.Sprintf("%s's http check expects status %d, which is not one", where, h.Status))
}
if h.Scheme != "" && h.Scheme != "http" && h.Scheme != "https" {
problems = append(problems, fmt.Sprintf("%s's http check is over %q; it is http or https", where, h.Scheme))
}
}
if (h.Command != "") != (h.Kind == HealthExec) {
if h.Kind == HealthExec {
problems = append(problems, fmt.Sprintf("%s's exec check says no command", where))
} else {
problems = append(problems, fmt.Sprintf("%s's %s check says a command, which only an exec check runs",
where, h.Kind))
}
}
if h.Kind == HealthTool {
switch {
case h.Tool == "":
problems = append(problems, fmt.Sprintf("%s's tool check names no tool", where))
case !tools[h.Tool]:
problems = append(problems, fmt.Sprintf("%s's tool check asks %q, which %s does not serve (its tools: %s)",
where, h.Tool, m.Module, orNoneWords(m.Tools)))
}
toolChecks = append(toolChecks, id)
} else {
if h.Tool != "" {
problems = append(problems, fmt.Sprintf("%s's %s check names a tool, which only a tool check asks", where, h.Kind))
}
if h.Kind != "" {
otherKinds++
}
}
if h.Needs != "" && !wants[h.Needs] {
problems = append(problems, fmt.Sprintf("%s's check needs %q, which %s does not require: a check names "+
"the provision it exercises, among those the module requires", where, h.Needs, m.Module))
}
problems = append(problems, healthTimingProblems(where, h)...)
}
if len(toolChecks) > 0 && otherKinds == 0 {
problems = append(problems, fmt.Sprintf("%s judges itself only by its own tool (%s): a tool check is for "+
"function no endpoint shows, and only beside a check of another kind the module does not run itself "+
"(ADR 0227 rule 8, ADR 0240 rule 2)", m.Module, strings.Join(toolChecks, ", ")))
}
return problems
}
// healthTimingProblems holds a check's timing to its bounds.
func healthTimingProblems(where string, h Health) []string {
var problems []string
if h.Interval < HealthIntervalFloor {
problems = append(problems, fmt.Sprintf("%s looks every %s; a check looks no more often than every %s — "+
"the mesh is a guest on the machine (ADR 0240 rule 2)", where, h.Interval, HealthIntervalFloor))
}
if h.Timeout <= 0 || h.Timeout >= h.Interval {
problems = append(problems, fmt.Sprintf("%s gives a look %s, which must be more than nothing and under its "+
"interval of %s", where, h.Timeout, h.Interval))
}
if h.Looks < HealthLooksFloor {
problems = append(problems, fmt.Sprintf("%s is unhealthy after %d failing look(s); it is at least %d — one "+
"look can be wrong (issue 277)", where, h.Looks, HealthLooksFloor))
}
if h.Grace < 0 {
problems = append(problems, fmt.Sprintf("%s has a grace of %s", where, h.Grace))
}
if h.Looks >= HealthLooksFloor && h.Interval >= HealthIntervalFloor {
if took := h.Grace + time.Duration(h.Looks)*h.Interval; took > HealthWithin {
problems = append(problems, fmt.Sprintf("%s is said unhealthy at the earliest %s after it starts (a grace "+
"of %s and %d looks every %s); it is at most %s, so a resource broken from its start is said within "+
"the gate's bound", where, took, h.Grace, h.Looks, h.Interval, HealthWithin))
}
}
return problems
}
func healthKindsWords() string {
return "a check is runtime (the image's own, adopted by name), http, tcp, exec, unit or tool"
}
func namedEndpointsWords(endpoints map[string]Listening) string {
if len(endpoints) == 0 {
return "it names none"
}
names := make([]string, 0, len(endpoints))
for n := range endpoints {
names = append(names, n)
}
sort.Strings(names)
return "it names " + strings.Join(names, ", ")
}
func orNoneWords(names []string) string {
if len(names) == 0 {
return "none"
}
return strings.Join(names, ", ")
}
// Undeclared is every long-running resource of the manifest without `health`, by id (ADR 0240 rule 8):
// what the catalogue's count counts.
func Undeclared(m Manifest) []string {
var out []string
for _, r := range m.Resources {
if !LongRunning(r) {
continue
}
if _, has := r[HealthField]; !has {
out = append(out, fmt.Sprint(r["id"]))
}
}
return out
}
// HealthChecks is every tool a module's health asks, as `<module>.<tool>`: what a machine's node-engine
// is granted to ask of its own node tools (to-be 48 §3).
func HealthChecks(m Manifest) []string {
var out []string
for _, r := range m.Resources {
h, has, _ := ReadHealth(r)
if has && h.Kind == HealthTool && h.Tool != "" && LongRunning(r) {
out = append(out, m.Module+"."+h.Tool)
}
}
sort.Strings(out)
return out
}
// healthInto composes a resource's `health` into the node-engine's words, or takes it away (to-be 48 §2,
// §3): for an engine that reads it, the endpoint becomes the port this machine published it on and the
// defaults are written out; for one that does not — older and strict — the field is not sent, and the
// resource is judged by liveness alone, as before.
func healthInto(resource map[string]any, m Manifest, with Rendering) {
if _, has := resource[HealthField]; !has {
return
}
if !with.ReadsHealth {
delete(resource, HealthField)
return
}
h, _, _ := ReadHealth(resource)
out := map[string]any{"kind": h.Kind, "interval": h.Interval.String(), "timeout": h.Timeout.String(),
"looks": h.Looks, "grace": h.Grace.String()}
if h.Endpoint != "" {
out["endpoint"] = h.Endpoint
if port, ok := EndpointPort(m, h.Endpoint); ok {
out["port"] = with.machinePort(m.Module, port)
}
}
if h.Kind == HealthHTTP {
path := h.Path
if path == "" {
path = "/"
}
out["path"] = path
if h.Status != 0 {
out["status"] = h.Status
}
if h.Body != "" {
out["body"] = h.Body
}
if h.Scheme != "" {
out["scheme"] = h.Scheme
}
}
if h.Command != "" {
out["command"] = h.Command
}
if h.Tool != "" {
out["tool"] = h.Tool
}
if h.Needs != "" {
out["needs"] = h.Needs
}
resource[HealthField] = out
}
// HealthWords is a declared check in a line, for `module check` and `node show`.
func HealthWords(h Health) string {
var what string
switch h.Kind {
case HealthHTTP:
what = "http " + orSlash(h.Path) + " on " + h.Endpoint
if h.Status != 0 {
what += " expecting " + strconv.Itoa(h.Status)
}
case HealthTCP:
what = "tcp on " + h.Endpoint
case HealthTool:
what = "its tool " + h.Tool
case HealthRuntime:
what = "its image's own check"
default:
what = h.Kind
}
return fmt.Sprintf("%s every %s", what, h.Interval)
}
func orSlash(p string) string {
if p == "" {
return "/"
}
return p
}
+217
View File
@@ -0,0 +1,217 @@
package catalogue
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/validate"
)
// A module says how each long-running resource is ready (novox/hq ADR 0240 rule 2, to-be 48 §8): `module
// check` refuses each part out of its bounds, each endpoint named by a port or an address, a tool the
// module does not serve, and a tool check alone — a test per refusal.
const healthDigest = "@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
// healthManifest is a module of one web container, one running service and one step, with the health
// given on the container (or on the resource named by on).
func healthManifest(t *testing.T, on string, health map[string]any, more ...map[string]any) []byte {
t.Helper()
resources := []map[string]any{
{"id": "server", "type": "container", "name": "app-server", "image": "registry.example/app" + healthDigest,
"ports": []any{"8080"}},
{"id": "daemon", "type": "service", "unit": "app.service", "state": "running"},
{"id": "seed", "type": "container", "name": "app-seed", "image": "registry.example/app" + healthDigest,
"run-once": true},
{"id": "sweep", "type": "container", "name": "app-sweep", "image": "registry.example/app" + healthDigest,
"schedule": "0 3 * * *"},
}
resources = append(resources, more...)
for _, r := range resources {
if r["id"] == on && health != nil {
r["health"] = health
}
}
m := map[string]any{"module": "app", "requires": []any{"postgres-database"}, "tools": []any{"app_status"},
"listens": []any{
map[string]any{"name": "web", "port": 8080, "from": "mesh"},
map[string]any{"name": "beacon", "port": 9999, "protocol": "udp", "from": "mesh"},
},
"resources": resources}
raw, err := json.Marshal(m)
if err != nil {
t.Fatal(err)
}
return raw
}
func TestAWellFormedHealthIsAccepted(t *testing.T) {
for _, h := range []map[string]any{
{"kind": "http", "endpoint": "web", "path": "/healthz", "status": 200, "body": "ok", "needs": "postgres-database"},
{"kind": "tcp", "endpoint": "web", "interval": "10s", "timeout": "2s", "looks": 2, "grace": "0s"},
{"kind": "runtime"},
{"kind": "exec", "command": "pg_isready -q", "grace": "4m", "looks": 2, "interval": "30s"},
} {
if _, err := ParseManifest(healthManifest(t, "server", h)); err != nil {
t.Errorf("%v was refused: %v", h, err)
}
}
if _, err := ParseManifest(healthManifest(t, "daemon", map[string]any{"kind": "unit"})); err != nil {
t.Errorf("a service's own readiness was refused: %v", err)
}
// A tool beside a check of another kind on the module.
raw := healthManifest(t, "server", map[string]any{"kind": "http", "endpoint": "web"},
map[string]any{"id": "admin", "type": "service", "unit": "app-admin.service", "state": "running",
"health": map[string]any{"kind": "tool", "tool": "app_status"}})
if _, err := ParseManifest(raw); err != nil {
t.Errorf("a tool check beside an http check was refused: %v", err)
}
}
func TestEveryOutOfBoundsHealthIsRefusedByName(t *testing.T) {
cases := []struct {
name string
on string
health map[string]any
says string
}{
{"no kind", "server", map[string]any{"endpoint": "web"}, "with no kind"},
{"an unknown kind", "server", map[string]any{"kind": "ping"}, `of kind "ping"`},
{"a port", "server", map[string]any{"kind": "tcp", "port": 8080}, "names a port"},
{"an address", "server", map[string]any{"kind": "http", "endpoint": "web", "address": "127.0.0.1"}, "names a address"},
{"a url", "server", map[string]any{"kind": "http", "url": "http://localhost:8080/"}, "names a url"},
{"an unknown key", "server", map[string]any{"kind": "tcp", "endpoint": "web", "retries": 3}, `"retries"`},
{"no endpoint", "server", map[string]any{"kind": "http"}, "names no endpoint"},
{"an undeclared endpoint", "server", map[string]any{"kind": "tcp", "endpoint": "admin"}, "does not declare"},
{"a udp endpoint", "server", map[string]any{"kind": "tcp", "endpoint": "beacon"}, "a check connects over tcp"},
{"an interval under the floor", "server", map[string]any{"kind": "tcp", "endpoint": "web", "interval": "5s", "timeout": "1s"}, "no more often than every 10s"},
{"a timeout of the interval", "server", map[string]any{"kind": "tcp", "endpoint": "web", "interval": "10s", "timeout": "10s"}, "under its interval"},
{"one failing look", "server", map[string]any{"kind": "tcp", "endpoint": "web", "looks": 1}, "at least 2"},
{"a grace and looks past five minutes", "server", map[string]any{"kind": "tcp", "endpoint": "web", "grace": "4m", "looks": 3, "interval": "30s"}, "at most 5m0s"},
{"a duration that is not one", "server", map[string]any{"kind": "tcp", "endpoint": "web", "interval": "often"}, "is a duration"},
{"looks that are not a number", "server", map[string]any{"kind": "tcp", "endpoint": "web", "looks": "three"}, "whole number"},
{"a path without a slash", "server", map[string]any{"kind": "http", "endpoint": "web", "path": "health"}, "starts with /"},
{"a status that is not one", "server", map[string]any{"kind": "http", "endpoint": "web", "status": 700}, "is not one"},
{"a scheme that is not one", "server", map[string]any{"kind": "http", "endpoint": "web", "scheme": "ftp"}, "http or https"},
{"a status on a tcp check", "server", map[string]any{"kind": "tcp", "endpoint": "web", "status": 200}, "only an http check has"},
{"an exec with no command", "server", map[string]any{"kind": "exec"}, "says no command"},
{"a command on an http check", "server", map[string]any{"kind": "http", "endpoint": "web", "command": "true"}, "only an exec check runs"},
{"a runtime check on a service", "daemon", map[string]any{"kind": "runtime"}, "only a container has"},
{"a unit check on a container", "server", map[string]any{"kind": "unit"}, "a service's or a process's"},
{"a tool the module does not serve", "server", map[string]any{"kind": "tool", "tool": "app_admin"}, "does not serve"},
{"a tool check alone", "server", map[string]any{"kind": "tool", "tool": "app_status"}, "judges itself only by its own tool"},
{"needs not required", "server", map[string]any{"kind": "tcp", "endpoint": "web", "needs": "redis"}, "does not require"},
{"health on a step", "seed", map[string]any{"kind": "runtime"}, "does not stay up"},
{"health on a schedule", "sweep", map[string]any{"kind": "runtime"}, "does not stay up"},
{"health that is not an object", "server", nil, "is an object"},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
raw := healthManifest(t, c.on, c.health)
if c.health == nil {
raw = []byte(strings.Replace(string(raw), `"name":"app-server"`, `"name":"app-server","health":"tcp"`, 1))
}
_, err := ParseManifest(raw)
if err == nil {
t.Fatalf("%s was accepted", c.name)
}
if !strings.Contains(err.Error(), c.says) {
t.Errorf("refused, but not for saying %q: %v", c.says, err)
}
})
}
}
func TestHealthIsComposedAsThePortThisMachineGaveTheEndpoint(t *testing.T) {
m, err := ParseManifest(healthManifest(t, "server", map[string]any{"kind": "http", "endpoint": "web", "path": "/healthz",
"needs": "postgres-database"}))
if err != nil {
t.Fatal(err)
}
compose := func(with Rendering) map[string]any {
out, err := Resolution{Node: "laptop", Modules: []Manifest{m}}.Declaration(with)
if err != nil {
t.Fatal(err)
}
at := indexOfID(out, "app.server")
if at < 0 {
t.Fatal("the container was lost")
}
return out[at]
}
// An engine older than the field is not sent it: it would refuse the whole declaration.
if h, sent := compose(Rendering{Ports: map[string]map[int]int{"app": {8080: 31001}}})["health"]; sent {
t.Fatalf("health was sent to an engine that does not read it: %v", h)
}
got := compose(Rendering{ReadsHealth: true, Ports: map[string]map[int]int{"app": {8080: 31001}}})["health"].(map[string]any)
if got["port"] != 31001 || got["endpoint"] != "web" || got["path"] != "/healthz" || got["needs"] != "postgres-database" {
t.Errorf("composed as %v", got)
}
if got["interval"] != "30s" || got["timeout"] != "5s" || got["looks"] != 3 || got["grace"] != "1m0s" {
t.Errorf("the defaults were not written out: %v", got)
}
// The port this machine gives the endpoint moves, and the check moves with it.
moved := compose(Rendering{ReadsHealth: true, Ports: map[string]map[int]int{"app": {8080: 31002}}})["health"].(map[string]any)
if moved["port"] != 31002 {
t.Errorf("after the port moved the check still dials %v", moved["port"])
}
// And the catalogue's manifest is untouched by composing it.
if _, ok := m.Resources[0]["health"].(map[string]any)["port"]; ok {
t.Error("composing wrote the port into the catalogue's own manifest")
}
}
func TestTheUndeclaredAreTheLongRunningWithoutHealth(t *testing.T) {
m, err := ParseManifest(healthManifest(t, "server", map[string]any{"kind": "runtime"}))
if err != nil {
t.Fatal(err)
}
if got := strings.Join(Undeclared(m), ","); got != "daemon" {
t.Errorf("undeclared: %q; the step and the schedule are not long-running, the server declares", got)
}
}
func TestATooledHealthIsGrantedToTheEngine(t *testing.T) {
raw := healthManifest(t, "server", map[string]any{"kind": "http", "endpoint": "web"},
map[string]any{"id": "admin", "type": "service", "unit": "app-admin.service", "state": "running",
"health": map[string]any{"kind": "tool", "tool": "app_status"}})
m, err := ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
if got := HealthChecks(m); len(got) != 1 || got[0] != "app.app_status" {
t.Errorf("checks %v", got)
}
}
// What the controller composes the node-engine takes: every kind, composed for an engine that reads it,
// passes the engine's own validator (mesh-host/validate) — one set of words on both sides.
func TestEveryComposedHealthIsOneTheNodeEngineTakes(t *testing.T) {
for _, h := range []map[string]any{
{"kind": "http", "endpoint": "web", "path": "/healthz", "status": 200, "body": "ok", "needs": "postgres-database"},
{"kind": "tcp", "endpoint": "web", "interval": "10s", "timeout": "2s", "looks": 2, "grace": "0s"},
{"kind": "runtime"},
{"kind": "exec", "command": "pg_isready -q"},
} {
m, err := ParseManifest(healthManifest(t, "server", h,
map[string]any{"id": "admin", "type": "service", "unit": "app-admin.service", "state": "running",
"health": map[string]any{"kind": "tool", "tool": "app_status"}}))
if err != nil {
t.Fatal(err)
}
m.Resources[1]["health"] = map[string]any{"kind": "unit"}
out, err := Resolution{Node: "laptop", Modules: []Manifest{m}}.Declaration(Rendering{ReadsHealth: true,
Ports: map[string]map[int]int{"app": {8080: 31001}}})
if err != nil {
t.Fatal(err)
}
body, err := json.Marshal(map[string]any{"declaration": validate.Version, "resources": out})
if err != nil {
t.Fatal(err)
}
if problems := validate.Declaration(body); len(problems) > 0 {
t.Errorf("%v composed into something the node-engine refuses: %v", h, problems)
}
}
}
+3
View File
@@ -2036,6 +2036,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
}
problems = append(problems, zoneProblems(m)...)
// How each long-running resource is ready (novox/hq ADR 0240 rule 2): said near its author, in the
// words the node-engine would refuse it in far away.
problems = append(problems, healthProblems(m)...)
if len(problems) > 0 {
sort.Strings(problems)
return Manifest{}, fmt.Errorf("this manifest cannot be used:\n - %s",
+8
View File
@@ -262,6 +262,14 @@ var ControllerVerbs = []Verb{
"cause": "the cause in a word, or a condition's kind — the word a second act for the same reason uses",
"condition": "the key of the condition it addressed, if any (optional)",
}, []string{"what", "why", "cause"})},
{Name: "drill", Description: "Record a drill — something broken on purpose to see the mesh raise and clear it: " +
"a module stopped, a process killed — with what it tests, in the hand-act log. A drill is a person's " +
"deliberate test, never a repair: no healer is wanted for it however often it is run (S15).",
Input: schema(map[string]string{
"what": "what was done on purpose, in a line",
"why": "what the drill tests",
"condition": "the key of the condition the drill is meant to raise, if any (optional)",
}, []string{"what", "why"})},
{Name: "hand-acts", Description: "What was done by hand lately — pushes, plans ended, consumers re-made, acts " +
"recorded — who, why and the cause of each, and which causes repeat: each repeat is a healer the mesh lacks.",
Input: schema(map[string]string{"days": "how many days back (default 14)"}, nil)},
+4 -1
View File
@@ -46,11 +46,14 @@ const (
ScopeMesh = "mesh"
// ScopeDelivery is a delivery mesh-delivery owns, by its id (novox/hq ADR 0239).
ScopeDelivery = "delivery"
// ScopeModule is a module on a machine, by `<module>.<machine>`: what it runs is not healthy there
// (novox/hq ADR 0240).
ScopeModule = "module"
)
// Scopes is every scope, in the order a person reads them.
var Scopes = []string{ScopeMachine, ScopePlan, ScopeCall, ScopeBuild, ScopeMerge, ScopeProvider,
ScopeSeat, ScopeBus, ScopeCore, ScopeProbe, ScopeMesh, ScopeDelivery}
ScopeSeat, ScopeBus, ScopeCore, ScopeProbe, ScopeMesh, ScopeDelivery, ScopeModule}
// Who resolves a condition.
const (
+2 -2
View File
@@ -64,7 +64,7 @@ type Facts struct {
Edges []Edge `json:"edges,omitempty"`
// Beside is the ref each repository is cloned at beside a merge check, by the directory it is found
// under — the commits the mesh runs, the catalogue's main — so a check run by hand reads the siblings
// the build seat reads (novox/hq issue 286).
// the build seat reads (novox/hq issue 283).
Beside map[string]string `json:"beside,omitempty"`
}
@@ -85,7 +85,7 @@ type Versions struct {
// Toolchains are the toolchain images a merge check runs in, by language, as the mesh holds them —
// with no address: the artifact store the snapshot is read from is where they are pulled from. What
// a repository's merge-check.sh runs in on the build seat, and so what it must run in anywhere else
// (novox/hq issue 286): two releases of one compiler disagree, down to how gofmt lays out a file.
// (novox/hq issue 283): two releases of one compiler disagree, down to how gofmt lays out a file.
Toolchains map[string]string `json:"toolchains,omitempty"`
}
+15
View File
@@ -172,3 +172,18 @@ func TestAWithheldPathStaysAPath(t *testing.T) {
t.Errorf("a key became %q", got)
}
}
// **Issue 288**: a repository was kept as the URL it was cloned from, the forge's address with it.
func TestARepositoryIsNamedWithoutTheForge(t *testing.T) {
for in, want := range map[string]string{
"http://forge.internal:3000/owner/repo.git": "owner/repo",
"ssh://git@forge.internal:222/owner/repo": "owner/repo",
"git@forge.internal:owner/repo.git": "owner/repo",
"owner/repo": "owner/repo",
"": "",
} {
if got := RepositoryName(in); got != want {
t.Errorf("%q is named %q, not %q", in, got, want)
}
}
}
+28
View File
@@ -321,3 +321,31 @@ func standIn(run string) string {
}
return run
}
// RepositoryName is a repository as `owner/repository`, without the forge's address it was cloned from:
// http://forge.internal:3000/owner/repo.git → owner/repo (novox/hq issue 288). What a check matches a
// pull request's repository by is its owner and name, never the forge's address, so nothing is lost.
func RepositoryName(repository string) string {
r := strings.TrimSuffix(strings.TrimSuffix(strings.TrimSpace(repository), "/"), ".git")
if r == "" {
return ""
}
if _, rest, found := strings.Cut(r, "://"); found {
r = rest
if _, path, found := strings.Cut(r, "/"); found {
r = path
} else {
return ""
}
} else if at := strings.Index(r, "@"); at >= 0 {
// scp-like: git@forge:owner/repo
if _, path, found := strings.Cut(r[at+1:], ":"); found {
r = path
}
}
parts := strings.Split(strings.Trim(r, "/"), "/")
if len(parts) >= 2 {
return parts[len(parts)-2] + "/" + parts[len(parts)-1]
}
return parts[len(parts)-1]
}
+2
View File
@@ -137,6 +137,8 @@ func declaredFor(m catalogue.Manifest, seats map[string]catalogue.SeatDeclaratio
// And the state it keeps and reads (novox/hq ADR 0201).
State: bucketsOf(m),
Reads: m.Reads,
// And the tools its health asks (novox/hq ADR 0240): the machine's node-engine is granted them.
Checks: catalogue.HealthChecks(m),
}
// Whether it can be given an account at all: delivered as its own secret named broker, so one
// that declares none has nowhere to read it (novox/hq issue 195).
+124
View File
@@ -0,0 +1,124 @@
package inventory
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
"github.com/jackc/pgx/v5"
)
// What each machine says of its long-running resources (novox/hq ADR 0240, to-be 48 §4): the newest
// statement per machine, and per module how many statements in a row said a resource of it was unhealthy.
// ResourceHealth is one long-running resource's state as the node-engine said it.
type ResourceHealth struct {
Module string `json:"module"`
Resource string `json:"resource"`
Kind string `json:"kind"`
Target string `json:"target"`
State string `json:"state"`
Reason string `json:"reason,omitempty"`
Since time.Time `json:"since"`
Streak int `json:"streak,omitempty"`
Restarts int `json:"restarts,omitempty"`
// Check is the declared check's kind, empty for liveness alone; Needs the provision it exercises
// (ADR 0240 Phase B, to-be 48 §6).
Check string `json:"check,omitempty"`
Needs string `json:"needs,omitempty"`
}
// NodeHealth is a machine's newest statement, as kept.
type NodeHealth struct {
Node string
Contract int
// SaidAt is when the engine looked (the machine's clock); HeardAt when this controller heard it.
SaidAt time.Time
HeardAt time.Time
Resources []ResourceHealth
// Streaks is, per module, how many statements in a row said a resource of it was unhealthy.
Streaks map[string]int
}
// HealthOf is a machine's newest statement; false when its node-engine has never stated one.
func (i *Inventory) HealthOf(ctx context.Context, nodeName string) (NodeHealth, bool, error) {
all, err := i.healths(ctx, nodeName)
if err != nil {
return NodeHealth{}, false, err
}
h, ok := all[nodeName]
return h, ok, nil
}
// Healths is every machine's newest statement, by machine. A machine absent never stated one: its
// node-engine is older than the judging, and its health is not known.
func (i *Inventory) Healths(ctx context.Context) (map[string]NodeHealth, error) {
return i.healths(ctx, "")
}
func (i *Inventory) healths(ctx context.Context, only string) (map[string]NodeHealth, error) {
rows, err := i.store.Pool().Query(ctx,
`select n.name, h.contract, h.said_at, h.heard_at, h.resources, h.streaks
from node_health h join node n on n.id = h.node
where $1 = '' or n.name = $1`, only)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string]NodeHealth{}
for rows.Next() {
var h NodeHealth
var resources, streaks []byte
if err := rows.Scan(&h.Node, &h.Contract, &h.SaidAt, &h.HeardAt, &resources, &streaks); err != nil {
return nil, err
}
if err := json.Unmarshal(resources, &h.Resources); err != nil {
return nil, fmt.Errorf("%s's health cannot be read: %w", h.Node, err)
}
if err := json.Unmarshal(streaks, &h.Streaks); err != nil {
return nil, fmt.Errorf("%s's health cannot be read: %w", h.Node, err)
}
out[h.Node] = h
}
return out, rows.Err()
}
// RecordHealth keeps a machine's statement in place of the one kept — unless the one kept is newer, by
// when the engine looked: then nothing is written, and false says it was refused as older.
func (i *Inventory) RecordHealth(ctx context.Context, h NodeHealth) (bool, error) {
if h.Resources == nil {
h.Resources = []ResourceHealth{}
}
if h.Streaks == nil {
h.Streaks = map[string]int{}
}
resources, err := json.Marshal(h.Resources)
if err != nil {
return false, err
}
streaks, err := json.Marshal(h.Streaks)
if err != nil {
return false, err
}
heard := h.HeardAt
if heard.IsZero() {
heard = time.Now()
}
var node string
err = i.store.Pool().QueryRow(ctx,
`insert into node_health (node, contract, said_at, heard_at, resources, streaks)
select id, $2, $3, $4, $5, $6 from node where name = $1
on conflict (node) do update set contract = excluded.contract, said_at = excluded.said_at,
heard_at = excluded.heard_at, resources = excluded.resources, streaks = excluded.streaks
where node_health.said_at <= excluded.said_at
returning node`, h.Node, h.Contract, h.SaidAt, heard, resources, streaks).Scan(&node)
if errors.Is(err, pgx.ErrNoRows) {
if _, nerr := i.NodeByName(ctx, h.Node); nerr != nil {
return false, nerr
}
return false, nil
}
return err == nil, err
}
@@ -0,0 +1,26 @@
-- A module says how it is healthy, and the node-engine judges it (novox/hq ADR 0240, to-be 48 Phase A).
--
-- Every machine's node-engine states the health of every long-running resource it runs for a module — a
-- container that stays up, a process that stays up, a service stated running — in every report and as an
-- event between reports. The controller keeps the newest statement per machine, here, so the release
-- gate, `node show` and a controller started again all read the same word; and with it, per module, how
-- many statements in a row said a resource of it was unhealthy — `module.<module>.<machine>.unhealthy` is
-- raised on the second (ADR 0240 §4).
--
-- One row per machine, replaced, as node_report is: the question is the machine's state now. A machine
-- whose node-engine is older than the judging has no row, and its health is not known — never healthy,
-- never unhealthy.
create table node_health (
node uuid primary key references node(id) on delete cascade,
-- The statement's version (the engine's liveness contract).
contract int not null,
-- When the node-engine looked, on the machine's clock: the order of its statements. An older one
-- than this is refused.
said_at timestamptz not null,
-- When this controller heard it, on its own: what "since the send" is judged by.
heard_at timestamptz not null default now(),
-- [{module, resource, kind, target, state, reason, since, streak, restarts}], as the engine said them.
resources jsonb not null default '[]',
-- {module: statements in a row saying a resource of it is unhealthy}.
streaks jsonb not null default '{}'
);
+8
View File
@@ -75,8 +75,16 @@ const (
// ToolsAliveSubjects is every machine's node tools saying they are there (novox/hq to-be 45 §3,
// S11): core NATS like the host's, for the same reason.
ToolsAliveSubjects = "mesh.control.*.tools-alive"
// HealthSubjects is every machine's health statement between its reports (novox/hq ADR 0240): core
// NATS like the heartbeat, because a statement lost is said again within a minute while anything is
// not healthy, and the next report carries it whatever happens.
HealthSubjects = "mesh.control.*.health"
)
// HealthSubject is one machine's health statement.
func HealthSubject(node string) string { return "mesh.control." + node + ".health" }
// ReportSubject is where one node says what it did. On the CONTROL stream, because it is the
// message the store-window guarantee is about (ADR 0083).
func ReportSubject(node string) string { return "mesh.control." + node + ".report" }
+13 -1
View File
@@ -29,6 +29,15 @@ import (
// then and with "still running as call <id>" when it does not; and every call is kept here, with
// what came of it, including an answer the bus refused, so `calls` can say it.
// ErrHandingOver is a call refused because the controller answering it is being replaced: stopping, or
// unable to run its own build because the node-engine has moved it aside (novox/hq issue 289). Nothing
// was done, and the controller after it answers the same call — so the answer carries
// `"retry": RetryHandingOver`, and a caller asks once more.
var ErrHandingOver = errors.New("the controller is handing over to the next one; nothing was done, ask again")
// RetryHandingOver is the `retry` mark of an answer refused by ErrHandingOver.
const RetryHandingOver = "handing-over"
// AnswerWithin is how long a call runs before its caller is answered that it is still running. Well
// inside the shortest wait of a caller the mesh ships (the console's thirty seconds) and the bus's
// own window for an answer (broker.ResponseTTL), so the one answer a call has is never late for
@@ -524,7 +533,10 @@ func (l *CallLog) serveCall(seat, verb string, args json.RawMessage, reply strin
var body []byte
result, err := handle(ctx, args)
failed := err != nil
if err != nil {
if errors.Is(err, ErrHandingOver) {
// Marked as well as said, so a caller asks again without reading the words (issue 289).
body, _ = json.Marshal(map[string]any{"error": err.Error(), "retry": RetryHandingOver})
} else if err != nil {
body, _ = json.Marshal(map[string]any{"error": err.Error()})
} else if body, err = json.Marshal(map[string]any{"result": result}); err != nil {
failed = true
+22
View File
@@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"strings"
"sync"
@@ -177,3 +178,24 @@ func recentOf(l *CallLog) []Call {
out, _ := l.Recent()
return out
}
// A call refused because its controller is handing over says so in a mark as well as in words, so the
// caller asks once more without parsing a sentence (novox/hq issue 289).
func TestAHandoverRefusalIsMarkedRetryable(t *testing.T) {
l, a := NewCallLog(), newAnswers(t)
l.serveCall("mesh-controller", "rotate", nil, "_INBOX.x.9", func(context.Context, json.RawMessage) (any, error) {
return nil, fmt.Errorf("%w: stopping", ErrHandingOver)
}, a.respond, nil)
got := a.only()
if got["retry"] != RetryHandingOver || !strings.Contains(fmt.Sprint(got["error"]), "handing over") {
t.Fatalf("answered %v", got)
}
l, a = NewCallLog(), newAnswers(t)
l.serveCall("mesh-controller", "rotate", nil, "_INBOX.x.10", func(context.Context, json.RawMessage) (any, error) {
return nil, errors.New("refused for its own reason")
}, a.respond, nil)
if _, marked := a.only()["retry"]; marked {
t.Fatal("an ordinary refusal was marked to be asked again")
}
}
+4
View File
@@ -35,6 +35,10 @@ var Contracts = map[string]Contract{
KindHeartbeat: {Unordered: "a word that the machine is there: the newest heard is the newest said, and one " +
"lost is the next one"},
KindToolsHeartbeat: {Unordered: "a word that the node tools are there, as a machine's heartbeat"},
KindHealth: {Ordered: "by the time the node-engine looked (`at`, on the machine's clock): a statement older " +
"than the one kept for that machine is refused, so an event arriving after a newer report does not undo it " +
"(novox/hq ADR 0240)",
Tests: []string{"TestAnOlderHealthStatementIsRefused"}},
KindEnrolment: {Unordered: "a request answered once, under a token spent once: a second presentation is " +
"refused by the token, not by an order (issue 083)",
Tests: []string{"TestAnEnrolmentMetByAHeldTokenIsAskedToTryAgain"}},
+1 -1
View File
@@ -16,7 +16,7 @@ import (
func TestEveryConsumedKindHasAContract(t *testing.T) {
// What the controller can be handed, from the subjects it is granted and the ones it derives.
subjects := []string{EnrolSubject, BuiltSubject, ReportSubject("anchor"), AliveSubject("anchor"),
ToolsAliveSubject("anchor"), "mesh.mod.postgres.event.provisioner.failing"}
ToolsAliveSubject("anchor"), HealthSubject("anchor"), "mesh.mod.postgres.event.provisioner.failing"}
subjects = append(subjects, broker.ControllerFollows...)
kinds := map[string]bool{}
for _, s := range subjects {
+56
View File
@@ -0,0 +1,56 @@
package link
import (
"context"
"encoding/json"
"strings"
)
// A machine's health statement (novox/hq ADR 0240, to-be 48 §4).
//
// **The node-engine owns every verdict; the controller keeps the last word per machine.** A statement
// arrives in every report and as an event between reports — on each change, and again every minute while
// a resource is not healthy. What the controller does with it — keep it, refuse an older one, raise
// `module.<module>.<machine>.unhealthy` on the second statement that says so — is the Healths it is given.
// Healths keeps what machines state of their long-running resources.
type Healths interface {
// Stated keeps one machine's statement, from a report or the event. An older statement than the one
// kept is refused there, by its time on the machine.
Stated(ctx context.Context, node string, h Health) error
}
// Hears says where the machines' health statements are kept. Their subscription is the heartbeats':
// core, and always made, so nothing is asked of the bus here.
func (s *Server) Hears(h Healths) { s.healths = h }
// healthSaid acts on one health event. Core NATS, so there is nothing to hold: a statement that could not
// be kept is said in the log, and the next one — a minute away while anything is not healthy — is kept.
func (s *Server) healthSaid(ctx context.Context, m Control) {
defer func() { _ = m.Took() }()
var said HealthSaid
if err := json.Unmarshal(m.Body(), &said); err != nil || said.Health.Contract == 0 {
return
}
// The machine is the one in the subject the bus let it publish on, never the body's.
node, ok := nodeOfHealth(m.Subject())
if !ok || s.healths == nil {
return
}
if err := s.healths.Stated(ctx, node, said.Health); err != nil {
s.log.Printf("could not keep what %s says of its resources' health: %v", node, err)
}
}
// nodeOfHealth is the machine a health statement names in its subject.
func nodeOfHealth(subject string) (string, bool) {
rest, ok := strings.CutPrefix(subject, "mesh.control.")
if !ok {
return "", false
}
node, kind, ok := strings.Cut(rest, ".")
if !ok || kind != "health" || node == "" || strings.Contains(node, ".") {
return "", false
}
return node, true
}
+37
View File
@@ -0,0 +1,37 @@
package link
import (
"context"
"testing"
"time"
)
// A health event is kept as the machine its subject names says it, whatever its body claims, and taken
// (novox/hq ADR 0240): core NATS, nothing to hold.
type keptHealths struct{ by map[string]Health }
func (k *keptHealths) Stated(_ context.Context, node string, h Health) error {
k.by[node] = h
return nil
}
func TestAHealthEventIsKeptAsTheMachineInItsSubject(t *testing.T) {
s, in := serving()
kept := &keptHealths{by: map[string]Health{}}
s.Hears(kept)
to := &settled{}
m := in.sends(t, to, KindHealth, HealthSaid{Node: "laptop", Health: Health{Contract: LivenessContract, At: time.Now(),
Resources: []ResourceHealth{{Module: "letta", Resource: "letta.server", State: StateUnhealthy}}}}).(*fakeControl)
m.subject = HealthSubject("anchor")
s.act(t.Context(), m)
if !to.acked {
t.Fatal("a health event was not taken")
}
if _, lied := kept.by["laptop"]; lied || len(kept.by["anchor"].Resources) != 1 {
t.Fatalf("kept %+v: the machine is the subject's, never the body's", kept.by)
}
if kind, ok := kindOfSubject(HealthSubject("anchor")); !ok || kind != KindHealth {
t.Fatalf("%s is not read as a health statement", HealthSubject("anchor"))
}
}
+58
View File
@@ -262,6 +262,13 @@ type Report struct {
// `witness` and `not-reversible` are sent only to a machine whose report carries it.
Witness int `json:"witness,omitempty"`
// Health is the machine's word on every long-running resource it runs for a module (novox/hq ADR
// 0240, to-be 48 §4; mesh-host's internal/liveness): its state, since when, its failing streak and
// the restarts its node-engine counted. **Absent from an engine older than the judging**, which is
// read as "not known" — never as healthy, never as a reason to raise anything; present with no
// resources from a machine that runs nothing long-lived.
Health *Health `json:"health,omitempty"`
// Rekey is a node taking a found tunnel's key as its overlay key after enrolment (novox/hq
// ADR 0105). A report carrying one is not an account of the machine: it moves the node's
// overlay key and tunnel and nothing else.
@@ -404,3 +411,54 @@ func EnrolProof(secret string, public []byte, overlay, sealing, serving string)
return []byte("novox-mesh-enrol\x00" + secret + "\x00" + base64.StdEncoding.EncodeToString(public) +
"\x00" + overlay + "\x00" + sealing + "\x00" + serving)
}
// LivenessContract is the version of the health statement this controller reads (ADR 0240 Phase A).
const LivenessContract = 1
// ReadinessContract is the statement of an engine that also reads a resource's declared `health` and
// judges it (ADR 0240 Phase B): only to an engine whose statement says this or later is the field sent,
// because an older one parses strictly and would refuse the whole declaration for it.
const ReadinessContract = 2
// Health is one statement of a machine's long-running resources (to-be 48 §4): in every report, as the
// event HealthSubject between reports on each change, and again every minute while one is not healthy.
// The node-engine's own (mesh-host internal/link Health); a test on each side holds the field names.
type Health struct {
Contract int `json:"contract"`
// At is when the engine looked, on the machine's clock: the order of its statements.
At time.Time `json:"at"`
Resources []ResourceHealth `json:"resources"`
}
// The states a resource is said in (ADR 0240 §4).
const (
StateHealthy = "healthy"
StateUnhealthy = "unhealthy"
StateStarting = "starting"
StateHeld = "held"
StateUnknown = "unknown"
)
// ResourceHealth is one long-running resource's state.
type ResourceHealth struct {
Module string `json:"module"`
Resource string `json:"resource"`
Kind string `json:"kind"`
Target string `json:"target"`
State string `json:"state"`
Reason string `json:"reason,omitempty"`
Since time.Time `json:"since"`
Streak int `json:"streak,omitempty"`
Restarts int `json:"restarts,omitempty"`
// Check is the declared check's kind (ADR 0240 Phase B), empty for a resource judged by liveness
// alone; Needs is the provision the check exercises (to-be 48 §6).
Check string `json:"check,omitempty"`
Needs string `json:"needs,omitempty"`
}
// HealthSaid is the health event's body: the machine and its statement. The machine is read from the
// subject the bus let it publish on, never from here.
type HealthSaid struct {
Node string `json:"node"`
Health Health `json:"health"`
}
+8
View File
@@ -25,6 +25,14 @@ func TestTheWireFormatIsExactlyTheseFieldNames(t *testing.T) {
[]string{"protocol", "address", "port", "by", "published", "container-port"}},
{EnrolRequest{Node: "n", Secret: "s", PublicKey: []byte("k")},
[]string{"node", "secret", "public_key"}},
// novox/hq ADR 0240: every long-running resource's health, in every report and in its own event.
{Report{Node: "n", Health: &Health{Contract: LivenessContract}}, []string{"node", "health"}},
{Health{Contract: LivenessContract, Resources: []ResourceHealth{}}, []string{"contract", "at", "resources"}},
{ResourceHealth{Module: "m", Resource: "m.r", Kind: "container", Target: "t", State: StateUnhealthy,
Reason: "restarting", Streak: 2, Restarts: 3, Check: "http", Needs: "postgres-database"},
[]string{"module", "resource", "kind", "target", "state", "reason", "since", "streak", "restarts",
"check", "needs"}},
{HealthSaid{Node: "n"}, []string{"node", "health"}},
} {
raw, err := json.Marshal(c.value)
if err != nil {
+4 -2
View File
@@ -30,8 +30,10 @@ const (
KindHeartbeat = "heartbeat"
// KindToolsHeartbeat is a machine's node tools saying they are there (novox/hq to-be 45 S11).
KindToolsHeartbeat = "tools-heartbeat"
KindBuilt = "built"
KindModuleMoved = "module-moved"
// KindHealth is a machine's statement of its long-running resources' health (novox/hq ADR 0240).
KindHealth = "health"
KindBuilt = "built"
KindModuleMoved = "module-moved"
// KindSourceMoved is the forge announcing a merge: a source moved, and what it produces is
// built without anybody telling the mesh (novox/hq 04-ISSUES/131).
KindSourceMoved = "source-moved"
+9
View File
@@ -108,6 +108,13 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
return fmt.Errorf("subscribing to the node tools' heartbeats: %w", err)
}
defer func() { _ = toolsAlive.Unsubscribe() }()
// And what each machine says of its long-running resources between reports (novox/hq ADR 0240): core
// too, and said again while it matters.
health, err := conn.ChanSubscribe(HealthSubjects, beats)
if err != nil {
return fmt.Errorf("subscribing to the machines' health: %w", err)
}
defer func() { _ = health.Unsubscribe() }()
// The events the controller follows, when something is listening for them.
var events chan *nats.Msg
@@ -239,6 +246,8 @@ func kindOfSubject(subject string) (string, bool) {
return KindHeartbeat, true
case "tools-alive":
return KindToolsHeartbeat, true
case "health":
return KindHealth, true
}
}
switch subject {
+4
View File
@@ -94,6 +94,8 @@ type Server struct {
retirements Retirements
// checker asks for a pull request's merge check (novox/hq to-be 45 §9).
checker Checker
// healths keeps what machines say of their long-running resources (novox/hq ADR 0240).
healths Healths
log *log.Logger
// giveUp is how long one message is held for the store; zero means GiveUpAfter.
@@ -194,6 +196,8 @@ func (s *Server) act(ctx context.Context, m Control) {
s.heartbeat(m)
case KindToolsHeartbeat:
s.toolsHeartbeat(m)
case KindHealth:
s.healthSaid(ctx, m)
case KindBuilt:
s.wasBuilt(ctx, m)
case KindModuleMoved:
+1
View File
@@ -60,6 +60,7 @@
"pause",
"resume",
"hand-act",
"drill",
"hand-acts",
"durations",
"conditions",
+41 -3
View File
@@ -590,6 +590,23 @@ type Process struct {
// For a process that stays up; a step or a scheduled run is not running a moment later by
// design, so there is nothing to hand over to.
Replaces []string `json:"replaces,omitempty"`
// Witness is how the node-engine judges a new build of this process, and restores the build
// before it when the new one is not healthy in bound (novox/hq to-be 45 §8): "lease" — the
// controller this machine started holds the controller's lease; "ping" — this machine's runtime
// answers the services protocol's PING; "none". Absent is the default for the process's name:
// the mesh's two core processes are judged, nothing else is. For a process that stays up.
Witness string `json:"witness,omitempty"`
// NotReversible says why this build may not be rolled back, when it may not: the build before it
// would run against what this one changes — a migration it runs that the older build cannot read
// (to-be 45 §8, rule 8). A build so declared that is not healthy in bound is left running and said
// as urgent; the build before it is never started against the newer data.
NotReversible string `json:"not-reversible,omitempty"`
// Health is how this resource is ready (novox/hq ADR 0240 rule 2, Phase B): one kind and its
// timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more.
Health *Health `json:"health,omitempty"`
}
func (d *Process) Identity() string { return d.ID }
@@ -621,7 +638,7 @@ func ProcessNameProblem(name string) string {
}
func (d *Process) validate(where string, _ bool) []string {
var problems []string
problems := d.Health.problems(where, false, !d.RunOnce && d.Schedule == "")
if problem := ProcessNameProblem(d.Name); problem != "" {
problems = append(problems, where+": "+problem)
}
@@ -637,6 +654,19 @@ func (d *Process) validate(where string, _ bool) []string {
if len(d.Run) == 0 {
problems = append(problems, where+": a process needs to say what to run")
}
switch d.Witness {
case "", "lease", "ping", "none":
default:
problems = append(problems, fmt.Sprintf("%s: witness %q is not one this host keeps: lease, ping or none",
where, d.Witness))
}
if d.Witness != "" && d.Witness != "none" && (d.RunOnce || d.Schedule != "") {
problems = append(problems, where+": a witness judges a process that stays up; a step or a "+
"scheduled run is not running between its runs")
}
if strings.ContainsAny(d.NotReversible, "\n\r") {
problems = append(problems, where+": not-reversible is one line")
}
for _, part := range d.Run {
if part == "" {
problems = append(problems, where+": a process command has an empty element")
@@ -756,6 +786,10 @@ type Service struct {
// said by the controller, which knows the found tunnel's key is this node's own: without that,
// starting this unit on the found one's port would drop every peer's packets.
TakesOver *TakeOver `json:"takes-over,omitempty"`
// Health is how this resource is ready (novox/hq ADR 0240 rule 2, Phase B): one kind and its
// timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more.
Health *Health `json:"health,omitempty"`
}
// TakeOver is a found tunnel a service replaces: its interface, the unit that raised it, and its
@@ -784,7 +818,7 @@ const (
func (s *Service) UserScoped() bool { return s.Scope == ScopeUser }
func (s *Service) validate(where string, _ bool) []string {
var problems []string
problems := s.Health.problems(where, false, s.State == "running")
if s.Unit == "" {
problems = append(problems, where+": a service needs a unit")
}
@@ -1096,6 +1130,10 @@ type Container struct {
// offline job says *before*, not *instead of*; a recurring window is the case order cannot
// express, and the only one this serves.
WhileStopped []string `json:"while-stopped,omitempty"`
// Health is how this resource is ready (novox/hq ADR 0240 rule 2, Phase B): one kind and its
// timing, judged by the node-engine beside liveness. Absent: judged alive or not, and nothing more.
Health *Health `json:"health,omitempty"`
}
func (c *Container) Identity() string { return c.ID }
@@ -1103,7 +1141,7 @@ func (c *Container) Kind() Type { return TypeContainer }
func (c *Container) Target() string { return c.Name }
func (c *Container) validate(where string, _ bool) []string {
var problems []string
problems := c.Health.problems(where, true, !c.RunOnce && c.Schedule == "")
if c.Name == "" {
problems = append(problems, where+": a container needs a name")
}
+188
View File
@@ -0,0 +1,188 @@
package declaration
import (
"bytes"
"encoding/json"
"fmt"
"strings"
"time"
)
// Health is how a long-running resource is ready, as the controller composed it from the module's
// `health` (novox/hq ADR 0240 rule 2, to-be 48 §2–§3, Phase B): one kind and its timing, the endpoint
// already the port this machine published it on.
//
// **The node-engine runs every kind and owns every verdict.** http and tcp it makes itself, from the
// machine to the port; unit it reads from the service manager it already reads; exec and runtime it hands
// to the container runtime as the container's own check, with this timing, and reads the state; tool it
// asks of its own node tools. Nothing else on the machine sets a container's check.
//
// Refused here as the controller refuses it near the author, in the same bounds: an engine that took a
// check it could not judge would say a module ready that nothing looked at.
type Health struct {
Kind string `json:"kind"`
// Endpoint is the module's name for what Port is: for the words a verdict is said in.
Endpoint string `json:"endpoint,omitempty"`
Port int `json:"port,omitempty"`
Path string `json:"path,omitempty"`
Status int `json:"status,omitempty"`
Body string `json:"body,omitempty"`
Scheme string `json:"scheme,omitempty"`
Command string `json:"command,omitempty"`
Tool string `json:"tool,omitempty"`
Interval string `json:"interval"`
Timeout string `json:"timeout"`
Looks int `json:"looks"`
Grace string `json:"grace"`
// Needs is the provision the check exercises (to-be 48 §6): said with every verdict, so the
// controller can hold what it finds under the provider's own condition.
Needs string `json:"needs,omitempty"`
}
// UnmarshalJSON reads a health strictly, as everything a declaration carries is read: a field this host
// does not know is a part of the check the controller believes it asked for, and nothing would look at it.
func (h *Health) UnmarshalJSON(raw []byte) error {
type plain Health
var p plain
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&p); err != nil {
return fmt.Errorf("health: %w", err)
}
*h = Health(p)
return nil
}
// The kinds.
const (
HealthRuntime = "runtime"
HealthHTTP = "http"
HealthTCP = "tcp"
HealthExec = "exec"
HealthUnit = "unit"
HealthTool = "tool"
)
// The bounds (ADR 0240 rule 2) — the controller's, held again here.
const (
HealthIntervalFloor = 10 * time.Second
HealthLooksFloor = 2
HealthWithin = 5 * time.Minute
)
// Every, Within and GraceOf are the timing, read. Validated on arrival, so a parse error here is
// impossible on a declaration that was accepted; it reads as zero.
func (h *Health) Every() time.Duration { d, _ := time.ParseDuration(h.Interval); return d }
func (h *Health) Within() time.Duration { d, _ := time.ParseDuration(h.Timeout); return d }
func (h *Health) GraceOf() time.Duration { d, _ := time.ParseDuration(h.Grace); return d }
// RunByRuntime says the container runtime runs this check as the container's own: exec and runtime.
func (h *Health) RunByRuntime() bool {
return h != nil && (h.Kind == HealthExec || h.Kind == HealthRuntime)
}
// Words is the check in a few words, as a verdict is said: "http /healthz on web".
func (h *Health) Words() string {
switch h.Kind {
case HealthHTTP:
return "http " + h.Path + " on " + orPort(h.Endpoint, h.Port)
case HealthTCP:
return "tcp on " + orPort(h.Endpoint, h.Port)
case HealthTool:
return "its tool " + h.Tool
case HealthRuntime:
return "its image's own check"
case HealthExec:
return "its command"
case HealthUnit:
return "its unit"
}
return h.Kind
}
func orPort(endpoint string, port int) string {
if endpoint != "" {
return endpoint
}
return fmt.Sprint(port)
}
// problems holds a resource's health to its kind and bounds. container says whether the resource is a
// container; longRunning whether it stays up.
func (h *Health) problems(where string, container, longRunning bool) []string {
if h == nil {
return nil
}
var problems []string
say := func(format string, args ...any) {
problems = append(problems, where+": "+fmt.Sprintf(format, args...))
}
if !longRunning {
say("health is judged on what stays up; a step or a scheduled run is judged by its own outcome")
}
switch h.Kind {
case HealthHTTP, HealthTCP:
if h.Port < 1 || h.Port > 65535 {
say("a %s check needs the port it looks at", h.Kind)
}
case HealthExec:
if !container {
say("an exec check runs inside a container")
}
if strings.TrimSpace(h.Command) == "" {
say("an exec check needs a command")
}
case HealthRuntime:
if !container {
say("a runtime check is a container image's own")
}
case HealthUnit:
if container {
say("a unit check is a service's or a process's own")
}
case HealthTool:
if strings.TrimSpace(h.Tool) == "" {
say("a tool check names the tool")
}
default:
say("health of kind %q; it is runtime, http, tcp, exec, unit or tool", h.Kind)
}
if h.Kind == HealthHTTP {
if !strings.HasPrefix(h.Path, "/") {
say("an http check asks a path starting with /")
}
if h.Status != 0 && (h.Status < 100 || h.Status > 599) {
say("an http check expects status %d, which is not one", h.Status)
}
if h.Scheme != "" && h.Scheme != "http" && h.Scheme != "https" {
say("an http check is over http or https, not %q", h.Scheme)
}
}
if strings.ContainsAny(h.Command, "\n\r") {
say("an exec check's command is one line")
}
every, everyErr := time.ParseDuration(h.Interval)
within, withinErr := time.ParseDuration(h.Timeout)
grace, graceErr := time.ParseDuration(h.Grace)
switch {
case everyErr != nil || withinErr != nil || graceErr != nil:
say("health's interval, timeout and grace are durations")
default:
if every < HealthIntervalFloor {
say("a check looks no more often than every %s, not every %s", HealthIntervalFloor, every)
}
if within <= 0 || within >= every {
say("a look takes more than nothing and less than its interval")
}
if grace < 0 {
say("a grace is not negative")
}
if h.Looks >= HealthLooksFloor && grace+time.Duration(h.Looks)*every > HealthWithin {
say("a grace and the failing looks take at most %s", HealthWithin)
}
}
if h.Looks < HealthLooksFloor {
say("a check is unhealthy after at least %d failing looks, not %d", HealthLooksFloor, h.Looks)
}
return problems
}
+2 -2
View File
@@ -75,7 +75,7 @@ github.com/nats-io/nkeys
# github.com/nats-io/nuid v1.0.1
## explicit
github.com/nats-io/nuid
# github.com/novox/mesh-host v0.0.0 => git.novox.be/novox/mesh-host v0.0.0-20261006095519-3e80b7ae325e
# github.com/novox/mesh-host v0.0.0 => git.novox.be/novox/mesh-host v0.0.0-20261007120832-bdd44154ccac
## explicit; go 1.26.0
github.com/novox/mesh-host/internal/declaration
github.com/novox/mesh-host/validate
@@ -133,4 +133,4 @@ golang.org/x/text/width
# golang.org/x/time v0.15.0
## explicit; go 1.25.0
golang.org/x/time/rate
# github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261006095519-3e80b7ae325e
# github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261007120832-bdd44154ccac