Compare commits

...
13 Commits
Author SHA1 Message Date
jochen 874a2670fa Pin the node-engine at the commit that refuses a system account as the agents'
mesh/repo-check pass: its merge-check.sh passed
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/delivery superseded: a newer head of the same pull request
The validator is unchanged; the pin follows the mesh-host pull request's head
so the two are judged together (hq ADR 0266).
2026-10-08 21:08:13 +02:00
jochen d71fde0d54 Refuse a node's accounts through any verb, and a stale or service-account agent verdict
The generic command verb ran node account and node agent-account, so an agent
could name itself the operator account and have the next send grant it root
(hq ADR 0266 review). Refuse every node subcommand but list and show through
any verb; refuse the operator account as the agent account in both
directions and well-known service accounts as an agent account; and count a
verdict heard more than 15 minutes ago as not judged, so stopping the
node-engine cannot freeze a healthy one. Re-pin mesh-host to its review head.
2026-10-08 21:08:13 +02:00
jochen 0e50ee446f Name the account agents run as on a node, and say whether it can become root
On the control node every agent ran as the operator's account, which has
passwordless sudo, so an agent could become root without a person (hq ADR
0266). A node now names an agent account at the controller's terminal only;
the agent's module declares it never to become root, the node-engine judges
that, and the self-check (DA) raises agent-can-become-root while it does not
hold, so ADR 0259's router can rest on it.
2026-10-08 21:08:13 +02:00
mesh-admin ca09a07fdf Merge pull request 'Keep what a consumer gives up on until a person delivers it again or drops it (hq issue 330, ADR 0264)' (#159) from fix/330-a-message-given-up-on-is-kept into main 2026-10-08 19:07:33 +00:00
mesh-admin 9b028b4212 Merge pull request 'Add the node-nfs-server and node-mounts seats (hq ADR 0263)' (#163) from feat/mounts-module into main 2026-10-08 18:39:52 +00:00
jochen d7fab82a89 Say the adopt switch is the string "true" and that reload only has the kernel reread its exports, as the holders do
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group feat/mounts-module delivered: every member is delivered
2026-10-08 20:11:33 +02:00
mesh-admin 7f65e62743 Merge pull request 'Keep a left-out module's provisions and backups, and refuse more identity keys (hq ADR 0262)' (#162) from feat/left-out-keeps-what-it-provides into main 2026-10-08 16:37:42 +00:00
jochen 2b01f8786e Let node-nfs-server.test take a client's address: the server knows the range, not the mesh's node names
mesh/delivery-group group feat/mounts-module rejected: a member's own check failed
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
2026-10-08 18:34:38 +02:00
jochen 909062e729 Deliver a dead letter again only where it is received, and never stop serving for the notices
mesh/delivery delivered
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group fix/330-a-message-given-up-on-is-kept delivered: every member is delivered
A dead letter was let go as delivered even when its consumer did not filter
its again subject; seat asks needed a grant over every seat's queue and left
the original stuck; a notices bind failure stopped the controller (review).
2026-10-08 18:32:58 +02:00
jochen 826dcb91b1 Keep what a consumer gives up on until a person delivers it again or drops it
Design 25 promised a dead-letter stream that did not exist: a message a
consumer gave up on stayed only in its source, which drops it after a week,
and its condition cleared when the advisories stopped (hq issue 330, ADR 0264).
2026-10-08 18:32:58 +02:00
jochen 193168e086 Place a left-out module's backup lines best effort, and refuse more identity keys (hq ADR 0262 review)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery delivered
An unplaceable line of a left-out module, such as an access nobody placed, failed the whole machine's
declaration. Say it among what could not be placed instead, never copy the definition's path past a
placement that does not read, and accept a removal only when the decoder is past it.
2026-10-08 18:27:50 +02:00
jochen 59fdffb979 Add the node-nfs-server and node-mounts seats, so a share and a mount have a role the mesh defines (hq ADR 0263)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery-group group feat/mounts-module ready: every member ready, and composed together they pass
mesh/delivery superseded: a newer head of the same pull request
A machine sharing folders and a machine mounting them each need one holder
per machine, with verbs an agent calls instead of exportfs, fstab edits or
zfs set. Both seats deliver nothing: nfs-share is provided at the mesh's
scope. The two adopt verbs are dry runs unless confirmed.
2026-10-08 18:24:38 +02:00
jochen 5d47e0bfd6 Keep a left-out module's provisions and backups, and refuse more identity keys (hq ADR 0262)
mesh/merge-gate pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus step; every machine composes with the change as it…
mesh/repo-check pass: its merge-check.sh passed
mesh/delivery superseded: a newer head of the same pull request
A module left out for an unknown key inside an entry lost its whole manifest, so every consumer of what
it provides was refused and its data stopped being copied. Read past only the unknown key, keep its
backup lines, and say in the condition what stops.
2026-10-08 18:03:46 +02:00
61 changed files with 3060 additions and 81 deletions
+175
View File
@@ -0,0 +1,175 @@
package main
// The account agents run as (novox/hq ADR 0266).
//
// On the control node every agent session ran as the operator's account, which may become root without a
// password — so any agent there could become root without a person, and ADR 0259 §8 (an answer from the
// operator's phone authorises an act) rests on that being false where the router and its channels run. The
// decision: a node may name an account its agents run as, of their own and without sudo; the operator's
// account keeps its sudo.
//
// - **Named at the controller's terminal only** (`node agent-account`): not a verb, not a setting, so no
// agent can name itself another account. Empty is a real state: agents run as the operator there.
// - **Composed** as `${machine:agent-account}`, `${machine:agent-home}` and `${machine:agent-root}` for the
// agent's module, which declares the account with `root: never`, and as MESH_AGENT_ACCOUNT and
// MESH_AGENT_HOME for its tools (catalogue/machine_into_files.go, runtime.go).
// - **Judged by the machine itself.** The node-engine reads, on every look, whether an account declared
// `root: never` can become root without a person — uid 0, a group that grants root, a sudo rule, a
// secret of the mesh it may read — and says it as the declaring module's account verdict, marked
// Root "never". agentConfined reads that verdict; the self-check (probe DA) raises
// `agent-can-become-root` while it does not hold, and `node show` says it.
//
// A verdict not given is never a pass: an engine older than the judging, an account not yet declared, a
// statement that says nothing of it — each is "not judged", and fails.
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// kindAgentCanBecomeRoot is the condition raised while a machine's agent account can become root without
// a person, or is not judged (ADR 0266).
const kindAgentCanBecomeRoot = "agent-can-become-root"
// agentAccountProbe is the self-check's probe of it.
const agentAccountProbe = "DA"
// agentConfined says whether the agents of a machine that names an agent account are confined: the
// machine's newest statement holds a healthy account verdict, judged for root, on that account. named is
// false for a machine that names none — agents run as the operator account there, which this does not
// judge. why is said either way, in the mesh's words; err is a store that could not be read.
//
// The one judgement: `node show`, the self-check, and ADR 0259's router honouring a verified sender read
// it here.
func agentConfined(ctx context.Context, inv *inventory.Inventory, node string) (named, confined bool, why string, err error) {
n, err := inv.NodeByName(ctx, node)
if err != nil {
return false, false, "", err
}
if n.AgentAccount == "" {
return false, false, fmt.Sprintf("%s names no agent account: agents run as the operator account (%s)",
node, orNoneKnown(n.Account)), nil
}
h, had, err := inv.HealthOf(ctx, node)
if err != nil {
return true, false, "", err
}
confined, why = judgedConfined(n.AgentAccount, h, had, time.Now())
return true, confined, why, nil
}
// verdictFreshFor is how old the statement holding the verdict may be, by this controller's clock. A
// node-engine states its health on every change and at least every five minutes (mesh-host's sayAnyway), so
// three statements missed is a node-engine stopped, or a machine away. **A stale verdict is not a pass**: an
// agent that stopped the node-engine must not leave "cannot become root" standing from before.
const verdictFreshFor = 15 * time.Minute
// judgedConfined is the judgement over one statement, without the store, at now.
func judgedConfined(agent string, h inventory.NodeHealth, had bool, now time.Time) (bool, string) {
if !had {
return false, fmt.Sprintf("the agent account %s is not judged: the machine's node-engine has stated "+
"nothing of what it runs", agent)
}
if age := now.Sub(h.HeardAt); age > verdictFreshFor {
return false, fmt.Sprintf("the agent account %s is not judged: the machine's newest statement was heard at "+
"%s, more than %d minutes ago, and a verdict that old is not a verdict on now", agent,
h.HeardAt.Local().Format("2006-01-02 15:04"), int(verdictFreshFor.Minutes()))
}
if h.Contract < link.RootContract {
return false, fmt.Sprintf("the agent account %s is not judged: the machine's node-engine is older than "+
"the judging of an account's root (its statement's contract is %d, the judging is %d)",
agent, h.Contract, link.RootContract)
}
var verdicts []inventory.ResourceHealth
for _, r := range h.Resources {
if r.Kind == link.KindAccount && r.Target == agent && r.Root == link.RootNever {
verdicts = append(verdicts, r)
}
}
if len(verdicts) == 0 {
return false, fmt.Sprintf("the agent account %s is not judged: the machine's newest statement holds no "+
"verdict on it — no module there declares it never to become root, or the declaration naming it "+
"has not been applied", agent)
}
sort.Slice(verdicts, func(i, j int) bool {
return verdicts[i].Module+verdicts[i].Resource < verdicts[j].Module+verdicts[j].Resource
})
for _, v := range verdicts {
switch v.State {
case link.StateHealthy:
case link.StateUnhealthy:
// The engine's own words, which start with link.ReasonRoot when it found a way to root.
return false, fmt.Sprintf("the agent account %s %s (said by %s's %s)", agent,
orNoneKnown(v.Reason), v.Module, v.Resource)
default:
return false, fmt.Sprintf("the agent account %s is not judged: %s (%s's %s, %s)", agent,
orNoneKnown(v.Reason), v.Module, v.Resource, v.State)
}
}
return true, fmt.Sprintf("the agent account %s cannot become root without a person (judged %s)", agent,
h.SaidAt.Local().Format("2006-01-02 15:04"))
}
// probeAgentAccounts is DA: every machine that names an agent account has it judged, on its node-engine's
// newest statement, unable to become root without a person (ADR 0266).
func probeAgentAccounts(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
inv := d.open.inventory
nodes, err := inv.Nodes(ctx)
if err != nil {
return nil, err
}
var out []conditions.Observation
for _, n := range nodes {
if n.AgentAccount == "" {
continue
}
h, had, err := inv.HealthOf(ctx, n.Name)
if err != nil {
return nil, err
}
confined, why := judgedConfined(n.AgentAccount, h, had, time.Now())
if confined {
continue
}
out = append(out, conditions.Observation{Scope: conditions.ScopeMachine, ID: n.Name, Token: "agent-root",
Machine: n.Name, Severity: conditions.Urgent,
Summary: fmt.Sprintf("on %s, %s (ADR 0266): an agent there may become root without a person, and "+
"no answer from a channel authorises an act there (ADR 0259 §8)", n.Name, why),
Said: why})
}
return sortedFound(out), nil
}
// agentAccountLines is what `node show` says of the account agents run as.
func agentAccountLines(ctx context.Context, inv *inventory.Inventory, n inventory.Node) []string {
if n.AgentAccount == "" {
return []string{fmt.Sprintf(" agents run as the operator account (%s); no agent account is named",
orNoneKnown(n.Account))}
}
_, confined, why, err := agentConfined(ctx, inv, n.Name)
if err != nil {
return []string{fmt.Sprintf(" agents run as %s (home %s); whether it can become root could NOT be read: %v",
n.AgentAccount, n.AgentHome(), err)}
}
verdict := "CAN become root, or is not judged: " + why
if confined {
verdict = why
}
return []string{fmt.Sprintf(" agents run as %s (home %s)", n.AgentAccount, n.AgentHome()),
" " + verdict}
}
// orNoneKnown is a value, or that none is known.
func orNoneKnown(s string) string {
if strings.TrimSpace(s) == "" {
return "none known"
}
return s
}
+199
View File
@@ -0,0 +1,199 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The agent account's judgement (novox/hq ADR 0266): confined only on a healthy account verdict judged for
// root, on the very account; every verdict not given — no statement, an older engine, no verdict on it, a
// verdict of unknown — is "not judged" and fails, never a pass.
func TestAnAgentAccountIsConfinedOnlyOnAHealthyVerdictJudgedForRoot(t *testing.T) {
at := time.Date(2026, 10, 8, 19, 21, 0, 0, time.UTC)
verdict := func(target, root, state, reason string) inventory.ResourceHealth {
return inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: target, State: state, Reason: reason, Root: root, Account: target}
}
statement := func(contract int, rs ...inventory.ResourceHealth) inventory.NodeHealth {
return inventory.NodeHealth{Node: "anchor", Contract: contract, SaidAt: at, HeardAt: at, Resources: rs}
}
for _, c := range []struct {
name string
h inventory.NodeHealth
had bool
confined bool
says string
}{
{"judged and unable", statement(link.RootContract, verdict("agent", link.RootNever, link.StateHealthy, "")),
true, true, "cannot become root without a person"},
{"judged and able", statement(link.RootContract, verdict("agent", link.RootNever, link.StateUnhealthy,
link.ReasonRoot+": in the group docker, which grants root")), true, false, "in the group docker"},
{"a verdict of unknown", statement(link.RootContract, verdict("agent", link.RootNever, link.StateUnknown,
"sudo could not be read")), true, false, "not judged"},
{"no statement", inventory.NodeHealth{}, false, false, "not judged"},
{"an older engine", statement(link.ReadinessContract, verdict("agent", "", link.StateHealthy, "")),
true, false, "older than the judging"},
{"a verdict on groups only", statement(link.RootContract, verdict("agent", "", link.StateHealthy, "")),
true, false, "no verdict on it"},
{"a verdict on another account", statement(link.RootContract, verdict("ops", link.RootNever, link.StateHealthy, "")),
true, false, "no verdict on it"},
} {
confined, why := judgedConfined("agent", c.h, c.had, at.Add(time.Minute))
if confined != c.confined || !strings.Contains(why, c.says) {
t.Errorf("%s: confined %v, %q; want %v saying %q", c.name, confined, why, c.confined, c.says)
}
}
// A verdict heard longer ago than the bound is no verdict: an agent that stopped the node-engine must not
// leave "healthy" standing.
fresh := statement(link.RootContract, verdict("agent", link.RootNever, link.StateHealthy, ""))
if ok, _ := judgedConfined("agent", fresh, true, at.Add(verdictFreshFor)); !ok {
t.Error("a verdict exactly at the bound is still one")
}
if ok, why := judgedConfined("agent", fresh, true, at.Add(verdictFreshFor+time.Second)); ok ||
!strings.Contains(why, "not judged") {
t.Errorf("a stale healthy verdict passed: %q", why)
}
}
// DA raises an urgent condition, with plain words, on a machine whose agent account is not judged unable to
// become root; a machine that names none is not its to judge.
func TestTheSelfCheckSaysAnAgentAccountThatCanBecomeRoot(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.NodeByName(ctx, n); err != nil {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAccount(ctx, n, "ops", ""); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
found = onlyMachine(found, "anchor")
if len(found) != 1 || found[0].Machine != "anchor" || found[0].Severity != conditions.Urgent ||
!strings.Contains(found[0].Said, "not judged") {
t.Fatalf("a named agent account with no verdict: %+v", found)
}
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if w.Headline == "" || w.Needs == "" || w.Resolved == "" {
t.Errorf("the condition has no plain words: %+v", w)
}
healthy := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: link.StateHealthy, Root: link.RootNever, Account: "agent"}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{healthy}}); err != nil {
t.Fatal(err)
}
if found, err = probeAgentAccounts(ctx, d); err != nil || len(onlyMachine(found, "anchor")) != 0 {
t.Fatalf("a judged agent account still fails: %+v %v", found, err)
}
if named, confined, why, err := agentConfined(ctx, inv, "anchor"); err != nil || !named || !confined {
t.Fatalf("agentConfined on anchor: %v %v %q %v", named, confined, why, err)
}
if named, _, why, err := agentConfined(ctx, inv, "laptop"); err != nil || named ||
!strings.Contains(why, "operator account") {
t.Fatalf("agentConfined on a machine naming none: %v %q %v", named, why, err)
}
}
func TestTheAgentRootWordsArePlain(t *testing.T) {
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if why, ok := conditions.PlainWords(w, "anchor"); !ok {
t.Fatalf("not plain: %s: %+v", why, w)
}
}
func onlyMachine(obs []conditions.Observation, machine string) []conditions.Observation {
var out []conditions.Observation
for _, o := range obs {
if o.Machine == machine {
out = append(out, o)
}
}
return out
}
// The snapshot a merge check composes from carries the agent account as a pseudonym (novox/hq ADR 0266),
// so a change is judged against machines that name one, and the name never leaves.
func TestTheFactsCarryTheAgentAccountAsAPseudonym(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
if err := open.inventory.SetAccount(ctx, "anchor", "keeper", ""); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetAgentAccount(ctx, "anchor", "warden", ""); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
body, _ := f.Encode()
if strings.Contains(string(body), "warden") {
t.Error("the agent account's name is in the snapshot")
}
m, ok := f.Machine(snapshot.Pseudonym("machine", "anchor"))
if !ok || m.AgentAccount != snapshot.Pseudonym("account", "warden") || m.Account == m.AgentAccount {
t.Fatalf("the anchor's agent account reads as %q (operator %q)", m.AgentAccount, m.Account)
}
}
// No verb runs a `node` command that sets something: through the generic `command` verb, `node account`,
// `node agent-account` and every other `node` subcommand but list and show are refused, naming the terminal.
func TestNoVerbSetsANodesAccounts(t *testing.T) {
for _, line := range []string{
"node agent-account novox --clear",
"node agent-account novox ops",
"node account novox agent",
"node account novox",
"node add intruder",
"node public-domain novox --clear",
"node",
"node frobnicate",
} {
argv, err := argvFor("command", map[string]any{"command": line})
if err == nil || !strings.Contains(err.Error(), "controller's terminal only") ||
!strings.Contains(err.Error(), "ADR 0266") {
t.Errorf("%q ran as %v (%v); want a refusal naming the terminal", line, argv, err)
}
}
for _, line := range []string{"node show novox", "node list --json", "status --json"} {
if _, err := argvFor("command", map[string]any{"command": line}); err != nil {
t.Errorf("%q, a read, was refused: %v", line, err)
}
}
if err := terminalOnly([]string{"node", "account", "a", "b"}); err == nil {
t.Error("the refusal is not only the command verb's")
}
}
// Naming the agent account is the controller's terminal's alone: the `node` verb only shows.
func TestTheNodeVerbOnlyShows(t *testing.T) {
argv, err := argvFor("node", map[string]any{"node": "anchor"})
if err != nil || strings.Join(argv, " ") != "node show anchor" {
t.Fatalf("the node verb runs %v (%v)", argv, err)
}
for _, v := range catalogue.ControllerVerbs {
if strings.Contains(v.Name, "agent") {
t.Errorf("a verb %q may name the agent account", v.Name)
}
}
}
+166
View File
@@ -0,0 +1,166 @@
package main
import (
"context"
"errors"
"fmt"
"strconv"
"strings"
"sync/atomic"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// What a consumer gave up on, answered by the serving controller (novox/hq issue 330).
//
// **In this process, on its own connection**: DEAD_LETTERS is read and changed on the bus, and the
// serving controller is already on it. A verb run as a fresh process would open a connection of its own
// for each call (novox/hq issue 327).
// deadLettersOn is the serving controller's bus, set when it starts watching the mesh; nil in a
// process that serves nothing.
var deadLettersOn atomic.Pointer[busHandles]
// busHandles are the serving controller's connection and JetStream handle.
type busHandles struct {
conn *nats.Conn
js nats.JetStreamContext
}
// defaultDeadLetters is how many the list says when not asked for more.
const defaultDeadLetters = 50
// causeDeadLetter is the cause a delivery or drop gives when the caller gives none.
const causeDeadLetter = "dead-letter"
// deadLettersAnswer is what `dead-letters` answers: the list, one whole, or what came of delivering one
// again or dropping it.
func deadLettersAnswer(ctx context.Context, a *verbArguments) (any, error) {
on := deadLettersOn.Load()
if on == nil {
return nil, errors.New("this controller is not serving, so it does not read DEAD_LETTERS: ask again, " +
"and the serving controller answers")
}
// The shape first, and every argument it reads; one given beside it is refused before anything is
// done, as every verb refuses what it would pass over (novox/hq issue 244).
var deliver, drop, why, cause, idText, consumer, limit string
switch {
case a.given["deliver"] != "" || a.given["drop"] != "":
deliver, drop, why, cause = a.str("deliver"), a.str("drop"), a.str("why"), a.str("cause")
case a.given["id"] != "":
idText = a.str("id")
default:
consumer, limit = a.str("consumer"), a.str("limit")
}
if unused := a.unused(); len(unused) > 0 {
return nil, fmt.Errorf("dead-letters did not use %s together with %s, and an argument a verb would pass "+
"over is refused: nothing was done", quoteAll(unused), quoteAll(a.usedGiven()))
}
switch {
case deliver != "" && drop != "":
return nil, errors.New("dead-letters delivers one again or drops one, not both. Nothing was done")
case deliver != "" || drop != "":
act, text := "deliver", deliver
if drop != "" {
act, text = "drop", drop
}
if strings.TrimSpace(why) == "" {
return nil, fmt.Errorf("dead-letters %s is a hand act, and says why: why is required and recorded in "+
"the hand-act log (novox/hq to-be 45 §7). Nothing was done", act)
}
id, err := deadLetterID(text)
if err != nil {
return nil, err
}
return actOnDeadLetter(ctx, on, act, id, why, cause)
case idText != "":
id, err := deadLetterID(idText)
if err != nil {
return nil, err
}
return link.DeadLetterNamed(on.js, id)
}
most := defaultDeadLetters
if limit != "" {
n, err := strconv.Atoi(limit)
if err != nil || n <= 0 {
return nil, fmt.Errorf("limit is a number of dead letters, not %q", limit)
}
most = n
}
held, total, err := link.DeadLetters(on.js, consumer, most)
if err != nil {
return nil, err
}
answer := map[string]any{"dead_letters": held, "held": total,
"note": "newest first; with id, one whole; deliver or drop one with why"}
if total == 0 {
answer["note"] = "no consumer gave up on a message that is still kept"
}
return answer, nil
}
// deadLetterID is a dead letter's id as a caller wrote it.
func deadLetterID(text string) (uint64, error) {
id, err := strconv.ParseUint(strings.TrimSpace(text), 10, 64)
if err != nil || id == 0 {
return 0, fmt.Errorf("a dead letter's id is its number in %s, as dead-letters lists it, not %q",
broker.DeadLettersStream, text)
}
return id, nil
}
// actOnDeadLetter delivers one again or drops it, recorded in the hand-act log before it is done. A log
// that cannot be written is said, and the act still happens: the log is never the reason a person's act
// is refused (handacts.go).
func actOnDeadLetter(ctx context.Context, on *busHandles, act string, id uint64, why, cause string) (any, error) {
d, err := link.DeadLetterNamed(on.js, id)
if err != nil {
return nil, err
}
if act == "deliver" {
// Refused before it is recorded: an act that cannot be done is not an act.
if _, err := link.AgainTo(on.js, d); err != nil {
return nil, err
}
}
if cause == "" {
cause = causeDeadLetter
}
by := link.CallerIn(ctx)
if by == "" {
by = "a seat call whose caller the bus did not name"
}
answer := map[string]any{"dead_letter": d.ID, "consumer": d.Who, "subject": d.Subject}
recorded, logErr := link.RecordHandAct(ctx, on.conn, link.HandAct{Verb: "dead-letters " + act,
Args: []string{strconv.FormatUint(id, 10), d.Stream + "." + d.Consumer}, Why: why, Cause: cause,
Condition: conditions.Key(conditions.ScopeBus, d.Stream+"."+d.Consumer, link.AdvisoryMaxDeliveries),
By: by + ", through the " + catalogue.ControllerSeatName + " seat"})
if logErr != nil {
answer["unrecorded"] = "the hand-act log could not be written, and the act was done all the same: " + logErr.Error()
} else {
answer["recorded"] = recorded.ID
}
switch act {
case "deliver":
_, to, err := link.DeliverAgain(on.js, id)
if err != nil {
return nil, err
}
answer["delivered_on"] = to
answer["done"] = fmt.Sprintf("dead letter %d was delivered again to %s, and nobody else; it is no longer kept",
id, consumerWho(d.Stream, d.Consumer))
case "drop":
if _, err := link.DropDeadLetter(on.js, id); err != nil {
return nil, err
}
answer["done"] = fmt.Sprintf("dead letter %d, which %s gave up on, was dropped for good", id,
consumerWho(d.Stream, d.Consumer))
}
return answer, nil
}
+132
View File
@@ -0,0 +1,132 @@
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// The serving controller's bus, with the mesh's streams, for the verb to read and act on.
func servingDeadLetters(t *testing.T) *broker.JetStream {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
before := deadLettersOn.Load()
deadLettersOn.Store(&busHandles{conn: js.Conn(), js: js.Context()})
t.Cleanup(func() { deadLettersOn.Store(before) })
return js
}
func askDeadLetters(t *testing.T, args map[string]any) (any, error) {
t.Helper()
a, err := readArguments("dead-letters", args)
if err != nil {
return nil, err
}
return deadLettersAnswer(context.Background(), a)
}
func TestDeadLettersListsDropsAndRecordsWhy(t *testing.T) {
js := servingDeadLetters(t)
if _, err := js.Context().Publish("mesh.mod.gitea.event.pull.merged", []byte(`{"n":1}`)); err != nil {
t.Fatal(err)
}
d, err := link.KeepDeadLetter(js.Context(), []byte(`{"stream":"EVENTS","consumer":"media_sonarr","stream_seq":1,"deliveries":5}`))
if err != nil {
t.Fatal(err)
}
answer, err := askDeadLetters(t, map[string]any{})
if err != nil {
t.Fatal(err)
}
listed := answer.(map[string]any)
if listed["held"] != 1 || len(listed["dead_letters"].([]link.DeadLetter)) != 1 {
t.Fatalf("listed %v", listed)
}
// Refused before anything is done: an act without why, an argument the shape passes over, both acts.
for _, args := range []map[string]any{
{"drop": "1"},
{"drop": "1", "why": "x", "limit": "3"},
{"drop": "1", "deliver": "1", "why": "x"},
{"why": "x"},
{"id": "nought"},
} {
if _, err := askDeadLetters(t, args); err == nil {
t.Errorf("%v was done", args)
}
}
if _, total, _ := link.DeadLetters(js.Context(), "", 0); total != 1 {
t.Fatalf("a refused call changed what is kept: %d left", total)
}
done, err := askDeadLetters(t, map[string]any{"drop": "1", "why": "the media server took the download in by hand"})
if err != nil {
t.Fatal(err)
}
if said := done.(map[string]any); said["recorded"] == nil || !strings.Contains(said["done"].(string), "dropped") {
t.Fatalf("answered %v", said)
}
acts, err := link.HandActs(context.Background(), js.Conn(), time.Now().Add(-time.Minute))
if err != nil || len(acts) != 1 || acts[0].Verb != "dead-letters drop" || acts[0].Cause != causeDeadLetter ||
!strings.Contains(acts[0].Condition, "media_sonarr") {
t.Fatalf("recorded %+v (%v)", acts, err)
}
if !personsDecision(acts[0]) {
t.Error("dropping a dead letter is counted as a repair, so S15 would want a healer for it")
}
if _, err := askDeadLetters(t, map[string]any{"id": "1"}); err == nil {
t.Errorf("dead letter %d is still answered after it was dropped", d.ID)
}
}
// Open while DEAD_LETTERS holds a message for the consumer, in words the operator reads in one pass:
// what is held, and where to act.
func TestAConsumersDeadLettersAreSaidUntilActedOn(t *testing.T) {
f := &signalFacts{now: time.Now(), deadLetters: map[string]int{"EVENTS.media_sonarr": 4, "EVENTS.controller": 1}}
found := watchDeadLetters(f)
if len(found) != 2 {
t.Fatalf("said %d conditions", len(found))
}
for _, o := range found {
if o.Kind != "max-deliveries" || o.Severity != conditions.Warning || o.Needs == "" {
t.Errorf("%+v", o)
}
if why, ok := conditions.PlainWords(conditions.Words{Headline: o.Headline, Needs: o.Needs,
Explanation: o.Explanation, Resolved: o.Resolved}, "media"); !ok {
t.Errorf("%q is not plain: %s", o.Headline, why)
}
if !strings.Contains(o.Needs, "mesh MCP server") {
t.Errorf("does not say where to act: %q", o.Needs)
}
}
sonarr := found[1]
if sonarr.ID != "EVENTS.media_sonarr" || sonarr.Machine != "media" ||
sonarr.Headline != "Sonarr on media could not handle 4 messages" ||
!strings.Contains(sonarr.Summary, "DEAD_LETTERS") {
t.Errorf("%+v", sonarr)
}
if found[0].Headline != "The controller could not handle a message" {
t.Errorf("%q", found[0].Headline)
}
// None held, none said: it clears when they are delivered again or dropped.
if left := watchDeadLetters(&signalFacts{now: time.Now()}); len(left) != 0 {
t.Fatalf("%v", left)
}
}
+5
View File
@@ -116,6 +116,11 @@ var probeRegistry = []probe{
{ID: probeDeliveriesID, Asserts: "no delivery is held past its state's bound unsaid: mesh-delivery's " +
"`stalled`, each with the transition its table lets healer H2 take", From: "ADR 0239",
Kind: kindDeliveryStalled, Phase: 3, run: probeDeliveries},
// The account agents run as (novox/hq ADR 0266): where a machine names one, its node-engine has judged it
// unable to become root without a person — what ADR 0259 §8 rests an authorised answer on.
{ID: agentAccountProbe, Asserts: "every machine that names an agent account has it judged, on its node-engine's " +
"newest statement, unable to become root without a person", From: "ADR 0266, ADR 0259 §8",
Kind: kindAgentCanBecomeRoot, Phase: 1, run: probeAgentAccounts},
{ID: "DW", Asserts: "the watchdogs of the signals table ran within three of their intervals",
From: "ADR 0227 rule 6: the watchers are watched", Kind: "watchdogs-silent", Phase: 1, run: probeWatchdogs},
// The core's health definitions (novox/hq to-be 45 §8, ADR 0236): what a core component's new build is
+2 -1
View File
@@ -269,7 +269,8 @@ func gatherFacts(ctx context.Context, open *stores, busVersion string) (snapshot
engines := map[string]bool{}
for _, n := range nodes {
m := snapshot.Machine{Name: scrub.Machine(n.Name), Length: len(n.Name), Adopted: n.Adopted,
AccountHome: scrub.Text(n.AccountHome), NodeEngine: n.HostVersion, PublicDomain: domains[n.Name]}
AccountHome: scrub.Text(n.AccountHome), NodeEngine: n.HostVersion, PublicDomain: domains[n.Name],
AgentAccount: scrub.Account(n.AgentAccount), AgentAccountHome: scrub.Text(n.AgentAccountHome)}
switch n.Account {
case "", "root":
m.Account = n.Account
+5
View File
@@ -82,6 +82,11 @@ var handActVerbs = []handActVerb{
{Verb: "retire approve", Decision: "nothing is retired past its bound without a person (ADR 0230)"},
{Verb: "retire reject", Decision: "keeping a consumer active is a person's word (ADR 0230)"},
{Verb: "cleanup delete", Decision: "nothing retired is deleted without a person (ADR 0230)"},
// What becomes of a message a consumer gave up on (novox/hq issue 330): kept until a person says.
{Verb: "dead-letters deliver", Decision: "a message a consumer gave up on is delivered again only on a " +
"person's word (issue 330)"},
{Verb: "dead-letters drop", Decision: "a message a consumer gave up on is let go only on a person's word " +
"(issue 330)"},
// The sweep run on a person's word rather than after a build: the same decision the records make, at
// a moment the person chose (ADR 0251) — never a repair.
{Verb: "collect", Decision: "letting the store go of what the records keep for no reason, now rather " +
+5
View File
@@ -1013,6 +1013,11 @@ func raiseFromFacts(ctx context.Context, open *stores, f snapshot.Facts, shelf m
return notes, err
}
}
if m.AgentAccount != "" {
if err := inv.SetAgentAccount(ctx, m.Name, m.AgentAccount, m.AgentAccountHome); err != nil {
return notes, err
}
}
if m.PublicDomain != "" {
if err := inv.SetPublicDomain(ctx, m.Name, m.PublicDomain); err != nil {
return notes, err
+1 -1
View File
@@ -73,7 +73,7 @@ func stateHealth(ctx context.Context, inv *inventory.Inventory, k *conditions.Ke
for _, r := range h.Resources {
kept := inventory.ResourceHealth{Module: r.Module, Resource: r.Resource, Kind: r.Kind, Target: r.Target,
State: r.State, Reason: r.Reason, Since: r.Since, Streak: r.Streak, Restarts: r.Restarts,
Check: r.Check, Needs: r.Needs, Account: r.Account}
Check: r.Check, Needs: r.Needs, Account: r.Account, Root: r.Root}
resources = append(resources, kept)
if r.State == link.StateUnhealthy && r.Module != "" {
unhealthy[r.Module] = append(unhealthy[r.Module], kept)
+62 -1
View File
@@ -98,6 +98,12 @@ func nodeCommand(ctx context.Context, args []string) error {
"containers reaching outward. The machine reports which of its links face outside; see " +
"`node show <name>`")
case "agent-account":
// The account agents run as on this machine, when it is not the operator's (novox/hq ADR 0266). Here,
// at the controller's terminal, and nowhere else: no verb and no setting names it, so no agent can
// name itself another account.
return nodeAgentAccount(ctx, inv, args[1:])
case "account":
// The operator's login on this machine (novox/hq to-be 29): what a home-scoped file is
// owned by and which account `ssh <node>` uses. Reports with no argument; sets with one;
@@ -105,7 +111,7 @@ func nodeCommand(ctx context.Context, args []string) error {
return nodeAccount(ctx, inv, args[1:])
default:
return fmt.Errorf("node has no %q; it has add, list, show, public-domain and account", args[0])
return fmt.Errorf("node has no %q; it has add, list, show, public-domain, account and agent-account", args[0])
}
}
@@ -176,6 +182,57 @@ func nodeAccount(ctx context.Context, inv *inventory.Inventory, positionals []st
return nil
}
const agentAccountUsage = "node agent-account <name> — what it is now; " +
"<name> <account> [home] to name the account agents run as (home defaults to /home/<account>); " +
"<name> --clear to have them run as the operator account again"
// nodeAgentAccount reports, names or clears the account agents run as on a node (novox/hq ADR 0266). Read-
// shaped with no account, like public-domain; clearing is asked for by name.
func nodeAgentAccount(ctx context.Context, inv *inventory.Inventory, args []string) error {
set := flag.NewFlagSet("node agent-account", flag.ContinueOnError)
clear := set.Bool("clear", false, "agents run as the operator account again")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) == 0 || len(positionals) > 3 {
return errors.New(agentAccountUsage)
}
node := positionals[0]
switch {
case *clear && len(positionals) > 1:
return fmt.Errorf("name an agent account for %s or --clear, not both", node)
case *clear:
if err := inv.SetAgentAccount(ctx, node, "", ""); err != nil {
return err
}
fmt.Printf("agents on %s run as the operator account again\n", node)
fmt.Printf(" run `push %s` to send it; the agent account itself is kept (the mesh never deletes a login)\n", node)
return nil
case len(positionals) >= 2:
home := ""
if len(positionals) == 3 {
home = positionals[2]
}
if err := inv.SetAgentAccount(ctx, node, positionals[1], home); err != nil {
return err
}
fmt.Printf("agents on %s run as %s\n", node, positionals[1])
fmt.Printf(" run `push %s` to send it; the self-check says once its node-engine has judged it "+
"unable to become root\n", node)
return nil
default:
n, err := inv.NodeByName(ctx, node)
if err != nil {
return err
}
for _, line := range agentAccountLines(ctx, inv, n) {
fmt.Println(strings.TrimPrefix(line, " "))
}
return nil
}
}
const publicDomainUsage = "node public-domain <name> — what it is now; " +
"<name> <domain> to set it; <name> --clear to take it away"
@@ -590,6 +647,10 @@ func showNode(ctx context.Context, inv *inventory.Inventory, name string) error
if err := showMode(ctx, inv, node); err != nil {
return err
}
// Whom agents run as here, and whether that account can become root without a person (ADR 0266).
for _, line := range agentAccountLines(ctx, inv, node) {
fmt.Println(line)
}
// The domain its routed names are composed under, when it has one (novox/hq ADR 0066). Shown
// only when set: a machine that serves nothing to the outside has no domain, and saying so of
+14 -3
View File
@@ -110,6 +110,15 @@ var plainWordings = map[string]func(conditions.Observation) words{
"reaches it. It keeps running what it has.", m),
Resolved: m + " can get new instructions again"}
}),
kindAgentCanBecomeRoot: worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: "Sessions on " + m + " could become root",
Needs: "take the sessions' own account out of every group and rule that grants root; the details say which.",
Explanation: fmt.Sprintf("Assistant sessions on %s run under an account of their own, so that none "+
"can take over the machine without you. The machine cannot show that this holds now, so an answer "+
"from your phone authorises nothing there until it does.", m),
Resolved: "Sessions on " + m + " cannot become root again"}
}),
"own-address-banned": worded(func(o conditions.Observation) words {
m := machineOr(o, "a machine")
return words{Headline: m + " has banned the mesh",
@@ -544,9 +553,11 @@ var plainWordings = map[string]func(conditions.Observation) words{
Resolved: "The listener keeps up again"}
}),
"max-deliveries": worded(func(o conditions.Observation) words {
return words{Headline: "A message could not be handled",
Explanation: "The bus gave up on a message after trying to hand it over too many times.",
Resolved: "Resolved: messages are handled again"}
return words{Headline: "A listener gave up on messages",
Needs: "deliver them again or drop them, from the mesh MCP server.",
Explanation: "A listener on the bus could not handle messages after several tries, so what they asked " +
"for was not done. The mesh keeps them until you deliver them again or drop them.",
Resolved: "Resolved: the messages given up on were delivered again or dropped"}
}),
"refused": worded(func(o conditions.Observation) words {
return words{Headline: "The bus refuses some messages",
+17 -1
View File
@@ -136,7 +136,8 @@ func planFor(ctx context.Context, open *stores, nodeName string) (catalogue.Reso
resolved, err := catalogue.Resolve(shelf, assigned,
catalogue.Node{Name: nodeName, Site: site, Capabilities: capabilities,
At: onNetwork[nodeName], PublicDomain: publicDomain,
Account: who.Account, AccountHome: who.AccountHome}, world)
Account: who.Account, AccountHome: who.AccountHome,
AgentAccount: who.AgentAccount, AgentAccountHome: who.AgentAccountHome}, world)
if err != nil {
// The node's own set does not compose. Marked, because this is the only failure here that
// a mesh-wide gatherer may pass over — see notResolvable.
@@ -885,8 +886,13 @@ func renderingFor(ctx context.Context, open *stores, node string,
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
judgesRoot, err := engineJudgesRoot(ctx, inv, node)
if err != nil {
return catalogue.Rendering{}, inventory.Node{}, err
}
return catalogue.Rendering{
ReadsHealth: readsHealth,
JudgesRoot: judgesRoot,
BusMembership: memberships[node],
Settings: settings, Generators: gens, Grants: grants, Needed: needed, Foreseen: foreseen, Ports: ports,
Certificate: certificate, Authority: authority, Mesh: private, Names: names,
@@ -909,6 +915,16 @@ func engineReadsHealth(ctx context.Context, inv *inventory.Inventory, node strin
return had && stated.Contract >= link.ReadinessContract, nil
}
// engineJudgesRoot says whether a machine's node-engine judges a user's declared `root` (novox/hq ADR 0266),
// by its own newest statement, for the same reason as engineReadsHealth: an older engine parses strictly.
func engineJudgesRoot(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.RootContract, nil
}
// zonesInTheMesh is every zone a module in the mesh declares, where the mesh placed it (novox/hq ADR
// 0199): the zone settled from that node's settings, the node's private address, the port the
// answering listen is published on there.
+10
View File
@@ -875,6 +875,16 @@ func streamDiffers(want broker.Stream, have nats.StreamConfig) string {
if perSubject != 0 && have.MaxMsgsPerSubject != perSubject {
differs = append(differs, fmt.Sprintf("keeps %d per subject, defined %d", have.MaxMsgsPerSubject, perSubject))
}
if want.MaxBytes > 0 && have.MaxBytes != want.MaxBytes {
differs = append(differs, fmt.Sprintf("holds up to %d bytes, defined %d", have.MaxBytes, want.MaxBytes))
}
if want.DuplicatesSeconds > 0 && have.Duplicates != time.Duration(want.DuplicatesSeconds)*time.Second {
differs = append(differs, fmt.Sprintf("keeps one of a message id for %s, defined %s", have.Duplicates,
time.Duration(want.DuplicatesSeconds)*time.Second))
}
if want.DiscardNew && have.Discard != nats.DiscardNew {
differs = append(differs, "drops what it holds when full, defined to refuse what comes next")
}
return strings.Join(differs, "; ")
}
+36 -1
View File
@@ -51,6 +51,9 @@ func argvFor(verb string, args map[string]any) ([]string, error) {
return nil, err
}
argv, err := a.commandLine()
if err == nil {
err = terminalOnly(argv)
}
if len(a.misread) > 0 {
// The table and the command line disagree: the verb reads an argument no caller can see
// in its schema, so no caller could ever pass it.
@@ -250,6 +253,8 @@ func (a *verbArguments) commandLine() ([]string, error) {
return argv, nil
case "tools":
return nil, errors.New("tools is answered from the records, not by a command")
case "dead-letters":
return nil, errors.New("dead-letters is answered by the serving controller, on its own connection, not by a command")
case "status":
return []string{"status", "--json"}, nil
case "nodes":
@@ -982,6 +987,9 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
if verb == "calls" {
return callsAnswer(link.Calls, a.given["call"])
}
if verb == "dead-letters" {
return deadLettersAnswer(ctx, a)
}
if verb == "doctor" {
// From the serving controller, which runs the self-check and hears the signals
// (novox/hq to-be 45 §4): the last verdict at once, or a run now.
@@ -1064,7 +1072,10 @@ func actsOnAPlan(args map[string]any) bool {
// inProcess are the verbs answered by this process rather than by a command it runs: `tools` from
// the records, `calls` from what this process served.
var inProcess = map[string]bool{"tools": true, "calls": true, "doctor": true}
var inProcess = map[string]bool{"tools": true, "calls": true, "doctor": true,
// What a consumer gave up on, read and changed on the serving controller's own connection (novox/hq
// issue 330).
"dead-letters": true}
// answersFirst is a command line whose caller is answered before it runs: a push, by its verb or
// through `command`. A push sends the machine holding the bus first when its user list changed, the
@@ -1253,3 +1264,27 @@ func seatAnnouncement(handlers map[string]link.ToolHandler) micro.Info {
Endpoints: endpoints,
}
}
// nodeReads are the `node` subcommands a verb may run: the ones that only read.
var nodeReads = map[string]bool{"list": true, "show": true}
// terminalOnly refuses, through any verb, a command that is the operator's at the controller's terminal
// alone (novox/hq ADR 0266). **Every `node` subcommand that is not a read**: `node account` and
// `node agent-account` above all. Whoever may call a verb includes agents, and an agent that named itself
// the operator account, or cleared the agent account, would have the next send grant it root through the
// sudo module's rule. An allow list, so a subcommand added later is refused until it is judged a read.
func terminalOnly(argv []string) error {
if len(argv) == 0 || argv[0] != "node" {
return nil
}
if len(argv) > 1 && nodeReads[argv[1]] {
return nil
}
sub := "node"
if len(argv) > 1 {
sub += " " + argv[1]
}
return fmt.Errorf("%s is run at the controller's terminal only, never through a verb: a node's accounts "+
"decide who may become root on it (novox/hq ADR 0266). A verb may run node list and node show. "+
"Nothing was done", sub)
}
+6 -6
View File
@@ -237,19 +237,19 @@ func TestAJSONVerbsAnswerIsItsStandardOutput(t *testing.T) {
}
}
// `command` is the generic verb: the command line as given, split as a shell would, nothing added —
// so an operator's `node account g14 jochen` is one call through the console rather than a shell on
// the control node (novox/hq ADR 0154, ADR 0175).
// `command` is the generic verb: the command line as given, split as a shell would, nothing added
// (novox/hq ADR 0154, ADR 0175). It once carried an operator's `node account g14 jochen` too; a node's
// accounts are the controller's terminal's alone since ADR 0266 (TestNoVerbSetsANodesAccounts).
func TestCommandRunsTheLineAsGiven(t *testing.T) {
argv, err := argvFor("command", map[string]any{"command": "node account g14 jochen"})
if err != nil || strings.Join(argv, " ") != "node account g14 jochen" {
argv, err := argvFor("command", map[string]any{"command": "node show g14"})
if err != nil || strings.Join(argv, " ") != "node show g14" {
t.Fatalf("a plain line: %v %v", argv, err)
}
argv, err = argvFor("command", map[string]any{"command": `settings set dnsmasq '{"a": "b c"}' --node ace`})
if err != nil || len(argv) != 6 || argv[3] != `{"a": "b c"}` {
t.Fatalf("a quoted word stays one word: %q %v", argv, err)
}
argv, err = argvFor("command", map[string]any{"command": `node add "the box" --adopted`})
argv, err = argvFor("command", map[string]any{"command": `module show "the box" --json`})
if err != nil || len(argv) != 4 || argv[2] != "the box" {
t.Fatalf("double quotes group: %q %v", argv, err)
}
+87 -4
View File
@@ -154,8 +154,9 @@ var signalsTable = []signalRow{
}},
{Row: "S9", Signal: "bus advisories: maximum deliveries, consumer deleted; the controller's own slow " +
"consumer and refused subjects", Emitter: "bus server's advisory subjects; the controller's connection",
Trigger: "any", Bound: "any occurrence; clears after an hour without another, and a deleted consumer " +
"once it exists again or the mesh no longer expects it",
Trigger: "any", Bound: "any occurrence; clears after an hour without another, a deleted consumer " +
"once it exists again or the mesh no longer expects it, and a message given up on once DEAD_LETTERS " +
"no longer holds it (novox/hq issue 330)",
Kind: "slow-consumer, max-deliveries, refused, consumer-lost", Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.advisoriesErr }, watch: watchAdvisories,
newest: func(f *signalFacts) time.Time {
@@ -485,12 +486,94 @@ func watchAdvisories(f *signalFacts) []conditions.Observation {
if a.ID == "controller" {
machine = f.host
}
out = append(out, conditions.Observation{Scope: conditions.ScopeBus, ID: a.ID, Kind: a.Kind,
Machine: machine, Severity: severity, Summary: a.Said + times, Said: a.Said})
o := conditions.Observation{Scope: conditions.ScopeBus, ID: a.ID, Kind: a.Kind, Token: a.Token,
Machine: machine, Severity: severity, Summary: a.Said + times, Said: a.Said}
if a.Kind == link.AdvisoryMaxDeliveries && a.Token == link.AdvisoryNotKept {
o.Machine = consumerMachine(a.Stream, a.Consumer)
o.Headline = clip(conditions.Capital(fmt.Sprintf("%s gave up on a message, not kept",
consumerWho(a.Stream, a.Consumer))), 60)
o.Explanation = "A listener on the bus could not handle a message, and the mesh could not keep it " +
"for you yet. It tries again every minute."
o.Resolved = "Resolved: the message is kept"
}
out = append(out, o)
}
return append(out, watchDeadLetters(f)...)
}
// watchDeadLetters says each consumer that DEAD_LETTERS holds a message for (novox/hq issue 330): open
// while it holds any, so it clears when they are delivered again or dropped, never because the server
// stopped saying it.
func watchDeadLetters(f *signalFacts) []conditions.Observation {
keys := make([]string, 0, len(f.deadLetters))
for k := range f.deadLetters {
keys = append(keys, k)
}
sort.Strings(keys)
var out []conditions.Observation
for _, key := range keys {
n := f.deadLetters[key]
stream, consumer, _ := strings.Cut(key, ".")
messages, them := "a message", "it"
if n > 1 {
messages, them = fmt.Sprintf("%d messages", n), "them"
}
who := consumerWho(stream, consumer)
out = append(out, conditions.Observation{Scope: conditions.ScopeBus, ID: key, Kind: link.AdvisoryMaxDeliveries,
Machine: consumerMachine(stream, consumer), Severity: conditions.Warning,
Summary: fmt.Sprintf("%s gave up on %s; %s kept in %s until delivered again or dropped, with why, "+
"through the controller's dead-letters verb", link.ConsumerInWords(stream, consumer), messages,
map[bool]string{true: "they are", false: "it is"}[n > 1], broker.DeadLettersStream),
Said: fmt.Sprintf("%d held for %s", n, key),
Headline: clip(conditions.Capital(fmt.Sprintf("%s could not handle %s", who, messages)), 60),
Needs: fmt.Sprintf("deliver %s again or drop %s, from the mesh MCP server.", them, them),
Explanation: conditions.Capital(fmt.Sprintf("%s was handed %s several times and gave up, so what %s "+
"asked for was not done. The mesh keeps %s until you deliver %s again or drop %s.", who, messages,
them, them, them, them)),
Resolved: "Resolved: the messages it gave up on were delivered again or dropped"})
}
return out
}
// consumerWho is a durable consumer's holder as the operator says it: a module on its machine, the
// controller, or a seat's holders.
func consumerWho(stream, consumer string) string {
switch {
case consumer == broker.ControllerName:
return "the controller"
case strings.HasPrefix(stream, "SEAT_") && strings.HasSuffix(consumer, "_worker"):
seat := strings.ToLower(strings.ReplaceAll(strings.TrimSuffix(strings.TrimPrefix(consumer, "SEAT_"), "_worker"), "_", "-"))
return "the holder of " + seat
case stream == broker.EventsStream:
if node, module, ok := strings.Cut(consumer, "_"); ok {
return module + " on " + node
}
}
return "a listener on the bus"
}
// consumerMachine is the machine a module's consumer is on; empty for the others.
func consumerMachine(stream, consumer string) string {
if stream == broker.EventsStream {
if node, _, ok := strings.Cut(consumer, "_"); ok {
return node
}
}
return ""
}
// clip is words at most n characters long, cut at a word.
func clip(s string, n int) string {
if len(s) <= n {
return s
}
cut := s[:n]
if i := strings.LastIndex(cut, " "); i > 0 {
cut = cut[:i]
}
return cut
}
func watchSelfCheck(f *signalFacts) []conditions.Observation {
every := f.selfCheck.every
if every <= 0 {
+1 -1
View File
@@ -38,7 +38,7 @@ func unknownFieldObservations(known map[string]catalogue.Manifest) []conditions.
Summary: catalogue.UnknownFieldReason(m),
Said: m.UnknownField(),
Headline: name + " is left out until the controller is updated",
Explanation: name + " uses a field this controller does not know, so it is left out of every machine it is on, and nothing of it changes there until the controller is updated.",
Explanation: name + " uses a field this controller does not know. Until the controller is updated, nothing of it changes on its machines, and what it adds to other modules and the ports opened for it stop. Its data is still backed up as this controller reads it, which may not be what its newer version asks.",
Needs: "update the controller, or register " + name + " again at a version this controller knows.",
Resolved: "the controller reads " + name + " again",
})
+10
View File
@@ -75,6 +75,9 @@ type signalFacts struct {
advisories []link.Advisory
lostConsumers map[string]bool
// deadLetters are how many messages DEAD_LETTERS holds per consumer, by `<stream>.<consumer>`
// (novox/hq issue 330): each consumer's max-deliveries condition is open while it holds any.
deadLetters map[string]int
advisoriesErr error
selfCheck selfCheckFacts
@@ -332,6 +335,9 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
}
f.advisories = link.Advisories.Since(now.Add(-advisoryQuiet))
f.lostConsumers, f.advisoriesErr = w.lostConsumers(ctx, f.advisories)
if f.advisoriesErr == nil && w.js != nil {
f.deadLetters, f.advisoriesErr = link.HeldDeadLetters(w.js.Context())
}
f.handActs, f.handActsErr = w.gatherHandActs(ctx, now)
f.facts.taken, _, f.facts.began, f.facts.err = exportedFacts.last()
return f
@@ -654,6 +660,8 @@ func watchTheMesh(ctx context.Context, open *stores, server *link.Server, bus li
return nil
}
conditionsFrom = keeper
// What a consumer gave up on is read and acted on by the verb in this process (novox/hq issue 330).
deadLettersOn.Store(&busHandles{conn: server.JetStream().Conn(), js: server.JetStream().Context()})
logf := func(format string, args ...any) { fmt.Printf(format+"\n", args...) }
stopHearing, err := server.HearAdvisories(logf)
if err != nil {
@@ -680,6 +688,8 @@ func watchTheMesh(ctx context.Context, open *stores, server *link.Server, bus li
return func() {
stop()
stopHearing()
// No longer serving: the verb answers that it does not read DEAD_LETTERS here (novox/hq issue 330).
deadLettersOn.Store(nil)
flushing, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
keeper.Close(flushing)
+1 -1
View File
@@ -35,4 +35,4 @@ require (
// committed. Every build (the build agent's `go build`, the Dockerfile) compiles from vendor/ and
// fetches nothing; go refuses to build when vendor/ and this file disagree, so a pin moved without
// `go mod vendor` fails loudly, at once, everywhere.
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261007162834-56e2ebec4bac
replace github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261008185244-76f3ca12b8b8
+6
View File
@@ -4,6 +4,12 @@ git.novox.be/novox/mesh-host v0.0.0-20261007120832-bdd44154ccac h1:KvnKtJ2rWeIE/
git.novox.be/novox/mesh-host v0.0.0-20261007120832-bdd44154ccac/go.mod h1:VlilMCRZ5yyNXg7SNigNBLr0Gt32jrGw5KSNq5JAVYs=
git.novox.be/novox/mesh-host v0.0.0-20261007162834-56e2ebec4bac h1:yLtFS0pDCCqIE9Zx8hgXEFG9fUWzf8L9WQoKV+Amk1E=
git.novox.be/novox/mesh-host v0.0.0-20261007162834-56e2ebec4bac/go.mod h1:VlilMCRZ5yyNXg7SNigNBLr0Gt32jrGw5KSNq5JAVYs=
git.novox.be/novox/mesh-host v0.0.0-20261008163010-8390fab5cb35 h1:3uag/9Tv4Y3ippY5Yr1rIIBh23Ur9RbhzOB4CLbKz0I=
git.novox.be/novox/mesh-host v0.0.0-20261008163010-8390fab5cb35/go.mod h1:VlilMCRZ5yyNXg7SNigNBLr0Gt32jrGw5KSNq5JAVYs=
git.novox.be/novox/mesh-host v0.0.0-20261008183646-5fc37b44a94e h1:+XxiuXJqWj7ZcGMB2b/WGPsC8zpmXgmwXnBvjw9C/CM=
git.novox.be/novox/mesh-host v0.0.0-20261008183646-5fc37b44a94e/go.mod h1:72ZATZjxMLaJfWdvlSDJrygIoBzCmKIjCDMhEXxVzTo=
git.novox.be/novox/mesh-host v0.0.0-20261008185244-76f3ca12b8b8 h1:T4Bu9ymmcNC0x57EnDkjfZlJ9dvEeK8BRuHZTqw+uuI=
git.novox.be/novox/mesh-host v0.0.0-20261008185244-76f3ca12b8b8/go.mod h1:72ZATZjxMLaJfWdvlSDJrygIoBzCmKIjCDMhEXxVzTo=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
+7 -3
View File
@@ -38,7 +38,8 @@ type Consumer struct {
Push bool
// AckWaitSeconds before an unacknowledged delivery is redelivered.
AckWaitSeconds int
// MaxDeliver before the message is dead-lettered; zero for the mesh's default.
// MaxDeliver is how often a message is handed over before the consumer gives it up; zero for no
// bound. What a consumer gives up is kept in DEAD_LETTERS by the controller (novox/hq issue 330).
MaxDeliver int
// MaxAckPending is how many deliveries the server lets stand unacknowledged at once; zero for
// the server's default, which is many. **One, for a consumer handled one at a time**
@@ -145,13 +146,16 @@ func ConsumerFor(p Principal) (Consumer, bool) {
return Consumer{}, false
}
sort.Strings(filters)
// And the events given up on and delivered again to this consumer alone (novox/hq issue 330).
filters = append(filters, AgainFilter(consumerDurable(p)))
return Consumer{
Name: consumerDurable(p),
Stream: consumerStream(p),
Filters: filters,
AckWaitSeconds: 30,
MaxDeliver: 5,
Why: "what " + p.Module + " declared it consumes; after max-deliver it dead-letters",
Why: "what " + p.Module + " declared it consumes; after max-deliver it gives an event up, and the " +
"controller keeps it in DEAD_LETTERS until a person delivers it again or drops it",
}, true
}
@@ -234,7 +238,7 @@ func HolderConsumerFor(node, module string, seat DeclaredSeat) (Consumer, bool)
//
// **No max-deliver, and a long ack wait.** A declaration is settled only after the node has applied
// it and reported, which is minutes on a machine pulling images; and a declaration the mesh cannot
// get a node to accept is not one to dead-letter, because the stream keeps only the newest per node
// get a node to accept is not one to give up on, because the stream keeps only the newest per node
// anyway — so there is exactly one message per node to redeliver, for as long as that node is away.
func NodeConsumer(node string) Consumer {
return Consumer{
+3 -2
View File
@@ -61,8 +61,9 @@ func TestAModuleGetsOneConsumerCarryingEveryFilter(t *testing.T) {
if !ok {
t.Fatal("a module that consumes got no consumer")
}
if len(c.Filters) != 2 {
t.Fatalf("expected both subjects as filters, got %v", c.Filters)
// Both, and its own share of what is delivered again (novox/hq issue 330).
if len(c.Filters) != 3 || c.Filters[2] != "mesh.again.one_audit.>" {
t.Fatalf("expected both subjects and its own again filter, got %v", c.Filters)
}
perms, _ := PermissionsFor(Principal{Kind: KindModule, Node: "one", Module: "audit",
Consumes: []string{"shop.order.placed"}, PasswordHash: "x"})
+9
View File
@@ -154,6 +154,15 @@ func (j *JetStream) EnsureStream(s Stream) error {
Description: s.Why,
}
want.AllowDirect = s.Direct
if s.MaxBytes > 0 {
want.MaxBytes = s.MaxBytes
}
if s.DiscardNew {
want.Discard = nats.DiscardNew
}
if s.DuplicatesSeconds > 0 {
want.Duplicates = time.Duration(s.DuplicatesSeconds) * time.Second
}
if s.Retention == RetentionLastPerSubject {
// Last-per-subject is a limits stream with one message kept per subject, not a
// retention policy of its own — the state shape, spelled the way the server spells it.
+5
View File
@@ -412,6 +412,11 @@ func PermissionsFor(p Principal) (Permissions, error) {
// both in the mesh's own account; the controller says each as a condition in the mesh's words.
// Named, not `$JS.EVENT.>`: the other advisories are every API call the mesh makes.
sub = append(sub, BusAdvisories...)
// **And what a consumer gave up on, kept and delivered again** (novox/hq issue 330): the notice
// acknowledged once the message is copied, the copy kept, and an event delivered again to the one
// consumer that gave it up. An ask to a seat is not delivered again, so no seat's queue is granted.
pub = append(pub, "$JS.ACK."+DeadLetterNoticesStream+"."+ControllerName+".>", deadLetterPrefix+">",
againPrefix+">")
case KindPerson:
// Tools, and nothing else. Every subject a person may publish is a tool call; a person
+134 -6
View File
@@ -3,11 +3,13 @@ package broker
import (
"fmt"
"sort"
"strings"
)
// The mesh's own streams.
//
// **These four and no more** (novox/hq ADR 0116 task 1.4, as revised by ADR 0118). An earlier
// **These and no more** (novox/hq ADR 0116 task 1.4, as revised by ADR 0118; the two that keep what a
// consumer gave up on added for issue 330). An earlier
// reading had the controller create *every* stream at genesis, from a fixed set. That is only the
// mesh's own half: a seat's streams are created when the module declaring it is registered, and a
// module's durable consumers when it is assigned — neither of which has happened at genesis. What
@@ -50,11 +52,80 @@ type Stream struct {
// Direct lets a client read a subject's last message without a consumer, which is how a
// runtime reads its own membership with no JetStream API beyond one request (ADR 0160).
Direct bool
// MaxBytes bounds the stream's size, zero for unbounded. With DiscardNew a full stream refuses
// what comes next rather than dropping what it holds: the publisher is told, and says so.
MaxBytes int64
DiscardNew bool
// DuplicatesSeconds is the window in which a message id published twice is kept once; zero for the
// server's default (two minutes).
DuplicatesSeconds int
}
// AssignmentsStream holds every assignment's membership, the newest per subject.
const AssignmentsStream = "ASSIGNMENTS"
// What a durable consumer gave up on is kept (novox/hq issue 330, design 25 §3).
//
// **The server says it and keeps it; the controller copies it.** A consumer that handed a message over
// as often as it may stops offering it and publishes `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES` with
// the stream and the message's sequence. DeadLetterNoticesStream captures those advisories as the
// server publishes them, so one said while no controller listens is still there when one starts. The
// controller's consumer on it fetches the given-up message by its sequence, while the source stream
// still holds it, and keeps a copy in DeadLettersStream under DeadLetterSubject, with the consumer,
// the subject, how often it was handed over and when it was given up. It stays there until a person
// delivers it again or drops it, with why; a condition is open for as long as it does.
const (
DeadLetterNoticesStream = "DEAD_LETTER_NOTICES"
DeadLettersStream = "DEAD_LETTERS"
// MaxDeliveriesAdvisories is the subject the server says a given-up message on.
MaxDeliveriesAdvisories = "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>"
deadLetterPrefix = "mesh.events.dead."
againPrefix = "mesh.again."
// DeadLettersBytes bounds the kept copies. Full, the stream refuses the next copy, which is said
// as the consumer's condition; it never drops one it holds.
DeadLettersBytes = 256 << 20
)
// DeadLetterSubject is where a message one consumer gave up on is kept: `mesh.events.dead.<stream>.<consumer>`.
func DeadLetterSubject(stream, consumer string) string {
return deadLetterPrefix + stream + "." + consumer
}
// DeadLetterOf is the stream and consumer a kept message's subject names; false for any other subject.
func DeadLetterOf(subject string) (stream, consumer string, ok bool) {
rest, found := strings.CutPrefix(subject, deadLetterPrefix)
if !found {
return "", "", false
}
stream, consumer, ok = strings.Cut(rest, ".")
return stream, consumer, ok && stream != "" && consumer != "" && !strings.Contains(consumer, ".")
}
// AgainSubject is where an event given up on is delivered again to the one consumer that gave it up,
// and to nobody else: `mesh.again.<consumer>.` and the original subject without its `mesh.`. Every
// consumer on EVENTS filters its own (AgainFilter), and a module's runtime reads the event's key from
// the tokens around `.event.`, so the handler sees the same key it saw the first time.
func AgainSubject(consumer, original string) string {
return againPrefix + consumer + "." + strings.TrimPrefix(original, "mesh.")
}
// AgainFilter is the one consumer's share of the subjects events are delivered again on.
func AgainFilter(consumer string) string { return againPrefix + consumer + ".>" }
// OriginalOfAgain is the subject an event delivered again was first published on; false for a subject
// that is not one delivered again.
func OriginalOfAgain(subject string) (string, bool) {
rest, found := strings.CutPrefix(subject, againPrefix)
if !found {
return "", false
}
_, original, ok := strings.Cut(rest, ".")
if !ok || original == "" {
return "", false
}
return "mesh." + original, true
}
// MeshStreams is the foundation set, in the order a person reads it.
//
// **CONTROL names its subjects rather than taking `mesh.control.>`**, because heartbeats live
@@ -97,7 +168,9 @@ func MeshStreams() []Stream {
// A seat's own events ride here too: they are 1:many like any event, and the
// `event` token keeps them clear of both the seat's work queue (`accept`) and its
// tools (`tool`), which must not be persisted.
Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>"},
// And an event given up on, delivered again to the one consumer that gave it up
// (novox/hq issue 330): under `mesh.again.<consumer>.`, which only that consumer filters.
Subjects: []string{"mesh.mod.*.event.>", "mesh.seat.*.event.>", againPrefix + ">"},
Retention: RetentionLimits,
MaxAge: 7 * 24 * 60 * 60,
MaxMsgsPerSubject: 10000,
@@ -105,6 +178,26 @@ func MeshStreams() []Stream {
"excluded by the event token; per-subject caps keep a noisy emitter from " +
"evicting a quiet one without splitting the stream",
},
{
Name: DeadLetterNoticesStream,
Subjects: []string{MaxDeliveriesAdvisories},
Retention: RetentionWorkQueue,
MaxAge: 7 * 24 * 60 * 60,
Why: "the server's word that a consumer gave up on a message, kept until the controller has " +
"copied the message into DEAD_LETTERS (novox/hq issue 330); a week, the longest the source " +
"streams keep what they are about",
},
{
Name: DeadLettersStream,
Subjects: []string{deadLetterPrefix + ">"},
Retention: RetentionLimits,
MaxBytes: DeadLettersBytes,
DiscardNew: true,
DuplicatesSeconds: 24 * 60 * 60,
Why: "every message a consumer gave up on, with its consumer, subject, deliveries and when, kept " +
"until a person delivers it again or drops it with why (novox/hq issue 330); no age, and full " +
"it refuses the next copy rather than drop one it holds",
},
}
}
@@ -298,7 +391,7 @@ const EventsStream = "EVENTS"
//
// **Unlimited redelivery on CONTROL, deliberately.** The store window's bound is the controller's,
// not the server's (window.go): a message is held with a nak-and-delay until the controller either
// takes it or gives up and says so. A max-deliver here would dead-letter a push that was being
// takes it or gives up and says so. A max-deliver here would give up on a push that was being
// held through a store restart — the exact message the stream exists to protect — some minutes
// before the controller had finished deciding about it.
func MeshConsumers() []Consumer {
@@ -314,7 +407,7 @@ func MeshConsumers() []Consumer {
{
Name: ControllerName,
Stream: "EVENTS",
Filters: ControllerFollows,
Filters: append(append([]string(nil), ControllerFollows...), AgainFilter(ControllerName)),
Push: true,
AckWaitSeconds: 30,
MaxDeliver: 5,
@@ -331,12 +424,47 @@ func MeshConsumers() []Consumer {
Resettable: "what it drops is caught up: merges by the catch-up pass (issue 266), build outcomes " +
"from the build records (issue 214), a provider's failing word said again (ADR 0224)",
Why: "the events the mesh's own controller reacts to, one at a time; after " +
"max-deliver it dead-letters, because an announcement it cannot act on will not " +
"become actionable",
"max-deliver it gives the event up, and the controller keeps it in DEAD_LETTERS until " +
"a person delivers it again or drops it",
},
// What the server said a consumer gave up on (novox/hq issue 330): copied into DEAD_LETTERS and
// acknowledged. No max-deliver: a notice the controller could not copy is offered again, and said.
{
Name: ControllerName,
Stream: DeadLetterNoticesStream,
Push: true,
AckWaitSeconds: 30,
Why: "the controller copies each message a consumer gave up on into DEAD_LETTERS; no max-deliver, " +
"because a notice it gave up on would lose the message it is about",
},
}
}
// NoticesConsumer is the controller's consumer on DEAD_LETTER_NOTICES (novox/hq issue 330).
func NoticesConsumer() Consumer {
for _, c := range MeshConsumers() {
if c.Stream == DeadLetterNoticesStream {
return c
}
}
panic("the mesh's consumers carry none on " + DeadLetterNoticesStream)
}
// AssertServingConsumers are the controller's own consumers its serving cannot go without: all but the
// one on DEAD_LETTER_NOTICES, which the keeper of dead letters asserts and retries by itself, so a fault
// there never stops the controller serving (novox/hq issue 330).
func AssertServingConsumers(e Ensurer) error {
for _, c := range MeshConsumers() {
if c.Stream == DeadLetterNoticesStream {
continue
}
if err := e.EnsureConsumer(c); err != nil {
return fmt.Errorf("asserting consumer %s on %s: %w", c.Name, c.Stream, err)
}
}
return nil
}
// Ensurer is the part of a JetStream connection consumer assertion needs, narrow for the reason
// Asserter is.
type Ensurer interface {
+4
View File
@@ -127,6 +127,10 @@ func TestEachStreamCarriesTheRetentionItsShapeNeeds(t *testing.T) {
"NODES": RetentionLastPerSubject,
"EVENTS": RetentionLimits,
"ASSIGNMENTS": RetentionLastPerSubject,
// What a consumer gave up on (novox/hq issue 330): the server's notice taken once, the message
// kept until somebody acts.
"DEAD_LETTER_NOTICES": RetentionWorkQueue,
"DEAD_LETTERS": RetentionLimits,
}
got := map[string]Retention{}
for _, s := range MeshStreams() {
+1 -1
View File
@@ -24,7 +24,7 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.assignment.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built"] }
allow_responses: { max: 1, ttl: "1m" }
} }
+5
View File
@@ -133,6 +133,11 @@ var WritersTable = []WriterRow{
Others: "—"},
{State: "the facts snapshot", Writer: "controller", KeptIn: "the artifact store, facts/latest",
Others: "the build seat reads"},
// What a consumer gave up on (novox/hq issue 330): copied by the controller from the server's notice,
// and delivered again or dropped only through its verb, with why.
{State: "a message a consumer gave up on", Writer: "controller", KeptIn: "the bus, the stream " + DeadLettersStream,
Others: "read, delivered again or dropped through the controller's dead-letters verb",
Subjects: []string{deadLetterPrefix + ">", againPrefix + ">"}, Writes: isController},
}
// CheckWriters refuses a grant that lets a principal publish on a subject the writers table gives
+1
View File
@@ -30,6 +30,7 @@ var designRows = []string{
"a provider's standing",
"the operator-channel's open messages",
"the facts snapshot",
"a message a consumer gave up on",
}
func TestTheWritersTableIsTheDesigns(t *testing.T) {
+113
View File
@@ -0,0 +1,113 @@
package catalogue
import (
"testing"
)
// The account agents run as (novox/hq ADR 0266): a module names it as a machine fact — the agent account
// where the node names one, the operator's otherwise — and asks the node-engine to judge it never to become
// root only where it is the agents' own.
func TestTheAgentAccountFactFallsBackToTheOperatorAndIsNeverRootOnlyWhenItsOwn(t *testing.T) {
facts := machineFacts(Resolution{Node: "anchor", Account: "ops"}, nil, "")
if facts["agent-account"] != "ops" || facts["agent-home"] != "/home/ops" || facts["agent-root"] != "" {
t.Errorf("with no agent account named, agents run as the operator: %v", facts)
}
facts = machineFacts(Resolution{Node: "anchor", Account: "ops", AccountHome: "/srv/ops"}, nil, "")
if facts["agent-home"] != "/srv/ops" {
t.Errorf("the operator's stated home is the agent's home when they are one account: %v", facts)
}
facts = machineFacts(Resolution{Node: "anchor", Account: "ops", AgentAccount: "agent"}, nil, "")
if facts["agent-account"] != "agent" || facts["agent-home"] != "/home/agent" || facts["agent-root"] != RootNever {
t.Errorf("a named agent account is the agents', never root: %v", facts)
}
if facts["account"] != "ops" {
t.Errorf("the operator account is still the operator's: %v", facts)
}
facts = machineFacts(Resolution{Node: "anchor", AgentAccount: "agent", AgentAccountHome: "/var/lib/agent"}, nil, "")
if facts["agent-home"] != "/var/lib/agent" || facts["agent-root"] != RootNever {
t.Errorf("an agent account with a stated home on a machine with no operator: %v", facts)
}
if _, has := machineFacts(Resolution{Node: "anchor"}, nil, "")["agent-account"]; has {
t.Error("a machine with no account at all names an agent account")
}
}
// The agent's module, in the shape the catalogue's declares it: the account, never root where it is its
// own; its directory under that home, owned by it.
const agentModule = `{"module": "agent", "version": "1", "resources": [
{"id": "account", "type": "user", "name": "${machine:agent-account}", "root": "${machine:agent-root}"},
{"id": "home", "type": "directory", "path": "${machine:agent-home}/.agent", "mode": "0700",
"owner": "${machine:agent-account}"}
]}`
func TestTheAgentAccountIsDeclaredNeverRootOnlyToAnEngineThatJudgesIt(t *testing.T) {
m, err := ParseManifest([]byte(agentModule))
if err != nil {
t.Fatal(err)
}
compose := func(r Resolution, with Rendering) (user, home map[string]any) {
t.Helper()
r.Node, r.Modules = "anchor", []Manifest{m}
out, err := r.Declaration(with)
if err != nil {
t.Fatal(err)
}
return fileNamed(out, "agent.account"), fileNamed(out, "agent.home")
}
user, home := compose(Resolution{Account: "ops", AgentAccount: "agent"}, Rendering{JudgesRoot: true})
if user["name"] != "agent" || user[RootField] != RootNever {
t.Errorf("an engine that judges root is sent the agent account never to become root: %v", user)
}
if home["path"] != "/home/agent/.agent" || home["owner"] != "agent" {
t.Errorf("the agent's directory is under its own home, its own: %v", home)
}
user, _ = compose(Resolution{Account: "ops", AgentAccount: "agent"}, Rendering{})
if _, sent := user[RootField]; sent || user["name"] != "agent" {
t.Errorf("an older engine, which parses strictly, is sent root: %v", user)
}
user, home = compose(Resolution{Account: "ops"}, Rendering{JudgesRoot: true})
if _, sent := user[RootField]; sent || user["name"] != "ops" {
t.Errorf("where agents run as the operator, root asserts nothing and is not sent: %v", user)
}
if home["path"] != "/home/ops/.agent" || home["owner"] != "ops" {
t.Errorf("with no agent account, the agent's directory is the operator's: %v", home)
}
}
func TestTheRuntimeIsToldTheAgentAccount(t *testing.T) {
with := Rendering{ArtifactStore: "anchor.internal:5101",
Needed: map[string]map[string]string{RuntimeModule: {"broker": "sealed-credential"}}}
envOf := func(r Resolution) map[string]string {
t.Helper()
r.Node, r.Modules = "anchor", []Manifest{aToolsModule(t, "nftables", "tools/index.js"), theRuntime(t)}
out, err := r.Declaration(with)
if err != nil {
t.Fatal(err)
}
process := fileNamed(out, RuntimeModule+"."+RuntimeProcessID())
if process == nil {
t.Fatal("no runtime process was composed")
}
return process["env"].(map[string]string)
}
env := envOf(Resolution{Account: "ops", AgentAccount: "agent"})
if env[RuntimeAgentAccount] != "agent" || env[RuntimeAgentHome] != "/home/agent" || env[RuntimeOperatorAccount] != "ops" {
t.Errorf("the runtime is not told whom agents run as: %v", env)
}
env = envOf(Resolution{Account: "ops"})
if env[RuntimeAgentAccount] != "ops" || env[RuntimeAgentHome] != "/home/ops" {
t.Errorf("with no agent account, agents run as the operator: %v", env)
}
env = envOf(Resolution{})
if _, set := env[RuntimeAgentAccount]; set {
t.Errorf("a machine with no account names an agent account: %v", env)
}
if problems := bundleEnvProblems("x", Artifact{Name: "b", Kind: ArtifactBundle, Loads: []string{"x"},
Env: map[string]string{RuntimeAgentAccount: "me"}}); len(problems) == 0 {
t.Error("a bundle may tell the runtime whom agents run as")
}
}
+1 -1
View File
@@ -399,7 +399,7 @@ func versionOf(digest string) string {
// telling the runtime what it is, which is the mesh's to say (novox/hq ADR 0192).
var bundleEnvWords = map[string]bool{
RuntimeToolModules: true, RuntimeBrokerFile: true, RuntimeOperatorAccount: true,
RuntimeOperatorHome: true, RuntimeToolEnv: true,
RuntimeOperatorHome: true, RuntimeToolEnv: true, RuntimeAgentAccount: true, RuntimeAgentHome: true,
}
// bundleEnvProblems says what is wrong with what a bundle says it is given (novox/hq ADR 0192):
+35 -1
View File
@@ -241,6 +241,31 @@ type Rendering struct {
// refuses a field it does not know, whole — so to it the field is not sent, and what it runs is
// judged by liveness alone.
ReadsHealth bool
// JudgesRoot says this machine's node-engine judges a user's declared `root` (novox/hq ADR 0266: its
// statement's contract is link.RootContract or later). To an older, strict engine the field is not
// sent, and the account it names is not judged — which the self-check says, as not judged.
JudgesRoot bool
}
// RootField is a user resource's field saying the account must never become root without a person
// (novox/hq ADR 0266).
const RootField = "root"
// rootInto composes a user's `root` for the node-engine: taken away when it asserts nothing (empty — a
// machine where agents run as the operator) or when the engine is older than the field and parses
// strictly; kept as "never" otherwise.
func rootInto(resource map[string]any, with Rendering) {
if resource["type"] != "user" {
return
}
value, has := resource[RootField]
if !has {
return
}
if s, _ := value.(string); s == "" || !with.JudgesRoot {
delete(resource, RootField)
}
}
// machinePort is where a module's port lives on this machine, or the port itself when the mesh has
@@ -369,11 +394,16 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// Before placing, because a placement is a setting too.
left := r.LeftOut(with.Settings, with.Adopted)
kept := make([]Manifest, 0, len(r.Modules))
var stillBackedUp []Manifest
for _, m := range r.Modules {
if why, isLeft := left[m.Module]; isLeft {
if leftOut != nil {
leftOut[m.Module] = why
}
// **Its data is still copied** (novox/hq ADR 0262): a module left out runs nothing new, and
// the data it already holds on the machine is the reason to keep copying it. Only its data,
// as the backup holder's lines are derived from it, and the directories they name.
stillBackedUp = append(stillBackedUp, backupView(m))
continue
}
kept = append(kept, m)
@@ -975,11 +1005,15 @@ func (r Resolution) compose(with Rendering, owner map[string]string,
// How it is ready, in the node-engine's words: its endpoint as the port this machine
// published it on — or not sent at all to an engine older than the field (ADR 0240).
healthInto(copied, m, with)
// And a user's `root` (novox/hq ADR 0266): sent only when it asserts something, to an engine
// that judges it.
rootInto(copied, with)
// The account's environment and every module's shell code, where this module holds the
// seat that places them (novox/hq ADR 0203, ADR 0204). Gathered from every module on
// the node, as the jails are, and **last of every placeholder pass**: shell code is a
// shell's own syntax, full of `${…}` no pass above should ever be shown.
if err := contributionsInto(copied, m, r.Modules, thisMachine, with, r.Capabilities, unplaced); err != nil {
contributing := append(append([]Manifest(nil), r.Modules...), stillBackedUp...)
if err := contributionsInto(copied, m, contributing, thisMachine, with, r.Capabilities, unplaced); err != nil {
return nil, err
}
copied["id"] = m.Module + "." + fmt.Sprint(resource["id"])
+42 -2
View File
@@ -78,9 +78,47 @@ func machineFacts(r Resolution, names map[string]string, meshRange string) map[s
out["account"] = r.Account
out["account-home"] = accountHomeOf(r.Account, r.AccountHome)
}
// The account agents run as here, and whether it must never become root (novox/hq ADR 0266). The agent
// account where the node names one; the operator account otherwise, so a module writing the agent's
// home names one fact on every machine. `agent-root` is "never" only for an account of the agents' own:
// the user resource naming it then asks the node-engine to judge it, and on a machine where agents run
// as the operator it is empty, asserting nothing — the operator's account may become root there.
if agent, home := r.agentAccount(); agent != "" {
out["agent-account"] = agent
out["agent-home"] = home
out["agent-root"] = ""
if r.AgentAccount != "" {
out["agent-root"] = RootNever
}
}
return out
}
// RootNever is what a user resource's `root` says of an account that must never become root without a
// person (novox/hq ADR 0266); the node-engine judges it.
const RootNever = "never"
// agentAccount is the account agents run as on this machine and its home: the agent account when the node
// names one (novox/hq ADR 0266), else the operator account; empty when neither is known.
func (r Resolution) agentAccount() (string, string) {
if r.AgentAccount != "" {
return r.AgentAccount, agentHomeOf(r.AgentAccount, r.AgentAccountHome)
}
if r.Account != "" {
return r.Account, accountHomeOf(r.Account, r.AccountHome)
}
return "", ""
}
// agentHomeOf is where the agent account's home is: what was stored, or /home/<account>. Never /root: the
// agent account is never root.
func agentHomeOf(account, home string) string {
if home != "" {
return home
}
return "/home/" + account
}
// accountHomeOf is where an account's home is: what was stored, or the derived default — /root for
// root, /home/<account> otherwise. The one place the default is written, so a fact and the store
// cannot disagree about it.
@@ -105,8 +143,10 @@ func machineInto(resource map[string]any, facts map[string]string, module string
// (novox/hq to-be 29), the same reason its content names ${machine:address}. And the name a
// `user` shape sets the login shell of, and the user a user-scoped unit or a process runs as:
// the shell module makes the operator's account its holder's login shell, and the desktop's
// watchers run as that account (novox/hq ADR 0176, ADR 0177) — neither can name the person.
for _, field := range []string{"path", "owner", "content", "name", "user"} {
// watchers run as that account (novox/hq ADR 0176, ADR 0177) — neither can name the person. And a
// user's `root`: the agent's module declares the account agents run as with ${machine:agent-root},
// "never" only where that account is the agents' own (novox/hq ADR 0266).
for _, field := range []string{"path", "owner", "content", "name", "user", "root"} {
s, ok := resource[field].(string)
if !ok {
continue
+16 -6
View File
@@ -471,6 +471,11 @@ type Manifest struct {
// machine's declaration by name until the controller is updated (LeftOut).
unknown string
// bestEffort marks the view of a left-out module that only its backup lines are made from
// (backupView, novox/hq ADR 0262): a line of it that cannot be placed is said in the plan's list of
// what could not be placed, never an error that would cost the whole machine its declaration.
bestEffort bool
// Data is every kind of data this module keeps — its own, by directory, and what it keeps for
// its consumers, by provision — each with a class the mesh protects and watches it by (novox/hq
// ADR 0233). One list: the backup holder's lines, the bindings that do not move, what an
@@ -1283,16 +1288,21 @@ func (m *Manifest) UnmarshalJSON(raw []byte) error {
if asUnknownField(err) == nil {
return err
}
// Read without it where the key is at the top; where it is inside a block, the block's own
// decoder refuses it again, and the manifest keeps its name and version alone. Either way the
// module is left out of every declaration by name (LeftOut), so nothing runs on a part-read
// manifest.
// Read without it, at whatever depth it is (prunedFields): the module is left out of every
// declaration by name (LeftOut), and still provides what it provides, and still has its data
// copied, so nothing that requires it is refused and nothing it holds goes uncopied.
unknown = err.Error()
fields = manifestFields{}
if json.Unmarshal(rest, &fields) != nil {
var pruned []string
var perr error
if fields, pruned, perr = prunedFields(keys); perr != nil {
// Not read past: the manifest keeps its name and version alone. It is left out and raised
// all the same, and one manifest never fails the whole catalogue.
fields = manifestFields{}
_ = json.Unmarshal(keys["module"], &fields.Module)
_ = json.Unmarshal(keys["version"], &fields.Version)
unknown += " (read no further: " + perr.Error() + ")"
} else {
unknown += " (read without " + strings.Join(pruned, ", ") + ")"
}
}
*m = Manifest(fields)
+11 -1
View File
@@ -32,6 +32,11 @@ type Node struct {
// to-be 29). What a home-scoped file is owned by and what ${machine:account} resolves to.
Account string
AccountHome string
// AgentAccount is the login agents run as here when it is not the operator's, AgentAccountHome its
// home when not derived (novox/hq ADR 0266). What ${machine:agent-account} resolves to; empty means
// agents run as the operator account.
AgentAccount string
AgentAccountHome string
}
// World is what the rest of the mesh already has.
@@ -134,6 +139,10 @@ type Resolution struct {
// here without a store lookup.
Account string
AccountHome string
// AgentAccount and AgentAccountHome are the account agents run as here when it is not the
// operator's, and its home (novox/hq ADR 0266); empty when agents run as the operator account.
AgentAccount string
AgentAccountHome string
// Capabilities are the machine's, as its profile reported them, carried from the node so a
// contribution placed only where the machine has something (`if-capability`, novox/hq ADR 0255)
// is decided here without a store lookup.
@@ -703,7 +712,8 @@ func Resolve(catalogue map[string]Manifest, assigned []string, node Node, world
}
resolution := Resolution{Node: node.Name, At: node.At, PublicDomain: node.PublicDomain,
Account: node.Account, AccountHome: node.AccountHome, Capabilities: node.Capabilities,
Account: node.Account, AccountHome: node.AccountHome, AgentAccount: node.AgentAccount,
AgentAccountHome: node.AgentAccountHome, Capabilities: node.Capabilities,
Because: because, Needs: needs, Unhostable: unhostable, Kept: kept}
for _, n := range providersFirst(order, catalogue) {
resolution.Modules = append(resolution.Modules, catalogue[n])
+9
View File
@@ -83,6 +83,11 @@ const (
RuntimeBrokerFile = "MESH_BROKER_FILE"
RuntimeOperatorAccount = "MESH_OPERATOR_ACCOUNT"
RuntimeOperatorHome = "MESH_OPERATOR_HOME"
// RuntimeAgentAccount and RuntimeAgentHome are the account agents run as on the machine and its home
// (novox/hq ADR 0266): the agent account where the node names one, the operator account otherwise.
// The agent's module writes the agent's home from them; absent where neither account is known.
RuntimeAgentAccount = "MESH_AGENT_ACCOUNT"
RuntimeAgentHome = "MESH_AGENT_HOME"
// RuntimeToolEnv is every served module's composed environment, as JSON (novox/hq ADR 0192):
// {"<module>": {"<word>": "<value>"}}. The runtime takes it at start, removes it from its own
// environment and hands each module's words to that module's bundles alone. In the unit, so a
@@ -215,6 +220,10 @@ func (r Resolution) runtimeProcess(with Rendering) (map[string]any, error) {
env[RuntimeOperatorHome] = accountHomeOf(r.Account, r.AccountHome)
process["user"] = r.Account
}
if agent, home := r.agentAccount(); agent != "" {
env[RuntimeAgentAccount] = agent
env[RuntimeAgentHome] = home
}
// Routed through the artifact store as this network reaches it now, like everything the mesh
// built; refused with the same words when there is no store to route through.
if err := artifactsInto(process, RuntimeModule, with); err != nil {
+33 -11
View File
@@ -212,9 +212,20 @@ func seatContributions(modules []Manifest, holder Manifest, placeholder string,
return shapedContributions(modules, holder, facts["name"], s, r, where, caps)
}
var failed error
var unplacedLines []string
var b strings.Builder
for _, m := range inModuleOrder(modules) {
named := false
// A left-out module's lines are best effort (novox/hq ADR 0262): what cannot be placed is said,
// and the machine is declared without it. A placement setting that does not read is not
// replaced by the definition's own path, which may not be where the data is.
if m.bestEffort && r.Dirs {
if _, err := Places(m, with.Settings[m.Module]); err != nil {
unplacedLines = append(unplacedLines, fmt.Sprintf("%s's %s for %s, kept while it is left out, "+
"is not placed: its placement setting does not read (%v)", m.Module, kind, s.Name, err))
continue
}
}
for _, c := range m.allContributions() {
if c.Kind != kind || !capable(c, caps) {
continue
@@ -222,15 +233,12 @@ func seatContributions(modules []Manifest, holder Manifest, placeholder string,
if cs, known := SeatNamed(c.Seat); !known || cs.Name != s.Name {
continue
}
if !named {
fmt.Fprintf(&b, "%s %s\n", r.Comment, m.Module)
named = true
}
content := c.Content
var lineFailed error
if r.Dirs {
filled, err := dirFill(content, dirsFor(m, with), m.Module)
if err != nil && failed == nil {
failed = err
if err != nil && lineFailed == nil {
lineFailed = err
}
// An operator's path the module was given, as an item of data on it (novox/hq ADR 0233).
if accessRef.MatchString(filled) {
@@ -238,15 +246,15 @@ func seatContributions(modules []Manifest, holder Manifest, placeholder string,
if err == nil {
filled, err = accessFill(filled, byID, m.Module)
}
if err != nil && failed == nil {
failed = err
if err != nil && lineFailed == nil {
lineFailed = err
}
}
for _, key := range machineUsed(filled) {
value, has := facts[key]
if !has {
if failed == nil {
failed = fmt.Errorf("%s's %s for %s says ${machine:%s}, and this machine says %s",
if lineFailed == nil {
lineFailed = fmt.Errorf("%s's %s for %s says ${machine:%s}, and this machine says %s",
m.Module, kind, s.Name, key, orNothing(namesOfFacts(facts)))
}
continue
@@ -255,11 +263,25 @@ func seatContributions(modules []Manifest, holder Manifest, placeholder string,
}
content = filled
}
if lineFailed != nil {
if m.bestEffort {
unplacedLines = append(unplacedLines, fmt.Sprintf("%s's %s for %s, kept while it is left "+
"out, is not placed: %v", m.Module, kind, s.Name, lineFailed))
continue
}
if failed == nil {
failed = lineFailed
}
}
if !named {
fmt.Fprintf(&b, "%s %s\n", r.Comment, m.Module)
named = true
}
b.WriteString(content)
if !strings.HasSuffix(content, "\n") {
b.WriteString("\n")
}
}
}
return b.String(), nil, failed
return b.String(), unplacedLines, failed
}
+5
View File
@@ -304,6 +304,11 @@ var defaultSeats = append([]Seat{
// Where it is held, its holder writes the resolver file and the uplink's holder steps back from it
// (node_resolver.go). It knows nothing of any VPN: its verbs route domains to servers over a link.
{Name: ResolverSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0247", Serves: resolverVerbs()},
// A machine's shares and the shares it mounts (novox/hq ADR 0263): the holder of node-nfs-server
// exports a machine's folders to the private network and provides each as `nfs-share`; the holder of
// node-mounts writes a mount and an automount unit per share on a machine that asks (shares.go).
{Name: NFSServerSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0263", Serves: nfsServerVerbs()},
{Name: MountsSeat, Scope: ScopeNode, Decision: "novox/hq ADR 0263", Serves: mountsVerbs()},
},
// The graphical session's roles (novox/hq ADR 0208), last because they are a workstation's.
graphicalSessionSeats()...)
+3 -3
View File
@@ -46,7 +46,7 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
delivered[s.Delivers] = s.Name
}
}
// Thirty-eight with node-resolver (novox/hq ADR 0247); thirty-seven with mesh-delivery (novox/hq ADR 0239); thirty-six since node-resolver-config retired
// Forty with node-nfs-server and node-mounts (novox/hq ADR 0263); thirty-eight with node-resolver (novox/hq ADR 0247); thirty-seven with mesh-delivery (novox/hq ADR 0239); thirty-six since node-resolver-config retired
// into node-uplink (novox/hq ADR 0223); thirty-seven
// since the retired node-dns-resolver went (novox/hq ADR 0220); thirty-eight with
// node-backup (novox/hq ADR 0214); thirty-seven with node-message-bus (novox/hq ADR 0215);
@@ -57,8 +57,8 @@ func TestTheSeatsAreAClosedSetAndEachNamesItsDecision(t *testing.T) {
// node-container-runtime (ADR 0207); nineteen with node-environment and node-login-shell (ADR 0203,
// ADR 0204); seventeen with node-build-agent (ADR 0190). One fewer once the retired
// mesh-build-machine row goes, when no registered manifest claims it.
if len(Seats()) != 38 {
t.Errorf("the mesh defines %d seats rather than 38; the set is closed, so a change here is "+
if len(Seats()) != 40 {
t.Errorf("the mesh defines %d seats rather than 40; the set is closed, so a change here is "+
"a decision (novox/hq ADR 0110): %s", len(Seats()), seatNames())
}
}
+23 -6
View File
@@ -57,20 +57,27 @@ var operatorsOwn = map[string]bool{
"identity": true, "login": true, "user": true, "username": true, "account": true, "owner": true,
"uid": true, "gid": true, "puid": true, "pgid": true,
"password": true, "pass": true, "passwd": true, "passphrase": true, "secret": true, "token": true,
"key": true, "apikey": true, "bearer": true, "cert": true, "credential": true,
"key": true, "apikey": true, "bearer": true, "cert": true, "certificate": true, "credential": true,
"nameserver": true, "gateway": true, "subnet": true, "sender": true, "recipient": true, "contact": true,
"trusted": true, "whitelist": true, "peer": true, "bind": true, "listen": true, "upstream": true,
"proxy": true, "admin": true, "mac": true,
}
// operatorsCompounds are names of two words that are the operator's though neither word alone says so
// at the end of a key: a client's identifier, and a name the world knows a site or server by.
var operatorsCompounds = map[string]bool{
"client-id": true, "site-name": true, "server-name": true, "public-name": true, "smtp-relay": true,
"host-name": true, "user-name": true, "domain-name": true, "dns-server": true,
// Whom a rule lets in or keeps out: a list of addresses or networks.
"allow-from": true, "deny-from": true, "allow-list": true,
}
// aboutAnAmount are first words that make a key about how many or whether, never about whom:
// `max-tokens` is a number, `show-hostname` a switch.
// `max-tokens` is a number, `show-hostname` a switch. Not `allow` or `use`: `allow-from` and
// `use-host` name whom.
var aboutAnAmount = map[string]bool{
"max": true, "min": true, "num": true, "count": true, "show": true, "hide": true, "enable": true,
"disable": true, "use": true, "allow": true,
"disable": true,
}
// operatorsWord is what in a key's name says its value is the operator's, or "". A key is about its
@@ -82,13 +89,23 @@ func operatorsWord(key string) string {
return ""
}
for i, w := range words {
if !operatorsOwn[w] && strings.HasSuffix(w, "s") && operatorsOwn[strings.TrimSuffix(w, "s")] {
switch {
case operatorsOwn[w]:
case strings.HasSuffix(w, "ies") && operatorsOwn[strings.TrimSuffix(w, "ies")+"y"]:
words[i] = strings.TrimSuffix(w, "ies") + "y"
case strings.HasSuffix(w, "s") && operatorsOwn[strings.TrimSuffix(w, "s")]:
words[i] = strings.TrimSuffix(w, "s")
}
}
if n := len(words); n > 1 {
if pair := words[n-2] + "-" + words[n-1]; operatorsCompounds[pair] {
return pair
// Where a secret or an identity is kept is the operator's too: `password-file`, `token-path`.
if (words[n-1] == "file" || words[n-1] == "path") && operatorsOwn[words[n-2]] {
return words[n-2] + "-" + words[n-1]
}
for _, last := range []string{words[n-1], strings.TrimSuffix(words[n-1], "s")} {
if pair := words[n-2] + "-" + last; operatorsCompounds[pair] {
return pair
}
}
}
if last := words[len(words)-1]; operatorsOwn[last] {
+118 -2
View File
@@ -223,7 +223,10 @@ func TestANodeCalledDefaultIsANodesLayer(t *testing.T) {
func TestAKeyIsTheOperatorsByWhatItIsAbout(t *testing.T) {
for _, key := range []string{"max-tokens", "show-hostname", "ghost-opacity", "users-per-page", "mailbox-size",
"font-size", "width", "keyboard-delay", "ipv6-preferred", "client-width", "user-agent", "url-timeout",
"site-title", "cert-renewal-days", "name", "font-name"} {
"site-title", "cert-renewal-days", "name", "font-name", "allow-resize", "use-gpu", "disable-sender-check",
"max-recipients", "gateway-timeout", "sender-delay", "upstream-resolvers", "mirror-countries",
"pool-region", "proxy-timeout", "listen-backlog", "peer-keepalive", "admin-theme", "log-file",
"cache-path"} {
if w := operatorsWord(key); w != "" {
t.Errorf("%s read as the operator's (%s)", key, w)
}
@@ -235,7 +238,17 @@ func TestAKeyIsTheOperatorsByWhatItIsAbout(t *testing.T) {
"allowed-hosts": "host", "admin-emails": "email", "tokens": "token", "hostname": "hostname",
"apikey": "apikey", "servername": "servername", "tls-cert": "cert", "site": "site", "timezone": "timezone",
"bearer": "bearer", "bind-ipv4": "ipv4", "listen-ipv6": "ipv6", "site-name": "site-name",
"server-name": "server-name", "public-name": "public-name", "smtp-relay": "smtp-relay"} {
"server-name": "server-name", "public-name": "public-name", "smtp-relay": "smtp-relay",
"host-name": "host-name", "user-name": "user-name", "domain-name": "domain-name",
"nameserver": "nameserver", "upstream-nameservers": "nameserver", "dns-server": "dns-server",
"dns-servers": "dns-server", "default-gateway": "gateway", "lan-subnet": "subnet", "sender": "sender",
"notify-recipients": "recipient", "contact": "contact", "tls-certificate": "certificate",
"allow-from": "allow-from", "deny-from": "deny-from", "allow-hosts": "host", "use-host": "host",
"trusted": "trusted", "allow-list": "allow-list", "ip-whitelist": "whitelist", "peer": "peer",
"wireguard-peers": "peer", "bind": "bind", "listen": "listen", "upstream": "upstream", "http-proxy": "proxy",
"trusted-proxies": "proxy", "admin": "admin", "notify-admin": "admin", "wake-mac": "mac",
"password-file": "password-file", "key-file": "key-file", "token-path": "token-path",
"secret-file": "secret-file", "cert-path": "cert-path"} {
if w := operatorsWord(key); w != word {
t.Errorf("%s: read %q, want %q", key, w, word)
}
@@ -332,3 +345,106 @@ func TestAStoredManifestWithAnUnknownKeyIsLeftOutAndRegistrationRefusesIt(t *tes
t.Fatal("a malformed stored manifest was read")
}
}
// A module left out for a key this controller does not know, inside an entry, is read past that key
// alone: it still provides what it provides, its other entries are whole, and a key of the same name
// that another entry knows is kept (novox/hq ADR 0262).
func TestALeftOutModuleStillProvidesWhatItProvides(t *testing.T) {
var m Manifest
raw := `{"module": "later", "version": "2",
"provides": [{"name": "db", "scope": "mesh"}, {"name": "cache", "a-field-from-later": 1}],
"state": [{"name": "s", "history": 3}],
"data": {"own": [{"id": "d", "path": "${dir:d}", "class": "valuable", "backup": {"dump": "x", "into": "d", "class": "later"}}]},
"resources": [{"id": "d", "type": "directory", "mode": "0700"}]}`
if err := json.Unmarshal([]byte(raw), &m); err != nil {
t.Fatal(err)
}
if len(m.Provides) != 2 || m.Provides[0].Name != "db" || m.Provides[1].Name != "cache" {
t.Fatalf("provides: %+v", m.Provides)
}
if len(m.State) != 1 || m.State[0].History != 3 {
t.Fatalf("state: %+v", m.State)
}
if m.Data == nil || len(m.Data.Own) != 1 || m.Data.Own[0].Class != "valuable" {
t.Fatalf("data: the item's own class was taken for the backup's unknown one: %+v", m.Data)
}
for _, want := range []string{"provides[1].a-field-from-later", "data.own[0].backup.class"} {
if !strings.Contains(m.UnknownField(), want) {
t.Errorf("unknown %q does not say it read without %s", m.UnknownField(), want)
}
}
if !strings.Contains(UnknownFieldReason(m), "left out") {
t.Fatal(UnknownFieldReason(m))
}
}
// A left-out module's data is still copied: the backup holder on its machine keeps its lines while every
// other thing of it is left out.
func TestALeftOutModulesDataIsStillBackedUp(t *testing.T) {
var later Manifest
if err := json.Unmarshal([]byte(`{"module": "later", "version": "2", "a-field-from-later": 1,
"resources": [{"id": "d", "type": "directory", "mode": "0700"}, {"id": "rc", "type": "file", "path": "/etc/later.conf", "content": "x\n"}],
"data": {"own": [{"id": "d", "path": "${dir:d}", "class": "valuable"}]}}`), &later); err != nil {
t.Fatal(err)
}
holder := Manifest{Module: "backups", Version: "1",
Claims: []Claim{{Name: BackupSeat, Scope: ScopeNode}},
Resources: []map[string]any{{"id": "list", "type": "file", "path": "/etc/backups.list", "mode": "0644",
"content": "${contribution:" + BackupSeat + ":backup}"}}}
r := anAdoptedAnchor()
r.Modules = append(r.Modules, holder, later)
composed, err := r.Compose(anchorRendering(false))
if err != nil {
t.Fatal(err)
}
if _, left := composed.LeftOut["later"]; !left {
t.Fatalf("not left out: %v", composed.LeftOut)
}
got := byID(composed.Resources)
if _, declared := got["later.rc"]; declared {
t.Fatal("the left-out module's file is still declared")
}
list, _ := got["backups.list"]["content"].(string)
if !strings.Contains(list, "# later") || !strings.Contains(list, "path /var/lib/later/d") {
t.Fatalf("the left-out module's data is no longer backed up:\n%s", list)
}
}
// A left-out module's backup lines are best effort: a line that cannot be placed — an access nobody
// placed, a placement setting that does not read — is said among what could not be placed, and the
// machine is declared (novox/hq ADR 0262).
func TestALeftOutModulesUnplaceableBackupLineCostsOnlyThatLine(t *testing.T) {
holder := Manifest{Module: "backups", Version: "1",
Claims: []Claim{{Name: BackupSeat, Scope: ScopeNode}},
Resources: []map[string]any{{"id": "list", "type": "file", "path": "/etc/backups.list", "mode": "0644",
"content": "${contribution:" + BackupSeat + ":backup}"}}}
media := Manifest{Module: "media", Version: "1",
Accesses: []Access{{ID: "library", Mode: "read"}},
Data: &Data{Own: []DataItem{{ID: "library", Path: "${access:library}", Class: "valuable"}}}}
placedBadly := Manifest{Module: "notes", Version: "1",
Resources: []map[string]any{{"id": "d", "type": "directory", "path": "/srv/notes", "mode": "0700"}},
Data: &Data{Own: []DataItem{{ID: "d", Path: "${dir:d}", Class: "valuable"}}}}
r := anAdoptedAnchor()
r.Modules = append(r.Modules, holder, media, placedBadly)
with := anchorRendering(false)
with.Settings["notes"] = []Layer{{From: "anchor", Values: map[string]any{PlacesSetting: "not a map"}}}
composed, err := r.Compose(with)
if err != nil {
t.Fatalf("an unplaceable line of a left-out module failed the machine: %v", err)
}
for _, m := range []string{"media", "notes"} {
if _, left := composed.LeftOut[m]; !left {
t.Errorf("%s is not left out: %v", m, composed.LeftOut)
}
}
unplaced := strings.Join(composed.Unplaced, "\n")
for _, want := range []string{"media's backup for node-backup", "notes's backup for node-backup", "placement setting does not read"} {
if !strings.Contains(unplaced, want) {
t.Errorf("not said among what could not be placed: %q in\n%s", want, unplaced)
}
}
list, _ := byID(composed.Resources)["backups.list"]["content"].(string)
if strings.Contains(list, "/srv/notes") || strings.Contains(list, "${") {
t.Fatalf("a line was placed from the definition's default or unfilled:\n%s", list)
}
}
+96
View File
@@ -0,0 +1,96 @@
package catalogue
// A machine's shares, and the shares a machine mounts (novox/hq ADR 0263).
//
// **Two roles, one per side of the wire.** A machine that shares a directory with the mesh does it
// through the holder of `node-nfs-server`: it writes the machine's export file, runs the NFS service,
// opens its port to the private network only, and provides each share as the provision `nfs-share`. A
// machine that wants the files gets them through the holder of `node-mounts`: it writes a mount unit and
// an automount unit per share, so nothing mounts at boot and nothing can fail a boot, and it says a
// device that comes and goes is absent rather than failed.
//
// **One holder per machine on each side.** Two modules writing one machine's export file, or two writing
// mount units for one mount point, is the conflict a seat exists to refuse. Both seats deliver nothing:
// `nfs-share` is provided at the mesh's scope, by the module holding `node-nfs-server` on the machine that
// shares, and a seat at a node's scope cannot be the answer for a provision at the mesh's.
//
// **The data is the operator's** (ADR 0051). Neither holder creates, chowns or removes anything under a
// shared path; the export maps every client to the path's owner, so no client acts as another account on
// the server. Their verbs read, and the ones that act take over what a person wrote by hand only on a
// person's word: a dataset's export property, an fstab line.
// NFSServerSeat is the role of the machine that shares directories over NFS (novox/hq ADR 0263).
const NFSServerSeat = "node-nfs-server"
// MountsSeat is the role that mounts a machine's shares and occasional sources (novox/hq ADR 0263).
const MountsSeat = "node-mounts"
// nfsServerVerbs is the contract every holder of node-nfs-server serves (novox/hq ADR 0263).
func nfsServerVerbs() []Verb {
return []Verb{
{Name: "exports", Description: "Every share this machine exports: its name, its path, read-write or " +
"read-only, the owner every client is mapped to (uid and gid), the clients it is exported to (the " +
"private network's range), and whether the kernel holds it now. Also the exports found that are " +
"not the mesh's: a dataset's sharenfs property, a line in /etc/exports.",
Input: schema(map[string]string{}, nil),
Replaces: []string{"exportfs -v", "cat /etc/exports", "zfs get sharenfs"}},
{Name: "clients", Description: "Which machines have mounted which share now, as the NFS server " +
"knows its clients.",
Input: schema(map[string]string{}, nil),
Replaces: []string{"ss -tn sport = :2049", "cat /proc/fs/nfsd/clients/*/info"}},
{Name: "test", Description: "Whether this machine exports one share for the mesh now, and whether its " +
"NFS service is up; with an address, also whether that address is inside the private network's " +
"range the share is exported to. What a machine mounting the share asks before it mounts.",
Input: schema(map[string]string{
"share": "the share, by its name",
"address": "a client's address on the private network (optional)",
}, []string{"share"}),
Replaces: []string{"showmount -e"}},
{Name: "reload", Description: "Have the kernel read this machine's export files again now (exportfs " +
"-ra) and answer the shares as it then holds them. The module's own process writes its export " +
"file; this is for after a change made outside it. Changes no shared path.",
Input: schema(map[string]string{}, nil),
Replaces: []string{"exportfs -ra"}},
{Name: "adopt", Description: "Take over an export a person made by hand: clear a dataset's sharenfs " +
"property for a path the mesh's own export now serves — only when that export is live. Without " +
"confirm, says what it would do and changes nothing.",
Input: schema(map[string]string{
"path": "the shared path whose hand-made export is taken over",
"confirm": "\"true\": change it (needs why); anything else is a dry run",
"why": "why, for the record",
}, []string{"path"}, "confirm"),
Replaces: []string{"zfs set sharenfs=off"}},
}
}
// mountsVerbs is the contract every holder of node-mounts serves (novox/hq ADR 0263).
func mountsVerbs() []Verb {
return []Verb{
{Name: "list", Description: "Every mount point on this machine: its fstab line, the mesh's mount and " +
"automount units for it, its state (armed, mounted, absent, unreachable, failed) and since when, " +
"and which settings asked for it. A password in a mount's options is never shown.",
Input: schema(map[string]string{}, nil),
Replaces: []string{"cat /etc/fstab", "findmnt", "systemctl list-units --type=mount"}},
{Name: "test", Description: "Whether one share's or occasional source's server answers from this " +
"machine now, without touching its mount point.",
Input: schema(map[string]string{"name": "the share or source, by its name"}, []string{"name"}),
Replaces: []string{"showmount -e", "ping"}},
{Name: "mount", Description: "Mount one share or occasional source now, rather than at its first " +
"access.",
Input: schema(map[string]string{"name": "the share or source, by its name"}, []string{"name"}),
Replaces: []string{"mount"}},
{Name: "unmount", Description: "Release one share or occasional source now; its automount stays " +
"armed, so the next access mounts it again.",
Input: schema(map[string]string{"name": "the share or source, by its name"}, []string{"name"}),
Replaces: []string{"umount"}},
{Name: "adopt", Description: "Take over a mount a person wrote by hand: comment out the /etc/fstab " +
"line for a mount point the mesh's own units now serve, keeping a copy of the file — only when " +
"the mesh's automount for it is armed. Without confirm, says what it would do and changes nothing.",
Input: schema(map[string]string{
"mountpoint": "the mount point whose fstab line is taken over",
"confirm": "\"true\": change it (needs why); anything else is a dry run",
"why": "why, for the record",
}, []string{"mountpoint"}, "confirm"),
Replaces: []string{"sed -i /etc/fstab"}},
}
}
+161
View File
@@ -0,0 +1,161 @@
package catalogue
import (
"strings"
"testing"
)
// Defends novox/hq ADR 0263: a machine's shares and the shares a machine mounts are two node seats, each
// with its verbs required of every holder, each delivering nothing — `nfs-share` is provided at the mesh's
// scope by the holder of node-nfs-server, and a node seat cannot answer for a provision at the mesh's.
func TestTheSharesAndMountsAreNodeSeatsWithTheirVerbs(t *testing.T) {
for seat, want := range map[string]string{
NFSServerSeat: "exports clients test reload adopt",
MountsSeat: "list test mount unmount adopt",
} {
s, ok := SeatNamed(seat)
if !ok {
t.Fatalf("%s is not in the mesh's set", seat)
}
if s.Scope != ScopeNode || s.Decision != "novox/hq ADR 0263" || s.Delivers != "" || s.Replicated {
t.Errorf("%s is %+v; a node seat under ADR 0263 that delivers nothing", seat, s)
}
var got []string
for _, v := range s.Serves {
got = append(got, v.Name)
if v.Optional {
t.Errorf("%s.%s is optional; its first holder serves it", seat, v.Name)
}
if v.Description == "" || v.Input["type"] != "object" || len(v.Replaces) == 0 {
t.Errorf("%s.%s has no description, no object schema or says it replaces nothing", seat, v.Name)
}
}
if strings.Join(got, " ") != want {
t.Errorf("%s serves %v, not %s", seat, got, want)
}
}
}
// Both acts that take over what a person wrote by hand are dry runs unless confirmed, and name the path.
func TestTheAdoptVerbsAreDryRunsUnlessConfirmed(t *testing.T) {
for seat, key := range map[string]string{NFSServerSeat: "path", MountsSeat: "mountpoint"} {
s, _ := SeatNamed(seat)
for _, v := range s.Serves {
if v.Name != "adopt" {
continue
}
props, _ := v.Input["properties"].(map[string]any)
confirm, _ := props["confirm"].(map[string]any)
if confirm == nil || confirm["enum"] == nil {
t.Errorf("%s.adopt has no confirm switch: %v", seat, v.Input)
}
if req, _ := v.Input["required"].([]string); len(req) != 1 || req[0] != key {
t.Errorf("%s.adopt requires %v, not %q alone", seat, v.Input["required"], key)
}
if !strings.Contains(v.Description, "Without confirm, says what it would do and changes nothing") {
t.Errorf("%s.adopt does not say a call without confirm changes nothing: %s", seat, v.Description)
}
}
}
}
// A holder claiming the seat at a node with every verb holds it; one naming a verb the seat does not
// promise, or leaving one out, is refused — as the catalogue's nfs-server and mounts modules claim them.
func TestAHolderOfTheShareSeatsServesEveryVerbAndNothingElse(t *testing.T) {
cases := []struct {
seat, module string
verbs []string
}{
{NFSServerSeat, "nfs-server", []string{"exports", "clients", "test", "reload", "adopt"}},
{MountsSeat, "mounts", []string{"list", "test", "mount", "unmount", "adopt"}},
}
for _, c := range cases {
seat, _ := SeatNamed(c.seat)
holder := Manifest{Module: c.module, Version: "1",
Claims: []Claim{{Name: c.seat, Scope: ScopeNode, Serves: c.verbs}}}
if c.seat == NFSServerSeat {
holder.Provides = []Offer{{Name: "nfs-share", Scope: ScopeMesh}}
}
if err := CanHold(holder, seat); err != nil {
t.Errorf("%s serving every verb is refused: %v", c.module, err)
}
typo := holder
typo.Claims = []Claim{{Name: c.seat, Scope: ScopeNode, Serves: append(append([]string{}, c.verbs...), "export")}}
if err := CanHold(typo, seat); err == nil || !strings.Contains(err.Error(), "does not promise") {
t.Errorf("%s naming a verb the seat does not promise was accepted: %v", c.module, err)
}
short := holder
short.Claims = []Claim{{Name: c.seat, Scope: ScopeNode, Serves: c.verbs[:len(c.verbs)-1]}}
if err := CanHold(short, seat); err == nil || !strings.Contains(err.Error(), "adopt") {
t.Errorf("%s leaving adopt out was accepted: %v", c.module, err)
}
mesh := holder
mesh.Claims = []Claim{{Name: c.seat, Scope: ScopeMesh, Serves: c.verbs}}
if err := CanHold(mesh, seat); err == nil {
t.Errorf("%s claiming a node seat at the mesh's scope was accepted", c.module)
}
}
}
// What each verb replaces is what an agent would type over ssh to read or change a share by hand.
func TestTheShareVerbsSayWhatTheyReplace(t *testing.T) {
want := map[string]string{
NFSServerSeat + ".exports": "exportfs -v",
NFSServerSeat + ".adopt": "zfs set sharenfs=off",
MountsSeat + ".list": "cat /etc/fstab",
MountsSeat + ".adopt": "sed -i /etc/fstab",
}
for _, s := range DefaultSeats() {
for _, v := range s.Serves {
cmd, ok := want[s.Name+"."+v.Name]
if !ok {
continue
}
delete(want, s.Name+"."+v.Name)
found := false
for _, r := range v.Replaces {
found = found || r == cmd
}
if !found {
t.Errorf("%s.%s does not say it replaces %q: %v", s.Name, v.Name, cmd, v.Replaces)
}
}
}
for verb := range want {
t.Errorf("%s is not a verb of the compiled seats", verb)
}
}
// The confirm switch is the seat's string "true", as every verb's switch is, and its description says so:
// a holder handed the boolean reading of "true to change it" would treat the string as a dry run.
func TestTheConfirmSwitchIsTheStringTrue(t *testing.T) {
for _, seat := range []string{NFSServerSeat, MountsSeat} {
s, _ := SeatNamed(seat)
for _, v := range s.Serves {
props, _ := v.Input["properties"].(map[string]any)
confirm, _ := props["confirm"].(map[string]any)
if confirm == nil {
continue
}
if confirm["type"] != "string" {
t.Errorf("%s.%s confirm is %v, not the string switch", seat, v.Name, confirm["type"])
}
if d, _ := confirm["description"].(string); !strings.Contains(d, `"true"`) {
t.Errorf("%s.%s confirm does not name the string \"true\": %q", seat, v.Name, d)
}
}
}
}
// reload has the kernel read the export files; the module's process writes its file. The description
// must not promise a write it does not do.
func TestReloadSaysWhatItDoes(t *testing.T) {
s, _ := SeatNamed(NFSServerSeat)
for _, v := range s.Serves {
if v.Name == "reload" && (strings.Contains(v.Description, "Write this machine's export file") ||
!strings.Contains(v.Description, "exportfs")) {
t.Errorf("reload's description promises a write or does not name exportfs: %s", v.Description)
}
}
}
+149 -1
View File
@@ -1,7 +1,10 @@
package catalogue
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"regexp"
)
@@ -54,5 +57,150 @@ func UnknownFieldReason(m Manifest) string {
return ""
}
return m.Module + " uses a field this controller does not know (" + m.unknown + "); it is left out " +
"until the controller is updated (novox/hq ADR 0262)"
"until the controller is updated: nothing of it is changed on its machines and its contributions to " +
"other modules and its open ports stop. Its data is still backed up as this controller reads it, " +
"which may not be what its newer manifest asks, and it still provides what it provides " +
"(novox/hq ADR 0262)"
}
// backupView is what of a left-out module still reaches its machine: its data, so the backup holder
// keeps copying it, and the directories and accesses its data items name. No contribution, shell code
// or environment of its own: those are what leaving it out stops.
func backupView(m Manifest) Manifest {
view := Manifest{Module: m.Module, Data: m.Data, Accesses: m.Accesses, bestEffort: true}
for _, r := range m.Resources {
if fmt.Sprint(r["type"]) == "directory" {
view.Resources = append(view.Resources, r)
}
}
return view
}
// prunedFields is a stored manifest's fields with every key this controller does not know taken out,
// and the keys taken out (novox/hq ADR 0262). A second pass, after the strict one refused: each
// top-level field is decoded alone, and where an entry inside it has an unknown key, the one
// occurrence whose removal moves the decoder past it is removed — never a key of the same name that
// the entry around it knows. So a left-out module still provides what it provides, and its data and
// directories are still read, which its backup lines are made from.
func prunedFields(keys map[string]json.RawMessage) (manifestFields, []string, error) {
var removed []string
tree := map[string]any{}
for k, raw := range keys {
var v any
dec := json.NewDecoder(bytes.NewReader(raw))
dec.UseNumber()
if err := dec.Decode(&v); err != nil {
return manifestFields{}, nil, err
}
tree[k] = v
}
for _, k := range sortedAnyKeys(tree) {
for tries := 0; ; tries++ {
err := decodesAlone(k, tree[k])
if err == nil {
break
}
unknown := asUnknownField(err)
if unknown == nil || tries > 64 {
return manifestFields{}, nil, err
}
if unknown.Field == k {
delete(tree, k)
removed = append(removed, k)
break
}
path, ok := removalThatHelps(k, tree[k], unknown.Field, err.Error())
if !ok {
// The same key unknown in two entries alike: no one removal changes the words.
// Every occurrence goes, and the field is judged again.
if removeEvery(tree[k], unknown.Field) == 0 {
return manifestFields{}, nil, err
}
path = "…." + unknown.Field
}
removed = append(removed, k+path)
}
}
raw, err := json.Marshal(tree)
if err != nil {
return manifestFields{}, nil, err
}
var fields manifestFields
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&fields); err != nil {
return manifestFields{}, nil, err
}
return fields, removed, nil
}
// decodesAlone is whether one top-level field decodes strictly on its own.
func decodesAlone(k string, v any) error {
raw, err := json.Marshal(map[string]any{k: v})
if err != nil {
return err
}
var fields manifestFields
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
return dec.Decode(&fields)
}
// removalThatHelps removes, from v, the one occurrence of key whose removal changes what the strict
// decoder says of field k, and says where it was. Every other occurrence is left as it was.
func removalThatHelps(k string, v any, key, said string) (string, bool) {
var found bool
var where string
var walk func(node any, at string) bool
walk = func(node any, at string) bool {
switch n := node.(type) {
case map[string]any:
if value, has := n[key]; has {
delete(n, key)
// Accepted only when the decoder is past it: nothing left, or an unknown key said
// elsewhere. A different kind of error means the removal broke the entry; the same
// words mean this was not the occurrence it refused.
err := decodesAlone(k, v)
if err == nil || (asUnknownField(err) != nil && err.Error() != said) {
found, where = true, at+"."+key
return true
}
n[key] = value
}
for _, sub := range sortedAnyKeys(n) {
if walk(n[sub], at+"."+sub) {
return true
}
}
case []any:
for i, item := range n {
if walk(item, fmt.Sprintf("%s[%d]", at, i)) {
return true
}
}
}
return false
}
walk(v, "")
return where, found
}
// removeEvery removes key from every object in v, and says how many it removed.
func removeEvery(v any, key string) int {
n := 0
switch node := v.(type) {
case map[string]any:
if _, has := node[key]; has {
delete(node, key)
n++
}
for _, sub := range node {
n += removeEvery(sub, key)
}
case []any:
for _, item := range node {
n += removeEvery(item, key)
}
}
return n
}
+19 -3
View File
@@ -268,9 +268,10 @@ var ControllerVerbs = []Verb{
"machine it is assigned to with where it comes from; module and node narrow it (novox/hq ADR 0262)",
}, nil, "clear", "replace", "history")},
{Name: "command", Description: "Run one command line of the controller's own, as you would type it at its " +
"shell — `node account g14 jochen`, `node show ace`, `module list` — and answer what it printed. The " +
"generic verb beside the named ones (novox/hq ADR 0154): everything the binary can do, without a verb " +
"per command. Any node may call any tool (ADR 0175), so nothing is held back here.",
"shell — `node show ace`, `module list` — and answer what it printed. The generic verb beside the named " +
"ones (novox/hq ADR 0154): what the binary can do, without a verb per command. Held back: every `node` " +
"command but `node list` and `node show` — a node's accounts decide who may become root on it, and are " +
"set at the controller's terminal only (ADR 0266).",
Input: schema(map[string]string{
"command": "the command line, as the controller's binary takes it; quotes group a word with spaces",
}, []string{"command"})},
@@ -424,6 +425,21 @@ var ControllerVerbs = []Verb{
"why": "with consumer or older-than: why — required, and recorded in the hand-act log",
"cause": "with consumer or older-than: the cause in a word (cleanup-waiting when absent)",
}, nil, "confirm")},
// What a consumer gave up on (novox/hq issue 330): kept in DEAD_LETTERS until a person acts on it.
{Name: "dead-letters", Description: "Every message a consumer on the bus gave up on after handing it over " +
"as often as it may, kept in DEAD_LETTERS: whose consumer, the subject, how often it was handed over and " +
"when it was given up, newest first. With id: that one whole, with what it said. With deliver: hand it " +
"again to the consumer that gave it up, and nobody else; with drop: let it go for good. Delivering and " +
"dropping are hand acts, which say why (novox/hq issue 330).",
Input: schema(map[string]string{
"id": "a dead letter's id, as the list gives it: that one whole, with its message",
"consumer": "a consumer's name, or <stream>.<consumer>: only what that one gave up on",
"limit": "how many to list, newest first (default 50); only when listing",
"deliver": "a dead letter's id: deliver it again to the consumer that gave it up (needs why)",
"drop": "a dead letter's id: drop it for good (needs why)",
"why": "with deliver or drop: why — required, and recorded in the hand-act log",
"cause": "with deliver or drop: the cause in a word (dead-letter when absent)",
}, nil)},
// The data every machine declares (novox/hq ADR 0233).
{Name: "data", Description: "Every item of data every machine declares, as the self-check last measured it: " +
"its class (irreplaceable, rebuildable, cache), where it is, its size, its newest write, its newest good " +
+4
View File
@@ -131,6 +131,10 @@ type Machine struct {
// home is when that is not the derived one.
Account string `json:"account,omitempty"`
AccountHome string `json:"account-home,omitempty"`
// AgentAccount is the account agents run as there when it is not the operator's (a pseudonym), and
// AgentAccountHome its home when not derived (novox/hq ADR 0266).
AgentAccount string `json:"agent-account,omitempty"`
AgentAccountHome string `json:"agent-account-home,omitempty"`
// PublicDomain is the domain it answers for, its labels replaced.
PublicDomain string `json:"public-domain,omitempty"`
// Assigned is every module assigned there.
+93
View File
@@ -0,0 +1,93 @@
package inventory
import (
"context"
"errors"
"strings"
"testing"
)
// The agent account (novox/hq ADR 0266): recorded and read back with every node, its home derived when
// not stated, cleared by an empty name — and refused when it is root, the operator's own account, or no
// login at all, because each of those would say agents have an account of their own while they do not.
func TestAgentAccountIsRecordedAndRefusedWhereItWouldNotConfine(t *testing.T) {
inv := ForTest(t)
ctx := context.Background()
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
if err := inv.SetAccount(ctx, "anchor", "operator", ""); err != nil {
t.Fatal(err)
}
n, err := inv.NodeByName(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
if n.AgentAccount != "" || n.AgentHome() != "" {
t.Fatalf("a node that names none has agent account %q, home %q", n.AgentAccount, n.AgentHome())
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
n, _ = inv.NodeByName(ctx, "anchor")
if n.AgentAccount != "agent" || n.AgentHome() != "/home/agent" {
t.Fatalf("agent account %q, home %q; want agent, /home/agent", n.AgentAccount, n.AgentHome())
}
all, err := inv.Nodes(ctx)
if err != nil || len(all) != 1 || all[0].AgentAccount != "agent" {
t.Fatalf("the listing does not carry the agent account: %+v %v", all, err)
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", "/srv/agent"); err != nil {
t.Fatal(err)
}
if n, _ = inv.NodeByName(ctx, "anchor"); n.AgentHome() != "/srv/agent" {
t.Fatalf("the stated home is %q", n.AgentHome())
}
for _, c := range []struct{ account, home, says string }{
{"root", "", "may not run as root"},
{"operator", "", "operator account"},
{"Agent", "", "not a login name"},
{"9agent", "", "not a login name"},
{"agent", "relative", "absolute"},
{"postgres", "", "service account"},
{"systemd-network", "", "service account"},
{"showcase", "", "service account"},
{"", "/home/x", "without an agent account"},
} {
err := inv.SetAgentAccount(ctx, "anchor", c.account, c.home)
if err == nil || !strings.Contains(err.Error(), c.says) {
t.Errorf("%q %q: %v; want a refusal saying %q", c.account, c.home, err, c.says)
}
}
if n, _ = inv.NodeByName(ctx, "anchor"); n.AgentAccount != "agent" {
t.Fatalf("a refusal changed the record: %q", n.AgentAccount)
}
// The other direction: the operator account may not be named as the agent account either.
if err := inv.SetAccount(ctx, "anchor", "agent", ""); err == nil || !strings.Contains(err.Error(), "agent account") {
t.Fatalf("the operator account named as the agent account: %v; want a refusal", err)
}
if n, _ = inv.NodeByName(ctx, "anchor"); n.Account != "operator" {
t.Fatalf("a refusal changed the operator account: %q", n.Account)
}
if err := inv.SetAgentAccount(ctx, "anchor", "", ""); err != nil {
t.Fatal(err)
}
if err := inv.SetAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatalf("with the agent account cleared, the name is free: %v", err)
}
if err := inv.SetAccount(ctx, "anchor", "operator", ""); err != nil {
t.Fatal(err)
}
if n, _ = inv.NodeByName(ctx, "anchor"); n.AgentAccount != "" || n.AgentAccountHome != "" {
t.Fatalf("clearing left %q %q", n.AgentAccount, n.AgentAccountHome)
}
if err := inv.SetAgentAccount(ctx, "nowhere", "agent", ""); !errors.Is(err, ErrNoSuchNode) {
t.Fatalf("an unknown node: %v", err)
}
}
+3
View File
@@ -31,6 +31,9 @@ type ResourceHealth struct {
// Account is the account whose own service manager runs it, or the account a resource of kind account
// is (novox/hq ADR 0254).
Account string `json:"account,omitempty"`
// Root is "never" on an account verdict that judged whether the account can become root without a
// person (novox/hq ADR 0266).
Root string `json:"root,omitempty"`
}
// NodeHealth is a machine's newest statement, as kept.
@@ -0,0 +1,13 @@
-- A node names the account its agents run as (novox/hq ADR 0266).
--
-- On the control node every agent session ran as the operator's account, which may become root without
-- a password: any agent there could become root without a person. The decision is an account of the
-- agents' own, without sudo, beside the operator's, who keeps theirs. Stated by the operator at the
-- controller's terminal, like the operator account (migration 0036), and never by a verb or a setting,
-- so no agent can change which account it is.
--
-- Empty rather than null, as the operator account is: empty is a real state, "agents run as the
-- operator's account here" — a workstation's today. The home is stored only when it is not
-- /home/<account>; empty means derive it.
alter table node add column agent_account text not null default '';
alter table node add column agent_account_home text not null default '';
+118 -2
View File
@@ -9,6 +9,7 @@ import (
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"strings"
"time"
@@ -69,6 +70,14 @@ type Node struct {
Account string
AccountHome string
// AgentAccount is the login agents run as on this machine when it is not the operator's — `agent`
// on the control node (novox/hq ADR 0266): an account of their own, without sudo, so no agent there
// can become root without a person. Empty means agents run as the operator account. Stated at the
// controller's terminal only, never by a verb or a setting. AgentAccountHome is its home when not
// /home/<account>; empty means derive it.
AgentAccount string
AgentAccountHome string
// HostVersion is the version of the host this machine reported running (novox/hq 04-ISSUES/087).
// Empty when it has not said since the mesh began keeping it — which is not the same as running
// no host, so nothing derives "behind" from an empty one.
@@ -91,6 +100,17 @@ func (n Node) Home() string {
}
}
// AgentHome is the agent account's home, derived when not stored; empty when no agent account is named.
func (n Node) AgentHome() string {
if n.AgentAccount == "" {
return ""
}
if n.AgentAccountHome != "" {
return n.AgentAccountHome
}
return "/home/" + n.AgentAccount
}
// Silent is how long since this node was last heard from, and whether it ever was.
func (n Node) Silent() (time.Duration, bool) {
if n.LastSeen.IsZero() {
@@ -147,14 +167,14 @@ func (i *Inventory) AddNodeAs(ctx context.Context, name string, adopted bool) (N
// nodeColumns and scanNode are the one reading of a node row, so every way of finding a node
// says whether it is adopted.
const nodeColumns = `id, name, created, last_seen, adopted, adopted_since, account, account_home,
host_version`
agent_account, agent_account_home, host_version`
func scanNode(row pgx.Row) (Node, error) {
var n Node
var seen, since *time.Time
var host *string
if err := row.Scan(&n.ID, &n.Name, &n.Created, &seen, &n.Adopted, &since,
&n.Account, &n.AccountHome, &host); err != nil {
&n.Account, &n.AccountHome, &n.AgentAccount, &n.AgentAccountHome, &host); err != nil {
return Node{}, err
}
if host != nil {
@@ -172,7 +192,23 @@ func scanNode(row pgx.Row) (Node, error) {
// SetAccount records the operator account on a node — its human login — and optionally where that
// account's home is (novox/hq to-be 29). An empty home means the mesh derives it. Clearing the
// account (empty name) is allowed: a machine may stop having a known operator.
//
// **Never the node's agent account** (novox/hq ADR 0266): the operator account may become root, and the
// agent account exists so agents cannot; naming the one as the other gives agents root. Refused here as
// SetAgentAccount refuses the other direction.
func (i *Inventory) SetAccount(ctx context.Context, node, account, home string) error {
account = strings.TrimSpace(account)
if account != "" {
n, err := i.NodeByName(ctx, node)
if err != nil {
return err
}
if n.AgentAccount != "" && n.AgentAccount == account {
return fmt.Errorf("%s is %s's agent account: the operator account may become root, and agents run as "+
"%s so that they cannot (novox/hq ADR 0266); clear the agent account first "+
"(node agent-account %s --clear) if the operator is to log in as it", account, node, account, node)
}
}
tag, err := i.store.Pool().Exec(ctx,
`update node set account = $1, account_home = $2 where name = $3`, account, home, node)
if err != nil {
@@ -184,6 +220,86 @@ func (i *Inventory) SetAccount(ctx context.Context, node, account, home string)
return nil
}
// loginName is what a login may be called: what useradd accepts by default, lower case, a letter or
// an underscore first.
var loginName = regexp.MustCompile(`^[a-z_][a-z0-9_-]{0,31}$`)
// SetAgentAccount records the account agents run as on a node, and optionally its home (novox/hq ADR
// 0266). An empty account clears it: agents run as the operator account again.
//
// **Refused, rather than recorded and judged later:** root, which is the very thing the account exists
// to keep agents from; the node's operator account, which may become root without a password and is
// what agents ran as before — naming it here would say the agents have an account of their own while
// they do not; and a name no machine would accept as a login.
func (i *Inventory) SetAgentAccount(ctx context.Context, node, account, home string) error {
account, home = strings.TrimSpace(account), strings.TrimSpace(home)
if account == "" && home != "" {
return errors.New("a home without an agent account says nothing; name the account too")
}
if account != "" {
if err := AgentAccountRefusal(account, home); err != nil {
return err
}
n, err := i.NodeByName(ctx, node)
if err != nil {
return err
}
if n.Account != "" && n.Account == account {
return fmt.Errorf("%s is %s's operator account: agents would run as the operator, who may become "+
"root; clear the agent account instead (node agent-account %s --clear)", account, node, node)
}
}
tag, err := i.store.Pool().Exec(ctx,
`update node set agent_account = $1, agent_account_home = $2 where name = $3`, account, home, node)
if err != nil {
return err
}
if tag.RowsAffected() == 0 {
return fmt.Errorf("%w: %s", ErrNoSuchNode, node)
}
return nil
}
// serviceAccounts are the system and service accounts a machine of the mesh has, or a module of the
// catalogue declares (showcase, and the accounts ADR 0259 gives the router and the channels). The controller
// cannot read a machine's user database, so this list is the controller's half; the node-engine's half is
// refusing to take an existing account below the first login uid as one that must never become root.
var serviceAccounts = map[string]bool{
"root": true, "bin": true, "daemon": true, "sys": true, "adm": true, "nobody": true, "mail": true,
"ftp": true, "http": true, "www-data": true, "git": true, "sshd": true, "dbus": true, "polkitd": true,
"postgres": true, "docker": true, "nats": true, "redis": true, "uuidd": true, "dnsmasq": true,
"avahi": true, "rtkit": true, "colord": true, "geoclue": true, "tss": true, "alpm": true, "usbmux": true,
"showcase": true, "messenger": true, "telegram": true,
}
// serviceAccount says whether a name is a system or service account: one of the list, or a name of
// systemd's own (systemd-…).
func serviceAccount(name string) bool {
return serviceAccounts[name] || strings.HasPrefix(name, "systemd-")
}
// AgentAccountRefusal is why an agent account cannot be named, or nil: root, a malformed login, or a
// home that is not an absolute path.
func AgentAccountRefusal(account, home string) error {
if account == "root" {
return errors.New("agents may not run as root: the agent account exists to keep them from it " +
"(novox/hq ADR 0266)")
}
if !loginName.MatchString(account) {
return fmt.Errorf("%q is not a login name: lower case letters, digits, _ and -, a letter or _ first, "+
"at most 32", account)
}
if serviceAccount(account) {
return fmt.Errorf("%q is a system or service account a machine or a module already has: the agent "+
"account is one the mesh creates for agents alone, which nothing else runs as or owns files as "+
"(novox/hq ADR 0266); name a new one, such as agent", account)
}
if home != "" && !strings.HasPrefix(home, "/") {
return fmt.Errorf("the agent account's home %q is not an absolute path", home)
}
return nil
}
// Nodes are every node record, oldest first.
func (i *Inventory) Nodes(ctx context.Context) ([]Node, error) {
rows, err := i.store.Pool().Query(ctx,
+15 -4
View File
@@ -34,6 +34,9 @@ const (
AdvisoryMaxDeliveries = "max-deliveries"
AdvisoryRefused = "refused"
AdvisoryConsumerLost = "consumer-lost"
// AdvisoryNotKept is the token of a max-deliveries advisory whose message could not be kept in
// DEAD_LETTERS (novox/hq issue 330): said from the log, since the stream does not hold it.
AdvisoryNotKept = "not-kept"
)
// Advisory is one thing the bus said, kept as its newest word and how often it was said.
@@ -43,6 +46,8 @@ type Advisory struct {
ID string
// Stream and Consumer are the consumer it is about, when it is about one.
Stream, Consumer string
// Token is the condition key's last part, when it is not Kind.
Token string
// Said is the newest saying, in the mesh's words.
Said string
First, Last time.Time
@@ -62,7 +67,7 @@ var Advisories = &AdvisoryLog{seen: map[string]*Advisory{}}
func (l *AdvisoryLog) Heard(a Advisory, at time.Time) {
l.mu.Lock()
defer l.mu.Unlock()
key := a.Kind + "/" + a.ID
key := a.Kind + "/" + a.ID + "/" + a.Token
if had, ok := l.seen[key]; ok {
had.Last, had.Said, had.Count = at, a.Said, had.Count+1
return
@@ -107,8 +112,9 @@ func ReadAdvisory(subject string, body []byte) (Advisory, bool) {
switch {
case strings.HasPrefix(subject, "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES."):
return Advisory{Kind: AdvisoryMaxDeliveries, ID: id, Stream: a.Stream, Consumer: a.Consumer,
Said: fmt.Sprintf("%s handed message %d over %d times and gave up on it: it will not be delivered "+
"again, and what it asked for was not done", who, a.StreamSeq, a.Deliveries)}, true
Said: fmt.Sprintf("%s handed message %d over %d times and gave up on it: what it asked for was not "+
"done, and the controller keeps it in %s until it is delivered again or dropped", who, a.StreamSeq,
a.Deliveries, broker.DeadLettersStream)}, true
case strings.HasPrefix(subject, "$JS.EVENT.ADVISORY.CONSUMER.DELETED."):
if !MeshNamed(a.Stream, a.Consumer) {
// A reader's own consumer, gone when it finished — every watch of a bucket and every
@@ -175,7 +181,12 @@ func (s *Server) HearAdvisories(logf func(string, ...any)) (func(), error) {
for _, subject := range broker.BusAdvisories {
sub, err := conn.Subscribe(subject, func(m *nats.Msg) {
if a, ok := ReadAdvisory(m.Subject, m.Data); ok {
Advisories.Heard(a, time.Now())
// A message given up on is said from DEAD_LETTERS, where it is kept, for as long as it is
// kept (novox/hq issue 330) — not for an hour after the server said it; one that could not
// be kept is recorded by whoever tried.
if a.Kind != AdvisoryMaxDeliveries {
Advisories.Heard(a, time.Now())
}
logf("the bus says: %s", a.Said)
}
})
+2 -1
View File
@@ -37,7 +37,8 @@ func TestOnlyTheMeshsOwnConsumersAreSaidLost(t *testing.T) {
[]byte(`{"stream":"EVENTS","consumer":"anchor_shop","stream_seq":7,"deliveries":5}`))
if !ok || a.Kind != AdvisoryMaxDeliveries || a.ID != "EVENTS.anchor_shop" ||
a.Said != "how shop on anchor hears what it consumes handed message 7 over 5 times and gave up on it: "+
"it will not be delivered again, and what it asked for was not done" {
"what it asked for was not done, and the controller keeps it in DEAD_LETTERS until it is delivered "+
"again or dropped" {
t.Fatalf("%+v", a)
}
}
+5 -2
View File
@@ -8,13 +8,16 @@ import (
// JetStream connection the caller has already raised the streams on. Nothing is declared here —
// the streams and the controller's consumers are asserted by Raise, before anything is served.
func ConnectNats(js *broker.JetStream, enroller Enroller, listener Listener) *Server {
logger := newLog()
inbound := Nats(js).(*natsInbound)
inbound.log = logger
return &Server{
inbound: Nats(js),
inbound: inbound,
bus: OverNATS{JS: js.Context(), Conn: js.Conn()},
js: js,
enroller: enroller,
listener: listener,
log: newLog(),
log: logger,
}
}
+372
View File
@@ -0,0 +1,372 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"slices"
"strconv"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
)
// What a consumer gave up on, kept until a person acts on it (novox/hq issue 330, design 25 §3).
//
// A durable consumer hands a message over as often as it may, gives up on it and says so on
// `$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES`. The server's notice is captured in its own stream
// (broker.DeadLetterNoticesStream), so one said while no controller listens waits for the next. The
// serving controller takes each notice, fetches the message it names by its sequence, while the source
// stream still holds it, and keeps a copy in DEAD_LETTERS with what the notice said: the consumer, the
// subject, how often it was handed over and when it was given up. Until then a message given up on was
// kept only by its source, which drops an event after a week, and nothing said which.
//
// The copy's headers are the message's own, but for the server's (`Nats-…`), which describe the first
// publish and would refuse or de-duplicate the copy; they are kept under DeadHeader names.
// The headers a kept message carries beside its own.
const (
DeadHeaderStream = "Mesh-Dead-Stream"
DeadHeaderConsumer = "Mesh-Dead-Consumer"
DeadHeaderSequence = "Mesh-Dead-Sequence"
DeadHeaderSubject = "Mesh-Dead-Subject"
DeadHeaderDeliveries = "Mesh-Dead-Deliveries"
DeadHeaderGaveUp = "Mesh-Dead-Gave-Up"
DeadHeaderPublished = "Mesh-Dead-Published"
DeadHeaderLost = "Mesh-Dead-Lost"
// DeadHeaderMsgID is the message's own de-duplication id, kept, since the copy carries its own.
DeadHeaderMsgID = "Mesh-Dead-Msg-Id"
// AgainHeader marks a message delivered again, with the kept copy's id it came from.
AgainHeader = "Mesh-Delivered-Again"
)
// DeadLetter is one message a consumer gave up on, as DEAD_LETTERS keeps it.
type DeadLetter struct {
// ID is its sequence in DEAD_LETTERS: what the verb names it by.
ID uint64 `json:"id"`
Stream string `json:"stream"`
Consumer string `json:"consumer"`
// Who is the consumer in the mesh's words: whose, and for what.
Who string `json:"who"`
Subject string `json:"subject"`
Sequence uint64 `json:"sequence"`
Deliveries uint64 `json:"deliveries"`
GaveUp time.Time `json:"gave_up"`
Published time.Time `json:"published,omitzero"`
// Lost says why the message itself is not kept: its source no longer held it when the notice was
// taken. The record of it is kept all the same, so it is said and dropped, never silently missing.
Lost string `json:"lost,omitempty"`
Size int `json:"size"`
// Body and Headers are the message itself, given only for one dead letter asked by its id; a header
// with several values keeps them all.
Body string `json:"body,omitempty"`
Headers map[string][]string `json:"headers,omitempty"`
}
// maxDeliveries is the part of the server's notice a kept message is made from.
type maxDeliveries struct {
Stream string `json:"stream"`
Consumer string `json:"consumer"`
StreamSeq uint64 `json:"stream_seq"`
Deliveries uint64 `json:"deliveries"`
Timestamp time.Time `json:"timestamp"`
}
// KeepDeadLetter keeps the message one maximum-deliveries notice is about. An error leaves nothing
// kept, and the notice is to be taken again: the source still holds the message, or a full
// DEAD_LETTERS has room again. A source that no longer holds it is no error: the record is kept with
// why, so it is said.
func KeepDeadLetter(js nats.JetStreamContext, notice []byte) (DeadLetter, error) {
var a maxDeliveries
if err := json.Unmarshal(notice, &a); err != nil || a.Stream == "" || a.Consumer == "" || a.StreamSeq == 0 {
return DeadLetter{}, fmt.Errorf("not a maximum-deliveries notice the mesh can read: %.200s", notice)
}
gaveUp := a.Timestamp
if gaveUp.IsZero() {
gaveUp = time.Now()
}
kept := &nats.Msg{Subject: broker.DeadLetterSubject(a.Stream, a.Consumer), Header: nats.Header{}}
raw, err := js.GetMsg(a.Stream, a.StreamSeq)
switch {
case err == nil:
for k, v := range raw.Header {
if strings.HasPrefix(k, "Nats-") {
continue
}
kept.Header[k] = v
}
if id := raw.Header.Get(nats.MsgIdHdr); id != "" {
kept.Header.Set(DeadHeaderMsgID, id)
}
kept.Header.Set(DeadHeaderSubject, raw.Subject)
kept.Header.Set(DeadHeaderPublished, raw.Time.UTC().Format(time.RFC3339Nano))
kept.Data = raw.Data
case errors.Is(err, nats.ErrMsgNotFound) || errors.Is(err, nats.ErrStreamNotFound):
kept.Header.Set(DeadHeaderLost, fmt.Sprintf("%s no longer held message %d when the notice was taken: %v",
a.Stream, a.StreamSeq, err))
default:
return DeadLetter{}, fmt.Errorf("message %d of %s could not be read: %w", a.StreamSeq, a.Stream, err)
}
kept.Header.Set(DeadHeaderStream, a.Stream)
kept.Header.Set(DeadHeaderConsumer, a.Consumer)
kept.Header.Set(DeadHeaderSequence, strconv.FormatUint(a.StreamSeq, 10))
kept.Header.Set(DeadHeaderDeliveries, strconv.FormatUint(a.Deliveries, 10))
kept.Header.Set(DeadHeaderGaveUp, gaveUp.UTC().Format(time.RFC3339Nano))
// One copy per message given up, however often its notice is taken: a controller that copied and
// stopped before acknowledging, or two controllers each taking it.
kept.Header.Set(nats.MsgIdHdr, fmt.Sprintf("%s.%s.%d", a.Stream, a.Consumer, a.StreamSeq))
ack, err := js.PublishMsg(kept)
if err != nil {
return DeadLetter{}, fmt.Errorf("message %d of %s could not be kept in %s: %w", a.StreamSeq, a.Stream,
broker.DeadLettersStream, err)
}
return deadLetterOf(ack.Sequence, kept.Subject, kept.Header, kept.Data, false), nil
}
// deadLetterOf reads a kept message.
func deadLetterOf(id uint64, subject string, h nats.Header, data []byte, whole bool) DeadLetter {
d := DeadLetter{ID: id, Stream: h.Get(DeadHeaderStream), Consumer: h.Get(DeadHeaderConsumer),
Subject: h.Get(DeadHeaderSubject), Lost: h.Get(DeadHeaderLost), Size: len(data)}
if d.Stream == "" || d.Consumer == "" {
d.Stream, d.Consumer, _ = broker.DeadLetterOf(subject)
}
d.Who = ConsumerInWords(d.Stream, d.Consumer)
d.Sequence, _ = strconv.ParseUint(h.Get(DeadHeaderSequence), 10, 64)
d.Deliveries, _ = strconv.ParseUint(h.Get(DeadHeaderDeliveries), 10, 64)
d.GaveUp, _ = time.Parse(time.RFC3339Nano, h.Get(DeadHeaderGaveUp))
d.Published, _ = time.Parse(time.RFC3339Nano, h.Get(DeadHeaderPublished))
if whole {
d.Body = string(data)
d.Headers = map[string][]string{}
for k, v := range h {
if !strings.HasPrefix(k, "Mesh-Dead-") && !strings.HasPrefix(k, "Nats-") {
d.Headers[k] = append([]string(nil), v...)
}
}
}
return d
}
// HeldDeadLetters is how many messages DEAD_LETTERS holds for each consumer, by `<stream>.<consumer>`.
func HeldDeadLetters(js nats.JetStreamContext) (map[string]int, error) {
info, err := js.StreamInfo(broker.DeadLettersStream, &nats.StreamInfoRequest{SubjectsFilter: ">"})
if err != nil {
return nil, fmt.Errorf("%s cannot be read: %w", broker.DeadLettersStream, err)
}
held := map[string]int{}
for subject, n := range info.State.Subjects {
if stream, consumer, ok := broker.DeadLetterOf(subject); ok && n > 0 {
held[stream+"."+consumer] += int(n)
}
}
return held, nil
}
// DeadLetters lists what DEAD_LETTERS holds, newest first, at most most of them (all when most is not
// positive); for one consumer when consumer names one (its name, or `<stream>.<consumer>`). The total is
// the stream's own count per consumer, so it is right however few are read.
func DeadLetters(js nats.JetStreamContext, consumer string, most int) ([]DeadLetter, int, error) {
held, err := HeldDeadLetters(js)
if err != nil {
return nil, 0, err
}
total := 0
for key, n := range held {
if consumer == "" || key == consumer || strings.HasSuffix(key, "."+consumer) {
total += n
}
}
if total == 0 {
return nil, 0, nil
}
info, err := js.StreamInfo(broker.DeadLettersStream)
if err != nil {
return nil, 0, fmt.Errorf("%s cannot be read: %w", broker.DeadLettersStream, err)
}
var out []DeadLetter
for seq := info.State.LastSeq; seq >= info.State.FirstSeq && seq > 0; seq-- {
if most > 0 && len(out) >= most || len(out) >= total {
break
}
raw, err := js.GetMsg(broker.DeadLettersStream, seq)
if errors.Is(err, nats.ErrMsgNotFound) {
continue // delivered again or dropped
}
if err != nil {
return out, total, fmt.Errorf("dead letter %d cannot be read: %w", seq, err)
}
d := deadLetterOf(seq, raw.Subject, raw.Header, raw.Data, false)
if consumer != "" && consumer != d.Consumer && consumer != d.Stream+"."+d.Consumer {
continue
}
out = append(out, d)
}
return out, total, nil
}
// ErrNoDeadLetter is a dead letter's id DEAD_LETTERS does not hold.
var ErrNoDeadLetter = errors.New("no such dead letter")
// DeadLetterNamed is one kept message whole: what it said, and the headers it said it with.
func DeadLetterNamed(js nats.JetStreamContext, id uint64) (DeadLetter, error) {
raw, err := js.GetMsg(broker.DeadLettersStream, id)
if errors.Is(err, nats.ErrMsgNotFound) {
return DeadLetter{}, fmt.Errorf("%w: %s holds no message %d — delivered again or dropped already, "+
"or never kept; dead-letters lists what it holds", ErrNoDeadLetter, broker.DeadLettersStream, id)
}
if err != nil {
return DeadLetter{}, fmt.Errorf("dead letter %d cannot be read: %w", id, err)
}
return deadLetterOf(id, raw.Subject, raw.Header, raw.Data, true), nil
}
// AgainTo is where a kept message is delivered again so that only the consumer that gave it up gets
// it: an event under that consumer's own again subject on EVENTS — and only when the consumer exists and
// filters that subject, so a message is never let go as delivered while nobody receives it. Any other
// stream's message is refused, with why: an ask given up on by a seat's worker is not delivered again yet
// (novox/hq issue 330's follow-up), since publishing it again leaves the original stuck in the queue.
func AgainTo(js nats.JetStreamContext, d DeadLetter) (string, error) {
switch {
case d.Lost != "":
return "", fmt.Errorf("dead letter %d holds no message to deliver: %s. Drop it", d.ID, d.Lost)
case d.Subject == "":
return "", fmt.Errorf("dead letter %d does not say the subject it was published on, so it cannot be "+
"delivered again. Drop it", d.ID)
case d.Stream != broker.EventsStream:
return "", fmt.Errorf("dead letter %d is from %s, and only an event is delivered again: publishing it "+
"again would reach every consumer of its subject, or leave the original in its queue. Drop it, and "+
"have its sender say it again", d.ID, d.Stream)
}
info, err := js.ConsumerInfo(d.Stream, d.Consumer)
if errors.Is(err, nats.ErrConsumerNotFound) {
return "", fmt.Errorf("dead letter %d was given up by %s, which is no longer on the bus, so nothing would "+
"receive it. Drop it, or deliver it again once the module is assigned there again", d.ID, d.Who)
}
if err != nil {
return "", fmt.Errorf("whether %s can receive dead letter %d cannot be read: %w", d.Who, d.ID, err)
}
want := broker.AgainFilter(d.Consumer)
filters := append([]string{info.Config.FilterSubject}, info.Config.FilterSubjects...)
if !slices.Contains(filters, want) {
return "", fmt.Errorf("%s does not yet take events delivered again (it does not filter %s): the controller "+
"sets that at the next send to its machine. Nothing was done, and dead letter %d is still kept",
d.Who, want, d.ID)
}
return broker.AgainSubject(d.Consumer, d.Subject), nil
}
// DeliverAgain hands a kept message to the consumer that gave it up, and nobody else, then removes it
// from DEAD_LETTERS. The message carries its own headers and AgainHeader; its de-duplication id is the
// kept copy's, so asking twice delivers it once.
func DeliverAgain(js nats.JetStreamContext, id uint64) (DeadLetter, string, error) {
d, err := DeadLetterNamed(js, id)
if err != nil {
return d, "", err
}
to, err := AgainTo(js, d)
if err != nil {
return d, "", err
}
again := &nats.Msg{Subject: to, Header: nats.Header{}, Data: []byte(d.Body)}
for k, v := range d.Headers {
again.Header[k] = append([]string(nil), v...)
}
again.Header.Set(AgainHeader, strconv.FormatUint(id, 10))
again.Header.Set(nats.MsgIdHdr, "again."+broker.DeadLettersStream+"."+strconv.FormatUint(id, 10))
if _, err := js.PublishMsg(again); err != nil {
return d, to, fmt.Errorf("dead letter %d could not be delivered again on %s: %w; it is still kept", id, to, err)
}
if err := js.DeleteMsg(broker.DeadLettersStream, id); err != nil && !errors.Is(err, nats.ErrMsgNotFound) {
return d, to, fmt.Errorf("dead letter %d was delivered again on %s and could not be removed from %s: %w",
id, to, broker.DeadLettersStream, err)
}
return d, to, nil
}
// DropDeadLetter removes a kept message for good.
func DropDeadLetter(js nats.JetStreamContext, id uint64) (DeadLetter, error) {
d, err := DeadLetterNamed(js, id)
if err != nil {
return d, err
}
if err := js.DeleteMsg(broker.DeadLettersStream, id); err != nil {
return d, fmt.Errorf("dead letter %d could not be dropped: %w", id, err)
}
return d, nil
}
// noticeRetry is how long a notice whose message could not be kept waits before it is taken again.
var noticeRetry = time.Minute
// keepingGivenUp keeps taking the notices until ctx ends; the returned function stops it. A subscription
// that cannot be made is said — in the log, and as the max-deliveries condition of the notices themselves
// — and tried again every noticeRetry, so it never stops the controller serving.
func keepingGivenUp(ctx context.Context, bus *broker.JetStream, logger *log.Logger) func() {
ctx, cancel := context.WithCancel(ctx)
done := make(chan struct{})
go func() {
defer close(done)
for {
sub, err := func() (*nats.Subscription, error) {
if err := bus.EnsureConsumer(broker.NoticesConsumer()); err != nil {
return nil, err
}
return keepGivenUp(bus.Context(), logger)
}()
if err == nil {
<-ctx.Done()
_ = sub.Unsubscribe()
return
}
logger.Printf("the notices of messages consumers gave up on cannot be taken, so none is kept until "+
"they can: %v", err)
Advisories.Heard(Advisory{Kind: AdvisoryMaxDeliveries, ID: broker.DeadLetterNoticesStream + "." +
broker.ControllerName, Stream: broker.DeadLetterNoticesStream, Consumer: broker.ControllerName,
Token: AdvisoryNotKept, Said: "the controller cannot take the notices of messages consumers gave " +
"up on, so none is kept: " + err.Error()}, time.Now())
select {
case <-ctx.Done():
return
case <-time.After(noticeRetry):
}
}
}()
return func() {
cancel()
<-done
}
}
// keepGivenUp takes the server's maximum-deliveries notices off their stream and keeps the message
// each is about, for as long as the subscription stands. A notice that could not be kept is offered
// again after noticeRetry, and said: in the log, and as the consumer's max-deliveries condition.
func keepGivenUp(js nats.JetStreamContext, logger *log.Logger) (*nats.Subscription, error) {
return js.Subscribe("", func(m *nats.Msg) {
d, err := KeepDeadLetter(js, m.Data)
if err != nil {
logger.Printf("a message a consumer gave up on could NOT be kept: %v", err)
var a maxDeliveries
if json.Unmarshal(m.Data, &a) == nil && a.Stream != "" && a.Consumer != "" {
Advisories.Heard(Advisory{Kind: AdvisoryMaxDeliveries, ID: a.Stream + "." + a.Consumer,
Stream: a.Stream, Consumer: a.Consumer, Token: AdvisoryNotKept,
Said: fmt.Sprintf("%s gave up on message %d, and it could not be kept: %v",
ConsumerInWords(a.Stream, a.Consumer), a.StreamSeq, err)}, time.Now())
}
_ = m.NakWithDelay(noticeRetry)
return
}
if d.Lost != "" {
logger.Printf("%s gave up on message %d of %s, which its stream no longer held; kept as dead letter %d "+
"without it", d.Who, d.Sequence, d.Stream, d.ID)
} else {
logger.Printf("%s gave up on message %d of %s (%s) after %d deliveries; kept as dead letter %d",
d.Who, d.Sequence, d.Stream, d.Subject, d.Deliveries, d.ID)
}
_ = m.Ack()
}, nats.Bind(broker.DeadLetterNoticesStream, broker.ControllerName), nats.ManualAck())
}
+336
View File
@@ -0,0 +1,336 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
)
// A module's consumer as the controller makes it, on a bus whose streams the controller asserted.
func aModuleConsumer(t *testing.T, js *broker.JetStream, node, module string, consumes ...string) jetstream.Consumer {
t.Helper()
c, ok := broker.ConsumerFor(broker.Principal{Kind: broker.KindModule, Node: node, Module: module,
Consumes: consumes, PasswordHash: "x"})
if !ok {
t.Fatal("a module that consumes got no consumer")
}
if err := js.EnsureConsumer(c); err != nil {
t.Fatal(err)
}
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
reading, err := api.Consumer(t.Context(), broker.EventsStream, c.Name)
if err != nil {
t.Fatal(err)
}
return reading
}
// next is the one message a consumer hands over within a second, or nil.
func next(t *testing.T, c jetstream.Consumer) jetstream.Msg {
t.Helper()
batch, err := c.Fetch(1, jetstream.FetchMaxWait(time.Second))
if err != nil {
t.Fatal(err)
}
for msg := range batch.Messages() {
return msg
}
return nil
}
// keeping takes the notices as the serving controller does, until the test ends.
func keeping(t *testing.T, js *broker.JetStream) {
t.Helper()
if err := broker.AssertMeshConsumers(js); err != nil {
t.Fatal(err)
}
sub, err := keepGivenUp(js.Context(), log.New(io.Discard, "", 0))
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
}
// givenUp hands one event to a consumer as often as it may, failing each time, and waits for it kept.
func givenUp(t *testing.T, js *broker.JetStream, c jetstream.Consumer) DeadLetter {
t.Helper()
for {
msg := next(t, c)
if msg == nil {
break
}
_ = msg.Nak()
}
var kept []DeadLetter
eventually(t, "the event kept", func() bool {
kept, _, _ = DeadLetters(js.Context(), c.CachedInfo().Name, 0)
return len(kept) == 1
})
return kept[0]
}
// **Delivered again to the consumer that gave it up, and nobody else**: another module consuming the
// same event handled it the first time and must not handle it twice.
func TestADeadLetterIsDeliveredAgainToItsConsumerAlone(t *testing.T) {
js := aBus(t)
keeping(t, js)
failing := aModuleConsumer(t, js, "media", "sonarr", "gitea.pull.merged")
handling := aModuleConsumer(t, js, "media", "radarr", "gitea.pull.merged")
const subject = "mesh.mod.gitea.event.pull.merged"
msg := &nats.Msg{Subject: subject, Data: []byte(`{"n":1}`), Header: nats.Header{}}
msg.Header.Set("x-node", "forge")
msg.Header.Set(nats.MsgIdHdr, "event-1")
if _, err := js.Context().PublishMsg(msg); err != nil {
t.Fatal(err)
}
if m := next(t, handling); m == nil {
t.Fatal("the other module was not handed the event")
} else {
_ = m.Ack()
}
d := givenUp(t, js, failing)
if d.Consumer != "media_sonarr" || d.Stream != "EVENTS" || d.Subject != subject || d.Deliveries != 5 ||
d.GaveUp.IsZero() || d.Published.IsZero() || d.Lost != "" {
t.Fatalf("kept as %+v", d)
}
held, err := HeldDeadLetters(js.Context())
if err != nil || held["EVENTS.media_sonarr"] != 1 {
t.Fatalf("held %v (%v)", held, err)
}
whole, err := DeadLetterNamed(js.Context(), d.ID)
if err != nil || whole.Body != `{"n":1}` || len(whole.Headers["x-node"]) != 1 || whole.Headers["x-node"][0] != "forge" {
t.Fatalf("one asked whole is %+v (%v)", whole, err)
}
_, to, err := DeliverAgain(js.Context(), d.ID)
if err != nil {
t.Fatal(err)
}
if to != "mesh.again.media_sonarr.mod.gitea.event.pull.merged" {
t.Fatalf("delivered again on %s", to)
}
again := next(t, failing)
if again == nil {
t.Fatal("the consumer that gave it up was not handed it again")
}
if string(again.Data()) != `{"n":1}` || again.Headers().Get("x-node") != "forge" ||
again.Headers().Get(AgainHeader) == "" {
t.Fatalf("handed again as %s %v", again.Data(), again.Headers())
}
if original, ok := broker.OriginalOfAgain(again.Subject()); !ok || original != subject {
t.Fatalf("delivered again on %s, which does not say the event's own subject", again.Subject())
}
_ = again.Ack()
if m := next(t, handling); m != nil {
t.Fatalf("the module that handled it was handed it twice: %s", m.Subject())
}
if _, err := DeadLetterNamed(js.Context(), d.ID); !errors.Is(err, ErrNoDeadLetter) {
t.Fatalf("still kept after it was delivered again: %v", err)
}
held, _ = HeldDeadLetters(js.Context())
if held["EVENTS.media_sonarr"] != 0 {
t.Fatalf("still held: %v", held)
}
// Asked twice, delivered once: the second finds nothing kept.
if _, _, err := DeliverAgain(js.Context(), d.ID); !errors.Is(err, ErrNoDeadLetter) {
t.Fatalf("a second delivery answered %v", err)
}
}
func TestADeadLetterDroppedIsGone(t *testing.T) {
js := aBus(t)
keeping(t, js)
failing := aModuleConsumer(t, js, "media", "sonarr", "gitea.pull.merged")
if _, err := js.Context().Publish("mesh.mod.gitea.event.pull.merged", []byte(`{}`)); err != nil {
t.Fatal(err)
}
d := givenUp(t, js, failing)
if _, err := DropDeadLetter(js.Context(), d.ID); err != nil {
t.Fatal(err)
}
if left, total, err := DeadLetters(js.Context(), "", 0); err != nil || total != 0 || len(left) != 0 {
t.Fatalf("after the drop: %v %d %v", left, total, err)
}
if m := next(t, failing); m != nil {
t.Fatalf("a dropped event was handed over: %s", m.Subject())
}
}
// A notice whose message its stream no longer holds is kept as a record that says so — said, and
// dropped by a person, never silently missing — and cannot be delivered again.
func TestANoticeWhoseMessageIsGoneIsKeptAndSaysSo(t *testing.T) {
js := aBus(t)
d, err := KeepDeadLetter(js.Context(), []byte(`{"stream":"EVENTS","consumer":"media_sonarr","stream_seq":42,`+
`"deliveries":5,"timestamp":"2026-10-06T15:51:00Z"}`))
if err != nil {
t.Fatal(err)
}
if !strings.Contains(d.Lost, "no longer held message 42") || d.Consumer != "media_sonarr" {
t.Fatalf("kept as %+v", d)
}
if _, _, err := DeliverAgain(js.Context(), d.ID); err == nil || !strings.Contains(err.Error(), "Drop it") {
t.Fatalf("a record without its message was delivered again: %v", err)
}
// The same notice taken twice is kept once.
if _, err := KeepDeadLetter(js.Context(), []byte(`{"stream":"EVENTS","consumer":"media_sonarr","stream_seq":42,`+
`"deliveries":5}`)); err != nil {
t.Fatal(err)
}
if _, total, _ := DeadLetters(js.Context(), "media_sonarr", 0); total != 1 {
t.Fatalf("one notice taken twice is kept %d times", total)
}
}
// What only the giving-up consumer filters is its own: a subject delivered again reads back as the
// event's, and a module's runtime reads the same key from it (the tokens around `.event.`).
func TestASubjectDeliveredAgainSaysTheEventsOwn(t *testing.T) {
for _, original := range []string{"mesh.mod.gitea.event.pull.merged", "mesh.seat.node-build-agent.event.built"} {
again := broker.AgainSubject("ace_sonarr", original)
if back, ok := broker.OriginalOfAgain(again); !ok || back != original {
t.Errorf("%s reads back as %s", again, back)
}
key := func(subject string) string {
before, event, _ := strings.Cut(subject, ".event.")
parts := strings.Split(before, ".")
return parts[len(parts)-1] + "." + event
}
if key(again) != key(original) {
t.Errorf("%s reads as key %s, the event as %s", again, key(again), key(original))
}
}
if _, ok := broker.OriginalOfAgain("mesh.mod.gitea.event.pull.merged"); ok {
t.Error("an event's own subject reads as delivered again")
}
}
// Only an event is delivered again, and only to a consumer that is on the bus and filters its again
// subject: a dead letter is never let go as delivered while nobody receives it.
func TestADeadLetterIsDeliveredAgainOnlyWhereItIsReceived(t *testing.T) {
js := aBus(t)
keeping(t, js)
failing := aModuleConsumer(t, js, "media", "sonarr", "gitea.pull.merged")
if _, err := js.Context().Publish("mesh.mod.gitea.event.pull.merged", []byte(`{}`)); err != nil {
t.Fatal(err)
}
d := givenUp(t, js, failing)
// A consumer as it was before this fix: its filters do not take what is delivered again.
c, _ := broker.ConsumerFor(broker.Principal{Kind: broker.KindModule, Node: "media", Module: "sonarr",
Consumes: []string{"gitea.pull.merged"}, PasswordHash: "x"})
c.Filters = c.Filters[:len(c.Filters)-1]
if err := js.EnsureConsumer(c); err != nil {
t.Fatal(err)
}
if _, _, err := DeliverAgain(js.Context(), d.ID); err == nil || !strings.Contains(err.Error(), "does not yet take") {
t.Fatalf("delivered to a consumer that does not filter its again subject: %v", err)
}
if _, err := DeadLetterNamed(js.Context(), d.ID); err != nil {
t.Fatalf("a refused delivery let the dead letter go: %v", err)
}
// The consumer gone: nothing would receive it.
if err := js.Context().DeleteConsumer(broker.EventsStream, c.Name); err != nil {
t.Fatal(err)
}
if _, _, err := DeliverAgain(js.Context(), d.ID); err == nil || !strings.Contains(err.Error(), "no longer on the bus") {
t.Fatalf("delivered to a consumer that is gone: %v", err)
}
if _, err := DeadLetterNamed(js.Context(), d.ID); err != nil {
t.Fatalf("a refused delivery let the dead letter go: %v", err)
}
// Not an event: refused, whatever stream it is from.
for _, other := range []DeadLetter{
{ID: 2, Stream: "SEAT_TELEGRAM_SENDER", Consumer: "SEAT_TELEGRAM_SENDER_worker", Subject: "mesh.seat.telegram-sender.accept.send"},
{ID: 3, Stream: "KV_x", Consumer: "y", Subject: "$KV.x.k"},
} {
if _, err := AgainTo(js.Context(), other); err == nil {
t.Errorf("%s was given a subject to be delivered again on", other.Stream)
}
}
}
// A total says how many are held, however few the list carries.
func TestTheListSaysTheTotalAndStopsAtItsLimit(t *testing.T) {
js := aBus(t)
for seq := 1; seq <= 5; seq++ {
if _, err := KeepDeadLetter(js.Context(), []byte(fmt.Sprintf(`{"stream":"EVENTS","consumer":"media_sonarr",`+
`"stream_seq":%d,"deliveries":5}`, seq))); err != nil {
t.Fatal(err)
}
}
list, total, err := DeadLetters(js.Context(), "media_sonarr", 2)
if err != nil || total != 5 || len(list) != 2 || list[0].Sequence != 5 {
t.Fatalf("%d of %d (%v): %+v", len(list), total, err, list)
}
if _, total, _ := DeadLetters(js.Context(), "another_one", 2); total != 0 {
t.Fatalf("another consumer's total is %d", total)
}
}
// An event the controller itself gave up on, delivered again by a person, is acted on as the event it
// was: it arrives under the controller's again subject, which its consumer filters.
func TestTheControllerActsOnAnEventDeliveredAgain(t *testing.T) {
js := aBus(t)
told := &toldAbout{}
s := &Server{inbound: Nats(js), bus: OverNATS{Conn: js.Conn(), JS: js.Context()}, log: quiet()}
if err := s.Follows(told); err != nil {
t.Fatal(err)
}
if err := s.Answers(replaysWith{}); err != nil {
t.Fatal(err)
}
ctx, stop := context.WithCancel(context.Background())
defer stop()
go func() { _ = s.Serve(ctx) }()
eventually(t, "the controller's event consumer being made", func() bool {
_, err := js.Context().ConsumerInfo("EVENTS", broker.ControllerName)
return err == nil
})
moved, _ := json.Marshal(Upgraded{Module: "gitea", Commit: "abcdef0123"})
if _, err := js.Context().Publish(broker.AgainSubject(broker.ControllerName, broker.ControllerFollows[0]), moved); err != nil {
t.Fatal(err)
}
eventually(t, "the upgrade delivered again reaching the controller", func() bool { return told.count() == 1 })
}
// The notices cannot be taken (their stream is gone): said as a condition, and the controller serves on.
func TestNoticesThatCannotBeTakenAreSaidAndServingGoesOn(t *testing.T) {
js := aBus(t)
if err := js.Context().DeleteStream(broker.DeadLetterNoticesStream); err != nil {
t.Fatal(err)
}
stop := keepingGivenUp(context.Background(), js, quiet())
defer stop()
eventually(t, "the failure said", func() bool {
for _, a := range Advisories.Since(time.Now().Add(-time.Minute)) {
if a.Token == AdvisoryNotKept && a.Stream == broker.DeadLetterNoticesStream {
return true
}
}
return false
})
held := &counted{}
_, stopServing := servingOn(t, js, held)
defer stopServing()
if _, err := js.Context().Publish("mesh.control.anchor.report", []byte(`{"node":"anchor"}`)); err != nil {
t.Fatal(err)
}
eventually(t, "a report heard while the notices cannot be taken", func() bool { return held.count() == 1 })
}
+18
View File
@@ -420,6 +420,20 @@ const LivenessContract = 1
// because an older one parses strictly and would refuse the whole declaration for it.
const ReadinessContract = 2
// RootContract is the statement of an engine that also judges a user's declared `root` (novox/hq ADR 0266):
// whether an account declared never to become root without a person can — uid 0, a group that grants root,
// a sudo rule, a secret of the mesh it may read. Only to such an engine is the field sent: an older one
// parses strictly and would refuse the whole declaration for it.
const RootContract = 3
// ReasonRoot starts the reason of an account verdict that found a way to root (ADR 0266); the node-engine's
// own words (mesh-host internal/accounts ReasonRoot).
const ReasonRoot = "can become root without a person"
// RootNever is the value of a user's `root`, and of a verdict's Root, that the account must never become
// root without a person (ADR 0266).
const RootNever = "never"
// Health is one statement of a machine's long-running resources (to-be 48 §4): in every report, as the
// event HealthSubject between reports on each change, and again every minute while one is not healthy.
// The node-engine's own (mesh-host internal/link Health); a test on each side holds the field names.
@@ -542,6 +556,10 @@ type ResourceHealth struct {
// manager, and the account itself for a resource of kind KindAccount (novox/hq ADR 0254). Empty from an
// engine older than that, and for anything the machine's own manager or runtime runs.
Account string `json:"account,omitempty"`
// Root is "never" on a verdict of kind KindAccount whose account is declared never to become root
// without a person (novox/hq ADR 0266): the engine judged that too, and a healthy verdict says it cannot.
// Empty from an engine older than RootContract, and on every other verdict.
Root string `json:"root,omitempty"`
}
// HealthSaid is the health event's body: the machine and its statement. The machine is read from the
+22 -1
View File
@@ -41,6 +41,15 @@ type natsInbound struct {
// that restarts loses these and starts the window again, which is correct — it is holding
// nothing, and the messages are all still on the server.
since map[uint64]time.Time
// log is where it says what it could not do; the standard logger when nobody gave one.
log *log.Logger
}
func (n *natsInbound) logger() *log.Logger {
if n.log != nil {
return n.log
}
return log.Default()
}
// Nats is the consume side of the bus being built.
@@ -70,7 +79,7 @@ func (n *natsInbound) Close() {}
// which message is held, and since when — is read and written without a lock because the AMQP loop
// never had two. A second goroutine would make that wrong in a way no test would catch.
func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Control)) error {
if err := broker.AssertMeshConsumers(n.js); err != nil {
if err := broker.AssertServingConsumers(n.js); err != nil {
return err
}
js, conn := n.js.Context(), n.js.Conn()
@@ -94,6 +103,13 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
holding.Store(true)
defer holding.Store(false)
// What a consumer gave up on, kept by the controller acting now (novox/hq issue 330): the server's
// notices wait in their stream for it, so one said while no controller listened is not lost.
// **Never the reason the controller stops serving**: a notice it cannot take yet waits in its stream,
// and the failure is said and tried again every minute.
stopKeeping := keepingGivenUp(ctx, n.js, n.logger())
defer stopKeeping()
// Heartbeats, on core NATS and off any stream (design 25 §3). Their own subscription because
// they are their own guarantee: a lost one is the next one.
beats := make(chan *nats.Msg, Prefetch)
@@ -167,6 +183,11 @@ func (n *natsInbound) Receive(ctx context.Context, act func(context.Context, Con
func (n *natsInbound) deliver(ctx context.Context, act func(context.Context, Control),
msg *nats.Msg, streamed bool) {
// An event the controller gave up on and a person delivered again (novox/hq issue 330) arrives under
// the controller's own again subject, and is the event it was: acted on as first published.
if original, ok := broker.OriginalOfAgain(msg.Subject); ok {
msg.Subject = original
}
kind, known := kindOfSubject(msg.Subject)
if !known {
if streamed {
+93
View File
@@ -0,0 +1,93 @@
package link
import (
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
)
// novox/hq issue 330, replayed with only what the link had before its fix, so it can be laid over the
// older commit. On 2026-10-06 a media server's event consumer gave up on several messages after five
// deliveries each. The controller raised a condition for each run and cleared it within minutes; the
// messages were kept only by EVENTS, which drops an event after a week, and nothing said which they
// were. Design 25 promised a dead-letter stream that does not exist. A consumer that never acknowledges
// an event: once it gives up, the event is kept with its consumer and how often it was handed over.
func TestReplay330(t *testing.T) {
js := aBus(t)
consumer, ok := broker.ConsumerFor(broker.Principal{Kind: broker.KindModule, Node: "media", Module: "sonarr",
Consumes: []string{"gitea.pull.merged"}, PasswordHash: "x"})
if !ok {
t.Fatal("a module that consumes got no consumer")
}
if err := js.EnsureConsumer(consumer); err != nil {
t.Fatal(err)
}
_, stop := servingOn(t, js, &counted{})
defer stop()
const subject = "mesh.mod.gitea.event.pull.merged"
if _, err := js.Context().Publish(subject, []byte(`{"repository":"novox/media"}`)); err != nil {
t.Fatal(err)
}
// The module's handler fails every time, as the media server's did.
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
reading, err := api.Consumer(t.Context(), broker.EventsStream, consumer.Name)
if err != nil {
t.Fatal(err)
}
for handed := 0; handed < consumer.MaxDeliver; handed++ {
batch, err := reading.Fetch(1, jetstream.FetchMaxWait(5*time.Second))
if err != nil {
t.Fatal(err)
}
n := 0
for msg := range batch.Messages() {
n++
_ = msg.Nak()
}
if n != 1 {
t.Fatalf("handed over %d times, then nothing: %v", handed, batch.Error())
}
}
// And it goes on reading, as a module's runtime does: the server gives the event up when it would
// hand it over a sixth time.
if batch, err := reading.Fetch(1, jetstream.FetchMaxWait(time.Second)); err == nil {
for msg := range batch.Messages() {
t.Fatalf("handed over a sixth time: %s", msg.Subject())
}
}
kept := "mesh.events.dead." + broker.EventsStream + "." + consumer.Name
var held *nats.RawStreamMsg
deadline := time.Now().Add(10 * time.Second)
for time.Now().Before(deadline) && held == nil {
if stream, err := js.Context().StreamNameBySubject(kept); err == nil {
held, _ = js.Context().GetLastMsg(stream, kept)
}
if held == nil {
time.Sleep(50 * time.Millisecond)
}
}
if held == nil {
t.Fatalf("%s gave up on the event and nothing on the bus keeps it under %s", consumer.Name, kept)
}
if string(held.Data) != `{"repository":"novox/media"}` {
t.Errorf("kept %q, not the event", held.Data)
}
for header, want := range map[string]string{"Mesh-Dead-Consumer": consumer.Name, "Mesh-Dead-Stream": "EVENTS",
"Mesh-Dead-Subject": subject, "Mesh-Dead-Deliveries": "5"} {
if got := held.Header.Get(header); got != want {
t.Errorf("the kept event's %s is %q, not %q", header, got, want)
}
}
if held.Header.Get("Mesh-Dead-Gave-Up") == "" {
t.Error("the kept event does not say when it was given up")
}
}
+1
View File
@@ -71,6 +71,7 @@
"bus",
"retire",
"cleanup",
"dead-letters",
"data",
"build",
"artifacts",
+20
View File
@@ -383,8 +383,25 @@ type User struct {
// has it not. On the account rather than on the unit, because it is the account's: two units of
// one account cannot disagree about it, and undeclaring one of them must not stop the other.
Linger *bool `json:"linger,omitempty"`
// Root says whether this account may become root without a person (novox/hq ADR 0266). "never"
// is the agents' own account: the login an agent session runs as on a machine where it must not
// reach root by itself. Empty asserts nothing, as Shell's does.
//
// **A statement the engine judges, never one it acts on.** The apply gives an account it creates
// no password, no sudo rule and no group beyond those declared, as it always has, and takes none
// away from one it finds: a sudo rule or a group granted by hand is a person's to remove, and a
// declaration that silently stripped them would be the mesh deciding what a person's machine
// grants. What "never" adds is the look: on every look the engine reads whether the account can
// become root by itself — by its uid, a group that grants root, any sudo rule, or a secret the mesh
// placed that it can read — and says it unhealthy while it can (internal/accounts), so the
// controller can tell a machine where it holds from one where it does not.
Root string `json:"root,omitempty"`
}
// RootNever is the one value Root takes besides empty: the account never becomes root without a person.
const RootNever = "never"
// Network is a named network on this machine.
//
// **A name and nothing else.** Not a driver, a subnet or a gateway: each of those is something a
@@ -478,6 +495,9 @@ func (u *User) validate(where string, _ bool) []string {
if u.Home != "" && !strings.HasPrefix(u.Home, "/") {
problems = append(problems, where+": a home directory is an absolute path")
}
if u.Root != "" && u.Root != RootNever {
problems = append(problems, fmt.Sprintf("%s: root is %q or absent, and %q is neither", where, RootNever, u.Root))
}
return problems
}
+2 -2
View File
@@ -75,7 +75,7 @@ github.com/nats-io/nkeys
# github.com/nats-io/nuid v1.0.1
## explicit
github.com/nats-io/nuid
# github.com/novox/mesh-host v0.0.0 => git.novox.be/novox/mesh-host v0.0.0-20261007162834-56e2ebec4bac
# github.com/novox/mesh-host v0.0.0 => git.novox.be/novox/mesh-host v0.0.0-20261008185244-76f3ca12b8b8
## explicit; go 1.26.0
github.com/novox/mesh-host/internal/declaration
github.com/novox/mesh-host/validate
@@ -133,4 +133,4 @@ golang.org/x/text/width
# golang.org/x/time v0.15.0
## explicit; go 1.25.0
golang.org/x/time/rate
# github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261007162834-56e2ebec4bac
# github.com/novox/mesh-host => git.novox.be/novox/mesh-host v0.0.0-20261008185244-76f3ca12b8b8