novox/mesh-host
18
0
Fork 0
Code Issues Pull Requests 1 Releases Wiki Activity

Compare commits

merge into: novox/mesh-host:fix/1945-adopt-hashes
Branches Tags
novox/mesh-host:main novox/mesh-host:feat/774-l3-reland novox/mesh-host:revert/774-l3 novox/mesh-host:fix/1126-924-held-and-judged novox/mesh-host:fix/924-judge-by-have novox/mesh-host:fix/1126-fold-watch novox/mesh-host:fix/1945-adopt-hashes novox/mesh-host:feat/r4g-finish-1935 novox/mesh-host:fix/1695-bucket-per-node novox/mesh-host:replay/r92-before novox/mesh-host:fix/567-an-access-is-judged-by-who-writes-it novox/mesh-host:fix/inspect-by-kind-not-the-ambiguous-bare-form
..
pull from: novox/mesh-host:replay/r92-before
Branches Tags
novox/mesh-host:feat/774-l3-reland novox/mesh-host:main novox/mesh-host:revert/774-l3 novox/mesh-host:fix/1126-924-held-and-judged novox/mesh-host:fix/924-judge-by-have novox/mesh-host:fix/1126-fold-watch novox/mesh-host:fix/1945-adopt-hashes novox/mesh-host:feat/r4g-finish-1935 novox/mesh-host:fix/1695-bucket-per-node novox/mesh-host:replay/r92-before novox/mesh-host:fix/567-an-access-is-judged-by-who-writes-it novox/mesh-host:fix/inspect-by-kind-not-the-ambiguous-bare-form
1 Commits
fix/1945-adopt-hashes .. replay/r92-before
Author SHA1 Message Date
jochen 9e233e1d87 Replay R92's commit before its fix: mesh-host main without genesis's registry trust (#91 reverted) 
Never to be merged. The replay of hq issue 92 runs genesis at the commit before its fix, and the fix's
own first parent stops at step 6 on issue 146; this is today's genesis without #91, so the bed
reaches the push it replays (mesh-lab replays/register.go R92).
 2026-10-13 02:57:43 +02:00
275 changed files with 9180 additions and 18307 deletions
Expand all files Collapse all files
README.md
+1 -14
View File
@@ -102,19 +102,6 @@ what it did not understand would apply most of a declaration and report success.
]}
```
**A directory or archive says how it treats files it did not write** (issue 1024,
[`novox/hq` ADR 0422](https://git.novox.be/novox/hq)). `unknown` is `move-out`, `module` or `program`;
absent, nothing about unknown paths is done. `knows` lists paths inside it, relative to its `path`, that
are not declared but belong there; a trailing `/` names a folder and everything in it. `knows` without
`unknown` is refused. For `move-out` the node-engine, after every apply and so on every reconcile, moves
each path inside the directory that nothing declared, no archive unpacked and `knows` does not name,
whole, into `<home>/.local/state/mesh-moved/<module>/<UTC time>/` beside a `<UTC time>.json` list. The
list names each move before it is made. Nothing is deleted. A path younger than a minute waits for the
next look; a socket, pipe or device is left and said. It is only for a directory under an account's
home on one file system with the kept folder, and otherwise the resource fails and nothing moves.
`module` and `program` are read by the module and the program that owns the path, and the node-engine
does nothing for them.
**It converges rather than executes.** Applying twice changes nothing the second time; applying
to a drifted machine returns it. A mode is *maintained*, not merely set — a permission applied
at creation is not a permission held.
@@ -257,7 +244,7 @@ without its neighbour checked out is a repository nobody can build.
```
mesh-controller: ./build/mesh-controller plan <node> --json > /tmp/d.json
mesh-sdk: MESH_EMITTED=/tmp/d.json go test ./declaration/ -v (run in mesh-sdk's go/ directory)
mesh-host: MESH_EMITTED=/tmp/d.json go test ./internal/declaration/ -v
```
**What this node says when it joins, read by the mesh:**
RELEASE-57.md
-6
View File
@@ -1,6 +0,0 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
cmd/mesh-bootstrap/main.go
-6
View File
@@ -116,8 +116,6 @@ const usage = `mesh-bootstrap — make a bare machine into a mesh
--tools-source the repository the shared base is built from
--tools-ref what of it to build (default main)
--tools-go-source the repository the Go toolchain is built from (default: mesh-tools-go beside --tools-source)
--tools-go-ref what of it to build (default main)
--catalog-source the catalogue REPOSITORY, for building its modules;
--catalog is the checkout that says what they are
--catalog-ref what of it to build (default main)
@@ -310,10 +308,6 @@ func newFlagSet(opts *bootstrap.Options, jsonOut *bool) *flag.FlagSet {
"the repository the shared base is built from")
set.StringVar(&opts.ToolsSource.Ref, "tools-ref", opts.ToolsSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.ToolsGoSource.Repository, "tools-go-source", opts.ToolsGoSource.Repository,
"the repository the Go toolchain is built from (default: mesh-tools-go beside --tools-source)")
set.StringVar(&opts.ToolsGoSource.Ref, "tools-go-ref", opts.ToolsGoSource.Ref,
"what of it to build (default main)")
set.StringVar(&opts.CatalogSource.Repository, "catalog-source", opts.CatalogSource.Repository,
"the catalogue REPOSITORY, for building its modules — --catalog is the checkout that says what they are")
set.StringVar(&opts.CatalogSource.Ref, "catalog-ref", opts.CatalogSource.Ref,
cmd/mesh-host/apply_context_test.go
-39
View File
@@ -1,39 +0,0 @@
package main
import (
"context"
"testing"
"time"
)
// An apply runs in a context the bus's loss cannot end: the link's context ends when the bus goes, and the bus may go
// because of the apply itself (it replaces the bus's own container), which then must finish (issue 1945, live on
// 2026-10-25 at 11:23: novox's node-engine removed the bus container, its link's context ended, and the docker run that
// would have made the new one was never made).
func TestAnApplyOutlivesTheLinksContext(t *testing.T) {
type key struct{}
link, cancel := context.WithCancel(context.WithValue(context.Background(), key{}, "v"))
ctx, done := applyContext(link)
defer done()
cancel()
if ctx.Err() != nil {
t.Fatalf("the apply's context ended with the link's: %v", ctx.Err())
}
if ctx.Value(key{}) != "v" {
t.Fatal("the apply's context lost the link's values")
}
}
// An apply is bounded all the same: a step that hangs (a docker call that never returns) cannot hold the node agent for
// ever (the review of issue 1945).
func TestAnApplyIsBounded(t *testing.T) {
ctx, done := applyContext(context.Background())
defer done()
at, ok := ctx.Deadline()
if !ok {
t.Fatal("an apply has no bound")
}
if left := time.Until(at); left > applyBound || left < applyBound-time.Minute {
t.Fatalf("the bound is %s away, want about %s", left, applyBound)
}
}
cmd/mesh-host/applywhole_test.go
-41
View File
@@ -1,41 +0,0 @@
package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// applyAndKeep applies a whole declaration the way the carried bundle is applied, and says what came of it as the facts
// the node says: what the tests of the apply's steps read. The node-engine applies no whole declaration on its own now
// (novox/hq ADR 0539 §9); this is the test's way to put a machine in a state.
func applyAndKeep(ctx context.Context, opts options, raw []byte, signed *store.Declared, sched *apply.Scheduler,
say link.Announce) link.Report {
declared, err := declaration.Parse(raw)
if err != nil {
return link.Report{Refused: err.Error()}
}
got := applyModuleHeld(ctx, opts, declared, nil, sched, say)
if got.Refused != "" {
return link.Report{Refused: got.Refused}
}
held := map[string]bool{}
for _, h := range got.Updated.Held {
held[h.ID] = true
}
report := factsOf(ctx, opts, declared, got.Updated, got.Outcome, held)
if signed != nil {
if err := store.SaveDeclared(store.DeclaredPath(opts.state), *signed); err != nil {
report.Failed = map[string]string{"keeping the declaration": err.Error()}
}
}
if got.Err != nil {
report.Failed = map[string]string{"apply": got.Err.Error()}
}
report.Failed = withUnanswered(report.Failed, got.Probe)
report.Unapplied = unapplied(declared, got.Outcome, got.Err)
return report
}
cmd/mesh-host/backup_event_test.go
-42
View File
@@ -1,42 +0,0 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
)
func backupEvent(node, module string, ok bool, at time.Time, why string) []byte {
raw, _ := json.Marshal(map[string]any{"node": node, "module": module, "ok": ok, "at": at, "error": why})
return raw
}
// A backup is done when the holder says so for this node and module, held on its words when it says it failed, and held
// when it says nothing within the bound; another node's or module's event, or one from before the ask, is not the answer
// (R6-A A5, novox/hq ADR 0429 §14).
func TestABackupEndsWhenItsEventSaysSo(t *testing.T) {
since := time.Now()
run := func(within time.Duration, events ...[]byte) (done bool, words string) {
ch := make(chan []byte, len(events))
for _, e := range events {
ch <- e
}
got := awaitBackup(context.Background(), ch, "n1", "db", since, within)
return got.Done, got.Words
}
if done, _ := run(time.Second,
backupEvent("n2", "db", true, since, ""), backupEvent("n1", "web", true, since, ""),
backupEvent("n1", "db", false, since.Add(-time.Hour), "an old night"),
backupEvent("n1", "db", true, since.Add(time.Second), "")); !done {
t.Error("the event of this node and module was not the answer")
}
if done, words := run(time.Second, backupEvent("n1", "db", false, since, "the repository is locked")); done ||
!strings.Contains(words, "repository is locked") {
t.Errorf("a failed backup: done=%v words=%q", done, words)
}
if done, words := run(50 * time.Millisecond); done || !strings.Contains(words, "did not say it was done") {
t.Errorf("silence within the bound: done=%v words=%q", done, words)
}
}
cmd/mesh-host/backup_nothing_test.go
-29
View File
@@ -1,29 +0,0 @@
package main
import (
"encoding/json"
"testing"
)
// A module that declares nothing to back up on this machine is not backed up before it moves, and so never held for a
// backup that cannot run (issue 1945, live on 2026-10-25: redis and searxng on ace were held: "redis declares nothing to
// back up on this machine"). What node-backup.backed-up answers decides it.
func TestAModuleWithNothingToBackUpIsNotBackedUp(t *testing.T) {
for _, c := range []struct {
said string
nothing bool
}{
{`[{"module":"postgres","runs":1,"paths":["/var/lib/postgres"]}]`, true},
{`[{"module":"redis","runs":0,"paths":null,"data":[{"item":"cache"}]}]`, true},
{`[]`, true},
{`[{"module":"redis","runs":1,"paths":["/var/lib/redis"]}]`, false},
} {
nothing, ok := declaresNothingToBackUp(json.RawMessage(c.said), "redis")
if !ok || nothing != c.nothing {
t.Errorf("%s: nothing %v (read %v), want %v", c.said, nothing, ok, c.nothing)
}
}
if _, ok := declaresNothingToBackUp(json.RawMessage(`{"error":"x"}`), "redis"); ok {
t.Error("an answer that is not the list was read")
}
}
cmd/mesh-host/busy_test.go
-34
View File
@@ -1,34 +0,0 @@
package main
import (
"context"
"sync/atomic"
"testing"
"time"
)
// While a container waits, a pass is asked for every so often, not at some long interval; with nothing waiting, none
// (novox/hq ADR 0335 §2, ADR 0360 §11).
func TestAWaitIsAskedAgainSoon(t *testing.T) {
if BusyAskEvery > time.Minute {
t.Fatalf("a waiting container is asked every %s", BusyAskEvery)
}
var asked atomic.Int32
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
busyWaiting.Store(false)
t.Cleanup(func() { busyWaiting.Store(false) })
go busyPasses(ctx, func() { asked.Add(1) }, 10*time.Millisecond)
time.Sleep(100 * time.Millisecond)
if n := asked.Load(); n != 0 {
t.Fatalf("%d passes were asked with nothing waiting", n)
}
busyWaiting.Store(true)
deadline := time.Now().Add(2 * time.Second)
for asked.Load() == 0 && time.Now().Before(deadline) {
time.Sleep(10 * time.Millisecond)
}
if asked.Load() == 0 {
t.Fatal("no pass was asked while a container waits")
}
}
cmd/mesh-host/empty_write_test.go
-33
View File
@@ -1,33 +0,0 @@
package main
import (
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A composition that names a file the provisioner fills without saying it is filled would have this node-engine write
// that file empty: its apply is refused, loudly, before anything is written (issue 1945, live on 2026-10-25 at 01:22,
// when ace, g14 and shanks wrote claude-code's, node-env's and power's files empty).
func TestAFileItCannotProduceIsNeverWrittenEmpty(t *testing.T) {
unit := json.RawMessage(`{"id":"power.after-boot-unit","type":"file","path":"/etc/systemd/system/mesh-power-after-boot.service","mode":"0644","owner-at-runtime":"owned"}`)
c := delivery.Composed{Node: "ace", Resources: []json.RawMessage{unit}}
d, err := declarationOf(c)
if err != nil {
t.Fatal(err)
}
if err := refuseEmptyWrites(d); err == nil || !strings.Contains(err.Error(), "power.after-boot-unit") {
t.Fatalf("a file the provisioner fills, not filled here: %v", err)
}
c.Filled = []string{"/etc/systemd/system/mesh-power-after-boot.service"}
if d, _ = declarationOf(c); refuseEmptyWrites(d) != nil {
t.Fatalf("a filled file was refused: %v", refuseEmptyWrites(d))
}
plain := delivery.Composed{Node: "ace", Resources: []json.RawMessage{json.RawMessage(
`{"id":"x.f","type":"file","path":"/etc/x","mode":"0644","content":"x"}`)}}
if d, _ = declarationOf(plain); refuseEmptyWrites(d) != nil {
t.Fatalf("a file with its content was refused: %v", refuseEmptyWrites(d))
}
}
cmd/mesh-host/epoch_test.go
+77
View File
@@ -0,0 +1,77 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
"path/filepath"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// **A declaration from an older lease epoch is refused before anything is applied** (novox/hq to-be
// 45 §6, ADR 0227 rule 2). The controller that sent it lost its lease and goes on sending; this node
// applied the holder's. Refused on the host's own apply path — the one the queue's worker runs — before
// the machine is read or touched, with a report naming what was refused and what is held, and the
// kept declaration left as it was.
func TestADeclarationFromAnOlderEpochIsRefusedBeforeAnythingIsApplied(t *testing.T) {
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
_, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "a.conf")
declared := func(epoch, sequence int64) store.Declared {
body, err := json.Marshal(map[string]any{"declaration": 1, "epoch": epoch, "sequence": sequence,
"resources": []any{map[string]any{"id": "a", "type": "file", "path": target, "content": "x\n"}}})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: ed25519.Sign(private, body)}
}
held := declared(57, 3)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), held); err != nil {
t.Fatal(err)
}
stale := declared(41, 12)
report := applyAndKeep(context.Background(), opts, stale.Declaration, &stale, nil, nil)
if report.OlderThan == nil || *report.OlderThan != (link.Order{Epoch: 57, Sequence: 3}) ||
report.Order != (link.Order{Epoch: 41, Sequence: 12}) || report.Declared != digestOf(stale.Declaration) ||
report.Refused == "" {
t.Fatalf("the refusal does not name what was refused and what is held: %+v", report)
}
if _, err := os.Stat(target); !errors.Is(err, os.ErrNotExist) {
t.Fatal("the refused declaration touched the machine")
}
kept, err := store.ReadDeclared(store.DeclaredPath(opts.state))
if err != nil || string(kept.Declaration) != string(held.Declaration) {
t.Fatal("the refused declaration replaced what this node kept")
}
}
// The queue's counts are kept beside the state and read back by the next host; unreadable is an
// error, never zero.
func TestTheNumbersSurviveTheHost(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
if sequence, refused, err := numbersBeside(state).Read(); err != nil || sequence != 0 || refused != 0 {
t.Fatalf("a node that never reported read %d, %d, %v", sequence, refused, err)
}
if err := numbersBeside(state).Save(41, 2); err != nil {
t.Fatal(err)
}
if sequence, refused, err := numbersBeside(state).Read(); err != nil || sequence != 41 || refused != 2 {
t.Fatalf("read back %d, %d, %v", sequence, refused, err)
}
if err := os.WriteFile(store.NumbersPath(state), []byte(`{"report_sequence":`), 0o600); err != nil {
t.Fatal(err)
}
if _, _, err := numbersBeside(state).Read(); err == nil {
t.Fatal("unreadable numbers were read as zero")
}
}
cmd/mesh-host/file_references_test.go
-116
View File
@@ -1,116 +0,0 @@
package main
import (
"context"
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/link"
)
// aFilesLink is a link that serves objects by name, or fails every read with err.
type aFilesLink struct {
objects map[string][]byte
err error
}
func (a aFilesLink) FileObject(_ context.Context, name string, _ int64) ([]byte, error) {
if a.err != nil {
return nil, a.err
}
if b, ok := a.objects[name]; ok {
return b, nil
}
return nil, os.ErrNotExist
}
func referenceDeclaration(path string, body []byte) []byte {
return []byte(`{"declaration":1,"resources":[{"id":"m.big","type":"file","path":"` + path + `","digest":"` +
declaration.DigestOf(body) + `","size":` + strconv.Itoa(len(body)) + `}]}`)
}
// 1695/AC1 and AC2, through the whole apply: a reference is fetched, verified and written; a differing object
// refuses the declaration whole and writes nothing.
func TestADeclarationWithAFileByDigestAppliesFromTheObject(t *testing.T) {
unansweringPacman(t)
body := []byte(strings.Repeat("a line of a big file\n", 4000))
name, _ := declaration.ObjectName(declaration.DigestOf(body))
dir := t.TempDir()
target := filepath.Join(dir, "big")
opts := options{state: filepath.Join(dir, "state.json"), out: &strings.Builder{}}
raw := referenceDeclaration(target, body)
bad := append([]byte(nil), body...)
bad[10] ^= 1
ctx := link.WithFiles(context.Background(), aFilesLink{objects: map[string][]byte{name: bad}})
report := applyAndKeep(ctx, opts, raw, nil, nil, nil)
if report.Refused == "" || !strings.Contains(report.Refused, "m.big") {
t.Errorf("1695/AC2: a differing object did not refuse the declaration: %+v", report)
}
if _, err := os.Stat(target); err == nil {
t.Error("1695/AC2: a file was written from an object that differs")
}
if failedWith(report.Refused).Transient {
t.Errorf("an object that differs is a real failure, and was taken as the bus not answering: %s", report.Refused)
}
// The bus answers no JetStream request while it reloads its accounts (2026-10-25 23:04:24, mesh-vault on the bus
// node): the failure is transient, so the agent tries it again, and the detail kept is its cause, the tail.
noResponders := fmt.Errorf("fetching the object 2bd98ebd of this node's own object bucket: reading the chunks of "+
"object %s: %w", strings.Repeat("x", 300), nats.ErrNoResponders)
ctx = link.WithFiles(context.Background(), aFilesLink{err: noResponders})
report = applyAndKeep(ctx, opts, raw, nil, nil, nil)
failed := failedWith("a file of this composition could not be had by its digest: file mesh-vault.kept: " + report.Refused)
if !failed.Transient || !strings.HasSuffix(failed.Detail, nats.ErrNoResponders.Error()) || len(failed.Detail) > 310 ||
!strings.HasPrefix(failed.Detail, "a file of this composition") {
t.Errorf("a fetch that met no responders: transient %v, detail %q (%d bytes); want transient, its head and its cause",
failed.Transient, failed.Detail, len(failed.Detail))
}
ctx = link.WithFiles(context.Background(), aFilesLink{objects: map[string][]byte{name: body}})
report = applyAndKeep(ctx, opts, raw, nil, nil, nil)
if report.Refused != "" {
t.Fatalf("1695/AC1: refused: %s", report.Refused)
}
got, err := os.ReadFile(target)
if err != nil || string(got) != string(body) {
t.Errorf("1695/AC1: the file holds %d bytes (%v), want %d", len(got), err, len(body))
}
}
// 1695/AC8: the hand-run `apply <file>` and `reconcile` have no bus: a reference whose file here verifies is applied
// as it is, any other refuses and writes nothing — never an empty file.
func TestTheHandApplyNeverWritesAReferenceEmpty(t *testing.T) {
unansweringPacman(t)
body := []byte(strings.Repeat("a line of a big file\n", 4000))
dir := t.TempDir()
target := filepath.Join(dir, "big")
opts := options{state: filepath.Join(dir, "state.json"), out: &strings.Builder{}}
raw := referenceDeclaration(target, body)
d, err := declaration.ParseFileTrusted(raw)
if err != nil {
t.Fatal(err)
}
if err := runApply(context.Background(), opts, d, raw, fromFile); err == nil || !strings.Contains(err.Error(), "digest") {
t.Errorf("1695/AC8: a reference with no file and no bus was not refused: %v", err)
}
if _, err := os.Stat(target); err == nil {
t.Error("1695/AC8: a file was written for a reference that could not be had")
}
if err := os.WriteFile(target, body, 0o644); err != nil {
t.Fatal(err)
}
d, _ = declaration.ParseFileTrusted(raw)
if err := runApply(context.Background(), opts, d, raw, fromFile); err != nil && !strings.Contains(err.Error(), "applied only in part") {
t.Fatalf("1695/AC8: %v", err)
}
if got, _ := os.ReadFile(target); string(got) != string(body) {
t.Errorf("1695/AC8: the file holds %d bytes, want %d", len(got), len(body))
}
}
cmd/mesh-host/filling.go
-95
View File
@@ -1,95 +0,0 @@
package main
import (
"context"
"crypto/ed25519"
"errors"
"fmt"
"path/filepath"
"sync/atomic"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// serveFilling writes filled files on the provisioner client's request for as long as this engine serves (novox/hq
// ADR 0349 §4). A socket that cannot be served is said once, and this engine goes on applying.
func serveFilling(ctx context.Context, opts options, signer ed25519.PublicKey, say func(string)) {
e := fillingEngine(opts, signer, say)
fillingNow.Store(e)
defer fillingNow.CompareAndSwap(e, nil)
if err := filling.Serve(ctx, filling.Socket, e, say); err != nil {
say("filled files are not written on request on this machine: " + err.Error())
}
}
// fillingNow is the filling this engine serves, for knockFilling; nil while none is served.
var fillingNow atomic.Pointer[filling.Engine]
// knockFilling tells the provisioner's client to fill again what the filling refused or failed and the declaration
// applied now may let through (filling.Engine.Knock), once a module's apply or removal settled (agent.Env.Settled).
// What it told, or why it could not, is said.
func knockFilling(module string, say func(string)) {
e := fillingNow.Load()
if e == nil {
return
}
told, err := e.Knock(filling.Socket)
switch {
case err != nil:
say(fmt.Sprintf("after %s settled: %v", module, err))
case told != "":
say(fmt.Sprintf("after %s settled, the provisioner's client is told to fill again: %s", module, told))
}
}
// fillingEngine is the write of a filled file as this engine does it: the declaration it keeps, verified, read under
// the apply's lock — the same two locks every apply takes, the process's and the one beside the state — and the file
// written through the apply (apply.WriteFilled), its outcome recorded in the node's state as an apply's is.
func fillingEngine(opts options, signer ed25519.PublicKey, say func(string)) *filling.Engine {
return &filling.Engine{
Declared: func() (*declaration.Declaration, error) {
return declaredNow(opts.state, signer)
},
UIDOf: filling.UIDOf,
Lock: func() (func(), error) {
applying.Lock()
unlock, err := store.Lock(opts.state, nil)
if err != nil {
applying.Unlock()
return nil, err
}
return func() { unlock(); applying.Unlock() }, nil
},
Place: func(ctx context.Context, d *declaration.Declaration, path string, content []byte) (string, string, error) {
built, err := system.For(builtFor)
if err != nil {
return "", "", err
}
known, err := store.Load(opts.state)
if err != nil {
return "", "", err
}
report, updated, writeErr := apply.WriteFilled(ctx, built, d, known, path, content, apply.ExecRunner,
announceOr(say), apply.KeepIn(filepath.Dir(opts.state)), apply.WindowsIn(filepath.Dir(opts.state)))
// Saved whichever way it went, as an apply's: what was written is on the machine either way.
if err := store.Save(opts.state, updated); err != nil {
return "", "", errors.Join(writeErr, errors.New("written, and the node's state could not be saved: "+
err.Error()))
}
action, digest := "", ""
for _, o := range report.Outcomes {
if o.Target == path && o.Type == string(declaration.TypeFile) {
action = o.Action
}
}
if rec, ok := updated.At(string(declaration.TypeFile), path); ok {
digest = rec.Wrote
}
return action, digest, writeErr
},
}
}
cmd/mesh-host/filling_test.go
-105
View File
@@ -1,105 +0,0 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"os"
"os/user"
"path/filepath"
"testing"
"time"
"github.com/novox/mesh-host/internal/filling"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §4 (issue 668): a write of a filled file and an apply are one at a time — the write
// takes the lock the apply holds, and writes only once the apply let it go — and the write is recorded in the
// node's state as an apply's outcome is.
func TestAWriteAndAnApplyOfTheSameFileSerialise(t *testing.T) {
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
me, err := user.Current()
if err != nil {
t.Fatal(err)
}
if me.Uid == "0" {
t.Skip("run as root, whose account is never the provisioner's")
}
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(dir, "m.conf")
raw, err := json.Marshal(map[string]any{"declaration": 1, "provisioner_account": me.Username,
"filled_files": []string{path}, "resources": []any{map[string]any{"id": "m.conf", "type": "file",
"path": path, "content": "from the declaration\n", "owner-at-runtime": "owned", "owner": me.Username}}})
if err != nil {
t.Fatal(err)
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state),
store.Declared{Declaration: raw, Signature: ed25519.Sign(private, raw)}); err != nil {
t.Fatal(err)
}
e := fillingEngine(opts, public, func(string) {})
uid := uint32(os.Getuid())
if err := e.Accepts(uid); err != nil {
t.Fatal(err)
}
// An apply holds the lock.
applying.Lock()
done := make(chan filling.Answer)
go func() {
done <- e.Write(context.Background(), uid, filling.Request{Path: path, Content: []byte("filled\n")})
}()
select {
case a := <-done:
applying.Unlock()
t.Fatalf("written while an apply held the lock: %+v", a)
case <-time.After(200 * time.Millisecond):
}
if _, err := os.Stat(path); err == nil {
applying.Unlock()
t.Fatal("the file was written while an apply held the lock")
}
applying.Unlock()
a := <-done
if a.Refused != "" || a.Failed != "" || a.Action != "created" {
t.Fatalf("the write after the apply: %+v", a)
}
got, err := os.ReadFile(path)
if err != nil || string(got) != "filled\n" {
t.Fatalf("the file holds %q (%v)", got, err)
}
known, err := store.Load(opts.state)
if err != nil {
t.Fatal(err)
}
if rec, ok := known.Find("m.conf"); !ok || rec.Wrote != a.Digest || rec.Target != path {
t.Errorf("the write is not recorded in the node's state: %+v, answered %q", rec, a.Digest)
}
// And the cross-process lock beside the state too: a write waits for a `reconcile` run by hand.
unlock, err := store.Lock(opts.state, nil)
if err != nil {
t.Fatal(err)
}
go func() {
done <- e.Write(context.Background(), uid, filling.Request{Path: path, Content: []byte("again\n")})
}()
select {
case a := <-done:
unlock()
t.Fatalf("written while another process held the apply's lock: %+v", a)
case <-time.After(200 * time.Millisecond):
}
unlock()
if a := <-done; a.Action != "updated" {
t.Fatalf("the second write: %+v", a)
}
}
cmd/mesh-host/generation_floor_test.go
+40
View File
@@ -0,0 +1,40 @@
package main
import (
"context"
"testing"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// **The generation applied is kept, and a put-back never lowers it**: after a put-back of generation 5
// over 7, a send from generation 6 is still older than what this machine applied.
func TestThePutBackDoesNotLowerTheGenerationApplied(t *testing.T) {
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
f := machineThatApplied(t, 7)
newer := f.declared(t, map[string]any{"sequence": 9, "generation": 8}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, newer.Declaration, &newer, nil, nil); r.Refused != "" {
t.Fatalf("a newer generation was refused: %s", r.Refused)
}
back := f.declared(t, map[string]any{"sequence": 10, "generation": 5, "put_back": true}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, back.Declaration, &back, nil, nil); r.Refused != "" {
t.Fatalf("a put-back was refused: %s", r.Refused)
}
state, err := store.Load(f.opts.state)
if err != nil || state.Generation != 8 {
t.Fatalf("the generation applied is %d after a put-back of 5 over 8 (%v)", state.Generation, err)
}
stale := f.declared(t, map[string]any{"sequence": 11, "generation": 7}, "docker.conf", f.stays)
if r := applyAndKeep(context.Background(), f.opts, stale.Declaration, &stale, nil, nil); r.OlderGeneration == nil {
t.Fatalf("a send from generation 7 after 8 was applied because a put-back came between: %+v", r)
}
}
cmd/mesh-host/generation_test.go
+170
View File
@@ -0,0 +1,170 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The assignment generation a declaration was composed from (novox/hq issue 234).
//
// On 2026-10-04 a declaration newer in sequence than every other, and composed from a view of the
// assignments older than the one this machine had applied, named four fewer modules — and the host,
// ordering by sequence alone, undeclared all four. The sequence orders arrival; it cannot tell a later
// send carrying earlier content. The generation can: the node-engine refuses a declaration composed from
// an older generation than the one it applied, before anything is read from the machine.
//
// These read what the report says through its JSON, as the controller does, so they run against a
// node-engine that has never heard of a generation and fail there for that reason.
// generationFixture is a machine that applied generation 7: a file the mesh declared, kept in its state,
// and the declaration it kept — signed — that named it.
type generationFixture struct {
opts options
private ed25519.PrivateKey
kept store.Declared
// stays is the file the older generation's declaration does not name.
stays string
}
func machineThatApplied(t *testing.T, generation int64) generationFixture {
t.Helper()
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
_, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
stays := filepath.Join(dir, "docker.conf")
if err := os.WriteFile(stays, []byte("kept\n"), 0o644); err != nil {
t.Fatal(err)
}
// The state as a node-engine that applied generation 7 keeps it: the file it owns, and the highest
// generation it applied. Written as JSON, so the fixture is the same bytes for any build under test.
state := map[string]any{"resources": []any{map[string]any{"id": "docker.conf", "type": "file",
"origin": store.OriginDeclared, "target": stays}}, "generation": generation}
raw, err := json.Marshal(state)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(opts.state, raw, 0o600); err != nil {
t.Fatal(err)
}
f := generationFixture{opts: opts, private: private, stays: stays}
f.kept = f.declared(t, map[string]any{"sequence": 3, "generation": generation}, "docker.conf", stays)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), f.kept); err != nil {
t.Fatal(err)
}
return f
}
// declared is a signed declaration with the envelope's keys given and one file.
func (f generationFixture) declared(t *testing.T, envelope map[string]any, id, path string) store.Declared {
t.Helper()
body := map[string]any{"declaration": 1,
"resources": []any{map[string]any{"id": id, "type": "file", "path": path, "content": "x\n"}}}
for k, v := range envelope {
body[k] = v
}
raw, err := json.Marshal(body)
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: raw, Signature: ed25519.Sign(f.private, raw)}
}
// reportJSON is a report as the controller reads it.
func reportJSON(t *testing.T, r link.Report) map[string]any {
t.Helper()
raw, err := json.Marshal(r)
if err != nil {
t.Fatal(err)
}
var out map[string]any
if err := json.Unmarshal(raw, &out); err != nil {
t.Fatal(err)
}
return out
}
// **A newer sequence from an older generation is refused, and nothing is undeclared** (issue 234).
func TestANewerSequenceFromAnOlderGenerationIsRefusedAndNothingIsUndeclared(t *testing.T) {
f := machineThatApplied(t, 7)
other := filepath.Join(filepath.Dir(f.opts.state), "sudo.conf")
stale := f.declared(t, map[string]any{"sequence": 9, "generation": 6}, "sudo.conf", other)
report := applyAndKeep(context.Background(), f.opts, stale.Declaration, &stale, nil, nil)
said := reportJSON(t, report)
older, _ := said["older_generation"].(map[string]any)
if older == nil || older["generation"] != float64(6) || older["applied"] != float64(7) {
t.Fatalf("the refusal does not name the generation refused and the one applied: %v", said)
}
if report.Refused == "" || !strings.Contains(report.Refused, "generation 6") ||
!strings.Contains(report.Refused, "generation 7") {
t.Errorf("the refusal is not said in words naming both generations: %q", report.Refused)
}
if report.Declared != digestOf(stale.Declaration) || report.Sequence != 9 {
t.Errorf("the refusal does not name the declaration refused, so the mesh cannot name who sent it: %+v", report)
}
if _, err := os.Stat(f.stays); err != nil {
t.Fatalf("a refused declaration undeclared what the machine applied: %v", err)
}
if _, err := os.Stat(other); err == nil {
t.Error("the refused declaration touched the machine")
}
state, err := store.Load(f.opts.state)
if err != nil {
t.Fatal(err)
}
if _, owned := state.Find("docker.conf"); !owned {
t.Error("the refused declaration took a resource out of what this machine owns")
}
kept, err := store.ReadDeclared(store.DeclaredPath(f.opts.state))
if err != nil || string(kept.Declaration) != string(f.kept.Declaration) {
t.Error("the refused declaration replaced what this node kept, so the next reconcile would apply it")
}
}
// refusedForItsGeneration says a report refused its declaration for the generation it came from.
func refusedForItsGeneration(t *testing.T, r link.Report) bool {
t.Helper()
return reportJSON(t, r)["older_generation"] != nil || strings.Contains(r.Refused, "generation")
}
// refusedAsUnread says a report refused its declaration as one this node-engine cannot read.
func refusedAsUnread(r link.Report) bool { return strings.Contains(r.Refused, "not a declaration") }
// **A gate's put-back is applied, whatever generation it carries**: it carries the generation of what it
// puts back, and says it is a put-back (issue 234).
func TestAPutBackFromAnOlderGenerationIsNotRefused(t *testing.T) {
f := machineThatApplied(t, 7)
back := f.declared(t, map[string]any{"sequence": 9, "generation": 5, "put_back": true}, "docker.conf", f.stays)
report := applyAndKeep(context.Background(), f.opts, back.Declaration, &back, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a put-back was refused: %s", report.Refused)
}
}
// **A declaration that carries no generation is applied as it always was**: an older controller sends
// none, and a node-engine that refused it would strand every machine the moment the controller is older.
func TestADeclarationWithoutAGenerationIsNotRefusedForOne(t *testing.T) {
f := machineThatApplied(t, 7)
older := f.declared(t, map[string]any{"sequence": 9}, "docker.conf", f.stays)
report := applyAndKeep(context.Background(), f.opts, older.Declaration, &older, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a declaration without a generation was refused: %s", report.Refused)
}
// And the same generation again is no older.
same := f.declared(t, map[string]any{"sequence": 10, "generation": 7}, "docker.conf", f.stays)
report = applyAndKeep(context.Background(), f.opts, same.Declaration, &same, nil, nil)
if refusedForItsGeneration(t, report) || refusedAsUnread(report) {
t.Fatalf("a declaration of the generation applied was refused: %s", report.Refused)
}
}
cmd/mesh-host/healthowed_test.go
-90
View File
@@ -1,90 +0,0 @@
package main
import (
"context"
"errors"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
)
// owedBus is a link that takes reports and statements and keeps a have bucket in memory.
type owedBus struct {
mu sync.Mutex
kv map[string][]byte
rv map[string]uint64
}
func (b *owedBus) Report(context.Context, string, []byte) error { return nil }
func (b *owedBus) Health(context.Context, string, []byte) error { return nil }
func (b *owedBus) HaveBind(context.Context) (link.HaveBucket, error) { return b, nil }
func (b *owedBus) Get(_ context.Context, key string) ([]byte, uint64, error) {
b.mu.Lock()
defer b.mu.Unlock()
v, ok := b.kv[key]
if !ok {
return nil, 0, link.ErrHaveAbsent
}
return v, b.rv[key], nil
}
func (b *owedBus) Set(_ context.Context, key string, value []byte, revision uint64) error {
b.mu.Lock()
defer b.mu.Unlock()
if b.rv[key] != revision {
return errors.New("wrong last sequence")
}
b.kv[key], b.rv[key] = value, revision+1
return nil
}
func (b *owedBus) have(key string) (delivery.Have, bool) {
b.mu.Lock()
defer b.mu.Unlock()
h, err := delivery.DecodeHave(b.kv[key])
return h, err == nil
}
// A module with nothing long-running (xorg) is ok within one look of its apply, not at the five-minute statement: the
// facts look inside the apply runs while it is applying and judges nothing, and the running write then owes a look,
// which the health loop makes at once (review of fix 15, B1).
func TestAModuleWithNothingLongRunningIsOkWithinALookOfItsApply(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
bus := &owedBus{kv: map[string][]byte{}, rv: map[string]uint64{}}
queue := &link.Session{Membership: link.Membership{Node: "n1"}, Timeout: time.Second}
queue.Attach(ctx, bus)
j, err := liveness.Open(filepath.Join(t.TempDir(), "liveness.json"), nil)
if err != nil {
t.Fatal(err)
}
fact := link.ModuleFact{Module: "xorg", FP: "src2:" + strings.Repeat("a", 64), Composed: "sha256:" + strings.Repeat("b", 64)}
applying := fact
applying.Phase = string(delivery.PhaseApplying)
queue.NoteModule(ctx, applying)
go judgeWhatRuns(ctx, j, queue, func(string) {}, nil)
// The facts look inside the apply, while the module is still applying.
queue.SayFacts(ctx, link.Report{Health: &link.Health{Contract: link.LivenessContract, At: time.Now().UTC()}})
if h, _ := bus.have("n1.xorg"); h.Health != delivery.HealthUnknown {
t.Fatalf("judged while applying: %+v", h)
}
queue.NoteModule(ctx, fact)
deadline := time.Now().Add(2 * time.Second) // well within one look (liveness.LookEvery, 15 s)
for {
h, _ := bus.have("n1.xorg")
if h.Phase == delivery.PhaseRunning && h.Health == delivery.HealthOK {
return
}
if time.Now().After(deadline) {
t.Fatalf("not ok within a look of its apply: %+v", h)
}
time.Sleep(10 * time.Millisecond)
}
}
cmd/mesh-host/judge_directories.go
+7 -2
View File
@@ -5,6 +5,7 @@ import (
"fmt"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
@@ -12,12 +13,16 @@ import (
// judgeDirectories judges every directory of the declaration this node keeps, read-only (novox/hq issue 358): which
// the walk that follows no link another account could make would refuse. Nothing is made, chmodded or chowned.
func judgeDirectories(statePath string) (string, error) {
d, err := declaredNow(statePath, nil)
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
if errors.Is(err, store.ErrNothingDeclared) {
return "", errors.New("this node keeps no declaration, so it is declared no directory")
}
if err != nil {
return "", fmt.Errorf("what runs on this node does not read: %w", err)
return "", err
}
d, err := declaration.Parse(kept.Declaration)
if err != nil {
return "", fmt.Errorf("the declaration this node keeps does not read: %w", err)
}
return apply.JudgeDirectories(d), nil
}
cmd/mesh-host/libraries.go
+7 -3
View File
@@ -11,8 +11,8 @@ import (
"sync/atomic"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
@@ -251,12 +251,16 @@ func startKeeping(ctx context.Context, say func(string)) {
// declaration this node keeps, and run beside the applies.
func accessActFrom(statePath string, timeout time.Duration) link.AccessActFunc {
return func(act link.AccessAct) (string, error) {
d, err := declaredNow(statePath, nil)
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
if errors.Is(err, store.ErrNothingDeclared) {
return "", errors.New("this node keeps no declaration, so it is declared no library")
}
if err != nil {
return "", fmt.Errorf("what runs on this node does not read: %w", err)
return "", err
}
d, err := declaration.Parse(kept.Declaration)
if err != nil {
return "", fmt.Errorf("the declaration this node keeps does not read: %w", err)
}
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
cmd/mesh-host/libraries_test.go
+1 -1
View File
@@ -8,8 +8,8 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
)
cmd/mesh-host/machine_refresh_test.go
+107
View File
@@ -0,0 +1,107 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-host/internal/link"
)
// Defends novox/hq issue 160: a machine nobody pushed to still says what it is made of once a day, so the
// newest of it the mesh holds is no older than a day. Before, the reconcile said it only with something
// else (a hold, a firewall, an outward link changed), and a converged machine's picture was as old as the
// last delivery to it.
func TestAMachineNobodyPushedToSaysWhatItIsOnceADay(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
quiet := link.Report{Outward: []string{"wlan0"},
Profile: map[string]any{"machine": map[string]any{"observed_at": at}}}
if !w.news(quiet) {
t.Fatal("the first reconcile after the engine started did not say what the machine is")
}
w.said(quiet)
for _, after := range []time.Duration{5 * time.Minute, 12 * time.Hour, 22*time.Hour + 55*time.Minute} {
at = time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC).Add(after)
if w.news(quiet) {
t.Errorf("%s after the mesh heard it, an unchanged reconcile was said again", after)
}
}
// The reconcile that falls past 23 hours says it, so it reaches the mesh inside the day.
at = time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC).Add(23*time.Hour + 5*time.Minute)
if !w.news(quiet) {
t.Fatal("a day on, the reconcile did not say what the machine is")
}
w.said(quiet)
at = at.Add(5 * time.Minute)
if w.news(quiet) {
t.Error("once said, it was said again at the next reconcile")
}
}
// A delivery's account carries the profile too: heard, it starts the day again.
func TestADeliveryHeardStartsTheDayAgain(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
report := link.Report{Outward: []string{"eth0"}, Profile: map[string]any{"capabilities": []any{}}}
w.said(report)
at = at.Add(20 * time.Hour)
w.said(report) // a delivery's apply
at = at.Add(20 * time.Hour)
if w.news(report) {
t.Error("said again unasked 20 hours after a delivery told the mesh")
}
}
// A refusal carries nothing about the machine, so it is never said for being a day old.
func TestARefusalIsNotSaidForTheMachinesSake(t *testing.T) {
at := time.Date(2026, 10, 12, 9, 0, 0, 0, time.UTC)
w := &adoptionWatch{now: func() time.Time { return at }}
if w.machineDue(link.Report{Refused: "no"}) {
t.Error("a refusal was due for what the machine is made of")
}
}
// A laptop that slept still says what it is within a day of wall time (review of mesh-host#92). Go's monotonic
// clock stops while a machine is suspended, and Sub between two times that both carry a monotonic reading uses
// it, so a watch that kept one would count a day asleep as almost nothing. Simulated with a clock that first
// reads now (a monotonic reading), and after the sleep reads the wall time a day on: the watch must keep no
// monotonic reading, and judge the day as the wall clock says it.
func TestALaptopThatSleptStillSaysWhatItIsWithinADay(t *testing.T) {
before := time.Now() // carries a monotonic reading
asleep := before.Add(3 * time.Second)
if asleep.Sub(before) != 3*time.Second {
t.Fatal("the monotonic reading is not what this test simulates with")
}
calls := 0
w := &adoptionWatch{now: func() time.Time {
calls++
if calls == 1 {
return before
}
// The wall clock a day on, as a machine's clock reads after waking.
return asleep.Round(0).Add(24 * time.Hour)
}}
report := link.Report{Outward: []string{"wlan0"}, Profile: map[string]any{"capabilities": []any{}}}
w.said(report)
if strings.Contains(w.machineSaid.String(), "m=") {
t.Fatalf("the watch kept a monotonic reading (%s): a sleep would not count toward the day", w.machineSaid)
}
if !w.news(report) {
t.Error("a day of wall time on, a laptop that slept did not say what it is")
}
}
// And the watch's own clock never carries a monotonic reading.
func TestTheWatchClockIsWallTime(t *testing.T) {
w := &adoptionWatch{}
if s := w.clock().String(); strings.Contains(s, "m=") {
t.Errorf("the watch's clock carries a monotonic reading: %s", s)
}
w = &adoptionWatch{now: time.Now}
if s := w.clock().String(); strings.Contains(s, "m=") {
t.Errorf("a given clock's time kept its monotonic reading: %s", s)
}
}
cmd/mesh-host/main.go
+643 -292
View File
File diff suppressed because it is too large Load Diff
cmd/mesh-host/main_test.go
+296 -5
View File
@@ -3,6 +3,7 @@ package main
import (
"bytes"
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"os"
@@ -11,8 +12,10 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/bundle"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
@@ -95,6 +98,26 @@ func TestNoCommandIsNotAnError(t *testing.T) {
}
}
func TestApplyNeedsExactlyOneDeclaration(t *testing.T) {
// `apply` takes a file where every other command takes nothing, so the leftover-argument
// rule has an exception — and an exception is where a parser stops refusing things it
// should. Both directions are checked.
if _, _, err := parseArgs([]string{"apply"}); err == nil {
t.Error("apply with no file was accepted")
}
if _, _, err := parseArgs([]string{"apply", "a.json", "b.json"}); err == nil {
t.Error("apply with two files was accepted")
}
command, opts, err := parseArgs([]string{"apply", "decl.json", "--dry-run"})
if err != nil {
t.Fatalf("unexpected error: %v", err)
}
if command != "apply" || opts.file != "decl.json" || !opts.dryRun {
t.Errorf("parsed as command=%q file=%q dry-run=%v", command, opts.file, opts.dryRun)
}
}
func TestTheStateHasADocumentedDefault(t *testing.T) {
// A host that wrote its state somewhere unexpected would forget what it owns on the next
// run, and then leave everything it had applied behind forever.
@@ -112,22 +135,93 @@ func TestAFlagAfterAPositionalIsRead(t *testing.T) {
// off the front fixed the flag after the COMMAND and not the flag after its ARGUMENT: the
// standard library stops at the first non-flag argument wherever that argument is.
for _, args := range [][]string{
{"hand-over", "decl.json", "--dry-run", "--json"},
{"hand-over", "--dry-run", "decl.json", "--json"},
{"hand-over", "--dry-run", "--json", "decl.json"},
{"apply", "decl.json", "--dry-run", "--json"},
{"apply", "--dry-run", "decl.json", "--json"},
{"apply", "--dry-run", "--json", "decl.json"},
} {
command, opts, err := parseArgs(args)
if err != nil {
t.Errorf("%v: unexpected error: %v", args, err)
continue
}
if command != "hand-over" || opts.file != "decl.json" || !opts.dryRun || !opts.json {
if command != "apply" || opts.file != "decl.json" || !opts.dryRun || !opts.json {
t.Errorf("%v parsed as file=%q dry-run=%v json=%v",
args, opts.file, opts.dryRun, opts.json)
}
}
}
// Defends novox/hq ADR 0100: a reconcile on an adopted node speaks unasked only when what it holds
// or its firewall changed — which is how a predecessor still writing is caught, without a report
// every five minutes saying nothing new.
func TestAReconcileSpeaksOnlyWhenWhatIsHeldChanged(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page"}, {ID: "hello-web.server"}}}
if !w.changed(held) {
t.Fatal("the first report of a hold was not said")
}
again := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.server"}, {ID: "hello-web.page"}}}
if w.changed(again) {
t.Error("the same holds in another order were said again")
}
rewritten := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}, {ID: "hello-web.server"}}}
if !w.changed(rewritten) {
t.Error("a held file rewritten by something else was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a changed firewall was not said")
}
// What filters the machine is part of it (novox/hq ADR 0168): a predecessor's chain removed by
// hand, or the found firewall enabled again, is said without being asked.
filtered := link.Report{Firewall: "none", Held: rewritten.Held,
Filters: []link.Filter{{Where: "chain HAL-MESH-ONLY (iptables-legacy)", Owner: "other", Refuses: "-j DROP"}}}
if !w.changed(filtered) {
t.Error("a filter appearing was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held}) {
t.Error("a filter removed was not said")
}
if !w.changed(link.Report{Firewall: "none", Held: rewritten.Held, FoundFirewall: &link.FoundFirewall{Kind: "ufw", Active: true}}) {
t.Error("the found firewall coming back was not said")
}
if !worthSaying(link.Report{Filters: filtered.Filters}) {
t.Error("a report carrying only what filters the machine is not worth saying")
}
}
// What the mesh heard — a delivery's account as much as a reconcile's — counts as said: the queue
// tells the watch every report the broker took (link.Queue.Heard).
func TestWhatTheMeshHeardCountsAsSaid(t *testing.T) {
w := &adoptionWatch{}
report := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "a"}}}
heard := w.said
heard(report)
if w.changed(report) {
t.Error("a reconcile repeated what the mesh had just heard")
}
}
func TestAReconcileSpeaksWhenWhatIsReachableChanged(t *testing.T) {
// The controller previews a flip from what the node last said is reachable; a port that opened
// since must reach it without waiting for the next delivery (novox/hq ADR 0100).
w := &adoptionWatch{}
before := link.Report{Firewall: "ufw", Reachable: []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}}
if !w.changed(before) {
t.Fatal("the first report was not said")
}
reordered := link.Report{Firewall: "ufw", Reachable: []link.Reach{
{Protocol: "tcp", Address: "0.0.0.0", Port: 22, By: "sshd"}}}
if w.changed(reordered) {
t.Error("the same reachable set was said again")
}
opened := link.Report{Firewall: "ufw", Reachable: append(before.Reachable,
link.Reach{Protocol: "tcp", Address: "0.0.0.0", Port: 8080, By: "hello-web", Published: true, ContainerPort: 80})}
if !w.changed(opened) {
t.Error("a newly published port was not said")
}
}
// Defends the node's own record: the link and the reconcile loop both apply, and each reads the
// state, acts, and writes it back — so they must not run at the same time, or the last save loses
// what the other recorded.
@@ -176,6 +270,24 @@ func TestOnlyOneApplyRunsAtATime(t *testing.T) {
}
}
// Defends novox/hq ADR 0100: a change is counted as said only once the mesh has been told. Queued
// and lost — the link down when the reconcile spoke — it must be said again.
func TestAChangeThatNeverReachedTheMeshIsSaidAgain(t *testing.T) {
w := &adoptionWatch{}
held := link.Report{Firewall: "ufw", Held: []link.Held{{ID: "hello-web.page", Changed: "rewritten"}}}
if !w.differs(held) {
t.Fatal("the first report of a change was not new")
}
// The link was down: nothing published it, so nothing says it was said.
if !w.differs(held) {
t.Error("a change that never reached the mesh was counted as said")
}
w.said(held)
if w.differs(held) {
t.Error("a change the mesh was told was said again")
}
}
// Defends novox/hq issue 104: a declaration for the other mode than this node is in is refused at
// the point of application, whichever command delivered it, naming both — an adopted control-node
// once applied its converged genesis bundle and closed itself for forty-five minutes.
@@ -356,6 +468,125 @@ func TestADryRunChangesNothingAndListsTheActions(t *testing.T) {
}
}
// Defends novox/hq issue 104: once the mesh has told this node anything, `reconcile` holds it to
// that — never to the bundle the host carries, which genesis consumed.
func TestReconcileAfterAControllerDeclarationDoesNotReapplyTheBundle(t *testing.T) {
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
opts := stateWithMode(t, store.ModeAdopted)
controllerPublic, controllerPrivate, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
said := []byte(`{"declaration":1,"adoption":{"taken":[]},"resources":[{"id":"a","type":"file","path":"/tmp/a","content":"x\n"}]}`)
if err := store.SaveDeclared(store.DeclaredPath(opts.state), store.Declared{Declaration: said,
Signature: ed25519.Sign(controllerPrivate, said)}); err != nil {
t.Fatal(err)
}
// Told, and unable to prove by whom: refused, and the bundle is not applied in its place.
_, _, _, err = reconcileSource(opts)
if err == nil || !strings.Contains(err.Error(), "not applied in its place") {
t.Errorf("a node that cannot prove what it was told fell back to something: %v", err)
}
mine, err := identity.Generate("workstation")
if err != nil {
t.Fatal(err)
}
mine.Membership = identity.Membership{Broker: "198.51.100.10:5671", Fingerprint: "sha256:0",
Signer: controllerPublic, Password: "issued"}
if err := identity.Save(identity.Path(opts.state), mine); err != nil {
t.Fatal(err)
}
d, raw, from, err := reconcileSource(opts)
if err != nil {
t.Fatal(err)
}
if from != fromDeclared || !bytes.Equal(raw, said) || d.Adoption == nil {
t.Errorf("reconcile chose %d with %d bytes, not what the mesh last said", from, len(raw))
}
// And before the mesh has said anything: the bundle, as it always was. A test binary carries
// only the placeholder, and asking for it is what proves the path.
if err := os.Remove(store.DeclaredPath(opts.state)); err != nil {
t.Fatal(err)
}
_, _, _, err = reconcileSource(opts)
if !errors.Is(err, bundle.ErrEmpty) {
t.Errorf("with nothing said, reconcile did not reach for the carried bundle: %v", err)
}
}
// **A machine says which links face outside without being asked.**
//
// The mesh composes no filter for a machine that has not said (novox/hq ADR 0140), and a machine only
// speaks unasked when this fingerprint changes. Left out of it, a machine that has just learnt to say
// could speak only when a declaration arrived — and a declaration cannot be composed until it has
// spoken. A machine waiting for a push that is waiting for the machine.
func TestANewOutwardLinkIsSaidUnasked(t *testing.T) {
w := &adoptionWatch{}
first := link.Report{Firewall: "none"}
if !w.differs(first) {
t.Fatal("the first report should differ from nothing")
}
w.said(first)
// Only the links changed, and nothing about adoption.
learnt := link.Report{Firewall: "none", Outward: []string{"eth0"}}
if !w.differs(learnt) {
t.Fatal("a machine that has just learnt which links face outside would never say so, " +
"and could then never be sent a filter")
}
w.said(learnt)
if w.differs(link.Report{Firewall: "none", Outward: []string{"eth0"}}) {
t.Fatal("the same links are reported as a change, so the machine would speak on every reconcile")
}
// And a link that changes — a laptop moving from a cable to a radio — is said too, because the
// filter is written around the old one until it is.
if !w.differs(link.Report{Firewall: "none", Outward: []string{"wlan0"}}) {
t.Fatal("a changed outward link is not said, so the filter stays written around the old one")
}
}
// **A converged machine can say which links face outside, unasked.**
//
// A reconcile is otherwise silent, and the condition deciding when it speaks asked only what an
// adopted node reports — what it holds, and the firewall it found. A converged node has neither, so
// it could speak only in reply to a declaration, and the mesh composes no declaration for a node
// that has not said which links face outside (novox/hq ADR 0140). Measured: three converged machines
// sat silent while the control plane refused to send them a filter.
func TestAConvergedMachineSaysItsOutwardLinksUnasked(t *testing.T) {
// A converged node's reconcile: nothing held, no found firewall, and the links it can see.
converged := link.Report{Outward: []string{"eth0"}}
if !worthSaying(converged) {
t.Fatal("a converged machine cannot say which links face outside, so it can never be " +
"sent a filter — a machine waiting for a push that is waiting for the machine")
}
// An adopted node's reasons still hold, because that is how a predecessor still writing is caught.
if !worthSaying(link.Report{Firewall: "ufw"}) {
t.Fatal("an adopted machine no longer says which firewall it found")
}
if !worthSaying(link.Report{Held: []link.Held{{ID: "a-file"}}}) {
t.Fatal("an adopted machine no longer says what it holds")
}
// And a reconcile with nothing to say stays silent, or every machine speaks every five minutes
// about nothing.
if worthSaying(link.Report{}) {
t.Fatal("a reconcile with nothing to say speaks anyway")
}
// A refused report says nothing about the machine; the refusal is for the console.
if worthSaying(link.Report{Outward: []string{"eth0"}, Refused: "not for this node"}) {
t.Fatal("a refused report is offered as news about the machine")
}
}
// **A host running as a service says what its apply did.**
//
// The serving path passed nil where the apply writes its detail, and nil is silence. The one-shot path
@@ -380,3 +611,63 @@ func TestTheApplysLogIsNeverNil(t *testing.T) {
t.Fatalf("the apply's detail did not reach the caller's announce: %v", said)
}
}
// Defends novox/hq issues 257 and 261: a reconcile due while the link applies a newer declaration
// applies that newer one once its turn comes — never the one kept when its timer fired. Read before
// waiting, it gave back a module assigned a second earlier, and reported a declaration the mesh no
// longer recorded as sent.
func TestAReconcileAppliesWhatWasKeptWhenItsTurnComes(t *testing.T) {
built, err := system.For("arch")
if err != nil || built.Confirm(context.Background(), apply.ExecRunner) != nil {
t.Skip("this machine is not one these tests can apply on")
}
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
dir := t.TempDir()
opts := options{state: filepath.Join(dir, "state.json")}
public, private, err := ed25519.GenerateKey(nil)
if err != nil {
t.Fatal(err)
}
target := filepath.Join(dir, "a.conf")
kept := func(content string) store.Declared {
body := []byte(`{"declaration":1,"resources":[{"id":"a","type":"file","path":"` + target +
`","content":"` + content + `\n"}]}`)
return store.Declared{Declaration: body, Signature: ed25519.Sign(private, body)}
}
if err := store.SaveDeclared(store.DeclaredPath(opts.state), kept("older")); err != nil {
t.Fatal(err)
}
// The link is applying: the reconcile's timer fires and it waits its turn.
applying.Lock()
done := make(chan error, 1)
go func() {
_, err := reapplyKept(context.Background(), opts, public, nil, nil)
done <- err
}()
time.Sleep(50 * time.Millisecond)
// The link's apply ends by keeping the newer declaration, and lets go.
if err := store.SaveDeclared(store.DeclaredPath(opts.state), kept("newer")); err != nil {
applying.Unlock()
t.Fatal(err)
}
applying.Unlock()
select {
case err := <-done:
if err != nil {
t.Fatal(err)
}
case <-time.After(10 * time.Second):
t.Fatal("the reconcile never ran once the node was free")
}
got, err := os.ReadFile(target)
if err != nil {
t.Fatal(err)
}
if string(got) != "newer\n" {
t.Errorf("the reconcile applied %q, what was kept before the newer declaration", got)
}
}
cmd/mesh-host/meshcli.go
+2 -2
View File
@@ -11,14 +11,14 @@ import (
// serveMeshCLI answers mesh-cli on this machine for as long as this engine serves (novox/hq ADR 0272): the account
// from the kernel, the line asked of the controller over the link open at the time. A socket that cannot be served
// is said once, and this engine goes on applying; mesh-cli then says the engine does not answer.
func serveMeshCLI(ctx context.Context, queue *link.Session, say func(string)) {
func serveMeshCLI(ctx context.Context, queue *link.Queue, say func(string)) {
if err := meshcli.Serve(ctx, cli.Socket, forwardOver(queue), say); err != nil {
say("mesh-cli is not answered on this machine: " + err.Error())
}
}
// forwardOver asks the controller over the queue's link, and follows the line to what came of it.
func forwardOver(queue *link.Session) meshcli.Forward {
func forwardOver(queue *link.Queue) meshcli.Forward {
return func(ctx context.Context, asked meshcli.Asked) cli.Answer {
return meshcli.AskTheController(ctx, queue.AskController, asked)
}
cmd/mesh-host/module_facts_test.go
-38
View File
@@ -1,38 +0,0 @@
package main
import (
"errors"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
)
// 1132/AC1, AC3: the report carries one fact per module the declaration names, with the build the controller sent for
// it; a module not yet taken on an adopted node is not one.
func TestTheReportCarriesOneFactPerModuleWithItsBuild(t *testing.T) {
d, err := declaration.Parse([]byte(`{"declaration":1,"builds":{"a":"b-1","b":"b-2"},"resources":[
{"id":"a.dir","type":"directory","path":"/srv/a","module":"a"},
{"id":"b.dir","type":"directory","path":"/srv/b","module":"b"}]}`))
if err != nil {
t.Fatal(err)
}
failure := &apply.Error{Resource: "b.dir", Err: errors.New("no space"), Failures: []apply.Failure{{Resource: "b.dir", Err: errors.New("no space")}}}
facts := moduleFacts(d, apply.Report{}, failure)
if len(facts) != 2 || facts[0].Module != "a" || facts[0].Build != "b-1" || facts[0].Failed ||
facts[1].Module != "b" || facts[1].Build != "b-2" || !facts[1].Failed || facts[1].Detail == "" {
t.Fatalf("facts %+v", facts)
}
adopted, err := declaration.Parse([]byte(`{"declaration":1,"adoption":{"taken":["a"],"untaken":{"b":["b.dir"]}},"resources":[
{"id":"a.dir","type":"directory","path":"/srv/a","module":"a"},
{"id":"b.dir","type":"directory","path":"/srv/b","module":"b"}]}`))
if err != nil {
t.Fatal(err)
}
if facts := moduleFacts(adopted, apply.Report{}, nil); len(facts) != 1 || facts[0].Module != "a" {
t.Fatalf("an untaken module is a fact: %+v", facts)
}
if moduleFacts(nil, apply.Report{}, nil) != nil {
t.Fatal("facts without a declaration")
}
}
cmd/mesh-host/move_out_test.go
-113
View File
@@ -1,113 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"os"
osuser "os/user"
"path/filepath"
"strconv"
"strings"
"syscall"
"testing"
"time"
"github.com/novox/mesh-host/internal/apply"
)
// TestAMoveOutDirectoryKeepsOnlyWhatItKnows drives the node-engine whole through its command (issue 1024, novox/hq
// ADR 0422): a directory under an account's home that says move-out keeps the file the mesh declared and moves a
// file nobody declared into the kept folder, named in a list, and the run says it moved it. It needs no bus: the
// apply is the one a delivery and the five-minute reconcile both run.
func TestAMoveOutDirectoryKeepsOnlyWhatItKnows(t *testing.T) {
me, err := osuser.Current()
if err != nil {
t.Fatal(err)
}
// The user database lists an account as a person's only at uid 1000 or more; root, as the forge's merge check
// runs, is listed under a number of its own (it can give files to it), by the same name.
listedUID := me.Uid
if uid, _ := strconv.Atoi(me.Uid); uid < 1000 || uid == 65534 {
listedUID = "4242"
}
root := t.TempDir()
home := filepath.Join(root, "home", "agent")
dir := filepath.Join(home, ".ssh")
if err := os.MkdirAll(dir, 0o755); err != nil {
t.Fatal(err)
}
passwd := filepath.Join(root, "passwd")
line := fmt.Sprintf("%s:x:%s:%s::%s:/bin/sh\n", me.Username, listedUID, me.Gid, home)
if err := os.WriteFile(passwd, []byte(line), 0o644); err != nil {
t.Fatal(err)
}
t.Cleanup(apply.UseAccountsFileForTest(passwd))
// Independent of the machine: an arch host with a pacman on PATH that answers and holds nothing, as
// TestReplay233 puts one there (it answers with an error); nothing here declares a package.
bin := t.TempDir()
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
wasBuilt := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = wasBuilt })
stray := filepath.Join(dir, "id_stray")
if err := os.WriteFile(stray, []byte("somebody else's key"), 0o600); err != nil {
t.Fatal(err)
}
old := time.Now().Add(-time.Hour)
if err := os.Chtimes(stray, old, old); err != nil {
t.Fatal(err)
}
pipe := filepath.Join(dir, "agent.sock")
if err := syscall.Mkfifo(pipe, 0o644); err != nil {
t.Fatal(err)
}
if err := os.Chtimes(pipe, old, old); err != nil {
t.Fatal(err)
}
opts := options{state: filepath.Join(root, "state.json")}
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"ssh.dir","type":"directory","path":"` + dir + `","owner":"` + me.Username + `","unknown":"move-out"},` +
`{"id":"ssh.config","type":"file","path":"` + dir + `/config","content":"Host *\n","owner":"` + me.Username + `"}]}`)
var said []string
report := applyAndKeep(context.Background(), opts, raw, nil, nil, func(s string) { said = append(said, s) })
if report.Refused != "" || len(report.Failed) > 0 {
t.Fatalf("refused or failed: %q %v", report.Refused, report.Failed)
}
if _, err := os.Stat(filepath.Join(dir, "config")); err != nil {
t.Errorf("the declared file is not there: %v", err)
}
if _, err := os.Lstat(stray); err == nil {
t.Error("the file nobody declared stayed in the directory")
}
lists, _ := filepath.Glob(filepath.Join(home, ".local", "state", "mesh-moved", "ssh", "*.json"))
if len(lists) != 1 {
t.Fatalf("one list expected: %v", lists)
}
rawList, err := os.ReadFile(lists[0])
if err != nil {
t.Fatal(err)
}
var entries []struct{ From, To string }
if err := json.Unmarshal(rawList, &entries); err != nil || len(entries) != 1 {
t.Fatalf("the list: %s %v", rawList, err)
}
if got, err := os.ReadFile(entries[0].To); err != nil || string(got) != "somebody else's key" {
t.Errorf("the moved file is not whole in the kept folder: %q %v", got, err)
}
if len(report.MovedOut) != 1 || report.MovedOut[0].From != stray || report.MovedOut[0].To != entries[0].To {
t.Errorf("the report does not carry the move: %+v", report.MovedOut)
}
if len(report.Left) != 1 || report.Left[0].Path != pipe {
t.Errorf("the report does not list the pipe left in place: %+v", report.Left)
}
if !strings.Contains(strings.Join(said, "\n"), "moved "+stray) {
t.Errorf("the run does not say it moved the file: %v", said)
}
}
cmd/mesh-host/movesecret.go
-131
View File
@@ -1,131 +0,0 @@
package main
import (
"context"
"crypto/rand"
"crypto/subtle"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"os"
"time"
"golang.org/x/crypto/nacl/box"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// The move of a value sealed to this machine into the vault (design 57's V3, issue 676; internal/link
// movesecret.go). The controller cannot open what it holds; this machine can open what was sealed to it, and so
// hands it to the vault, sealed to a key the vault made for the call. The value is in this process for that one
// call and written nowhere.
// vaultAsk asks one of the vault's verbs in this engine's own name and answers its result.
type vaultAsk func(verb string, args any) (json.RawMessage, error)
// moveSecretFrom is the engine's move: this machine's sealing key, read for each ask, and the vault asked over
// the link open now.
func moveSecretFrom(statePath string, queue *link.Session) link.MoveSecretFunc {
return func(m link.MoveSecret) (link.MovedSecret, error) {
key, err := identity.LoadSealingKey(identity.SealingKeyPath(statePath))
if err != nil {
return link.MovedSecret{}, err
}
ask := func(verb string, args any) (json.RawMessage, error) {
return queue.AskVault(context.Background(), verb, args)
}
moved, err := moveIntoTheVault(ask, key, m)
if err != nil {
fmt.Fprintf(os.Stderr, "mesh-host: a value was not moved into the vault: %v\n", err)
}
return moved, err
}
}
// moveIntoTheVault opens the row's value with this machine's sealing key, hands it to the vault's move-in in its two
// steps, and answers what the vault kept once its read-back — sealed to a key made here for this one call — opens
// to the same value.
func moveIntoTheVault(ask vaultAsk, key identity.SealingKey, m link.MoveSecret) (link.MovedSecret, error) {
if m.SealedTo != "" && m.SealedTo != key.Public {
return link.MovedSecret{}, errors.New("the controller's store says this value was sealed to a sealing key this " +
"machine no longer has, so it cannot be opened here")
}
value, err := key.Unseal(m.Sealed)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the value does not open with this machine's sealing key: %w", err)
}
defer wipe(value)
// This machine's sealing key, stated in its own name: the key the vault seals the core's credentials to for the
// controller to carry here (novox/hq ADR 0349 §16, issue 763), never a key the controller names.
first := map[string]any{"origin": m.Origin, "generation": m.Generation, "sealing_key": key.Public}
if !m.MadeAt.IsZero() {
first["made_at"] = m.MadeAt.UTC().Format(time.RFC3339Nano)
}
if m.Own() {
first["node"], first["module"], first["name"] = m.Node, m.Module, m.Name
} else {
first["provider"], first["provision"] = m.Provider, m.Provision
first["consumer"], first["consumer_module"] = m.Consumer, m.ConsumerModule
if m.Local != "" {
first["local"] = m.Local
}
}
raw, err := ask("move-in", first)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault did not open the move: %w", err)
}
var opened struct {
Call string `json:"call"`
SealTo string `json:"seal_to"`
Secret string `json:"secret"`
}
if err := json.Unmarshal(raw, &opened); err != nil || opened.Call == "" || opened.SealTo == "" {
return link.MovedSecret{}, errors.New("the vault answered no call and no key to seal to")
}
sealed, err := identity.Seal(opened.SealTo, value)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault's key for the call is not a sealing key: %w", err)
}
backPub, backPriv, err := box.GenerateKey(rand.Reader)
if err != nil {
return link.MovedSecret{}, err
}
defer wipe(backPriv[:])
raw, err = ask("move-in", map[string]any{"call": opened.Call, "sealed": sealed,
"read_back_to": base64.StdEncoding.EncodeToString(backPub[:])})
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault did not take %s: %w", opened.Secret, err)
}
var kept struct {
Name string `json:"name"`
Generation int `json:"generation"`
Unchanged bool `json:"unchanged"`
ReadBack string `json:"read_back"`
}
if err := json.Unmarshal(raw, &kept); err != nil || kept.Name == "" || kept.ReadBack == "" {
return link.MovedSecret{}, fmt.Errorf("the vault answered %s with no read-back", opened.Secret)
}
blob, err := base64.StdEncoding.DecodeString(kept.ReadBack)
if err != nil {
return link.MovedSecret{}, fmt.Errorf("the vault's read-back of %s is not sealed", kept.Name)
}
back, ok := box.OpenAnonymous(nil, blob, backPub, backPriv)
if !ok {
return link.MovedSecret{}, fmt.Errorf("the vault's read-back of %s does not open with the key made for it", kept.Name)
}
defer wipe(back)
if subtle.ConstantTimeCompare(back, value) != 1 {
return link.MovedSecret{}, fmt.Errorf("the vault keeps %s, but it read back another value than the one handed "+
"in: the move of it did not hold", kept.Name)
}
return link.MovedSecret{Secret: kept.Name, Generation: kept.Generation, Unchanged: kept.Unchanged}, nil
}
func wipe(b []byte) {
for i := range b {
b[i] = 0
}
}
cmd/mesh-host/movesecret_test.go
-134
View File
@@ -1,134 +0,0 @@
package main
import (
"crypto/rand"
"encoding/base64"
"encoding/json"
"errors"
"strings"
"testing"
"golang.org/x/crypto/nacl/box"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// standInVault answers move-in's two steps as the vault does: a key made for the call, the value opened with it,
// and the value kept read back sealed to the key the second step sent. lie makes its read-back another value.
type standInVault struct {
t *testing.T
pub *[32]byte
priv *[32]byte
opened map[string]any
kept string
lie string
refuse string
carried []string
}
func (v *standInVault) ask(verb string, args any) (json.RawMessage, error) {
raw, _ := json.Marshal(args)
v.carried = append(v.carried, string(raw))
if verb != "move-in" {
v.t.Fatalf("asked %s", verb)
}
if v.refuse != "" {
return nil, errors.New(v.refuse)
}
var a map[string]any
_ = json.Unmarshal(raw, &a)
if a["call"] == nil {
v.opened = a
pub, priv, _ := box.GenerateKey(rand.Reader)
v.pub, v.priv = pub, priv
return json.Marshal(map[string]string{"call": "c1", "seal_to": base64.StdEncoding.EncodeToString(pub[:]),
"secret": "own/anchor/postgres/superuser-password"})
}
blob, _ := base64.StdEncoding.DecodeString(a["sealed"].(string))
value, ok := box.OpenAnonymous(nil, blob, v.pub, v.priv)
if !ok {
v.t.Fatal("what the engine sealed does not open with the call's key")
}
v.kept = string(value)
back := v.kept
if v.lie != "" {
back = v.lie
}
key, _ := base64.StdEncoding.DecodeString(a["read_back_to"].(string))
var to [32]byte
copy(to[:], key)
boxed, _ := box.SealAnonymous(nil, []byte(back), &to, rand.Reader)
return json.Marshal(map[string]any{"name": "own/anchor/postgres/superuser-password", "generation": 1,
"read_back": base64.StdEncoding.EncodeToString(boxed)})
}
const theValue = "the-superusers-password"
func aMoveOf(t *testing.T, key identity.SealingKey) link.MoveSecret {
t.Helper()
sealed, err := identity.Seal(key.Public, []byte(theValue))
if err != nil {
t.Fatal(err)
}
return link.MoveSecret{Node: "anchor", By: "the move", MoveSecretAsk: link.MoveSecretAsk{Module: "postgres",
Name: "superuser-password", Origin: "made", Generation: "g-1", Sealed: sealed, SealedTo: key.Public}}
}
// The engine opens what was sealed to its machine, hands it to the vault sealed to the call's key — never in the
// clear — and answers what the vault kept only once the read-back opens to the same value.
func TestTheEngineMovesAValueSealedToItAndChecksTheReadBack(t *testing.T) {
key, err := identity.GenerateSealingKey()
if err != nil {
t.Fatal(err)
}
vault := &standInVault{t: t}
moved, err := moveIntoTheVault(vault.ask, key, aMoveOf(t, key))
if err != nil {
t.Fatal(err)
}
if moved.Secret != "own/anchor/postgres/superuser-password" || moved.Generation != 1 || vault.kept != theValue {
t.Fatalf("moved %+v, the vault kept %q", moved, vault.kept)
}
if vault.opened["module"] != "postgres" || vault.opened["name"] != "superuser-password" || vault.opened["node"] != "anchor" ||
vault.opened["origin"] != "made" || vault.opened["generation"] != "g-1" ||
vault.opened["sealing_key"] != key.Public {
t.Errorf("the first step named %v", vault.opened)
}
if strings.Contains(strings.Join(vault.carried, " "), theValue) {
t.Fatal("the value crossed the bus in the clear")
}
}
// A read-back that is not the value handed in is a failed move, said; so is a value this machine cannot open, and
// the vault's refusal.
func TestTheEngineSaysAMoveThatDidNotHold(t *testing.T) {
key, _ := identity.GenerateSealingKey()
vault := &standInVault{t: t, lie: "something-else"}
if _, err := moveIntoTheVault(vault.ask, key, aMoveOf(t, key)); err == nil ||
!strings.Contains(err.Error(), "read back") {
t.Fatalf("a read-back of another value: %v", err)
}
other, _ := identity.GenerateSealingKey()
if _, err := moveIntoTheVault((&standInVault{t: t}).ask, key, aMoveOf(t, other)); err == nil ||
!strings.Contains(err.Error(), "sealing key") {
t.Fatalf("a value sealed to another key: %v", err)
}
refusing := &standInVault{t: t, refuse: "the move never replaces a value the vault holds"}
if _, err := moveIntoTheVault(refusing.ask, key, aMoveOf(t, key)); err == nil ||
!strings.Contains(err.Error(), "never replaces") {
t.Fatalf("the vault's refusal: %v", err)
}
// A pair credential names both its ends.
pair := aMoveOf(t, key)
pair.Module, pair.Name = "", ""
pair.Provider, pair.Provision, pair.Consumer, pair.ConsumerModule = "anchor", "database", "shanks", "gitea"
ends := &standInVault{t: t}
if _, err := moveIntoTheVault(ends.ask, key, pair); err != nil {
t.Fatal(err)
}
if ends.opened["provider"] != "anchor" || ends.opened["consumer"] != "shanks" || ends.opened["consumer_module"] != "gitea" ||
ends.opened["node"] != nil {
t.Errorf("the pair's first step named %v", ends.opened)
}
}
cmd/mesh-host/network_test.go
+1 -1
View File
@@ -4,7 +4,7 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/network"
)
cmd/mesh-host/no_check_script_test.go
-15
View File
@@ -1,15 +0,0 @@
package main
import (
"os"
"path/filepath"
"testing"
)
// The checker runs gofmt, go vet and both test levels by Go's convention, and the build module starts the bus the
// integration level needs (novox/hq ADR 0529): no script of the repository's own runs beside them.
func TestTheRepositoryHoldsNoCheckScript(t *testing.T) {
if _, err := os.Stat(filepath.Join("..", "..", "merge-check.sh")); err == nil {
t.Fatal("merge-check.sh is back: the checker's levels run the tests")
}
}
cmd/mesh-host/nodeagent.go
-750
View File
@@ -1,750 +0,0 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The node agent's wiring (novox/hq ADR 0429 §4, ADR 0539 §9): the buckets it watches, the haves it writes, the backup
// it asks and the apply of one module's composition.
// bucketWatch is a bucket of the bus as the agent watches it, through the session's link.
type bucketWatch struct {
session *link.Session
bucket string
}
func (w bucketWatch) Watch(ctx context.Context, pattern string) (<-chan agent.Update, error) {
in, err := w.session.KVWatch(ctx, w.bucket, pattern)
if err != nil {
return nil, err
}
out := make(chan agent.Update, 64)
go func() {
defer close(out)
for u := range in {
select {
case out <- agent.Update{Key: u.Key, Value: u.Value, Revision: u.Revision, Deleted: u.Deleted, Caught: u.Caught}:
case <-ctx.Done():
return
}
}
}()
return out, nil
}
// sessionHaves writes the haves the agent says, through the session.
type sessionHaves struct{ session *link.Session }
func (h sessionHaves) Read(ctx context.Context, module string) (agent.Have, bool) {
got, ok := h.session.ReadHave(ctx, module)
if !ok {
return agent.Have{}, false
}
return agent.Have{Module: module, FP: got.FP, Composed: got.Composed, Phase: string(got.Phase), Detail: got.Detail}, true
}
func (h sessionHaves) Write(ctx context.Context, x agent.Have) {
if x.Module == agent.NodeModule {
// The node's own key holds its facts and the digest of the composition it runs.
if x.Phase == "" || x.Phase == string(delivery.PhaseRunning) {
h.session.NoteNode(ctx, x.Composed)
}
return
}
phase := x.Phase
if phase == string(delivery.PhaseRunning) {
phase = ""
}
h.session.NoteModule(ctx, link.ModuleFact{Module: x.Module, FP: x.FP, Composed: x.Composed, Phase: phase,
Failed: phase == string(delivery.PhaseFailed), Detail: shorten(x.Detail)})
}
func (h sessionHaves) Removed(ctx context.Context, module string) {
h.session.NoteRemoved(ctx, module, "")
}
// shorten keeps a have's detail within what a have value holds: its head, which says what failed, and its tail, which is
// the cause (an error wrapped five times names the bus's "no responders" only at its end: 2026-10-25 23:04, mesh-vault on
// the bus node said only which object it read).
func shorten(s string) string {
const most, head = 300, 100
if len(s) <= most {
return s
}
const gap = " … "
tail := s[len(s)-(most-head-len(gap)):]
return strings.ToValidUTF8(s[:head], "") + gap + strings.ToValidUTF8(tail, "")
}
// failedWith is a failed apply that says why, transient when the words are the bus not answering for a moment
// (link.TransientWords): the agent then tries it again without a new revision.
func failedWith(why string) agent.Result {
return agent.Result{Failed: true, Transient: link.TransientWords(why), Detail: shorten(why)}
}
// backupWithin is how long a backup is given from its ask to say it is done (ADR 0429 §14).
var backupWithin = 10 * time.Minute
// backedUpEvent is where the holder of node-backup says a backup taken by hand ended, also one that could not start.
const backedUpEvent = "mesh.mod.restic.event.backed-up"
// backupNode asks the holder of node-backup on this node to back a module up now, and waits for its event to say how it
// went: the holder answers `now` when it has started, and says `backed-up` when the backup ends.
func backupNode(session *link.Session) func(ctx context.Context, module string) agent.Backup {
return func(ctx context.Context, module string) agent.Backup {
// Heard before the ask, so an event that comes at once is not lost.
events, stop, err := session.Hear(backedUpEvent)
if err != nil {
return agent.Backup{Words: "the end of the backup cannot be heard: " + err.Error()}
}
defer stop()
// A module this machine backs nothing of up moves without a backup: never held for one that cannot run (issue
// 1945). An answer that cannot be had or read asks for the backup as before.
if said, err := session.AskSeat(ctx, "node-backup", "backed-up", map[string]any{"module": module}, 30*time.Second); err == nil {
if nothing, read := declaresNothingToBackUp(said, module); read && nothing {
return agent.Backup{Done: true, Words: module + " declares nothing to back up on this machine"}
}
}
asked := time.Now()
if _, err := session.AskSeat(ctx, "node-backup", "now", map[string]any{"module": module}, 30*time.Second); err != nil {
return agent.Backup{Words: err.Error()}
}
return awaitBackup(ctx, events, session.Membership.Node, module, asked, backupWithin)
}
}
// awaitBackup waits for the event that says the backup of a module on this node ended, within the bound: done when it
// says ok, held on its words when it does not, and held when it says nothing within the bound. An event of another node
// or module, or from before the ask, is not the answer.
func awaitBackup(ctx context.Context, events <-chan []byte, node, module string, since time.Time, within time.Duration) agent.Backup {
bound := time.NewTimer(within)
defer bound.Stop()
for {
select {
case <-ctx.Done():
return agent.Backup{Words: "the node agent was stopped while the backup ran"}
case <-bound.C:
return agent.Backup{Words: fmt.Sprintf("the backup did not say it was done within %s", within)}
case raw := <-events:
var e struct {
Node string `json:"node"`
Module string `json:"module"`
OK bool `json:"ok"`
At time.Time `json:"at"`
Error string `json:"error"`
}
if json.Unmarshal(raw, &e) != nil || e.Node != node || e.Module != module || e.At.Before(since.Add(-time.Second)) {
continue
}
if e.OK {
return agent.Backup{Done: true}
}
words := "the backup failed"
if e.Error != "" {
words += ": " + e.Error
}
return agent.Backup{Words: words}
}
}
}
// declarationOf is the declaration the compositions make together: their resources, and the node's own parts when the
// node's composition is among them. It is what the node judges and says its health by, and what one module's apply is
// made of (with a single composition).
//
// **A resource the node's own composition shares with a module's is the module's** (issue 1945): from 2026-10-25 12:43
// the controller put modules it knew no fingerprint of the new form for into the node's own composition while their
// wants named their own, and the declaration joined of both declared each such file twice, which the filling refuses
// ("2 resources declare X") on every node. The node's own copy is left out, and which were left out is said once.
func declarationOf(cs ...delivery.Composed) (*declaration.Declaration, error) {
d := &declaration.Declaration{Version: declaration.Version}
var modules []delivery.Composed
for _, c := range cs {
if c.Module != "" {
modules = append(modules, c)
}
}
if len(modules) > 0 {
joined := make([]delivery.Composed, 0, len(cs))
for _, c := range cs {
if c.Module == "" {
var dropped []string
c, dropped = ownWithout(c, modules)
sayDroppedOnce(dropped)
}
joined = append(joined, c)
}
cs = joined
}
for _, c := range cs {
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
return nil, fmt.Errorf("a resource of %q: %w", c.Module, err)
}
d.Resources = append(d.Resources, r)
}
d.Filled = append(d.Filled, c.Filled...)
if c.Module == "" {
d.Adoption = c.Own.Adoption
d.Identities = append(d.Identities, c.Own.Identities...)
d.Groups = append(d.Groups, c.Own.Groups...)
d.Libraries = append(d.Libraries, c.Own.Libraries...)
d.Provisioner = c.Own.Provisioner
}
}
return d, nil
}
// ownWithout is the node's own composition without the resources a module's composition among modules also declares:
// the same kind and id. A resource that shares only its target (a unit or container of the same name, another id) is
// not the module's, and stays: dropped, its record would be an orphan the apply removes, stopping the module's unit. It
// answers what it left out, as "<id> (<module>)".
func ownWithout(own delivery.Composed, modules []delivery.Composed) (delivery.Composed, []string) {
byID := map[string]string{}
for _, c := range modules {
if c.Module == "" {
continue
}
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
continue
}
byID[string(r.Kind())+"\x00"+r.Identity()] = c.Module
}
}
if len(byID) == 0 {
return own, nil
}
var dropped []string
kept := make([]json.RawMessage, 0, len(own.Resources))
for _, raw := range own.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
kept = append(kept, raw) // refused where it is parsed, with its words
continue
}
module, dup := byID[string(r.Kind())+"\x00"+r.Identity()]
if dup {
dropped = append(dropped, fmt.Sprintf("%s (%s)", r.Identity(), module))
continue
}
kept = append(kept, raw)
}
if len(dropped) == 0 {
return own, nil
}
own.Resources = kept
return own, dropped
}
// droppedSaid is every set of the node's own resources left out that was said, so each is said once.
var droppedSaid sync.Map
// sayDroppedOnce says once which of the node's own resources were left out because a module's composition declares them.
func sayDroppedOnce(dropped []string) {
if len(dropped) == 0 {
return
}
sort.Strings(dropped)
key := strings.Join(dropped, ", ")
if _, said := droppedSaid.LoadOrStore(key, true); said {
return
}
fmt.Fprintf(os.Stderr, "mesh-host: the node's own composition also declares %d resource(s) a module's composition "+
"declares, and they are left out of it: %s (issue 1945)\n", len(dropped), key)
}
func scopeName(c delivery.Composed) string {
if c.Module == "" {
return agent.NodeModule
}
return c.Module
}
func resourcesOf(cs []delivery.Composed) []declaration.Resource {
var out []declaration.Resource
for _, c := range cs {
for _, raw := range c.Resources {
if r, err := declaration.ParseResource(raw); err == nil {
out = append(out, r)
}
}
}
return out
}
// operatorKeysFile is the file beside the node's state that holds the operator's public keys, one ed25519 key in base64 a
// line (a line that starts with # is a note). The switch's last declaration writes it as a file resource, beside the
// controller's key the node was enrolled with (novox/hq ADR 0539 §3, §11); a node adopted later has it at its adoption.
const operatorKeysFile = "operator-keys"
// trustedKeys are the keys a composition may be signed by: the controller's, which the node was enrolled with, and the
// operator's, read from operatorKeysFile each time it is asked, so a key that arrives while the agent runs is trusted at
// the next composition. A file that cannot be read trusts the controller's alone, and says so.
func trustedKeys(opts options, controller ed25519.PublicKey) func() []ed25519.PublicKey {
return func() []ed25519.PublicKey {
keys := []ed25519.PublicKey{}
if len(controller) == ed25519.PublicKeySize {
keys = append(keys, controller)
}
raw, err := os.ReadFile(filepath.Join(filepath.Dir(opts.state), operatorKeysFile))
if err != nil {
return keys
}
for _, line := range strings.Split(string(raw), "\n") {
line = strings.TrimSpace(line)
if line == "" || strings.HasPrefix(line, "#") {
continue
}
key, err := base64.StdEncoding.DecodeString(line)
if err != nil || len(key) != ed25519.PublicKeySize {
fmt.Fprintf(os.Stderr, "mesh-host: a line of %s is not an ed25519 public key in base64, and is not trusted\n", operatorKeysFile)
continue
}
keys = append(keys, ed25519.PublicKey(key))
}
return keys
}
}
// keptOnce reads the kept declaration once and splits it by module: its resources (the node's own are those that say
// none, ADR 0539 §9) and its filled paths.
func keptOnce(opts options, signer ed25519.PublicKey) func() (map[string][]json.RawMessage, map[string][]string) {
var once sync.Once
var by map[string][]json.RawMessage
var filled map[string][]string
return func() (map[string][]json.RawMessage, map[string][]string) {
once.Do(func() {
raw, err := store.LoadDeclared(store.DeclaredPath(opts.state), signer)
if err != nil {
return
}
by, filled = splitKept(raw)
})
return by, filled
}
}
// splitKept is a kept declaration's resources by the module they say (the node's own under agent.NodeModule) and its
// filled paths by the module whose file is at each path; nil for a declaration that does not read.
func splitKept(raw []byte) (map[string][]json.RawMessage, map[string][]string) {
var kept struct {
Resources []json.RawMessage `json:"resources"`
Filled []string `json:"filled_files"`
}
if json.Unmarshal(raw, &kept) != nil {
return nil, nil
}
isFilled := make(map[string]bool, len(kept.Filled))
for _, p := range kept.Filled {
isFilled[p] = true
}
by, filled := map[string][]json.RawMessage{}, map[string][]string{}
named := map[string]bool{}
for _, r := range kept.Resources {
var head struct {
Module string `json:"module"`
Type string `json:"type"`
Path string `json:"path"`
}
_ = json.Unmarshal(r, &head)
name := head.Module
if name == "" {
name = agent.NodeModule
}
by[name] = append(by[name], r)
if head.Type == string(declaration.TypeFile) && isFilled[head.Path] && !named[name+"\x00"+head.Path] {
named[name+"\x00"+head.Path] = true
filled[name] = append(filled[name], head.Path)
}
}
return by, filled
}
// nodeAgent runs the node agent under the session's links for as long as ctx lasts: it waits for a link, follows the
// node's wants and compositions on it, and begins again with the next link. A watch it cannot make (a bucket this node's
// account may not read) is said, and tried again with a growing wait, as the link is.
func nodeAgent(ctx context.Context, opts options, signer ed25519.PublicKey, session *link.Session, sched *apply.Scheduler,
say link.Announce, afterApply func(), current *agentNow, sayFacts func(ctx context.Context, ag *agent.Agent)) {
wait := 2 * time.Second
for ctx.Err() == nil {
linked, cancel, err := session.WhenLinked(ctx)
if err != nil {
return
}
began := time.Now()
node := session.Membership.Node
composedBucket, err := delivery.ComposedBucketFor(node)
if err != nil {
cancel()
say("the node agent cannot name this node's composition bucket: " + err.Error())
return
}
var ag *agent.Agent
kept := keptOnce(opts, signer)
// The bus's user list, written alone from the newest composition of the bus module (ADR 0547 §8): on every
// node, and a node whose compositions carry no list writes nothing.
users := userListFollower(opts, session, composedBucket, trustedKeys(opts, signer), say, func() {
// This engine wrote the list, so the bus reloads its accounts and answers no JetStream request for a few
// seconds: what failed in them is tried again once that is over, not at its next wait.
time.AfterFunc(reloadSettles, func() {
if now := current.get(); now != nil {
now.RetryTransient()
}
})
})
// A vault key heard (or heard again changed) looks at the list again, so its hashes are handed (ADR 0547 §9).
vaultKey.following(users.Again)
go func() {
if err := users.Run(linked); err != nil && linked.Err() == nil {
say("the bus's user list is not followed on this link: " + err.Error())
}
}()
env := agent.Env{
Node: node,
Wants: bucketWatch{session, delivery.AssignerWantsBucket},
Composed: bucketWatch{session, composedBucket},
Haves: sessionHaves{session},
Kept: func() map[string][]json.RawMessage { by, _ := kept(); return by },
// The same kept declaration's filled paths, so a module still running from it keeps its files filled.
KeptFilled: func() map[string][]string { _, filled := kept(); return filled },
Recorded: recordedIn(opts.state, say),
Revisions: func(ctx context.Context, key string) ([]agent.Update, error) {
got, err := session.KVHistory(ctx, composedBucket, key)
if err != nil {
return nil, err
}
out := make([]agent.Update, 0, len(got))
for _, u := range got {
out = append(out, agent.Update{Key: u.Key, Value: u.Value, Revision: u.Revision, Deleted: u.Deleted})
}
return out, nil
},
Trusted: trustedKeys(opts, signer),
Backup: backupNode(session),
// What the filling refused before this apply may be named now: the provisioner's client is told to fill
// again (ace, 2026-10-25 21:17).
Settled: func(module string) { knockFilling(module, say) },
Say: say,
// A failure of the bus not answering for a moment is tried again: 2 s, 5 s, 15 s, 60 s, then every 5 min.
Backoff: link.Backoff,
Apply: func(ctx context.Context, c delivery.Composed, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := applyComposition(actx, opts, c, others, sched, say)
done()
// An apply of the bus module may write an older list than the newest composition holds: looked at again.
if module, _ := users.State(); module != "" && module == c.Module {
users.Again()
}
afterApply()
sayFacts(ctx, ag)
return res
},
Remove: func(ctx context.Context, module string, others []delivery.Composed) agent.Result {
actx, done := applyContext(session.WithLink(ctx))
res := removeComposition(actx, opts, module, others, sched, say)
done()
afterApply()
sayFacts(ctx, ag)
return res
},
}
ag = agent.New(env)
current.set(ag)
err = ag.Run(linked)
cancel()
current.set(nil)
if ctx.Err() != nil {
return
}
if time.Since(began) > 30*time.Second {
wait = 2 * time.Second
}
if err != nil {
say(fmt.Sprintf("the node agent stopped: %v — beginning again in %s", err, wait))
}
select {
case <-ctx.Done():
return
case <-time.After(wait):
}
if wait *= 2; wait > 2*time.Minute {
wait = 2 * time.Minute
}
}
}
// reloadSettles is how long after this engine wrote the bus's user list the bus is taken to have reloaded its accounts:
// the bus module's entrypoint looks at the list every 5 s and then signals the server, whose reload of the accounts
// answers no JetStream request for about three seconds (on the bus node, 2026-10-25).
var reloadSettles = 10 * time.Second
// agentNow is the agent running now, which the node's facts are built from.
type agentNow struct {
mu sync.Mutex
ag *agent.Agent
}
func (c *agentNow) set(a *agent.Agent) {
c.mu.Lock()
c.ag = a
c.mu.Unlock()
}
func (c *agentNow) get() *agent.Agent {
c.mu.Lock()
defer c.mu.Unlock()
return c.ag
}
// applyComposition applies one composition: a module's, or the node's own.
func applyComposition(ctx context.Context, opts options, c delivery.Composed, others []delivery.Composed,
sched *apply.Scheduler, say link.Announce) agent.Result {
name := scopeName(c)
if c.Module == "" {
// What another module's composition declares is that module's, never the node's own apply's (issue 1945).
var dropped []string
c, dropped = ownWithout(c, others)
sayDroppedOnce(dropped)
}
d, err := declarationOf(c)
if err != nil {
return failedWith(err.Error())
}
if err := refuseEmptyWrites(d); err != nil {
return failedWith(err.Error())
}
scope := apply.Scope{Module: name, Others: resourcesOf(others)}
got := applyModuleHeld(ctx, opts, d, &scope, sched, say)
if got.Refused != "" {
return failedWith(got.Refused)
}
results := apply.ModuleResults(d, got.Outcome, got.Err)
if r, ok := results[name]; ok && r.Failed {
return failedWith(r.Detail)
}
if got.Err != nil {
return failedWith(got.Err.Error())
}
if got.Probe != nil {
return failedWith(got.Probe.partial().Error())
}
return agent.Result{}
}
// removeComposition removes one module's own resources and nothing of another's.
func removeComposition(ctx context.Context, opts options, module string, others []delivery.Composed,
sched *apply.Scheduler, say link.Announce) agent.Result {
scope := apply.Scope{Module: module, Others: resourcesOf(others)}
got := applyModuleHeld(ctx, opts, &declaration.Declaration{Version: declaration.Version}, &scope, sched, say)
switch {
case got.Refused != "":
return agent.Result{Failed: true, Detail: shorten(got.Refused)}
case got.Err != nil:
return agent.Result{Failed: true, Detail: shorten(got.Err.Error())}
}
return agent.Result{}
}
// busyModules are the modules whose last apply left a container running because it said busy.
func busyModules() []string {
busyMu.Lock()
defer busyMu.Unlock()
var out []string
for m, w := range busyBy {
if w && m != "" {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// keptDeclaration is the declaration the node kept before the switch, verified with the mesh's key when it is given.
func keptDeclaration(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
var raw []byte
if len(signer) > 0 {
var err error
if raw, err = store.LoadDeclared(store.DeclaredPath(statePath), signer); err != nil {
return nil, err
}
} else {
kept, err := store.ReadDeclared(store.DeclaredPath(statePath))
if err != nil {
return nil, err
}
raw = kept.Declaration
}
return declaration.Parse(raw)
}
// whatRuns is what runs on this node as the declaration of it (novox/hq ADR 0539 §9): the compositions the agent runs
// and, for every module the agent does not hold yet, what the kept declaration has for it, with the node's own parts
// (adoption, identities, groups, libraries) from the node's composition once the agent holds it and from the kept
// declaration until then. What the node judges its health by, and what the filling, the take act and the judgement of
// directories read.
func whatRuns(everything []delivery.Composed, holdsNode bool, kept func() (*declaration.Declaration, error)) (*declaration.Declaration, error) {
d, err := declarationOf(everything...)
if err != nil {
return nil, err
}
if holdsNode {
return d, nil
}
old, err := kept()
if err != nil {
if len(everything) > 0 {
return d, nil
}
return nil, err
}
d.Adoption, d.Identities, d.Groups, d.Libraries = old.Adoption, old.Identities, old.Groups, old.Libraries
d.Provisioner, d.Builds = old.Provisioner, old.Builds
d.Filled = append(d.Filled, old.Filled...)
return d, nil
}
// runsHere says what runs on this node; the running serve sets it to the agent's view, and a command run by hand reads
// the kept declaration.
var runsHere struct {
mu sync.Mutex
fn func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error)
}
func setRunsHere(fn func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error)) {
runsHere.mu.Lock()
runsHere.fn = fn
runsHere.mu.Unlock()
}
// declaredNow is the declaration of what runs on this node now.
func declaredNow(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
runsHere.mu.Lock()
fn := runsHere.fn
runsHere.mu.Unlock()
if fn != nil {
return fn(statePath, signer)
}
return keptDeclaration(statePath, signer)
}
// runsFromAgent is what runs now as the agent in hand sees it.
func runsFromAgent(current *agentNow) func(string, ed25519.PublicKey) (*declaration.Declaration, error) {
return func(statePath string, signer ed25519.PublicKey) (*declaration.Declaration, error) {
kept := func() (*declaration.Declaration, error) { return keptDeclaration(statePath, signer) }
ag := current.get()
if ag == nil {
return kept()
}
return whatRuns(ag.Everything(), ag.HoldsNode(), kept)
}
}
// refuseEmptyWrites refuses a declaration that names a file the provisioner fills (owned at runtime, no content, no
// bytes, no digest, no sealed value) without saying it is filled: applied, the file would be written empty over what the
// provisioner wrote (issue 1945, live on 2026-10-25 at 01:22 on ace, g14 and shanks). The apply fails, naming the file,
// and nothing is written.
func refuseEmptyWrites(d *declaration.Declaration) error {
filled := make(map[string]bool, len(d.Filled))
for _, p := range d.Filled {
filled[p] = true
}
var refused []string
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || f.OwnerAtRuntime != declaration.RuntimeOwned || filled[f.Path] {
continue
}
if f.Content == "" && f.Bytes == "" && f.Digest == "" && f.Sealed == "" && f.VaultSealed == nil {
refused = append(refused, fmt.Sprintf("%s (%s)", f.ID, f.Path))
}
}
if len(refused) > 0 {
return fmt.Errorf("the composition names %s with no content and does not say the provisioner fills it, so it "+
"would be written empty; nothing was applied", strings.Join(refused, ", "))
}
return nil
}
// recordedIn says the engine's applied state holds a record of every resource of a composition: what the agent believes
// of a have or the kept declaration only when it does, so a resource removed since is made again (issue 1945). A state
// that cannot be read says yes: the agent then believes what it was told, as before.
func recordedIn(statePath string, say func(string)) func(ctx context.Context, c delivery.Composed) bool {
return func(_ context.Context, c delivery.Composed) bool {
known, err := store.Load(statePath)
if err != nil {
if say != nil {
say(fmt.Sprintf("the applied state cannot be read, so what runs is believed as told: %v", err))
}
return true
}
filled := make(map[string]bool, len(c.Filled))
for _, p := range c.Filled {
filled[p] = true
}
for _, raw := range c.Resources {
r, err := declaration.ParseResource(raw)
if err != nil {
return true
}
// A file the provisioner client fills is left to it, and has no record until it writes it.
if f, ok := r.(*declaration.File); ok && filled[f.Path] {
continue
}
if _, ok := known.Find(r.Identity()); !ok {
return false
}
}
return true
}
}
// declaresNothingToBackUp reads node-backup.backed-up's answer for a module: true when this machine backs nothing of it
// up (no runs, no paths), and whether the answer could be read at all.
func declaresNothingToBackUp(said json.RawMessage, module string) (nothing bool, read bool) {
var list []struct {
Module string `json:"module"`
Runs int `json:"runs"`
Paths []string `json:"paths"`
}
if err := json.Unmarshal(said, &list); err != nil {
return false, false
}
for _, m := range list {
if m.Module == module && (m.Runs > 0 || len(m.Paths) > 0) {
return false, true
}
}
return true, true
}
// applyBound is how long one apply or removal may take: long enough for the slowest image pull and backup step, short
// enough that a step that hangs cannot hold the node agent for ever.
const applyBound = 30 * time.Minute
// applyContext is the context an apply or a removal runs in: the link's values, never its end, and a bound of its own.
// The link's context ends when the bus goes, and the bus may go because of the apply itself (it replaces the bus's own
// container), which must then finish: a container removed and never made again took the mesh down (issue 1945, live on
// 2026-10-25 at 11:23). applyBound keeps a hung step from blocking the agent for ever.
func applyContext(ctx context.Context) (context.Context, context.CancelFunc) {
return context.WithTimeout(context.WithoutCancel(ctx), applyBound)
}
cmd/mesh-host/nodeagent_integration_test.go
-313
View File
@@ -1,313 +0,0 @@
//go:build integration
package main
import (
"context"
"crypto/ed25519"
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/nats-io/nats-server/v2/server"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-host/internal/agent"
"github.com/novox/mesh-host/internal/link"
)
// The node agent whole, against a real bus with JetStream started in the test: the wants bucket, the node's composition
// bucket and the have bucket, a session on the bus, and the real apply of a module's composition into a directory of the
// test. R6-A AC1, AC5 and AC9 (novox/hq ADR 0539).
const itNode = "it1"
type itBed struct {
t *testing.T
conn *nats.Conn
js nats.JetStreamContext
wants nats.KeyValue
composed nats.KeyValue
haves nats.KeyValue
dir string
opts options
session *link.Session
controll ed25519.PrivateKey
operator ed25519.PrivateKey
said []string
mu sync.Mutex
}
func aServer(t *testing.T) *nats.Conn {
t.Helper()
srv, err := server.NewServer(&server.Options{Port: -1, JetStream: true, StoreDir: t.TempDir(), NoLog: true, NoSigs: true})
if err != nil {
t.Fatal(err)
}
go srv.Start()
if !srv.ReadyForConnections(10 * time.Second) {
t.Fatal("the embedded server did not start")
}
t.Cleanup(srv.Shutdown)
conn, err := nats.Connect(srv.ClientURL())
if err != nil {
t.Fatal(err)
}
t.Cleanup(conn.Close)
return conn
}
func newItBed(t *testing.T) *itBed {
t.Helper()
// Not skipped off Arch: a package manager that answers is put on PATH, so the apply of files runs on any build node.
bin := t.TempDir()
if err := os.WriteFile(filepath.Join(bin, "pacman"), []byte("#!/bin/sh\nexit 0\n"), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", bin+string(os.PathListSeparator)+os.Getenv("PATH"))
was := builtFor
builtFor = "arch"
t.Cleanup(func() { builtFor = was })
b := &itBed{t: t, conn: aServer(t), dir: t.TempDir()}
var err error
if b.js, err = b.conn.JetStream(); err != nil {
t.Fatal(err)
}
composedBucket, _ := delivery.ComposedBucketFor(itNode)
for _, c := range []struct {
name string
into *nats.KeyValue
history uint8
}{{delivery.AssignerWantsBucket, &b.wants, 8}, {composedBucket, &b.composed, delivery.ComposedHistory},
{delivery.HaveBucket, &b.haves, delivery.HaveHistory}} {
kv, err := b.js.CreateKeyValue(&nats.KeyValueConfig{Bucket: c.name, History: c.history})
if err != nil {
t.Fatal(err)
}
*c.into = kv
}
b.opts = options{state: filepath.Join(b.dir, "state", "state.json"), timeout: 2 * time.Second}
if err := os.MkdirAll(filepath.Dir(b.opts.state), 0o755); err != nil {
t.Fatal(err)
}
_, b.controll, _ = ed25519.GenerateKey(nil)
_, b.operator, _ = ed25519.GenerateKey(nil)
return b
}
// start runs the node agent under a session on the bus, trusting the controller's key, and the operator's when asked.
func (b *itBed) start(trustOperator bool) (stop func()) {
b.t.Helper()
if trustOperator {
pub := b.operator.Public().(ed25519.PublicKey)
if err := os.WriteFile(filepath.Join(filepath.Dir(b.opts.state), operatorKeysFile),
[]byte("# the operator\n"+base64.StdEncoding.EncodeToString(pub)+"\n"), 0o600); err != nil {
b.t.Fatal(err)
}
}
b.session = &link.Session{Membership: link.Membership{Node: itNode}, Timeout: 2 * time.Second,
Say: b.say}
b.session.Attach(context.Background(), link.OverNATS{Conn: b.conn, JS: b.js, Node: itNode})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
go func() {
defer close(done)
nodeAgent(ctx, b.opts, b.controll.Public().(ed25519.PublicKey), b.session, nil, b.say, func() {},
&agentNow{}, func(context.Context, *agent.Agent) {})
}()
return func() { cancel(); <-done }
}
func (b *itBed) say(line string) {
b.mu.Lock()
b.said = append(b.said, line)
b.mu.Unlock()
b.t.Log(line)
}
func fpIt(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
// compose writes a module's signed composition of one file into the node's bucket and returns its digest.
func (b *itBed) compose(module, fp, content string, data bool, signer ed25519.PrivateKey) string {
b.t.Helper()
path := filepath.Join(b.dir, module+".conf")
res, _ := json.Marshal(map[string]any{"id": module + ".conf", "type": "file", "path": path, "content": content, "module": module})
c := delivery.Composed{Node: itNode, Module: module, FP: fp, Data: data, Resources: []json.RawMessage{res}}
signed, err := delivery.SignComposed(c, signer)
if err != nil {
b.t.Fatal(err)
}
raw, err := delivery.EncodeComposed(signed)
if err != nil {
b.t.Fatal(err)
}
dig, _ := delivery.ComposedDigestOf(signed)
key, _ := delivery.ComposedKey(itNode, module, fp)
if _, err := b.composed.Put(key, raw); err != nil {
b.t.Fatal(err)
}
return dig
}
func (b *itBed) want(module, fp, digest, backup string) {
b.t.Helper()
raw, err := delivery.EncodeWant(delivery.Want{FP: fp, Composed: digest, Backup: backup})
if err != nil {
b.t.Fatal(err)
}
key, _ := delivery.WantKey(itNode, module)
if _, err := b.wants.Put(key, raw); err != nil {
b.t.Fatal(err)
}
}
func (b *itBed) file(module string) string {
got, _ := os.ReadFile(filepath.Join(b.dir, module+".conf"))
return string(got)
}
func (b *itBed) have(module string) delivery.Have {
key, _ := delivery.HaveKey(itNode, module)
e, err := b.haves.Get(key)
if err != nil {
return delivery.Have{}
}
h, _ := delivery.DecodeHave(e.Value())
return h
}
func within(t *testing.T, d time.Duration, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(d)
for !ok() {
if time.Now().After(deadline) {
t.Fatal(what)
}
time.Sleep(10 * time.Millisecond)
}
}
// R6-A AC1: given a running agent on a bus, when a want naming a digest and that composition are written, the module is
// applied within one second and its have names the fingerprint and the digest; a composition whose digest no want names
// is not applied.
func TestNatsAWantAndItsCompositionAreAppliedWithinASecond(t *testing.T) {
b := newItBed(t)
stop := b.start(false)
defer stop()
time.Sleep(300 * time.Millisecond) // the watches are up
fp := fpIt("a")
dig := b.compose("web", fp, "one\n", false, b.controll)
time.Sleep(300 * time.Millisecond)
if b.file("web") != "" {
t.Fatal("a composition no want names was applied")
}
began := time.Now()
b.want("web", fp, dig, "")
within(t, time.Second, "the module was not applied within one second of its want", func() bool { return b.file("web") == "one\n" })
t.Logf("applied %s after the want", time.Since(began))
within(t, 2*time.Second, "the have does not name the fingerprint and the digest", func() bool {
h := b.have("web")
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
})
// A want naming a digest the composition at its fingerprint does not have applies nothing.
fp2 := fpIt("b")
b.compose("web", fp2, "two\n", false, b.controll)
b.want("web", fp2, "sha256:"+strings.Repeat("0", 64), "")
time.Sleep(300 * time.Millisecond)
if b.file("web") != "one\n" {
t.Fatalf("a want naming another digest changed the module: %q", b.file("web"))
}
}
// R6-A AC5: a data module that ran before asks its backup before any resource changes; a backup that fails holds the
// module (nothing applied, the have says held with the backup's words); when its want then says skip, it is applied.
func TestNatsADataModuleAsksItsBackupFirstAndIsHeldWhenItFails(t *testing.T) {
b := newItBed(t)
var mu sync.Mutex
var asked []string
var onDiskWhenAsked string
if _, err := b.conn.Subscribe("mesh.seat.node-backup.tool.now."+itNode, func(m *nats.Msg) {
mu.Lock()
asked = append(asked, "now")
onDiskWhenAsked = b.file("db")
mu.Unlock()
_ = m.Respond([]byte(`{"result":{"started":"db"}}`))
// The holder says the backup ended on its event.
ev, _ := json.Marshal(map[string]any{"node": itNode, "module": "db", "ok": false, "at": time.Now().UTC(),
"error": "restic: the repository is locked"})
_ = b.conn.Publish(backedUpEvent, ev)
}); err != nil {
t.Fatal(err)
}
stop := b.start(false)
defer stop()
time.Sleep(300 * time.Millisecond)
fp1, fp2 := fpIt("a"), fpIt("b")
b.want("db", fp1, b.compose("db", fp1, "v1\n", true, b.controll), "")
within(t, 3*time.Second, "the first composition was not applied", func() bool { return b.file("db") == "v1\n" })
mu.Lock()
if len(asked) != 0 {
t.Fatalf("a module that never ran was backed up: %v", asked)
}
mu.Unlock()
dig2 := b.compose("db", fp2, "v2\n", true, b.controll)
b.want("db", fp2, dig2, "")
within(t, 3*time.Second, "the module was not held behind the failed backup", func() bool {
h := b.have("db")
return h.Phase == delivery.PhaseHeld && strings.Contains(h.Detail, "repository is locked")
})
mu.Lock()
if len(asked) != 1 || onDiskWhenAsked != "v1\n" {
t.Fatalf("the backup was asked %v, with %q on disk: it is asked before any resource changes", asked, onDiskWhenAsked)
}
mu.Unlock()
if b.file("db") != "v1\n" {
t.Fatalf("something was applied behind a failed backup: %q", b.file("db"))
}
if h := b.have("db"); h.FP != fp1 {
t.Fatalf("a held module should still name what it runs: %+v", h)
}
b.want("db", fp2, dig2, delivery.BackupSkip)
within(t, 3*time.Second, "the module was not applied once its want said skip", func() bool { return b.file("db") == "v2\n" })
within(t, 2*time.Second, "the have does not say running at the new fingerprint", func() bool {
h := b.have("db")
return h.Phase == delivery.PhaseRunning && h.FP == fp2 && h.Composed == dig2
})
}
// R6-A AC9, the hand path end to end: with no controller running, a composition signed by the operator's key and a want
// naming it are applied and the have written; the same signed by a key nobody trusts is not.
func TestNatsTheHandPathAppliesACompositionTheOperatorSigned(t *testing.T) {
b := newItBed(t)
stop := b.start(true)
defer stop()
time.Sleep(300 * time.Millisecond)
_, stranger, _ := ed25519.GenerateKey(nil)
fpS := fpIt("c")
b.want("evil", fpS, b.compose("evil", fpS, "no\n", false, stranger), "")
time.Sleep(300 * time.Millisecond)
if b.file("evil") != "" {
t.Fatal("a composition signed by a key nobody trusts was applied")
}
fp := fpIt("d")
dig := b.compose("rescue", fp, "back\n", false, b.operator)
b.want("rescue", fp, dig, "")
within(t, 2*time.Second, "the operator's composition was not applied", func() bool { return b.file("rescue") == "back\n" })
within(t, 2*time.Second, "the have was not written", func() bool {
h := b.have("rescue")
return h.FP == fp && h.Composed == dig && h.Phase == delivery.PhaseRunning
})
}
cmd/mesh-host/nodeclare_test.go
-42
View File
@@ -1,42 +0,0 @@
package main
import (
"io/fs"
"os"
"path/filepath"
"strings"
"testing"
)
// R6-A AC8: nothing in the node-engine receives, orders or reconciles a whole declaration (novox/hq ADR 0539 §9, ADR
// 0429 §8): no package subscribes to a node's declare subject or names the deleted symbols.
func TestNoPackageHearsOrOrdersAWholeDeclaration(t *testing.T) {
deleted := []string{
"DeclareSubject", "mesh.node.\" + node + \".declare", "ReconcileDue", "ReconcileEvery", "olderThanKept",
"refuseOlder", "olderGeneration", "appliedGeneration", "reconcileKept", "reapplyKept", "adoptionWatch",
"unsaidFile", "numbersFile", "func gather(", "func handleBody(", "Supersedes(", "ReportSequence",
"RefusedOlder", "ReadsGeneration", "ReadsModules", "GenerationRefusal", "LeftOutModuleOf",
}
root := filepath.Join("..", "..")
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "nodeclare_test.go") {
return nil
}
body, err := os.ReadFile(path)
if err != nil {
return err
}
for _, word := range deleted {
if strings.Contains(string(body), word) {
t.Errorf("%s still names %q", path, word)
}
}
return nil
})
if err != nil {
t.Fatal(err)
}
}
cmd/mesh-host/nodekey.go
-142
View File
@@ -1,142 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"time"
"github.com/novox/mesh-host/internal/identity"
)
// This machine's sealing key, stated to the vault in the engine's own name (novox/hq ADR 0372, issue 786). The vault
// records it from the machine's own account alone and, when it is a first or a changed key, says the machine's core
// secrets changed so the controller asks for them again sealed to it. Without this a machine that never moved a
// secret in (adopted after feature 57) has no key at the vault. Nothing but the public half crosses.
const (
// nodeKeyRetryFrom and nodeKeyRetryUpTo bound the wait between tries while the vault does not answer: it starts
// after the bus and may not serve the verb yet.
nodeKeyRetryFrom = 5 * time.Second
nodeKeyRetryUpTo = 5 * time.Minute
// nodeKeyLook is how often the key file is read to see whether it changed.
nodeKeyLook = 30 * time.Second
)
// heardKey is the vault's own public key as its answer to node-key gave it (ADR 0547 clause 9, issue 1945): the key the
// bus node hands the loaded list's hashes sealed to. One for the process, as the key stated is; kick is the follower of
// the user list on the link open now, told when the key is first heard or changes.
type heardKey struct {
mu sync.Mutex
pem string
kick func()
}
// vaultKey is the vault's key this engine heard.
var vaultKey = &heardKey{}
func (h *heardKey) get() string {
h.mu.Lock()
defer h.mu.Unlock()
return h.pem
}
// heard keeps the key a node-key answer gave, and tells the follower when it is a first or changed one. An answer
// with no key (a vault before this one) changes nothing.
func (h *heardKey) heard(answer json.RawMessage) {
var a struct {
VaultKey string `json:"vault_key"`
}
if json.Unmarshal(answer, &a) != nil || !strings.Contains(a.VaultKey, "PUBLIC KEY") {
return
}
h.mu.Lock()
changed := a.VaultKey != h.pem
h.pem = a.VaultKey
kick := h.kick
h.mu.Unlock()
if changed && kick != nil {
kick()
}
}
// following names the follower told when the key changes, and tells it at once when a key is known.
func (h *heardKey) following(kick func()) {
h.mu.Lock()
h.kick = kick
known := h.pem != ""
h.mu.Unlock()
if known && kick != nil {
kick()
}
}
// nodeKeyReask is how often the key is stated again though it did not change, so a changed vault key is heard: the
// vault takes the same key again and changes nothing (ADR 0547 clause 9).
const nodeKeyReask = 10 * time.Minute
// keyHearing is what the keeper does with the vault's own key in node-key's answers (ADR 0547 clause 9): heard takes
// it from an answer; held says whether one is held, and while none is the key is stated again at every look, so a vault
// delivered after this engine is heard within a look; every is how often it is stated again all the same, so a changed
// vault key is heard.
type keyHearing struct {
heard func(json.RawMessage)
held func() bool
// look is how often it is stated again while no vault key is held; every how often once one is.
look, every time.Duration
}
// keepNodeKeyStated states the sealing key load answers to the vault at start, retrying with a growing wait from
// retryFrom to retryUpTo while it does not answer, and states it again whenever the key changes. It says one line
// when a key is stated and one when a try fails for another reason than the last, and returns when ctx ends. With
// hearing, each answer goes to it, and the key is stated again while no vault key is held and at its interval.
func keepNodeKeyStated(ctx context.Context, load func() (identity.SealingKey, error), ask vaultAsk, say func(string),
retryFrom, retryUpTo time.Duration, hearing *keyHearing) {
stated, lastFailure := "", ""
var askedAt time.Time
wait := retryFrom
fail := func(why string) {
if why != lastFailure {
say("mesh-host: the sealing key was not stated to the vault, trying again: " + why)
lastFailure = why
}
wait = min(wait*2, retryUpTo)
}
look := min(nodeKeyLook, retryFrom)
for {
delay := look
key, err := load()
again := false
if hearing != nil && stated != "" {
since := time.Since(askedAt)
held := hearing.held == nil || hearing.held()
again = (!held && since >= hearing.look) || (held && hearing.every > 0 && since >= hearing.every)
}
switch {
case err != nil:
fail(err.Error())
delay = wait
case key.Public != stated || again:
if answer, err := ask("node-key", map[string]string{"key": key.Public}); err != nil {
fail(err.Error())
delay = wait
} else {
askedAt = time.Now()
if hearing != nil && hearing.heard != nil {
hearing.heard(answer)
}
if key.Public != stated {
say(fmt.Sprintf("mesh-host: this machine's sealing key %s is stated to the vault", key.Public))
}
stated, lastFailure, wait = key.Public, "", retryFrom
}
}
select {
case <-ctx.Done():
return
case <-time.After(delay):
}
}
}
cmd/mesh-host/nodekey_test.go
-190
View File
@@ -1,190 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"sync"
"testing"
"time"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/link"
)
// Issue 786 (novox/hq ADR 0372): the engine states its own sealing key to the vault's node-key at start, retries
// while the vault does not answer, and states it again when the key changes; the same key is not stated twice.
type keyVault struct {
mu sync.Mutex
stated []string
fail int
verbs []string
// keyless is how many answers carry no vault key (a vault before ADR 0547 clause 9's, or not yet started); key the
// vault's key the others carry.
keyless int
key string
}
func (v *keyVault) ask(verb string, args any) (json.RawMessage, error) {
v.mu.Lock()
defer v.mu.Unlock()
v.verbs = append(v.verbs, verb)
if v.fail > 0 {
v.fail--
return nil, errors.New("the vault does not answer")
}
raw, _ := json.Marshal(args)
var a struct {
Key string `json:"key"`
}
_ = json.Unmarshal(raw, &a)
v.stated = append(v.stated, a.Key)
if v.key == "" || v.keyless > 0 {
v.keyless--
return json.RawMessage(`{"recorded":true}`), nil
}
answer, _ := json.Marshal(map[string]any{"recorded": true, "vault_key": v.key})
return answer, nil
}
func (v *keyVault) said() []string {
v.mu.Lock()
defer v.mu.Unlock()
return append([]string(nil), v.stated...)
}
func waitFor(t *testing.T, what string, ok func() bool) {
t.Helper()
for i := 0; i < 400; i++ {
if ok() {
return
}
time.Sleep(5 * time.Millisecond)
}
t.Fatalf("timed out: %s", what)
}
func TestTheEngineStatesItsSealingKeyToTheVaultAtStartRetriesAndRestatesAChange(t *testing.T) {
first, _ := identity.GenerateSealingKey()
second, _ := identity.GenerateSealingKey()
var mu sync.Mutex
current := first
load := func() (identity.SealingKey, error) {
mu.Lock()
defer mu.Unlock()
return current, nil
}
vault := &keyVault{fail: 2}
var lines []string
var lmu sync.Mutex
say := func(l string) { lmu.Lock(); lines = append(lines, l); lmu.Unlock() }
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
var hmu sync.Mutex
var answers []string
heard := func(a json.RawMessage) { hmu.Lock(); answers = append(answers, string(a)); hmu.Unlock() }
go keepNodeKeyStated(ctx, load, vault.ask, say, time.Millisecond, 5*time.Millisecond, &keyHearing{heard: heard})
waitFor(t, "the first key stated after two refusals", func() bool { return len(vault.said()) == 1 })
// Its answer is handed on, so the vault's own key in it is heard (ADR 0547 clause 9); refusals hand nothing.
waitFor(t, "the answer heard", func() bool { hmu.Lock(); defer hmu.Unlock(); return len(answers) == 1 })
if vault.said()[0] != first.Public {
t.Errorf("stated %q, want the engine's own key %q", vault.said()[0], first.Public)
}
vault.mu.Lock()
verbs := append([]string(nil), vault.verbs...)
vault.mu.Unlock()
for _, v := range verbs {
if v != "node-key" {
t.Errorf("asked %s", v)
}
}
// The same key again is not stated again.
time.Sleep(60 * time.Millisecond)
if n := len(vault.said()); n != 1 {
t.Errorf("the same key was stated %d times", n)
}
mu.Lock()
current = second
mu.Unlock()
waitFor(t, "the changed key stated", func() bool { return len(vault.said()) == 2 })
if vault.said()[1] != second.Public {
t.Errorf("stated %q after the change, want %q", vault.said()[1], second.Public)
}
lmu.Lock()
defer lmu.Unlock()
if len(lines) == 0 {
t.Error("the engine said nothing")
}
asksAgainUntilItHearsTheVaultsKey(t)
}
// ADR 0547 clause 9: the vault's key is heard though the engine's own key never changes. While none is held the key
// is stated again at each look, so a vault delivered after the engine (or answering no key yet) is heard; once one is
// held, again only at the slow interval, so a changed vault key is heard; never more.
func asksAgainUntilItHearsTheVaultsKey(t *testing.T) {
sealing, _ := identity.GenerateSealingKey()
load := func() (identity.SealingKey, error) { return sealing, nil }
pem := func(c string) string { return "-----BEGIN PUBLIC KEY-----\n" + c + "\n-----END PUBLIC KEY-----\n" }
for _, c := range []struct {
name string
keyless int
every time.Duration
change bool
wantAtLeast int
wantAtMost int
}{
{name: "a vault with no key yet is asked again until it answers one, then not", keyless: 3, wantAtLeast: 4, wantAtMost: 4},
{name: "a held key is asked again at the slow interval, and a changed one heard", every: 20 * time.Millisecond,
change: true, wantAtLeast: 3, wantAtMost: 1 << 20},
} {
t.Run(c.name, func(t *testing.T) {
vault := &keyVault{keyless: c.keyless, key: pem("AAAA")}
h := &heardKey{}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go keepNodeKeyStated(ctx, load, vault.ask, func(string) {}, time.Millisecond, 5*time.Millisecond,
&keyHearing{heard: h.heard, held: func() bool { return h.get() != "" }, look: time.Millisecond, every: c.every})
waitFor(t, "the vault's key heard", func() bool { return h.get() == pem("AAAA") })
if c.change {
vault.mu.Lock()
vault.key = pem("BBBB")
vault.mu.Unlock()
waitFor(t, "the changed vault key heard", func() bool { return h.get() == pem("BBBB") })
}
time.Sleep(60 * time.Millisecond)
n := len(vault.said())
if n < c.wantAtLeast || n > c.wantAtMost {
t.Errorf("node-key asked %d times, want %d to %d", n, c.wantAtLeast, c.wantAtMost)
}
})
}
}
// The statement a host makes names this contract (issue 786), one above the vault's box. And the vault's key its
// answer gives is kept, told to the follower once when first heard and again only when changed; an answer without one
// (an older vault) or with no public key changes nothing (ADR 0547 clause 9).
func TestTheNodeKeyContractIsThirteen(t *testing.T) {
if link.NodeKeyContract != 13 {
t.Errorf("NodeKeyContract %d", link.NodeKeyContract)
}
h := &heardKey{}
kicks := 0
h.following(func() { kicks++ })
h.heard(json.RawMessage(`{"recorded":true}`))
h.heard(json.RawMessage(`{"recorded":true,"vault_key":"not a key"}`))
if h.get() != "" || kicks != 0 {
t.Fatalf("an answer with no key was kept: %q, %d kicks", h.get(), kicks)
}
pem := "-----BEGIN PUBLIC KEY-----\nAAAA\n-----END PUBLIC KEY-----\n"
answer, _ := json.Marshal(map[string]any{"recorded": true, "vault_key": pem})
h.heard(answer)
h.heard(answer)
if h.get() != pem || kicks != 1 {
t.Fatalf("key %q, %d kicks, want the key and one", h.get(), kicks)
}
h.following(func() { kicks += 10 })
if kicks != 11 {
t.Fatalf("a follower named after the key was heard was not told: %d", kicks)
}
}
cmd/mesh-host/order_test.go
+95
View File
@@ -0,0 +1,95 @@
package main
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// A declaration carries no order, so a host cannot tell an older one from a newer (novox/hq
// 04-ISSUES/107). Its only identity was the digest of its bytes: "not the last" could be said,
// "older" could not.
func keptWith(t *testing.T, sequence int64) store.Declared {
t.Helper()
return keptAt(t, link.Order{Sequence: sequence})
}
func keptAt(t *testing.T, order link.Order) store.Declared {
t.Helper()
body, err := json.Marshal(map[string]any{
"declaration": 1, "resources": []any{}, "owns_nothing": true, "sequence": order.Sequence,
"epoch": order.Epoch,
})
if err != nil {
t.Fatal(err)
}
return store.Declared{Declaration: body, Signature: []byte("x")}
}
func TestAnOlderDeclarationFromTheMeshIsRefused(t *testing.T) {
err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 5})
if err == nil {
t.Fatal("sequence 5 was accepted over a kept 7")
}
if !strings.Contains(err.Error(), "older") || !strings.Contains(err.Error(), "nothing was applied") {
t.Fatalf("the refusal does not say what it is: %v", err)
}
}
func TestANewerOrEqualDeclarationIsNot(t *testing.T) {
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 8}); err != nil {
t.Fatalf("sequence 8 was refused over a kept 7: %v", err)
}
// Equal is the same declaration again, which reconciling is for.
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 7}); err != nil {
t.Fatalf("the same sequence was refused: %v", err)
}
}
func TestNoOrderClaimedMeansNoOrderCompared(t *testing.T) {
// An older controller sends none; a host that received before it understood them kept none.
// Refusing on a guess would strand a node the moment the controller is older than the host.
if err := refuseOlder(keptWith(t, 7), nil, link.Order{Sequence: 0}); err != nil {
t.Fatalf("a declaration claiming no order was refused: %v", err)
}
if err := refuseOlder(keptWith(t, 0), nil, link.Order{Sequence: 3}); err != nil {
t.Fatalf("a declaration was refused against a kept one that claimed no order: %v", err)
}
if err := refuseOlder(store.Declared{}, store.ErrNothingDeclared, link.Order{Sequence: 3}); err != nil {
t.Fatalf("a first declaration was refused: %v", err)
}
}
// **The lease epoch decides first** (novox/hq to-be 45 §6): a declaration from a controller that lost
// its lease is refused whatever its sequence, and a new lease holder's is taken whatever its sequence.
func TestAnOlderEpochIsRefusedAndANewerTaken(t *testing.T) {
kept := keptAt(t, link.Order{Epoch: 57, Sequence: 3})
err := refuseOlder(kept, nil, link.Order{Epoch: 41, Sequence: 12})
if err == nil || !strings.Contains(err.Error(), "epoch 41") || !strings.Contains(err.Error(), "epoch 57") {
t.Fatalf("an older epoch was not refused by name: %v", err)
}
if err := refuseOlder(kept, nil, link.Order{Epoch: 58, Sequence: 1}); err != nil {
t.Fatalf("a new lease holder's first declaration was refused for its sequence: %v", err)
}
if err := refuseOlder(kept, nil, link.Order{Epoch: 57, Sequence: 2}); err == nil {
t.Fatal("a lower sequence under the same epoch was accepted")
}
// A controller with no lease — older, or rolled back to a build from before it — is today's
// behaviour: no epoch compared.
if err := refuseOlder(kept, nil, link.Order{Sequence: 4}); err != nil {
t.Fatalf("a declaration with no epoch was refused for one: %v", err)
}
if held, older := olderThanKept(kept, nil, link.Order{Epoch: 41, Sequence: 12}); !older ||
held != (link.Order{Epoch: 57, Sequence: 3}) {
t.Fatalf("olderThanKept said %v, holding %+v", older, held)
}
// What is kept cannot be read: no order to compare, so nothing is refused on a guess.
if _, older := olderThanKept(store.Declared{Declaration: []byte("garbled")}, nil,
link.Order{Epoch: 41, Sequence: 1}); older {
t.Fatal("refused against a kept declaration nobody can read")
}
}
cmd/mesh-host/probe.go
+1 -1
View File
@@ -6,8 +6,8 @@ import (
"os/exec"
"sort"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
cmd/mesh-host/probe_test.go
+82 -1
View File
@@ -8,7 +8,8 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/system"
)
@@ -126,3 +127,83 @@ func TestAPackageManagerThatIsNotThereRefusesWhole(t *testing.T) {
t.Error("the refused declaration touched the machine")
}
}
// Defends the review of novox/hq issue 233: **a re-apply whose result changed is said unasked**. A machine
// reconciles every five minutes and speaks only when its account is news; how the apply went was not part of
// it, so a package manager that stopped answering between deliveries was never said, and one that answered
// again never cleared. Now a failure is said once, the same failure again is not, and a clean re-apply after it
// is — which is what clears the mesh's condition.
func TestAReapplyWhoseResultChangedIsSaidUnasked(t *testing.T) {
dir := t.TempDir()
conf := filepath.Join(dir, "elsewhere", "pacman.conf") // not declared: the apply cannot make it answer
pacmanAnsweringOnce(t, conf)
opts := options{state: filepath.Join(dir, "state.json")}
raw := []byte(`{"declaration":1,"resources":[` +
`{"id":"motd","type":"file","path":"` + filepath.Join(dir, "motd") + `","content":"hello\n"},` +
`{"id":"pacman.base","type":"package","package":"base"}]}`)
w := &adoptionWatch{}
// Clean once, as it was before: base installed while pacman answered.
if err := os.MkdirAll(filepath.Dir(conf), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(conf, []byte("[options]\n"), 0o644); err != nil {
t.Fatal(err)
}
clean := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if faulty(clean) {
t.Fatalf("the first apply was not clean: %+v", clean)
}
w.said(clean)
if w.news(applyAndKeep(context.Background(), opts, raw, nil, nil, nil)) {
t.Error("a clean re-apply after a clean one was said, so every machine speaks every five minutes")
}
// The configuration goes: the re-apply fails, and that is news once.
if err := os.Remove(conf); err != nil {
t.Fatal(err)
}
failed := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if !faulty(failed) {
t.Fatalf("the re-apply did not fail: %+v", failed)
}
if !w.news(failed) {
t.Fatal("a re-apply that failed after a clean one was not said, so the mesh never hears the machine fails")
}
w.said(failed)
if w.news(applyAndKeep(context.Background(), opts, raw, nil, nil, nil)) {
t.Error("the same failure was said again on the next re-apply")
}
// It comes back: the clean re-apply is news, and is what clears the mesh's condition.
if err := os.WriteFile(conf, []byte("[options]\n"), 0o644); err != nil {
t.Fatal(err)
}
again := applyAndKeep(context.Background(), opts, raw, nil, nil, nil)
if faulty(again) {
t.Fatalf("the re-apply did not come back clean: %+v", again)
}
if !w.news(again) {
t.Fatal("a clean re-apply after a failed one was not said, so the mesh's condition never clears")
}
}
// A refusal is news too, once, and a reconcile with nothing to say after a clean one stays silent.
func TestARefusalIsSaidOnceAndNothingIsNotSaid(t *testing.T) {
w := &adoptionWatch{}
w.said(link.Report{})
if w.news(link.Report{}) {
t.Error("nothing was said again")
}
refused := link.Report{Refused: "pacman does not answer here"}
if !w.news(refused) {
t.Fatal("a refusal was not said")
}
w.said(refused)
if w.news(link.Report{Refused: "pacman does not answer here, at another time"}) {
t.Error("the same refusal in other words was said again")
}
if !w.news(link.Report{}) {
t.Error("a clean apply after a refusal was not said")
}
}
cmd/mesh-host/reads-outside
-1
View File
@@ -1 +0,0 @@
These tests read files outside this directory (the whole catalogue, the repository, or its root files): mesh-build-go never keeps this package's verdict (issue 1935).
cmd/mesh-host/recorded_test.go
-94
View File
@@ -1,94 +0,0 @@
package main
import (
"context"
"encoding/json"
"path/filepath"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/store"
)
// The record check reads the engine's applied state: a composition one of whose resources has no record there is not
// recorded (issue 1945).
func TestACompositionIsRecordedOnlyWhenEveryResourceIs(t *testing.T) {
path := filepath.Join(t.TempDir(), "state.json")
var s store.State
s.Record(store.Applied{ID: "ca-trust.trust", Type: "file", Target: "/etc/ca"})
if err := store.Save(path, s); err != nil {
t.Fatal(err)
}
c := delivery.Composed{Node: "ace", Module: "ca-trust", Resources: []json.RawMessage{
json.RawMessage(`{"id":"ca-trust.trust","type":"file","module":"ca-trust","path":"/etc/ca","mode":"0644","content":"x"}`)}}
recorded := recordedIn(path, func(string) {})
if !recorded(context.Background(), c) {
t.Fatal("a composition whose every resource has a record was not recorded")
}
c.Resources = append(c.Resources, json.RawMessage(
`{"id":"ca-trust.more","type":"file","module":"ca-trust","path":"/etc/ca2","mode":"0644","content":"y"}`))
if recorded(context.Background(), c) {
t.Fatal("a composition with a resource the state holds no record of was recorded")
}
}
// The modules applied again are those with a missing resource, each once; one down but there is not among them.
func TestTheModulesAppliedAgainAreThoseWithAMissingResource(t *testing.T) {
st := liveness.Statement{Resources: []liveness.State{
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.homeserver"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "matrix", ID: "matrix.db"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "ca-trust", ID: "ca-trust.trust"}, State: liveness.Unhealthy, Missing: true},
{Resource: liveness.Resource{Module: "web", ID: "web.server"}, State: liveness.Unhealthy},
}}
if got := missingModules(st); len(got) != 2 || got[0] != "ca-trust" || got[1] != "matrix" {
t.Fatalf("applied again: %v", got)
}
}
// A module is applied again only for a resource newly missing: one still missing at the next change of the statement is
// not acted on again, so a resource that cannot be made is not applied (and backed up) at every unrelated change. One
// that comes back and goes again is newly missing again (the review of issue 1945).
func TestOnlyANewlyMissingResourceAppliesItsModuleAgain(t *testing.T) {
missing := func(ids ...string) liveness.Statement {
var st liveness.Statement
for _, id := range ids {
st.Resources = append(st.Resources, liveness.State{Resource: liveness.Resource{Module: "matrix", ID: id},
State: liveness.Unhealthy, Missing: true})
}
return st
}
var last map[string]bool
var got []string
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 1 {
t.Fatalf("first missing: %v", got)
}
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 0 {
t.Fatalf("still missing was acted on again: %v", got)
}
got, last = newlyMissing(last, missing())
got, last = newlyMissing(last, missing("matrix.homeserver"))
if len(got) != 1 {
t.Fatalf("missing again after it came back: %v", got)
}
}
// A file the provisioner client fills is left to it by the apply and has no record until the client writes it: it does
// not make its composition unrecorded (the review of issue 1945).
func TestAFilledFileDoesNotMakeItsCompositionUnrecorded(t *testing.T) {
path := filepath.Join(t.TempDir(), "state.json")
var s store.State
s.Record(store.Applied{ID: "ca-trust.trust", Type: "file", Target: "/etc/ca"})
if err := store.Save(path, s); err != nil {
t.Fatal(err)
}
c := delivery.Composed{Node: "ace", Module: "ca-trust", Filled: []string{"/etc/ca-filled"}, Resources: []json.RawMessage{
json.RawMessage(`{"id":"ca-trust.trust","type":"file","module":"ca-trust","path":"/etc/ca","mode":"0644","content":"x"}`),
json.RawMessage(`{"id":"ca-trust.filled","type":"file","module":"ca-trust","path":"/etc/ca-filled","mode":"0644","owner-at-runtime":"owned"}`)}}
if !recordedIn(path, func(string) {})(context.Background(), c) {
t.Fatal("a filled file the client has not written made its composition unrecorded")
}
}
cmd/mesh-host/replay233_test.go
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
cmd/mesh-host/units_test.go
+5 -8
View File
@@ -5,8 +5,8 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/accounts"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/liveness"
"github.com/novox/mesh-host/internal/units"
@@ -87,13 +87,10 @@ func TestTheStatementSaysAnAccountJudgedForRoot(t *testing.T) {
Groups: []string{"openrazer"}}, State: accounts.Healthy, Since: at},
}}
h := withAccounts(healthAsReported(liveness.Statement{At: at}, nil), as)
// At RootContract or later: an engine that states its sealing key to the vault says NodeKeyContract (ADR 0372, issue 786), one above
// VaultBoxContract (it opens the vault's box for the core's credentials, ADR 0349 §16, issue 755), one above HandedContract (design 57 P7), one above ProvisionerContract (novox/hq ADR
// 0349 §4), one above ADR 0349 §5's OwnershipContract.
if h.Contract < link.RootContract || link.RootContract != 3 || h.Contract != link.NodeKeyContract ||
link.NodeKeyContract != link.VaultBoxContract+1 || link.VaultBoxContract != link.HandedContract+1 ||
link.HandedContract != link.ProvisionerContract+1 || link.ProvisionerContract != link.OwnershipContract+1 || link.OwnershipContract != link.BusyContract+1 ||
link.BusyContract != link.OwnGroupContract+1 {
// At RootContract or later: an engine that also reads an identity's runner says OwnGroupContract (novox/hq ADR
// 0334), one above ADR 0328's LibraryContract.
if h.Contract < link.RootContract || link.RootContract != 3 || h.Contract != link.OwnGroupContract ||
link.OwnGroupContract != link.LibraryContract+1 {
t.Fatalf("contract %d", h.Contract)
}
got := map[string]string{}
cmd/mesh-host/unsaid_test.go
+64
View File
@@ -0,0 +1,64 @@
package main
import (
"os"
"path/filepath"
"reflect"
"testing"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
)
// The last apply's report, kept beside the state until the mesh takes it (novox/hq issue 264).
// What a host that died between applying and reporting kept is read back by the next one exactly —
// the digest and the outcome, never a new account of the machine.
func TestTheKeptReportSurvivesTheHost(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
made := link.Report{Declared: "sha256:d1", Applied: []string{"a"}, Failed: map[string]string{"b": "no room"},
Host: "v2"}
if err := unsaidBeside(state).Keep(made); err != nil {
t.Fatal(err)
}
kept, ok, err := unsaidBeside(state).Pending()
if err != nil || !ok {
t.Fatalf("nothing kept: %v", err)
}
if !reflect.DeepEqual(kept, made) {
t.Fatalf("kept %+v, made %+v", kept, made)
}
// Taken about another declaration, it stays; taken about this one, it goes.
if err := unsaidBeside(state).Said("sha256:other"); err != nil {
t.Fatal(err)
}
if _, ok, _ := unsaidBeside(state).Pending(); !ok {
t.Fatal("a report about another declaration cleared this one")
}
if err := unsaidBeside(state).Said("sha256:d1"); err != nil {
t.Fatal(err)
}
if _, err := os.Stat(store.UnsaidPath(state)); !os.IsNotExist(err) {
t.Fatalf("the report was taken and is still kept: %v", err)
}
}
// A node that never applied anything has nothing to say again; and an apply refused before its
// declaration was read clears an older report, which would otherwise be re-said after it.
func TestNothingKeptWhenNothingWasApplied(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
if _, ok, err := unsaidBeside(state).Pending(); ok || err != nil {
t.Fatalf("a node that applied nothing has a report to say: %v %v", ok, err)
}
if err := unsaidBeside(state).Keep(link.Report{Declared: "sha256:d1"}); err != nil {
t.Fatal(err)
}
if err := unsaidBeside(state).Keep(link.Report{Refused: "forged"}); err != nil {
t.Fatal(err)
}
if _, ok, _ := unsaidBeside(state).Pending(); ok {
t.Fatal("an older report is kept after a later declaration was refused")
}
}
cmd/mesh-host/userlist.go
-127
View File
@@ -1,127 +0,0 @@
package main
import (
"context"
"crypto/ed25519"
"encoding/json"
"errors"
"fmt"
"os"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/link"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
"github.com/novox/mesh-host/internal/userlist"
)
// The bridge of novox/hq ADR 0547 §8, wired: the bus node's engine writes the bus's user list alone from the newest
// composition of the bus module, under the apply's two locks as a filled file is written (filling.go), and says what
// it wrote in the bus module's have.
// userListFollower is the follower of the bus's user list for one link: the node's composition bucket, the keys the
// agent trusts, the write alone, and the have.
func userListFollower(opts options, session *link.Session, composedBucket string, trusted func() []ed25519.PublicKey,
say link.Announce, wrote func()) *userlist.Follower {
return userlist.New(userlist.Env{
Node: session.Membership.Node,
Composed: bucketWatch{session, composedBucket},
Trusted: trusted,
Resolve: func(ctx context.Context, raw json.RawMessage) ([]byte, error) {
return resolveUserList(session.WithLink(ctx), raw)
},
OnDisk: userListOnDisk,
Write: func(ctx context.Context, module string, raw json.RawMessage, body []byte) (string, bool, error) {
digest, changed, err := writeUserListAlone(session.WithLink(ctx), opts, raw, body, say)
if changed && wrote != nil {
wrote()
}
return digest, changed, err
},
Report: func(ctx context.Context, module string, w userlist.Written) {
session.NoteUserList(ctx, module, w)
},
// The vault's key from its answer to node-key: the loaded list's hashes are handed sealed to it (ADR 0547
// clause 9).
VaultKey: vaultKey.get,
Say: announceOr(say),
})
}
// userListFile is a user-list resource as the file it declares.
func userListFile(raw []byte) (*declaration.File, error) {
r, err := declaration.ParseResource(raw)
if err != nil {
return nil, fmt.Errorf("the user list is not a resource: %w", err)
}
f, ok := r.(*declaration.File)
if !ok {
return nil, fmt.Errorf("the user list is a %s, not a file", r.Kind())
}
return f, nil
}
// resolveUserList is the list a user-list resource holds: its content, or the object its digest names in this node's
// file bucket, fetched and verified (apply.ResolveFileReferences, ADR 0486 §1).
func resolveUserList(ctx context.Context, raw []byte) ([]byte, error) {
f, err := userListFile(raw)
if err != nil {
return nil, err
}
d := &declaration.Declaration{Version: declaration.Version, Resources: []declaration.Resource{f}}
if err := apply.ResolveFileReferences(ctx, d, link.FilesFrom(ctx)); err != nil {
return nil, fmt.Errorf("the user list could not be had by its digest: %w", err)
}
return []byte(d.Resources[0].(*declaration.File).Content), nil
}
// userListOnDisk is what the user list's file holds on this machine now; nil when there is none.
func userListOnDisk(raw json.RawMessage) ([]byte, error) {
f, err := userListFile(raw)
if err != nil {
return nil, err
}
body, err := os.ReadFile(f.Path)
if errors.Is(err, os.ErrNotExist) {
return nil, nil
}
return body, err
}
// writeUserListAlone writes one user-list resource alone (apply.WriteAlone) with body as its content, the node's state
// read and saved under the apply's locks, and answers the digest of the file on the machine after.
func writeUserListAlone(ctx context.Context, opts options, raw []byte, body []byte, say link.Announce) (string, bool, error) {
declared, err := userListFile(raw)
if err != nil {
return "", false, err
}
file := *declared
file.Content, file.Digest, file.Size = string(body), "", 0
applying.Lock()
defer applying.Unlock()
unlock, err := store.Lock(opts.state, nil)
if err != nil {
return "", false, err
}
defer unlock()
known, err := store.Load(opts.state)
if err != nil {
return "", false, err
}
built, err := system.For(builtFor)
if err != nil {
return "", false, err
}
out, updated, writeErr := apply.WriteAlone(ctx, built, &file, known, apply.ExecRunner, announceOr(say))
// Saved whichever way it went, as an apply's: what was written is on the machine either way.
if err := store.Save(opts.state, updated); err != nil {
writeErr = errors.Join(writeErr, errors.New("written, and the node's state could not be saved: "+err.Error()))
}
digest := ""
if body, err := os.ReadFile(file.Path); err == nil {
digest = userlist.Digest(body)
}
return digest, writeErr == nil && out.Action != "unchanged", writeErr
}
cmd/mesh-host/vaultbox_nats_test.go
-153
View File
@@ -1,153 +0,0 @@
package main
// Issue 755's proof on a real bus (novox/hq ADR 0349 §16): a core credential the vault sealed to this machine crosses
// the bus inside the declaration, which the controller carries and cannot open, and this engine opens it with its
// sealing key and writes the value. A listener on every subject keeps every byte that crossed; the value is never
// among them. Run against the throwaway bus the build module starts for the integration level (MESH_TEST_NATS).
import (
"context"
"encoding/base64"
"encoding/json"
"os"
"path/filepath"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/seal"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/apply"
"github.com/novox/mesh-host/internal/identity"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
const coreCredential = "the-provisioner-clients-bus-password-2025"
func TestNatsTheEngineFillsACoreCredentialFromTheVaultsBoxWithNoValueInTheClear(t *testing.T) {
url := os.Getenv("MESH_TEST_NATS")
if url == "" {
t.Skip("MESH_TEST_NATS unset")
}
// This machine's sealing key, where the engine reads it.
state := filepath.Join(t.TempDir(), "state.json")
key, err := identity.GenerateSealingKey()
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(identity.SealingKeyPath(state), []byte(key.Private), 0o600); err != nil {
t.Fatal(err)
}
// Every byte that crosses.
tap, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer tap.Close()
var mu sync.Mutex
var crossed []string
if _, err := tap.Subscribe(">", func(m *nats.Msg) {
mu.Lock()
crossed = append(crossed, m.Subject+" "+string(m.Data))
mu.Unlock()
}); err != nil {
t.Fatal(err)
}
_ = tap.Flush()
// The engine hears the composition on a subject of the test (the node agent reads it from its bucket).
const subject = "mesh.test.vaultbox.composition"
engine, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer engine.Close()
heard, err := engine.SubscribeSync(subject)
if err != nil {
t.Fatal(err)
}
_ = engine.Flush()
// The vault's answer to the controller: the value in the ADR 0206 box, sealed to this machine's key. The
// controller carries it in the declaration as it is.
pub, err := key.PublicPEM()
if err != nil {
t.Fatal(err)
}
box, err := seal.Seal(coreCredential, pub)
if err != nil {
t.Fatal(err)
}
path := filepath.Join(t.TempDir(), "broker")
body, _ := json.Marshal(map[string]any{"declaration": 1, "resources": []map[string]any{
{"id": "need-provisioner-broker", "type": "file", "path": path, "vault-sealed": box}}})
controller, err := nats.Connect(url)
if err != nil {
t.Fatal(err)
}
defer controller.Close()
if err := controller.Publish(subject, body); err != nil {
t.Fatal(err)
}
_ = controller.Flush()
msg, err := heard.NextMsg(5 * time.Second)
if err != nil {
t.Fatalf("the declaration did not reach the engine: %v", err)
}
d, err := declaration.Parse(msg.Data)
if err != nil {
t.Fatal(err)
}
sys, err := system.For("arch")
if err != nil {
t.Fatal(err)
}
if _, _, err := apply.Apply(context.Background(), sys, d, store.State{}, store.OriginCarried,
func(context.Context, string, ...string) (string, error) { return "", nil }, nil, sealOpener(state)); err != nil {
t.Fatal(err)
}
on, err := os.ReadFile(path)
if err != nil || string(on) != coreCredential {
t.Fatalf("the engine wrote %q, %v", on, err)
}
_ = tap.Flush()
time.Sleep(100 * time.Millisecond)
mu.Lock()
all := strings.Join(crossed, "\n")
mu.Unlock()
if !strings.Contains(all, subject) || !strings.Contains(all, "vault-sealed") {
t.Fatalf("the listener heard nothing of the declaration:\n%s", all)
}
for _, form := range []string{coreCredential, base64.StdEncoding.EncodeToString([]byte(coreCredential))} {
if strings.Contains(all, form) {
t.Fatal("the core credential crossed the bus in the clear")
}
}
}
// The engine opens a value the controller sealed (the NaCl box) as before, and the vault's box, with one key.
func TestTheEnginesOpenerOpensBothBoxes(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
key, _ := identity.GenerateSealingKey()
if err := os.WriteFile(identity.SealingKeyPath(state), []byte(key.Private), 0o600); err != nil {
t.Fatal(err)
}
open := sealOpener(state)
nacl, _ := identity.Seal(key.Public, []byte("old"))
if got, err := open(nacl); err != nil || string(got) != "old" {
t.Errorf("the controller's box opened to %q, %v", got, err)
}
pub, _ := key.PublicPEM()
box, _ := seal.Seal("new", pub)
raw, _ := json.Marshal(box)
if got, err := open(string(raw)); err != nil || string(got) != "new" {
t.Errorf("the vault's box opened to %q, %v", got, err)
}
}
cmd/mesh-host/whatruns_test.go
-139
View File
@@ -1,139 +0,0 @@
package main
import (
"crypto/ed25519"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-host/internal/agent"
)
func aFileOf(module, id string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": module + "." + id, "type": "file", "path": "/tmp/r6/" + module + "/" + id,
"content": "x", "module": module})
return raw
}
// What runs is the agent's compositions and the kept declaration's resources for each module the agent does not hold;
// the node's own parts are the kept declaration's until the agent holds the node's composition (ADR 0539 §9). One module
// under the agent must not make the rest invisible to the liveness, units, accounts, libraries, filling and the take act.
func TestWhatRunsIsTheAgentsCompositionsAndTheKeptRest(t *testing.T) {
kept := &declaration.Declaration{Identities: []declaration.Identity{{Module: "db", Name: "mesh-db"}},
Libraries: []declaration.Library{{Library: "media"}}}
everything := []delivery.Composed{
{Node: "n1", Module: "web", Resources: []json.RawMessage{aFileOf("web", "a")}},
{Node: "n1", Module: "db", Resources: []json.RawMessage{aFileOf("db", "a")}}, // still the kept declaration's
}
d, err := whatRuns(everything, false, func() (*declaration.Declaration, error) { return kept, nil })
if err != nil {
t.Fatal(err)
}
if len(d.Resources) != 2 || len(d.Identities) != 1 || len(d.Libraries) != 1 {
t.Fatalf("%d resources, %d identities, %d libraries: the kept modules and the node's kept parts are missing",
len(d.Resources), len(d.Identities), len(d.Libraries))
}
// Once the agent holds the node's composition, its parts are the node's.
node := delivery.Composed{Node: "n1", Own: delivery.NodeOwn{Identities: []declaration.Identity{{Module: "web", Name: "mesh-web"}}}}
d, err = whatRuns(append(everything, node), true, func() (*declaration.Declaration, error) { return nil, errors.New("not read") })
if err != nil {
t.Fatal(err)
}
if len(d.Identities) != 1 || d.Identities[0].Name != "mesh-web" || len(d.Libraries) != 0 {
t.Fatalf("the node's parts: %+v %+v", d.Identities, d.Libraries)
}
}
// What the take act, the judgement of directories and the filling read is what runs, not the frozen kept declaration.
func TestDeclaredNowReadsWhatRunsWhenTheAgentIsRunning(t *testing.T) {
setRunsHere(func(string, ed25519.PublicKey) (*declaration.Declaration, error) {
return &declaration.Declaration{Version: 1, Resources: nil, Provisioner: "from-agent"}, nil
})
defer setRunsHere(nil)
d, err := declaredNow("/nowhere", nil)
if err != nil || d.Provisioner != "from-agent" {
t.Fatalf("declaredNow read %+v (%v), not what runs", d, err)
}
}
// 1945/AC17 (unit): a resource the node's own composition shares with a running module's composition is declared once,
// the module's: the declaration joined of both names each file once, so the filling writes it; the node's own resources
// no module declares stay. (Live from 2026-10-25 12:43: "2 resources declare X" on every node.)
func TestTheNodesOwnCopyOfAModulesResourceIsLeftOut(t *testing.T) {
own := delivery.Composed{Node: "n1", Resources: []json.RawMessage{aFileOf("zsh", "rc"), aFileOf("mesh-wireguard", "conf")},
Filled: []string{"/tmp/r6/zsh/rc"}}
zsh := delivery.Composed{Node: "n1", Module: "zsh", FP: "src2:aa", Resources: []json.RawMessage{aFileOf("zsh", "rc")},
Filled: []string{"/tmp/r6/zsh/rc"}}
d, err := declarationOf(own, zsh)
if err != nil {
t.Fatal(err)
}
at := map[string]int{}
for _, r := range d.Resources {
at[r.Target()]++
}
if at["/tmp/r6/zsh/rc"] != 1 || at["/tmp/r6/mesh-wireguard/conf"] != 1 || len(d.Resources) != 2 {
t.Errorf("the joined declaration declares %v, want each path once", at)
}
if !d.Fills("/tmp/r6/zsh/rc") {
t.Errorf("the filled path is no longer named")
}
// The same at the node's own apply: what the others declare is theirs.
trimmed, dropped := ownWithout(own, []delivery.Composed{zsh})
if len(trimmed.Resources) != 1 || len(dropped) != 1 || dropped[0] != "zsh.rc (zsh)" {
t.Errorf("the node's own without zsh's: %d resources, dropped %v", len(trimmed.Resources), dropped)
}
if len(own.Resources) != 2 {
t.Errorf("ownWithout changed its argument")
}
// A node's own composition alone is all of it.
if d, err := declarationOf(own); err != nil || len(d.Resources) != 2 {
t.Errorf("the node's own alone: %v (%v)", d, err)
}
}
// 1945/AC18 (unit): the kept declaration's filled paths go to the module whose file is at each path, the node's own to
// the node's; and what runs from it, joined with the node's composition, names them filled.
func TestTheKeptDeclarationsFilledPathsGoToTheirModules(t *testing.T) {
raw := []byte(`{"resources":[
{"id":"claude-code.settings","type":"file","module":"claude-code","path":"/etc/cc.json"},
{"id":"screen-lock.conf","type":"file","module":"screen-lock","path":"/etc/lock.conf"},
{"id":"host.motd","type":"file","path":"/etc/motd"},
{"id":"zsh.rc","type":"file","module":"zsh","path":"/etc/zshrc"}],
"filled_files":["/etc/cc.json","/etc/lock.conf","/etc/motd"]}`)
by, filled := splitKept(raw)
if len(by["claude-code"]) != 1 || len(by[agent.NodeModule]) != 1 {
t.Fatalf("resources by module: %v", by)
}
if strings.Join(filled["claude-code"], ",") != "/etc/cc.json" || strings.Join(filled["screen-lock"], ",") != "/etc/lock.conf" ||
strings.Join(filled[agent.NodeModule], ",") != "/etc/motd" || len(filled["zsh"]) != 0 {
t.Fatalf("filled by module: %v", filled)
}
cc := delivery.Composed{Node: "n1", Module: "claude-code", Resources: by["claude-code"], Filled: filled["claude-code"]}
node := delivery.Composed{Node: "n1"}
d, err := whatRuns([]delivery.Composed{cc, node}, true, func() (*declaration.Declaration, error) { return nil, errors.New("not read") })
if err != nil || !d.Fills("/etc/cc.json") {
t.Fatalf("what runs does not name claude-code's file filled: %v (%v)", d, err)
}
if splitBy, _ := splitKept([]byte("not json")); splitBy != nil {
t.Errorf("a kept declaration that does not read split into %v", splitBy)
}
}
// 1945/AC17 (unit, review): a resource of the node's own that shares only its target with a module's (another id: a
// unit or a container of the same name) is not the module's, and is neither dropped nor left to be removed.
func TestASharedTargetWithAnotherIDIsNotDropped(t *testing.T) {
ownFile, _ := json.Marshal(map[string]any{"id": "host.zshrc", "type": "file", "path": "/tmp/r6/zsh/rc", "content": "x"})
own := delivery.Composed{Node: "n1", Resources: []json.RawMessage{ownFile}}
zsh := delivery.Composed{Node: "n1", Module: "zsh", FP: "src2:aa", Resources: []json.RawMessage{aFileOf("zsh", "rc")}}
trimmed, dropped := ownWithout(own, []delivery.Composed{zsh})
if len(dropped) != 0 || len(trimmed.Resources) != 1 {
t.Errorf("a resource sharing only its target was dropped: %v", dropped)
}
}
cmd/mesh-host/witnessing.go
+11 -10
View File
@@ -41,9 +41,8 @@ func rolledBackHosts(statePath string) []string {
return upgrade.RolledBackVersions(verdicts)
}
// proving is this host waiting to write a have key under its own build — the evidence its launcher's trial waits for
// (novox/hq ADR 0539 §9, ADR 0227 §8: the component replaced is never the only witness of its successor) — and acting
// on it once.
// proving is this host waiting for the mesh to take a report it made under its own build — the
// evidence its launcher's trial waits for — and acting on it once.
type proving struct {
statePath string
version string
@@ -57,14 +56,16 @@ func unproved(statePath, version string) bool {
return err != nil || known != version
}
// haveWritten is told when this build has written a have key, or found the bucket already holding what it would write.
// The first proves it. **A declaration's report does not**: a node-engine that receives none is not on trial for it, and
// one that cannot write its haves is what the launcher keeps rolling back.
func (p *proving) haveWritten() {
// heard is told every account of the machine the mesh has taken. The first one about a declaration,
// made by this build, proves it.
func (p *proving) heard(r link.Report) {
if r.Declared == "" || r.Refused != "" || r.Host != p.version {
return
}
p.once.Do(func() {
retired, err := upgrade.Proved(p.statePath, upgrade.VersionsDir(""), p.version)
if err != nil {
p.say(fmt.Sprintf("this host wrote its first have under its own build %s, and proving it was not complete: %v", p.version, err))
p.say(fmt.Sprintf("this host reported under its own build %s, and proving it was not complete: %v", p.version, err))
}
if len(retired) > 0 {
p.say(fmt.Sprintf("host %s is proved; retired the host version(s) %s", p.version, strings.Join(retired, ", ")))
@@ -145,7 +146,7 @@ func launcherReplaced(pid int, proc, path string) bool {
}
// watchWhatThisHostPlaced judges the core builds this host placed, until ctx ends (to-be 45 §8).
func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Session, say func(string)) {
func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Queue, say func(string)) {
host, _ := os.Hostname()
w := &witness.Watcher{
Root: apply.DaemonRoot(), Node: node, Host: host,
@@ -158,7 +159,7 @@ func watchWhatThisHostPlaced(ctx context.Context, node string, queue *link.Sessi
},
// Said now, not at the next report: the verdict is on every report from here, and the mesh
// is asked for one at once.
Concluded: func(link.Rollback) { queue.FactsAsked(ctx) },
Concluded: func(link.Rollback) { queue.ReportAsked() },
Say: say,
}
w.Watch(ctx)
cmd/mesh-host/witnessing_test.go
+23 -15
View File
@@ -13,9 +13,9 @@ import (
"github.com/novox/mesh-host/internal/upgrade"
)
// The first have this build writes proves it to the launcher; nothing else does — not a report about a declaration, not
// a facts event, not the haves of another build (novox/hq ADR 0539 §9, ADR 0227 §8). R6-A AC7.
func TestOnlyAHaveUnderThisBuildProvesIt(t *testing.T) {
// The first account the mesh takes from this build, about a declaration, proves it to the launcher;
// nothing else does — a refusal, a report about no declaration, another build's report (to-be 45 §8).
func TestOnlyAnAccountUnderThisBuildProvesIt(t *testing.T) {
state := filepath.Join(t.TempDir(), "state.json")
t.Setenv("MESH_HOST_LIBEXEC", t.TempDir())
// Never the machine's own engine placed by hand: proving renews it (issue 356).
@@ -27,20 +27,22 @@ func TestOnlyAHaveUnderThisBuildProvesIt(t *testing.T) {
var said []string
p := &proving{statePath: state, version: "2.0", say: func(s string) { said = append(said, s) }}
// Facts said, a link opened, a declaration of the old kind reported: none of it is a have.
if !unproved(state, "2.0") {
t.Fatal("a build that wrote no have is proved")
for _, r := range []link.Report{
{Host: "2.0", Refused: "no"},
{Host: "2.0"},
{Host: "1.0", Declared: "abc"},
} {
p.heard(r)
if !unproved(state, "2.0") {
t.Fatalf("%+v proved this build", r)
}
if got, _ := os.ReadFile(fallback); string(got) != "a stale engine placed by hand" {
t.Fatalf("%+v renewed the engine placed by hand", r)
}
}
if got, _ := os.ReadFile(fallback); string(got) != "a stale engine placed by hand" {
t.Fatal("the engine placed by hand was renewed before any have")
}
// The session tells the proof when it has a have key saying what it would write.
session := &link.Session{HaveWritten: p.haveWritten}
_ = session
p.haveWritten()
p.heard(link.Report{Host: "2.0", Declared: "abc"})
if unproved(state, "2.0") {
t.Fatalf("the first have under this build did not prove it (%v)", said)
t.Fatalf("an account of a declaration under this build did not prove it (%v)", said)
}
if got, _ := upgrade.ReadKnownGood(upgrade.KnownGoodPath(state)); got != "2.0" {
t.Fatalf("known-good is %q", got)
@@ -76,6 +78,12 @@ func TestAStandingRollbackIsSaidOnEveryReport(t *testing.T) {
got[0].Outcome != link.RolledBack {
t.Fatalf("what the report says is %+v", got)
}
if !worthSaying(link.Report{Rollbacks: got}) {
t.Fatal("a reconcile with a rollback standing does not say it")
}
if adoptionFingerprint(link.Report{Rollbacks: got}) == adoptionFingerprint(link.Report{}) {
t.Fatal("a rollback reached or ended is not news to the reconcile")
}
}
// The launcher running this host is known to have been replaced only when it certainly was: a shell
examples/bundle_test.go
+2 -2
View File
@@ -12,7 +12,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
func bundle(t *testing.T) *declaration.Declaration {
@@ -49,7 +49,7 @@ func TestTheBundleCarriesTheFoundationAndTheControlPlaneAndNothingElse(t *testin
}
sort.Strings(images)
want := []string{"mesh-controller", "nats", "pgvector"}
want := []string{"mesh-controller", "nats", "postgres"}
if strings.Join(images, ",") != strings.Join(want, ",") {
t.Fatalf("the bundle carries %v; expected exactly %v.\n\n"+
"Adding one is a change to what every first node becomes, and to ADR 0006's "+
examples/foundation-first-node.lock
+19 -34
View File
@@ -28,12 +28,11 @@
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
// registry prefix is the lab's; the digests are upstream's.
//
// **The store is raised where the `postgres` module keeps it** (novox/hq ADR 0338): the container
// `postgres`, on the module's pgvector image, its data in the directory /var/lib/mesh-store owned by
// the image's postgres user, its superuser password read from the module's own-secret file — so the
// store step adopts it as it runs. Pinned upstream by the module's own digest, not through the lab's
// registry prefix, because the two must be the same reference (mesh-host internal/bootstrap holds
// them to each other against the module's manifest).
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
// creates is owned by root, and the database runs as somebody else inside the container — so it
// could not write, and the container crash-looped. A named volume lets the image set up its own
// ownership, and outlives the container, which is what you want for the thing holding the mesh's
// state.
{
"declaration": 1,
@@ -84,28 +83,17 @@
"boot": "enabled",
"restart-on": ["base-filter"]
},
// The store's data is a directory owned by the image's postgres user (999), made before the
// container starts: a directory the runtime made would be root's, and the database could not
// write it. Measured on the module's image 2026-10-13: it initialises a new store on a directory
// made so (novox/hq ADR 0338 §5).
{
"id": "store-data",
"type": "directory",
"path": "/var/lib/mesh-store",
"mode": "0700",
"owner": "999:70"
},
{
"id": "store",
"type": "container",
"name": "postgres",
"image": "pgvector/pgvector@sha256:cf134a767f474095eeba57e0117be8e568e011a63f33fbf252f14c9b760f8e6f",
"name": "mesh-store",
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
"env": {
"POSTGRES_PASSWORD": "bootstrap",
"PGDATA": "/var/lib/postgresql/data/pgdata"
},
"ports": ["5432:5432"],
"volumes": ["/var/lib/mesh-store:/var/lib/postgresql/data"]
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
},
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
@@ -114,21 +102,21 @@
{
"id": "store-ready",
"type": "action",
"in": "postgres",
"in": "mesh-store",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
},
{
"id": "inventory-database",
"type": "action",
"in": "postgres",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
},
{
"id": "identity-database",
"type": "action",
"in": "postgres",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
},
@@ -138,20 +126,20 @@
{
"id": "licences-database",
"type": "action",
"in": "postgres",
"in": "mesh-store",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
},
{
"id": "context-schemas",
"type": "action",
"command": ["docker", "run", "--rm", "--network", "container:postgres",
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"migrate"],
"verify": ["sh", "-c", "docker exec postgres psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec postgres psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec postgres psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
},
// **The bus is raised where the `nats` module keeps it** (novox/hq ADR 0333 §1): the container
// `mesh-broker-nats`, its configuration and user list in /var/lib/nats-module/conf, its streams in
@@ -221,20 +209,17 @@
"path": "/var/lib/nats-module/conf/accounts.conf",
"mode": "0600",
"create-once": true,
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_assignments.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_declared-config.>\", \"$KV.mesh-controller_lease.>\", \"$KV.mesh-controller_predefined.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.assignment-changed\", \"mesh.seat.mesh-controller.event.declared-config-changed\", \"mesh.seat.mesh-controller.event.predefined-changed\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.operator-channel.tool.tell\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
// The upstream server, until the broker step adopts it: the `nats` module's own image, built by the
// mesh, takes its place there once, on these same directories (ADR 0333 §3), as the bus's planned
// step (ADR 0236). The release is the one the module's image is built on (its NATS_BASE, 2.11.17),
// pinned upstream by that same digest: the adoption changes no server release, so it can be undone
// by putting the old image back and is said reversible. The published port is the
// mesh, takes its place there once, on these same directories (ADR 0333 §3). The published port is the
// node's bus port, 5671 unless given, which every enrolment token names; the installer gives the
// module the setting that maps its declared 4222 to it (ADR 0333 §2).
{
"id": "broker",
"type": "container",
"name": "mesh-broker-nats",
"image": "nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00",
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
"volumes": ["/var/lib/mesh-broker-nats:/data", "/var/lib/nats-module/conf:/etc/nats:ro", "/var/lib/mesh-broker-tls:/tls:ro"],
"args": ["-c", "/etc/nats/nats.conf", "-js"]
@@ -242,8 +227,8 @@
{
"id": "broker-ready",
"type": "action",
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671'"]
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
},
{
"id": "control-plane",
go.mod
+1 -7
View File
@@ -3,20 +3,14 @@ module github.com/novox/mesh-host
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936
github.com/nats-io/nats-server/v2 v2.11.17
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c
github.com/nats-io/nats.go v1.54.0
golang.org/x/crypto v0.57.0
golang.org/x/sys v0.48.0
)
require (
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op // indirect
github.com/google/go-tpm v0.9.8 // indirect
github.com/klauspost/compress v1.20.0 // indirect
github.com/minio/highwayhash v1.0.4 // indirect
github.com/nats-io/jwt/v2 v2.8.1 // indirect
github.com/nats-io/nkeys v0.4.16 // indirect
github.com/nats-io/nuid v1.0.1 // indirect
golang.org/x/time v0.15.0 // indirect
)
go.sum
+6 -21
View File
@@ -1,23 +1,11 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f h1:lUEyNPPF3w4Q0LIgcUUD6ofkKVhCF0n7dVjMoixQmXo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261024234636-10b6e202320f/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456 h1:He2o9vSKjgKbWOCOgOJY0WlaDh1n8RcGhvGY/dwXsBQ=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204213-f63fa86c7456/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e h1:k/1NjjW5HuuuHW2ldyWTvjp+1u3fN0RO18Q57Iup6Dk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261025204730-0be0ffd6159e/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936 h1:HGBoWimZXYZp+Qn9VCstYDitOP84t9EUQYd1yM9j2eg=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261026102838-174789941936/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
github.com/google/go-tpm v0.9.8/go.mod h1:h9jEsEECg7gtLis0upRBQU+GhYVH6jMjrFxI8u6bVUY=
git.novox.be/novox/mesh-sdk/go v0.1.11-0.20261009143344-f047d0a4a970 h1:9tFDQsgmI+4X7/BpZGXIr+HemPKE7YddYGqWV0lINAI=
git.novox.be/novox/mesh-sdk/go v0.1.11-0.20261009143344-f047d0a4a970/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010190644-622b6b4f556d h1:BR8uwv4BqyLaBwITuLylrd9DRUnu0D5DjJw0RuQSXbk=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010190644-622b6b4f556d/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c h1:l5onJwoIeH8yE/PjVlEeoPyXBsHp2NQiWLllz2fwX7s=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261010191001-33917f91ac3c/go.mod h1:GFuZUElBZ9A++mxgIKo97aXXo+kV0uJ/UkbhQPPIbrY=
github.com/klauspost/compress v1.20.0 h1:a3C1ke2ohxFymNlb2HWAHjDeKCI90scRskErZkR0ezA=
github.com/klauspost/compress v1.20.0/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
github.com/minio/highwayhash v1.0.4 h1:asJizugGgchQod2ja9NJlGOWq4s7KsAWr5XUc9Clgl4=
github.com/minio/highwayhash v1.0.4/go.mod h1:GGYsuwP/fPD6Y9hMiXuapVvlIUEhFhMTh0rxU3ik1LQ=
github.com/nats-io/jwt/v2 v2.8.1 h1:V0xpGuD/N8Mi+fQNDynXohVvp7ZztevW5io8CUWlPmU=
github.com/nats-io/jwt/v2 v2.8.1/go.mod h1:nWnOEEiVMiKHQpnAy4eXlizVEtSfzacZ1Q43LIRavZg=
github.com/nats-io/nats-server/v2 v2.11.17 h1:GKEghcFK6A+aFx11Yf1LjgLC3txAwvyhnYzhBIQZA8I=
github.com/nats-io/nats-server/v2 v2.11.17/go.mod h1:B1sFVz4StNosQ903ak4N1G01Fl/9f8e06mXpFIE2K24=
github.com/nats-io/nats.go v1.54.0 h1:vsXoOxjHp/GmPUN+EcI7uOf/uB+iAP+kEsAFNQN0yzA=
github.com/nats-io/nats.go v1.54.0/go.mod h1:y+DZoD1oBOYfZTU681eTUiUjI0vbqYGixNVFHcjHJ0k=
github.com/nats-io/nkeys v0.4.16 h1:rd5oAuLOb8mnAycB0xleuEBNS1pVVnN0fv/FF34Eypg=
@@ -26,8 +14,5 @@ github.com/nats-io/nuid v1.0.1 h1:5iA8DT8V7q8WK2EScv2padNa/rTESc1KdnPw4TC2paw=
github.com/nats-io/nuid v1.0.1/go.mod h1:19wcPz3Ph3q0Jbyiqsd0kePYG7A95tJPxeL+1OSON2c=
golang.org/x/crypto v0.57.0 h1:3ZVCjf8Ggz7zneR/EHRVx68Ctf+2pmIMP2UFhh9cC6M=
golang.org/x/crypto v0.57.0/go.mod h1:Fdz0i5U6CoizGwLda9DttjSk6qlZo25zYNtR+ycvuZA=
golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
internal/accounts/accounts.go
+2 -2
View File
@@ -45,7 +45,7 @@ import (
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
// The states, in the words liveness says them (the controller reads them alike).
@@ -119,7 +119,7 @@ func secretsOf(d *declaration.Declaration, account string) []string {
var out []string
for _, r := range d.Resources {
f, ok := r.(*declaration.File)
if !ok || f.Path == "" || (!f.Secret() && len(f.Secrets) == 0) || f.Owner == account {
if !ok || f.Path == "" || (f.Sealed == "" && len(f.Secrets) == 0) || f.Owner == account {
continue
}
out = append(out, f.Path)
internal/accounts/accounts_test.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
// The account judge (novox/hq ADR 0252, "how it is checked"): an account in a group its running session
internal/accounts/root_test.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
// An account declared never to become root without a person (novox/hq ADR 0266, "how it is checked"): each
internal/agent/agent.go
-927
View File
@@ -1,927 +0,0 @@
// Package agent is the node agent (novox/hq ADR 0429 §4, ADR 0539): the node-engine's own process that follows the
// node's wants and applies one module at a time.
//
// **It watches, it never polls.** A want `want.<node>.<module>` names a fingerprint and the digest of the one composition
// the assigner allows the node to apply for it; the composition itself waits under `<node>.<module>.<fingerprint>` in
// the node's bucket `composed-<node>`. When both are there and the composition's digest is the want's, and what runs
// differs, the agent applies that module's composition and writes its have with the fingerprint and the digest. A want
// `removed` removes that module's own resources. Nothing is applied for a composition no want names, and a failed apply
// is retried by the next change of either key, never by a clock (research 063 §9.1) — except an apply that failed only
// because the bus did not answer for a moment (Result.Transient: no responders while the bus reloads its accounts, a
// reconnect): that one is retried with a bounded backoff (Env.Backoff), and at once when the engine is told the bus is
// back (RetryTransient), since no key will change for it.
package agent
import (
"bytes"
"context"
"crypto/ed25519"
"encoding/json"
"fmt"
"sort"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Update is one change of a key in a watched bucket. Caught marks the end of the keys that were there when the watch
// began; every Update after it is a change.
type Update struct {
Key string
Value []byte
Revision uint64
Deleted bool
Caught bool
}
// Watcher follows one bucket: the keys matching the pattern now, a Caught update, then every change, until ctx ends.
type Watcher interface {
Watch(ctx context.Context, pattern string) (<-chan Update, error)
}
// NodeModule is the scope of the node's own composition, `<node>.node`.
const NodeModule = "node"
// Have is what a module's have key says, as the agent reads and writes it.
type Have struct {
Module string
FP string
Composed string
// Phase is "running", "failed", "held" or "removed"; empty reads as running.
Phase string
Detail string
}
// Haves is the node's have bucket as the agent uses it.
type Haves interface {
// Read is what the node-engine last wrote for a module, and whether it wrote anything.
Read(ctx context.Context, module string) (Have, bool)
// Write writes a module's have where it differs from the bucket's.
Write(ctx context.Context, h Have)
// Removed writes the module's have with the phase removed, once, if a key of it exists.
Removed(ctx context.Context, module string)
}
// Result is how one apply or removal came out.
type Result struct {
Failed bool
// Transient says the failure was the bus not answering for a moment (no responders, a timeout, a reconnect, a
// JetStream API error while the bus reloads), not the module's: the apply is tried again without a new revision.
Transient bool
Detail string
}
// Backup is the answer of the backup asked before a module moves.
type Backup struct {
Done bool
Words string
}
// Env is everything the agent needs of the node.
type Env struct {
Node string
Wants Watcher
Composed Watcher
Haves Haves
// Apply applies one module's composition (scope NodeModule for the node's own), given the compositions the node's
// other modules run so a path, unit or group they hold is not removed or taken. It is called with one module at a
// time.
Apply func(ctx context.Context, c delivery.Composed, others []delivery.Composed) Result
// Remove removes the module's own resources and nothing of another's.
Remove func(ctx context.Context, module string, others []delivery.Composed) Result
// Backup backs a module up before a new composition of it is applied; nil backs up nothing.
Backup func(ctx context.Context, module string) Backup
// Kept is the resources the node last applied, by module, from the declaration it kept before the switch (ADR 0539
// §9), as JSON; the node's own are under NodeModule. Nil when the node kept none. A module the agent does not hold yet
// still runs from it.
Kept func() map[string][]json.RawMessage
// KeptFilled is the paths the kept declaration names as filled by the provisioner, by the module whose file is at
// each (the node's own under NodeModule). A module that still runs from the kept declaration keeps its filled files
// filled: without them the filling refused each one as "not named as filled" (issue 1945). Nil names none.
KeptFilled func() map[string][]string
// Trusted is the public keys a composition may be signed by: the controller's and the operator's (ADR 0539 §3). A
// composition unsigned, signed by another key, or changed after it was signed is refused and nothing of it is
// applied. Nil trusts no key, so nothing is applied.
Trusted func() []ed25519.PublicKey
// Recorded says the engine holds a record of every resource of the composition, so it was applied and not removed
// since; nil says yes. A composition the kept declaration or a have says runs is applied again when it says no.
Recorded func(ctx context.Context, c delivery.Composed) bool
// Revisions are a composition key's revisions the bucket keeps, oldest first; nil reads none. The composition a want
// names is taken from them when a newer one replaced it under the same key (ADR 0539 clause 1).
Revisions func(ctx context.Context, key string) ([]Update, error)
// Settled is told a module's apply or removal that succeeded, once what runs holds it (Everything), outside the
// agent's lock: the declaration the node applies now names what that composition names, its filled files among
// them. The node-engine's filling tells the provisioner's client then to fill again what it refused (mesh-host
// internal/filling Knock; ace, 2026-10-25 21:17). Nil tells nothing.
Settled func(module string)
// Backoff is how long to wait before the try-th retry (1, 2, …) of an apply that failed transiently. Nil retries a
// transient failure only on RetryTransient or a change of either key.
Backoff func(try int) time.Duration
Say func(string)
}
// transientFor is how many transient retries say the phase applying before the have says failed: past them the bus has
// not answered for long enough that it is no longer a moment, and the failure is loud while the retries go on.
const transientFor = 4
// Agent follows the wants.
type Agent struct {
env Env
mu sync.Mutex
wants map[string]wantNow // by module
comps map[string]compNow // by composed key
running map[string]delivery.Have
// applied is the composition each module was last applied from: a revision a want named from the key's history is
// not the key's latest value, and what it holds is still held out of the others' applies.
applied map[string]compNow
// attempt is the last apply of a module that failed or is held, with the revisions it was made from.
attempt map[string]attempt
checked map[string]bool
// prior is a module the node ran before this agent started, so it has something to back up.
prior map[string]bool
said map[string]string
retry chan []string
// tries counts the transient failures in a row of each module; again carries a retry whose wait ran out, and
// transient asks for every transiently failed module at once. stopped is closed when Run returns.
tries map[string]int
gen uint64
again chan againAsk
transient chan struct{}
stopped chan struct{}
}
type againAsk struct {
module string
gen uint64
}
type wantNow struct {
Want delivery.Want
Node bool // the node's own want
Present bool
Rev uint64
}
type compNow struct {
Raw []byte
C delivery.Composed
Dig string
Rev uint64
Err error
}
type attempt struct {
FP, Composed string
WantRev, CompRev uint64
Held bool
AnsweredRev uint64
// Transient marks a failure of the bus, retried by the clock; Gen names the retry that is due for it.
Transient bool
Gen uint64
}
// New is an agent for a node.
func New(env Env) *Agent {
if env.Say == nil {
env.Say = func(string) {}
}
return &Agent{env: env, wants: map[string]wantNow{}, comps: map[string]compNow{}, running: map[string]delivery.Have{}, applied: map[string]compNow{},
attempt: map[string]attempt{}, checked: map[string]bool{}, prior: map[string]bool{}, said: map[string]string{},
retry: make(chan []string, 8), tries: map[string]int{}, again: make(chan againAsk, 64),
transient: make(chan struct{}, 1), stopped: make(chan struct{})}
}
// Run follows the wants until ctx ends. The keys there when it starts are taken first, the node's own composition
// before any module's, then each change as it comes.
func (a *Agent) Run(ctx context.Context) error {
defer close(a.stopped)
wants, err := a.env.Wants.Watch(ctx, "want."+a.env.Node+".>")
if err != nil {
return fmt.Errorf("the agent cannot watch its wants: %w", err)
}
comps, err := a.env.Composed.Watch(ctx, a.env.Node+".>")
if err != nil {
return fmt.Errorf("the agent cannot watch its compositions: %w", err)
}
var wantsCaught, compsCaught bool
dirty := map[string]bool{}
for {
select {
case <-ctx.Done():
return nil
case modules := <-a.retry:
// Applied again though the keys did not change: a container that waited because it said busy (ADR 0360
// §11) is replaced once it is idle.
a.mu.Lock()
for _, m := range modules {
delete(a.running, m)
delete(a.attempt, m)
dirty[m] = true
}
a.mu.Unlock()
case ask := <-a.again:
// The wait of a transiently failed apply ran out: tried again from the keys it has, if nothing tried it since.
a.mu.Lock()
if att, ok := a.attempt[ask.module]; ok && att.Transient && att.Gen == ask.gen {
delete(a.attempt, ask.module)
dirty[ask.module] = true
}
a.mu.Unlock()
case <-a.transient:
a.mu.Lock()
for m, att := range a.attempt {
if att.Transient {
delete(a.attempt, m)
dirty[m] = true
}
}
a.mu.Unlock()
case u, ok := <-wants:
if !ok {
return fmt.Errorf("the watch of the wants ended")
}
if u.Caught {
wantsCaught = true
} else if m := a.takeWant(u); m != "" {
dirty[m] = true
}
case u, ok := <-comps:
if !ok {
return fmt.Errorf("the watch of the compositions ended")
}
if u.Caught {
compsCaught = true
} else if m := a.takeComposition(u); m != "" {
dirty[m] = true
// The node's own may wait for this one to read (unreadableModules): looked at again with it.
dirty[NodeModule] = true
}
}
// Nothing is applied until what was there at the start is all in: a module's composition may arrive after its
// want, and the node's own composition comes before any module's.
if !(wantsCaught && compsCaught) || len(dirty) == 0 {
continue
}
a.settle(ctx, dirty)
dirty = map[string]bool{}
}
}
// Retry asks for the modules to be applied again from the keys they have, though nothing changed: what ADR 0360 §11
// needs for a container that waits because it said busy. It never blocks; a retry asked while eight wait is dropped, and
// asked again by its caller's next look.
func (a *Agent) Retry(modules ...string) {
select {
case a.retry <- modules:
default:
}
}
// RetryTransient asks for every module whose last apply failed transiently to be applied again now: the engine says it
// when it knows the bus answers again (after it wrote the bus's user list and the bus reloaded its accounts). It never
// blocks.
func (a *Agent) RetryTransient() {
select {
case a.transient <- struct{}{}:
default:
}
}
// Running is the compositions the node runs now: each module's that the agent applied or found already running, and
// the node's own. What the node judges its health by and says of itself is built from them.
func (a *Agent) Running() []delivery.Composed {
a.mu.Lock()
defer a.mu.Unlock()
mods := make([]string, 0, len(a.running))
for m := range a.running {
mods = append(mods, m)
}
sort.Strings(mods)
var out []delivery.Composed
for _, m := range mods {
h := a.running[m]
if h.Phase == delivery.PhaseRemoved {
continue
}
if c, ok := a.compositionFor(m, h.FP); ok && c.Err == nil && c.Dig == h.Composed {
out = append(out, c.C)
} else if c, ok := a.applied[m]; ok && c.Dig == h.Composed {
out = append(out, c.C)
}
}
return out
}
// takeWant takes a change of a want key and returns the module it concerns, or "" for a key that is not a want.
func (a *Agent) takeWant(u Update) string {
a.mu.Lock()
defer a.mu.Unlock()
if _, err := delivery.ParseWantNodeKey(u.Key); err == nil {
w := wantNow{Node: true, Rev: u.Revision}
if !u.Deleted {
nw, err := delivery.DecodeNodeWant(u.Value)
if err != nil {
a.sayOnce("want:node", fmt.Sprintf("the node's own want is not one: %v", err))
return ""
}
w.Present, w.Want = true, delivery.Want{FP: "", Composed: nw.Composed}
}
a.wants[NodeModule] = w
return NodeModule
}
node, module, err := delivery.ParseWantKey(u.Key)
if err != nil || node != a.env.Node {
return ""
}
w := wantNow{Rev: u.Revision}
if !u.Deleted {
want, err := delivery.DecodeWant(u.Value)
if err != nil {
a.sayOnce("want:"+module, fmt.Sprintf("the want of %s is not one: %v", module, err))
return ""
}
w.Present, w.Want = true, want
}
a.wants[module] = w
return module
}
// takeComposition takes a change of a composition key and returns the module it concerns.
func (a *Agent) takeComposition(u Update) string {
node, module, _, isNode, err := delivery.ParseComposedKey(u.Key)
if err != nil || node != a.env.Node {
return ""
}
a.mu.Lock()
defer a.mu.Unlock()
if isNode {
module = NodeModule
}
if u.Deleted {
delete(a.comps, u.Key)
return module
}
now := compNow{Raw: u.Value, Rev: u.Revision}
c, err := delivery.DecodeComposed(u.Value)
if err == nil {
var trusted []ed25519.PublicKey
if a.env.Trusted != nil {
trusted = a.env.Trusted()
}
// The signature before anything of the composition is used: its digest is what the want names.
now.C = c
now.Dig, err = delivery.VerifyComposed(c, trusted)
if err != nil {
now.C = delivery.Composed{}
a.sayOnce("verify:"+u.Key, fmt.Sprintf("the composition %s is refused and nothing of it is applied: %v", u.Key, err))
err = fmt.Errorf("refused: %w", err)
}
} else {
a.sayOnce("decode:"+u.Key, fmt.Sprintf("the composition %s cannot be read, and nothing of it is applied: %v", u.Key, err))
}
now.Err = err
a.comps[u.Key] = now
return module
}
func (a *Agent) sayOnce(key, line string) {
if a.said[key] == line {
return
}
a.said[key] = line
a.env.Say(line)
}
// settle applies what the dirty modules' wants allow: the node's own first, then each module in name order.
func (a *Agent) settle(ctx context.Context, dirty map[string]bool) {
order := make([]string, 0, len(dirty))
for m := range dirty {
order = append(order, m)
}
sort.Slice(order, func(i, j int) bool {
if (order[i] == NodeModule) != (order[j] == NodeModule) {
return order[i] == NodeModule
}
return order[i] < order[j]
})
for _, m := range order {
if ctx.Err() != nil {
return
}
a.settleOne(ctx, m)
}
}
func (a *Agent) compositionFor(module, fp string) (compNow, bool) {
var key string
var err error
if module == NodeModule {
key, err = delivery.ComposedNodeKey(a.env.Node)
} else {
key, err = delivery.ComposedKey(a.env.Node, module, fp)
}
if err != nil {
return compNow{}, false
}
c, ok := a.comps[key]
return c, ok
}
// revisionNamed is the revision of a module's composition key whose digest is the one given, signed by a trusted key,
// from the revisions the bucket keeps; false when none is.
func (a *Agent) revisionNamed(ctx context.Context, module, fp, digest string) (compNow, bool) {
if a.env.Revisions == nil {
return compNow{}, false
}
var key string
var err error
if module == NodeModule {
key, err = delivery.ComposedNodeKey(a.env.Node)
} else {
key, err = delivery.ComposedKey(a.env.Node, module, fp)
}
if err != nil {
return compNow{}, false
}
revs, err := a.env.Revisions(ctx, key)
if err != nil {
a.mu.Lock()
a.sayOnce("history:"+key, fmt.Sprintf("the revisions of %s cannot be read: %v", key, err))
a.mu.Unlock()
return compNow{}, false
}
var trusted []ed25519.PublicKey
if a.env.Trusted != nil {
trusted = a.env.Trusted()
}
for i := len(revs) - 1; i >= 0; i-- {
u := revs[i]
if u.Deleted || u.Key != key {
continue
}
c, err := delivery.DecodeComposed(u.Value)
if err != nil {
continue
}
dig, err := delivery.VerifyComposed(c, trusted)
if err != nil || dig != digest {
continue
}
return compNow{Raw: u.Value, C: c, Dig: dig, Rev: u.Revision}, true
}
return compNow{}, false
}
// others are what the node's other modules run now, so what they hold is not taken: the compositions the agent runs and,
// for every module the agent does not hold yet, the resources the kept declaration has for it (ADR 0429 §4).
func (a *Agent) others(except string) []delivery.Composed {
var out []delivery.Composed
mods := make([]string, 0, len(a.running))
for m := range a.running {
if m != except {
mods = append(mods, m)
}
}
sort.Strings(mods)
for _, m := range mods {
h := a.running[m]
if h.Phase == delivery.PhaseRemoved {
continue
}
if c, ok := a.compositionFor(m, h.FP); ok && c.Err == nil && c.Dig == h.Composed {
out = append(out, c.C)
} else if c, ok := a.applied[m]; ok && c.Dig == h.Composed {
out = append(out, c.C)
}
}
if a.env.Kept != nil {
kept := a.env.Kept()
names := make([]string, 0, len(kept))
for m := range kept {
if _, held := a.running[m]; !held && m != except {
names = append(names, m)
}
}
sort.Strings(names)
for _, m := range names {
c := delivery.Composed{Node: a.env.Node, Module: m, Resources: kept[m]}
if a.env.KeptFilled != nil {
c.Filled = a.env.KeptFilled()[m]
}
if m == NodeModule {
c.Module = ""
}
out = append(out, c)
}
}
return out
}
// unreadableModules are the keys of the module compositions in the node's bucket that cannot be read or are refused, of a
// module the assigner has not removed: for each module only the composition its want names, else its newest, since a
// stale one nobody applies (written before a rule the SDK now holds) must not hold the node's own back. Called holding
// a.mu.
func (a *Agent) unreadableModules() []string {
type pick struct {
key string
c compNow
}
current := map[string]pick{}
for key, c := range a.comps {
_, m, fp, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode {
continue
}
w := a.wants[m]
if w.Present && w.Want.Removed {
continue
}
if w.Present && w.Want.FP != "" {
if w.Want.FP == fp {
current[m] = pick{key, c}
}
continue
}
if prev, ok := current[m]; !ok || c.Rev > prev.c.Rev {
current[m] = pick{key, c}
}
}
var out []string
for _, p := range current {
if p.c.Err != nil {
out = append(out, p.key)
}
}
sort.Strings(out)
return out
}
// heldOut is what an apply or a removal holds out: others, and every module whose own composition the node's bucket holds
// and the agent does not run yet. A module that left the node's own composition for one of its own keeps its resources
// until that one is applied, so the node's own apply never removes them in between (issue 1945, live on 2026-10-25:
// shanks lost systemd-resolved's guard while its composition waited for a credential). The composition its want names,
// else its newest. Not part of Everything: what has not been applied is not what runs.
func (a *Agent) heldOut(except string) []delivery.Composed {
out := a.others(except)
held := map[string]bool{}
for _, c := range out {
held[c.Module] = true
}
waiting := map[string]compNow{}
for key, c := range a.comps {
_, m, fp, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode || c.Err != nil || m == except || held[m] {
continue
}
if h, ok := a.running[m]; ok && h.Phase != delivery.PhaseRemoved {
continue
}
if w := a.wants[m]; w.Present && w.Want.Removed {
continue // a module the assigner removed holds nothing back
}
if w := a.wants[m]; w.Present && !w.Want.Removed && w.Want.FP == fp {
waiting[m] = c
continue
}
if w := a.wants[m]; w.Present && !w.Want.Removed && waiting[m].Rev != 0 && waiting[m].C.FP == w.Want.FP {
continue
}
if prev, ok := waiting[m]; !ok || c.Rev > prev.Rev {
waiting[m] = c
}
}
names := make([]string, 0, len(waiting))
for m := range waiting {
names = append(names, m)
}
sort.Strings(names)
for _, m := range names {
out = append(out, waiting[m].C)
}
return out
}
// Everything is what the node runs: the compositions the agent runs, and the kept declaration's resources for each module
// the agent does not hold yet. What the node judges its health by, and what the filling, the take act and the judgement
// of directories read, is built from it.
func (a *Agent) Everything() []delivery.Composed {
a.mu.Lock()
defer a.mu.Unlock()
return a.others("\x00")
}
// HoldsNode says whether the agent runs the node's own composition, so the node's parts are its and not the kept
// declaration's.
func (a *Agent) HoldsNode() bool {
a.mu.Lock()
defer a.mu.Unlock()
h, ok := a.running[NodeModule]
return ok && h.Phase != delivery.PhaseRemoved
}
func (a *Agent) settleOne(ctx context.Context, module string) {
a.mu.Lock()
w := a.wants[module]
a.mu.Unlock()
if !w.Present {
return
}
if w.Want.Removed {
a.remove(ctx, module, w)
return
}
a.mu.Lock()
comp, have := a.compositionFor(module, w.Want.FP)
a.mu.Unlock()
if (!have || comp.Err != nil || comp.Dig != w.Want.Composed) && w.Want.Composed != "" {
// The composition the want names may be an older revision of the key, replaced by a newer one at the same
// fingerprint (a want held there keeps its digest): taken from the revisions the bucket keeps (ADR 0539 clause 1).
if old, found := a.revisionNamed(ctx, module, w.Want.FP, w.Want.Composed); found {
comp, have = old, true
}
}
if !have || comp.Err != nil {
return // waits for the composition; a change of its key settles the module again
}
if comp.Dig != w.Want.Composed {
// A composition whose digest no want names is not applied.
a.mu.Lock()
a.sayOnce("digest:"+module, fmt.Sprintf("the composition of %s at %s has digest %s, and its want names %s; nothing is applied until they agree",
module, w.Want.FP, comp.Dig, w.Want.Composed))
a.mu.Unlock()
return
}
a.mu.Lock()
if !a.checked[module] {
a.checked[module] = true
a.mu.Unlock()
if a.baseline(ctx, module, w, comp) {
return
}
} else {
a.mu.Unlock()
}
a.mu.Lock()
if r, ok := a.running[module]; ok && r.FP == w.Want.FP && r.Composed == w.Want.Composed && r.Phase != delivery.PhaseRemoved {
a.mu.Unlock()
return
}
att, tried := a.attempt[module]
same := tried && att.FP == w.Want.FP && att.Composed == w.Want.Composed
if same && !att.Held && att.WantRev == w.Rev && att.CompRev == comp.Rev {
a.mu.Unlock()
return // failed already for exactly these keys: retried by the next change of either
}
if same && att.Held && att.AnsweredRev >= w.Rev && att.CompRev == comp.Rev {
a.mu.Unlock()
return // held for a failed backup, and no answer since
}
if module == NodeModule {
// Not while a module's composition cannot be read or is refused: what it carries may be what left the node's own,
// and the node's own apply would remove it, held out by nothing (issue 1945, the review's gap 2).
if bad := a.unreadableModules(); len(bad) > 0 {
a.sayOnce("node-waits:"+strings.Join(bad, ","), fmt.Sprintf("the node's own composition is not applied while "+
"%s cannot be read or is refused: what it carries would be removed", strings.Join(bad, ", ")))
a.mu.Unlock()
return
}
}
others := a.heldOut(module)
a.mu.Unlock()
a.apply(ctx, module, w, comp, others)
}
// baseline decides whether what runs already is what the want names, once per module at the agent's start: the node's
// own have (what this node-engine last applied), else the declaration it kept before the switch (ADR 0539 §9). True when
// it is, with the have written; nothing is applied.
func (a *Agent) baseline(ctx context.Context, module string, w wantNow, comp compNow) bool {
// What a have or the kept declaration says runs is believed only when the engine still holds a record of each of its
// resources: a resource removed since (a node's own apply took it, issue 1945) is made again by an apply.
if a.env.Recorded != nil && !a.env.Recorded(ctx, comp.C) {
if module != NodeModule {
if _, ok := a.env.Haves.Read(ctx, module); ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
}
if a.env.Kept != nil {
if _, ok := a.env.Kept()[module]; ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
}
a.mu.Lock()
a.sayOnce("unrecorded:"+module+":"+comp.Dig, fmt.Sprintf("%s's composition %s is applied again: this engine holds "+
"no record of some of its resources", module, short(comp.Dig)))
a.mu.Unlock()
return false
}
if module != NodeModule {
if h, ok := a.env.Haves.Read(ctx, module); ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
if h.Composed != "" {
// This node-engine wrote it: what it says is what runs, and the kept declaration says nothing more. A
// failed apply or a different composition is applied again.
if h.FP == w.Want.FP && h.Composed == w.Want.Composed &&
(h.Phase == "" || h.Phase == string(delivery.PhaseRunning)) {
a.mu.Lock()
a.running[module] = delivery.Have{FP: h.FP, Composed: h.Composed, Phase: delivery.PhaseRunning}
a.mu.Unlock()
// Said again, so the session holds the module and its key carries the health this engine judges,
// not the one an older engine left there (xorg's unknown on g14 and shanks, 2026-10-25).
a.env.Haves.Write(ctx, Have{Module: module, FP: h.FP, Composed: h.Composed,
Phase: string(delivery.PhaseRunning), Detail: h.Detail})
return true
}
return false
}
}
}
if a.env.Kept == nil {
return false
}
kept, ok := a.env.Kept()[module]
if ok {
a.mu.Lock()
a.prior[module] = true
a.mu.Unlock()
}
if !ok || !SameResources(kept, comp.C.Resources) {
return false
}
a.mu.Lock()
a.running[module] = delivery.Have{FP: w.Want.FP, Composed: w.Want.Composed, Phase: delivery.PhaseRunning}
a.mu.Unlock()
a.env.Haves.Write(ctx, Have{Module: module, FP: w.Want.FP, Composed: w.Want.Composed, Phase: string(delivery.PhaseRunning)})
return true
}
func (a *Agent) apply(ctx context.Context, module string, w wantNow, comp compNow, others []delivery.Composed) {
fp, dig := w.Want.FP, w.Want.Composed
a.mu.Lock()
_, ranBefore := a.running[module]
ranBefore = ranBefore || a.prior[module]
a.mu.Unlock()
// A data module is backed up before a new composition of it moves (ADR 0429 §14), unless the want says to go on
// without one (ADR 0539 §16).
if comp.C.Data && ranBefore && a.env.Backup != nil && w.Want.Backup != delivery.BackupSkip {
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseBackingUp)})
if got := a.env.Backup(ctx, module); !got.Done {
a.env.Say(fmt.Sprintf("%s is held: its backup before %s failed: %s", module, fp, got.Words))
a.mu.Lock()
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev, Held: true, AnsweredRev: w.Rev}
prev := a.running[module]
a.mu.Unlock()
// It keeps running what it has: the have says what it holds and why it waits.
a.env.Haves.Write(ctx, Have{Module: module, FP: prev.FP, Composed: prev.Composed, Phase: string(delivery.PhaseHeld),
Detail: got.Words})
return
}
}
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseApplying)})
res := a.env.Apply(ctx, comp.C, others)
a.mu.Lock()
what := fmt.Sprintf("%s at %s", module, fp)
if module == NodeModule {
what = "the node's own composition"
}
if res.Failed && res.Transient {
a.tries[module]++
try := a.tries[module]
a.gen++
gen := a.gen
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev, Transient: true, Gen: gen}
a.mu.Unlock()
phase, when := delivery.PhaseApplying, "when the bus answers again"
if a.env.Backoff != nil {
wait := a.env.Backoff(try)
when = "in " + wait.String()
time.AfterFunc(wait, func() {
select {
case a.again <- againAsk{module, gen}:
case <-a.stopped:
case <-ctx.Done():
}
})
}
if try > transientFor {
phase = delivery.PhaseFailed // still retried, and said as failed: no longer a moment
}
a.env.Say(fmt.Sprintf("applying %s failed because the bus did not answer (try %d), retried %s: %s", what, try, when, res.Detail))
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(phase),
Detail: fmt.Sprintf("retrying %s (try %d): %s", when, try, res.Detail)})
return
}
delete(a.tries, module)
if res.Failed {
a.attempt[module] = attempt{FP: fp, Composed: dig, WantRev: w.Rev, CompRev: comp.Rev}
a.mu.Unlock()
a.env.Say(fmt.Sprintf("applying %s failed: %s", what, res.Detail))
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed), Detail: res.Detail})
return
}
delete(a.attempt, module)
a.running[module] = delivery.Have{FP: fp, Composed: dig, Phase: delivery.PhaseRunning}
a.applied[module] = comp
a.mu.Unlock()
if a.env.Settled != nil {
a.env.Settled(module)
}
a.env.Haves.Write(ctx, Have{Module: module, FP: fp, Composed: dig, Phase: string(delivery.PhaseRunning), Detail: res.Detail})
}
func (a *Agent) remove(ctx context.Context, module string, w wantNow) {
a.mu.Lock()
att := a.attempt[module]
r, ran := a.running[module]
if att.FP == "removed" && att.WantRev == w.Rev {
a.mu.Unlock()
return
}
if ran && r.Phase == delivery.PhaseRemoved {
a.mu.Unlock()
return
}
others := a.heldOut(module)
a.mu.Unlock()
res := a.env.Remove(ctx, module, others)
a.mu.Lock()
if res.Failed {
a.attempt[module] = attempt{FP: "removed", WantRev: w.Rev}
a.mu.Unlock()
a.env.Say(fmt.Sprintf("removing %s failed: %s", module, res.Detail))
return
}
delete(a.attempt, module)
a.running[module] = delivery.Have{Phase: delivery.PhaseRemoved}
a.mu.Unlock()
if a.env.Settled != nil {
a.env.Settled(module)
}
a.env.Haves.Removed(ctx, module)
}
// SameResources says whether two lists of resources hold the same resources, each by its id and its canonical JSON.
func SameResources(a, b []json.RawMessage) bool {
ma, ok := canonByID(a)
if !ok {
return false
}
mb, ok := canonByID(b)
if !ok || len(ma) != len(mb) {
return false
}
for id, x := range ma {
if y, there := mb[id]; !there || x != y {
return false
}
}
return true
}
func canonByID(rs []json.RawMessage) (map[string]string, bool) {
out := map[string]string{}
for _, raw := range rs {
var head struct {
ID string `json:"id"`
}
if json.Unmarshal(raw, &head) != nil || head.ID == "" {
return nil, false
}
canon, err := Canonical(raw)
if err != nil {
return nil, false
}
out[head.ID] = string(canon)
}
return out, true
}
// Canonical is raw with its object keys sorted and its spacing dropped, its numbers kept as written.
func Canonical(raw json.RawMessage) (json.RawMessage, error) {
var v any
d := json.NewDecoder(bytes.NewReader(raw))
d.UseNumber()
if err := d.Decode(&v); err != nil {
return nil, err
}
return json.Marshal(v)
}
func short(s string) string {
if i := strings.IndexByte(s, ':'); i >= 0 {
s = s[i+1:]
}
if len(s) > 12 {
s = s[:12]
}
return s
}
internal/agent/agent_test.go
-954
View File
@@ -1,954 +0,0 @@
package agent
import (
"context"
"crypto/ed25519"
"encoding/json"
"slices"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
const node = "n1"
// The keys of the tests: the controller's and the operator's are trusted, a stranger's is not.
var controllerPub, controllerKey = newKey()
var operatorPub, operatorKey = newKey()
var _, strangerKey = newKey()
func newKey() (ed25519.PublicKey, ed25519.PrivateKey) {
pub, key, err := ed25519.GenerateKey(nil)
if err != nil {
panic(err)
}
return pub, key
}
func fpOf(c string) string { return delivery.SourcePrefix2 + strings.Repeat(c, 64) }
// A watcher the test writes to; it says Caught once the keys written before the watch began are out.
type feed struct {
mu sync.Mutex
ch chan Update
rev uint64
first []Update
}
func newFeed() *feed { return &feed{ch: make(chan Update, 256)} }
func (f *feed) Watch(_ context.Context, _ string) (<-chan Update, error) {
f.mu.Lock()
defer f.mu.Unlock()
for _, u := range f.first {
f.ch <- u
}
f.ch <- Update{Caught: true}
return f.ch, nil
}
// before writes a key as if it was there when the watch began.
func (f *feed) before(key string, value []byte) {
f.mu.Lock()
defer f.mu.Unlock()
f.rev++
f.first = append(f.first, Update{Key: key, Value: value, Revision: f.rev})
}
func (f *feed) put(key string, value []byte) {
f.mu.Lock()
f.rev++
r := f.rev
f.mu.Unlock()
f.ch <- Update{Key: key, Value: value, Revision: r}
}
type haves struct {
mu sync.Mutex
stored map[string]Have
log []Have
removed []string
}
func (h *haves) Read(_ context.Context, module string) (Have, bool) {
h.mu.Lock()
defer h.mu.Unlock()
x, ok := h.stored[module]
return x, ok
}
func (h *haves) Write(_ context.Context, x Have) {
h.mu.Lock()
defer h.mu.Unlock()
if h.stored == nil {
h.stored = map[string]Have{}
}
h.stored[x.Module] = x
h.log = append(h.log, x)
}
func (h *haves) Removed(_ context.Context, m string) {
h.mu.Lock()
defer h.mu.Unlock()
h.removed = append(h.removed, m)
}
func (h *haves) last(module string) Have {
h.mu.Lock()
defer h.mu.Unlock()
return h.stored[module]
}
type bed struct {
t *testing.T
wants *feed
comps *feed
haves *haves
mu sync.Mutex
applied []string
removed []string
backups []string
failWith string
// transient marks failWith as the bus not answering; failTimes > 0 fails only the first that many applies.
transient bool
failTimes int
backup func(module string) Backup
kept map[string][]json.RawMessage
others [][]delivery.Composed
said []string
// settled is each module Settled was told, and settledRan whether what runs held it then.
settled []string
settledRan []bool
ag *Agent // the agent run() runs
env Env
}
func newBed(t *testing.T) *bed {
b := &bed{t: t, wants: newFeed(), comps: newFeed(), haves: &haves{}}
b.env = Env{Node: node, Wants: b.wants, Composed: b.comps, Haves: b.haves,
Apply: func(_ context.Context, c delivery.Composed, others []delivery.Composed) Result {
b.mu.Lock()
defer b.mu.Unlock()
b.others = append(b.others, others)
name := c.Module
if name == "" {
name = NodeModule
}
b.applied = append(b.applied, name)
if b.failWith != "" && (b.failTimes == 0 || len(b.applied) <= b.failTimes) {
return Result{Failed: true, Transient: b.transient, Detail: b.failWith}
}
return Result{}
},
Remove: func(_ context.Context, m string, _ []delivery.Composed) Result {
b.mu.Lock()
defer b.mu.Unlock()
b.removed = append(b.removed, m)
return Result{}
},
Backup: func(_ context.Context, m string) Backup {
b.mu.Lock()
defer b.mu.Unlock()
b.backups = append(b.backups, m)
if b.backup != nil {
return b.backup(m)
}
return Backup{Done: true}
},
Settled: func(m string) {
ran := false
if b.ag != nil {
for _, c := range b.ag.Everything() {
ran = ran || c.Module == m
}
}
b.mu.Lock()
defer b.mu.Unlock()
b.settled = append(b.settled, m)
b.settledRan = append(b.settledRan, ran)
},
Kept: func() map[string][]json.RawMessage { return b.kept },
Trusted: func() []ed25519.PublicKey { return []ed25519.PublicKey{controllerPub, operatorPub} },
Say: func(s string) {
t.Log(s)
b.mu.Lock()
b.said = append(b.said, s)
b.mu.Unlock()
},
}
return b
}
func (b *bed) run() (stop func()) {
ctx, cancel := context.WithCancel(context.Background())
done := make(chan struct{})
b.ag = New(b.env)
go func() { defer close(done); _ = b.ag.Run(ctx) }()
return func() { cancel(); <-done }
}
func file(module, id, content string) json.RawMessage {
raw, _ := json.Marshal(map[string]any{"id": module + "." + id, "type": "file", "path": "/tmp/r6/" + module + "/" + id,
"content": content, "module": module})
return raw
}
// composition is a module's composition at a fingerprint: its key, its bytes and its digest, from the writer's code.
func composition(t *testing.T, module, fp string, data bool, rs ...json.RawMessage) (key string, raw []byte, digest string) {
t.Helper()
c := delivery.Composed{Node: node, Module: module, FP: fp, Data: data, Resources: rs}
var err error
if module == NodeModule {
c.Module, c.FP = "", ""
key, err = delivery.ComposedNodeKey(node)
} else {
key, err = delivery.ComposedKey(node, module, fp)
}
if err != nil {
t.Fatal(err)
}
return signedBy(t, c, key, controllerKey)
}
// signedBy is c signed with signer, as the writer's code signs it: its key, its bytes and its digest.
func signedBy(t *testing.T, c delivery.Composed, key string, signer ed25519.PrivateKey) (string, []byte, string) {
t.Helper()
signed, err := delivery.SignComposed(c, signer)
if err != nil {
t.Fatal(err)
}
raw, err := delivery.EncodeComposed(signed)
if err != nil {
t.Fatal(err)
}
digest, err := delivery.ComposedDigestOf(signed)
if err != nil {
t.Fatal(err)
}
return key, raw, digest
}
func wantFor(t *testing.T, module, fp, digest, backup string) (string, []byte) {
t.Helper()
key, err := delivery.WantKey(node, module)
if err != nil {
t.Fatal(err)
}
raw, err := delivery.EncodeWant(delivery.Want{FP: fp, Composed: digest, Backup: backup})
if err != nil {
t.Fatal(err)
}
return key, raw
}
func eventually(t *testing.T, what string, ok func() bool) {
t.Helper()
deadline := time.Now().Add(3 * time.Second)
for !ok() {
if time.Now().After(deadline) {
t.Fatal(what)
}
time.Sleep(5 * time.Millisecond)
}
}
func (b *bed) appliedList() []string {
b.mu.Lock()
defer b.mu.Unlock()
return append([]string(nil), b.applied...)
}
// R6-A AC1 (the agent, with a bus that is a fake): given a want naming a digest and that composition, the module is
// applied and its have names the fingerprint and the digest; a composition whose digest no want names is not applied.
func TestAWantAndItsCompositionApplyTheModuleAndTheHaveNamesBoth(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
// A composition no want names yet is not applied.
b.comps.put(ck, craw)
time.Sleep(100 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a composition no want names was applied: %v", got)
}
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "the module was not applied", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "the have does not name both", func() bool {
h := b.haves.last("web")
return h.FP == fp && h.Composed == dig && h.Phase == string(delivery.PhaseRunning)
})
// Settled is told once what runs holds the module, so a declaration read then names its files (the filling's Knock;
// ace, 2026-10-25 21:17).
eventually(t, "Settled was not told web with web in what runs", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return slices.Equal(b.settled, []string{"web"}) && b.settledRan[0]
})
// A want naming a digest the composition at that fingerprint does not have: nothing is applied.
fp2 := fpOf("b")
ck2, craw2, _ := composition(t, "web", fp2, false, file("web", "x", "2"))
b.comps.put(ck2, craw2)
wk2, wraw2 := wantFor(t, "web", fp2, "sha256:"+strings.Repeat("0", 64), "")
b.wants.put(wk2, wraw2)
time.Sleep(100 * time.Millisecond)
if got := b.appliedList(); len(got) != 1 {
t.Fatalf("a want naming another digest applied the composition: %v", got)
}
}
// R6-A AC2: given a kept declaration and wants and compositions equal to it, when the agent starts, nothing is applied
// and every have names its fingerprint and digest.
func TestStartingOnWhatTheKeptDeclarationHoldsAppliesNothing(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
rs := []json.RawMessage{file("web", "x", "1"), file("web", "y", "2")}
ck, craw, dig := composition(t, "web", fp, false, rs...)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
// The kept declaration holds the same resources, in another order and spacing.
b.kept = map[string][]json.RawMessage{"web": {json.RawMessage(strings.ReplaceAll(string(rs[1]), ",", " , ")), rs[0]}}
stop := b.run()
defer stop()
eventually(t, "the have was not written", func() bool { return b.haves.last("web").Composed == dig })
if h := b.haves.last("web"); h.FP != fp || h.Phase != string(delivery.PhaseRunning) {
t.Fatalf("the have: %+v", h)
}
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that already runs what the want names was applied: %v", got)
}
}
// What the node-engine itself last applied is what runs after a restart, not the kept declaration.
func TestStartingOnItsOwnHaveAppliesNothing(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: dig, Phase: "running"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
time.Sleep(150 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that runs what the want names was applied after a restart: %v", got)
}
}
// A module found running on its own have after a restart is said again, once, through the haves: the session then holds
// it and writes the health this engine judges. Not said, its key kept the health an older engine wrote: xorg's have on
// g14 and shanks said unknown from before the engine said a module with nothing to judge ok, and the assigner held every
// module requiring x11-display back for it (2026-10-25).
func TestStartingOnItsOwnHaveSaysTheHaveAgainSoItsHealthIsThisEngines(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "xorg", fp, false, file("xorg", "x", "1"))
wk, wraw := wantFor(t, "xorg", fp, dig, "")
b.haves.stored = map[string]Have{"xorg": {Module: "xorg", FP: fp, Composed: dig, Phase: "running", Detail: "kept"}}
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
stop := b.run()
defer stop()
eventually(t, "the have found running was not said again", func() bool {
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
return len(b.haves.log) > 0
})
time.Sleep(50 * time.Millisecond)
b.haves.mu.Lock()
log := append([]Have(nil), b.haves.log...)
b.haves.mu.Unlock()
if len(log) != 1 {
t.Fatalf("the have was said %d times, want once: %+v", len(log), log)
}
if h := log[0]; h.Module != "xorg" || h.FP != fp || h.Composed != dig || h.Phase != string(delivery.PhaseRunning) || h.Detail != "kept" {
t.Fatalf("the have said again: %+v", h)
}
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a module that runs what the want names was applied: %v", got)
}
}
// A module whose kept resources differ from the composition is applied, and a failed apply is retried only by a change
// of either key.
func TestAFailedApplyIsRetriedOnlyByAChangeOfEitherKey(t *testing.T) {
// A real failure is not retried by the clock, however short the backoff: only by a change of either key.
t.Run("a real failure", func(t *testing.T) {
b := newBed(t)
b.failWith = "web.x: no space"
b.env.Backoff = func(int) time.Duration { return time.Millisecond }
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
eventually(t, "the have does not say failed", func() bool {
h := b.haves.last("web")
return h.Phase == string(delivery.PhaseFailed) && strings.Contains(h.Detail, "no space") && h.FP == fp
})
// An unrelated key changes: not a retry.
ck9, craw9, _ := composition(t, "web", fpOf("9"), false, file("web", "x", "9"))
b.comps.put(ck9, craw9)
time.Sleep(150 * time.Millisecond)
if n := len(b.appliedList()); n != 1 {
t.Fatalf("a failed apply was retried by an unrelated key or the clock (%d applies)", n)
}
// The want is written again: the retry.
b.mu.Lock()
b.failWith = ""
b.mu.Unlock()
b.wants.put(wk, wraw)
eventually(t, "the change of the want did not retry", func() bool { return len(b.appliedList()) == 2 })
eventually(t, "not running", func() bool { return b.haves.last("web").Phase == string(delivery.PhaseRunning) })
})
// The bus did not answer for a moment (mesh-vault on the bus node, 2026-10-25 23:04:24): tried again with no new
// revision of either key, by the backoff or by RetryTransient once the engine knows the bus is back; the have says
// retrying meanwhile, never failed.
for name, backoff := range map[string]func(int) time.Duration{
"a transient failure, by the backoff": func(int) time.Duration { return time.Millisecond },
"a transient failure, by RetryTransient": nil,
} {
t.Run(name, func(t *testing.T) {
b := newBed(t)
b.failWith, b.transient, b.failTimes = "fetching the object: nats: no responders available for request", true, 1
b.env.Backoff = backoff
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "vault", fp, false, file("vault", "x", "1"))
wk, wraw := wantFor(t, "vault", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) >= 1 })
if backoff == nil {
eventually(t, "the have does not say retrying", func() bool {
return strings.HasPrefix(b.haves.last("vault").Detail, "retrying")
})
b.ag.RetryTransient()
}
eventually(t, "not applied again without a new revision", func() bool { return len(b.appliedList()) == 2 })
eventually(t, "not running", func() bool { return b.haves.last("vault").Phase == string(delivery.PhaseRunning) })
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
for _, h := range b.haves.log {
if h.Phase == string(delivery.PhaseFailed) {
t.Errorf("a transient failure was written failed: %+v", h)
}
}
})
}
}
// R6-A AC5: a data module that ran before asks its backup before any resource changes; a refusal holds it, nothing is
// applied and the have says held; when the want then says skip, the module is applied.
func TestADataModuleIsBackedUpFirstAndHeldWhenItFails(t *testing.T) {
b := newBed(t)
fp1, fp2 := fpOf("a"), fpOf("b")
ck1, craw1, dig1 := composition(t, "db", fp1, true, file("db", "x", "1"))
wk1, wraw1 := wantFor(t, "db", fp1, dig1, "")
b.comps.before(ck1, craw1)
b.wants.before(wk1, wraw1)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}}
refuse := true
b.backup = func(string) Backup {
if refuse {
return Backup{Words: "restic: the repository is locked"}
}
return Backup{Done: true}
}
stop := b.run()
defer stop()
eventually(t, "baseline", func() bool { return b.haves.last("db").Composed == dig1 })
ck2, craw2, dig2 := composition(t, "db", fp2, true, file("db", "x", "2"))
b.comps.put(ck2, craw2)
wk2, wraw2 := wantFor(t, "db", fp2, dig2, "")
b.wants.put(wk2, wraw2)
eventually(t, "the backup was not asked", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.backups) == 1
})
eventually(t, "the have does not say held", func() bool {
h := b.haves.last("db")
return h.Phase == string(delivery.PhaseHeld) && strings.Contains(h.Detail, "repository is locked") && h.FP == fp1
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("something was applied behind a failed backup: %v", got)
}
// The answer arrives in the module's own want: back up again (it fails again), then skip.
wk, again := wantFor(t, "db", fp2, dig2, delivery.BackupAgain)
b.wants.put(wk, again)
eventually(t, "back up again did not ask the backup", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.backups) == 2
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied behind a second failed backup: %v", got)
}
_, skip := wantFor(t, "db", fp2, dig2, delivery.BackupSkip)
b.wants.put(wk, skip)
eventually(t, "skip did not apply the module", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
n := len(b.backups)
b.mu.Unlock()
if n != 2 {
t.Fatalf("a backup was asked after the answer was to skip: %d asks", n)
}
eventually(t, "not running at the new fingerprint", func() bool {
h := b.haves.last("db")
return h.Phase == string(delivery.PhaseRunning) && h.FP == fp2 && h.Composed == dig2
})
}
// A data module with a good backup is applied after it; one that never ran has nothing to back up.
func TestABackupDoneIsFollowedByTheApplyAndANewModuleIsNotBackedUp(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("c")
ck, craw, dig := composition(t, "db", fp, true, file("db", "x", "1"))
wk, wraw := wantFor(t, "db", fp, dig, "")
b.comps.put(ck, craw)
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
n := len(b.backups)
b.mu.Unlock()
if n != 0 {
t.Fatalf("a module that never ran was backed up (%d)", n)
}
fp2 := fpOf("d")
ck2, craw2, dig2 := composition(t, "db", fp2, true, file("db", "x", "2"))
wk2, wraw2 := wantFor(t, "db", fp2, dig2, "")
b.comps.put(ck2, craw2)
b.wants.put(wk2, wraw2)
eventually(t, "second not applied", func() bool { return len(b.appliedList()) == 2 })
b.mu.Lock()
n = len(b.backups)
b.mu.Unlock()
if n != 1 {
t.Fatalf("a module that ran before was backed up %d times, want once", n)
}
}
// R6-A AC4 at the agent: a removed want removes that module only, and its have is written removed.
func TestARemovedWantRemovesThatModuleOnly(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
gone, _ := delivery.EncodeWant(delivery.Want{Removed: true})
b.wants.put(wk, gone)
eventually(t, "not removed", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
return len(b.removed) == 1 && b.removed[0] == "web"
})
eventually(t, "the have was not written removed", func() bool {
b.haves.mu.Lock()
defer b.haves.mu.Unlock()
return len(b.haves.removed) == 1
})
}
// The node's own composition is applied before any module's, whichever key came first.
func TestTheNodesOwnCompositionComesBeforeAnyModules(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, err := delivery.WantNodeKey(node)
if err != nil {
t.Fatal(err)
}
nwraw, err := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
if err != nil {
t.Fatal(err)
}
b.comps.before(ck, craw)
b.comps.before(nk, nraw)
b.wants.before(wk, wraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "both not applied", func() bool { return len(b.appliedList()) == 2 })
if got := b.appliedList(); got[0] != NodeModule || got[1] != "web" {
t.Fatalf("the order was %v, the node's own first", got)
}
}
// R6-A AC9: a composition signed by either trusted key is applied; one signed by another key, one changed after it was
// signed and one unsigned are refused and nothing of them is applied (ADR 0539 §3).
func TestOnlyACompositionSignedByATrustedKeyIsApplied(t *testing.T) {
b := newBed(t)
stop := b.run()
defer stop()
plain := func(module, fp, content string) delivery.Composed {
return delivery.Composed{Node: node, Module: module, FP: fp, Resources: []json.RawMessage{file(module, "x", content)}}
}
put := func(c delivery.Composed, signer ed25519.PrivateKey, tamper func([]byte) []byte) {
key, _ := delivery.ComposedKey(node, c.Module, c.FP)
var raw []byte
var dig string
if signer == nil {
raw, _ = delivery.EncodeComposed(c)
dig, _ = delivery.ComposedDigestOf(c)
} else {
_, raw, dig = signedBy(t, c, key, signer)
}
if tamper != nil {
raw = tamper(raw)
}
b.comps.put(key, raw)
wk, wraw := wantFor(t, c.Module, c.FP, dig, "")
b.wants.put(wk, wraw)
}
// Refused: signed by a stranger, changed after signing, unsigned.
put(plain("stranger", fpOf("1"), "x"), strangerKey, nil)
put(plain("changed", fpOf("2"), "before"), controllerKey, func(raw []byte) []byte {
return []byte(strings.Replace(string(raw), "before", "after!", 1))
})
put(plain("unsigned", fpOf("3"), "x"), nil, nil)
time.Sleep(200 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("a composition that is not signed by a trusted key was applied: %v", got)
}
// Applied: signed by the controller, signed by the operator.
put(plain("bycontroller", fpOf("4"), "x"), controllerKey, nil)
put(plain("byoperator", fpOf("5"), "x"), operatorKey, nil)
eventually(t, "the trusted signers' compositions were not applied", func() bool { return len(b.appliedList()) == 2 })
got := b.appliedList()
if !(got[0] == "bycontroller" && got[1] == "byoperator") && !(got[0] == "byoperator" && got[1] == "bycontroller") {
t.Fatalf("applied %v", got)
}
if h := b.haves.last("byoperator"); h.Phase != string(delivery.PhaseRunning) || h.FP != fpOf("5") {
t.Fatalf("the operator's module has no running have: %+v", h)
}
}
// With no key trusted, nothing is applied: a node that does not know its signers fails closed.
func TestWithNoTrustedKeyNothingIsApplied(t *testing.T) {
b := newBed(t)
b.env.Trusted = nil
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
time.Sleep(200 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied with no trusted key: %v", got)
}
}
// What a module's apply must not take is every other module that runs, those the agent holds and those still running from
// the kept declaration (ADR 0429 §4): the resources of the kept declaration reach Apply as the others.
func TestApplyingOneModuleHoldsOutTheModulesStillRunningFromTheKeptDeclaration(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}, NodeModule: {file("node", "n", "n")}}
stop := b.run()
defer stop()
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
b.comps.put(ck, craw)
wk, wraw := wantFor(t, "web", fp, dig, "")
b.wants.put(wk, wraw)
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
defer b.mu.Unlock()
var names []string
for _, c := range b.others[0] {
names = append(names, c.Module)
}
if len(names) != 2 || !strings.Contains(strings.Join(names, ","), "db") {
t.Fatalf("the others of web were %v: db and the node's own run from the kept declaration", names)
}
}
// Everything is what runs: the agent's compositions and the kept resources of each module it does not hold.
func TestEverythingIsWhatTheAgentRunsAndWhatIsKeptForTheRest(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"db": {file("db", "x", "1")}, "web": {file("web", "x", "old")}}
a := New(b.env)
if got := a.Everything(); len(got) != 2 {
t.Fatalf("before the agent holds anything: %d compositions, want the 2 kept", len(got))
}
a.running["web"] = delivery.Have{FP: fpOf("a"), Composed: "sha256:" + strings.Repeat("1", 64), Phase: delivery.PhaseRunning}
ck, craw, dig := composition(t, "web", fpOf("a"), false, file("web", "x", "new"))
a.running["web"] = delivery.Have{FP: fpOf("a"), Composed: dig, Phase: delivery.PhaseRunning}
a.takeComposition(Update{Key: ck, Value: craw, Revision: 1})
got := a.Everything()
if len(got) != 2 {
t.Fatalf("%d compositions", len(got))
}
for _, c := range got {
if c.Module == "web" && !strings.Contains(string(c.Resources[0]), "new") {
t.Fatal("web is the kept declaration's, not the composition the agent runs")
}
}
if a.HoldsNode() {
t.Fatal("the agent holds no node composition")
}
}
// A failed apply is applied again after a restart even when the kept declaration holds the composition: once this
// node-engine wrote its haves, they are what it knows (the kept declaration is only for the first start after the switch).
func TestAFailedApplyIsAppliedAgainAfterARestartWhateverIsKept(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
b.kept = map[string][]json.RawMessage{"web": {file("web", "x", "1")}}
b.haves.stored = map[string]Have{"web": {Module: "web", FP: fp, Composed: dig, Phase: string(delivery.PhaseFailed)}}
stop := b.run()
defer stop()
eventually(t, "a failed apply was marked running from the kept declaration", func() bool { return len(b.appliedList()) == 1 })
}
// A have written before the switch carries no composition: then the kept declaration says what runs.
func TestAHaveWithNoCompositionDefersToTheKeptDeclaration(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, craw, dig := composition(t, "web", fp, false, file("web", "x", "1"))
wk, wraw := wantFor(t, "web", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
b.kept = map[string][]json.RawMessage{"web": {file("web", "x", "1")}}
b.haves.stored = map[string]Have{"web": {Module: "web", Phase: "running"}}
stop := b.run()
defer stop()
eventually(t, "the have was not written", func() bool { return b.haves.last("web").Composed == dig })
time.Sleep(50 * time.Millisecond)
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied %v", got)
}
}
// A module that leaves the node's own composition for a composition of its own keeps its resources until that one is
// applied: the node's own apply holds out every module whose composition the node's bucket holds and the agent does not
// run yet (issue 1945, live on 2026-10-25 at 02:32: shanks' node-engine applied a node's own composition that no longer
// carried systemd-resolved, whose own composition waited for its credential, and removed its guard, so the machine lost
// its resolver).
func TestTheNodesOwnApplyHoldsOutAModuleThatMovedToItsOwnComposition(t *testing.T) {
b := newBed(t)
fp := fpOf("b")
// systemd-resolved's own composition is written; no want names it yet.
rk, rraw, _ := composition(t, "systemd-resolved", fp, false, file("systemd-resolved", "guard", "g"))
b.comps.before(rk, rraw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, err := delivery.WantNodeKey(node)
if err != nil {
t.Fatal(err)
}
nwraw, err := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
if err != nil {
t.Fatal(err)
}
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
b.mu.Lock()
defer b.mu.Unlock()
var names []string
for _, c := range b.others[0] {
names = append(names, c.Module)
}
if !strings.Contains(strings.Join(names, ","), "systemd-resolved") {
t.Fatalf("the others of the node's own apply were %v: systemd-resolved's resources would be removed", names)
}
}
// A module the assigner removed holds nothing back, its compositions still in the bucket or not: its resources are its
// removal's to take (issue 1945).
func TestARemovedModulesCompositionHoldsNothingBack(t *testing.T) {
b := newBed(t)
fp := fpOf("b")
rk, rraw, _ := composition(t, "gone", fp, false, file("gone", "x", "1"))
b.comps.before(rk, rraw)
gk, err := delivery.WantKey(node, "gone")
if err != nil {
t.Fatal(err)
}
graw, err := delivery.EncodeWant(delivery.Want{Removed: true})
if err != nil {
t.Fatal(err)
}
b.wants.before(gk, graw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool {
for _, m := range b.appliedList() {
if m == NodeModule {
return true
}
}
return false
})
b.mu.Lock()
defer b.mu.Unlock()
for _, others := range b.others {
for _, c := range others {
if c.Module == "gone" {
t.Fatal("a removed module's composition was held out")
}
}
}
}
// The node's own composition is not applied while a module's composition in the node's bucket cannot be read or is not
// signed by a trusted key: what that one carries may be what left the node's own, and the node's own apply would remove
// it (issue 1945, the review's gap 2). It is said, naming the composition, and applied once that one reads.
func TestTheNodesOwnWaitsWhileAModulesCompositionCannotBeRead(t *testing.T) {
b := newBed(t)
bad, err := delivery.ComposedKey(node, "broken", fpOf("c"))
if err != nil {
t.Fatal(err)
}
b.comps.before(bad, []byte(`{"v":1,"not":"a composition"}`))
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the wait not said", func() bool {
b.mu.Lock()
defer b.mu.Unlock()
for _, s := range b.said {
if strings.Contains(s, bad) {
return true
}
}
return false
})
if got := b.appliedList(); len(got) != 0 {
t.Fatalf("applied %v while a composition could not be read", got)
}
// Read again, the node's own is applied.
gk, graw, _ := composition(t, "broken", fpOf("c"), false, file("broken", "x", "1"))
b.comps.put(gk, graw)
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
}
// A stale refused composition beside a newer readable one of the same module does not hold the node's own apply back:
// only the composition the module's want names, else its newest, counts (the review of issue 1945: the bucket keeps
// compositions written before the filled paths, which the SDK now refuses, and nobody applies them).
func TestAStaleRefusedCompositionDoesNotHoldTheNodesOwnBack(t *testing.T) {
b := newBed(t)
stale, err := delivery.ComposedKey(node, "power", fpOf("a"))
if err != nil {
t.Fatal(err)
}
b.comps.before(stale, []byte(`{"v":1,"not":"a composition"}`))
gk, graw, _ := composition(t, "power", fpOf("b"), false, file("power", "x", "1"))
b.comps.before(gk, graw)
nk, nraw, ndig := composition(t, NodeModule, "", false)
nwk, _ := delivery.WantNodeKey(node)
nwraw, _ := delivery.EncodeNodeWant(delivery.NodeWant{Composed: ndig})
b.comps.before(nk, nraw)
b.wants.before(nwk, nwraw)
stop := b.run()
defer stop()
eventually(t, "the node's own not applied", func() bool { return len(b.appliedList()) == 1 })
}
// A composition the kept declaration or the node's own have says runs is applied all the same when the engine holds no
// record of its resources: they were removed since (issue 1945, live on 2026-10-25: ace's and g14's node-engines removed
// ca-trust's, power's and matrix's resources in a node's own apply, and the wants that named those modules' own
// compositions were judged already running from the kept declaration, so nothing made them again).
func TestACompositionWhoseResourcesWereRemovedIsAppliedAgain(t *testing.T) {
for _, from := range []string{"kept", "have"} {
t.Run(from, func(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
rs := []json.RawMessage{file("ca-trust", "trust", "1")}
ck, craw, dig := composition(t, "ca-trust", fp, false, rs...)
wk, wraw := wantFor(t, "ca-trust", fp, dig, "")
b.comps.before(ck, craw)
b.wants.before(wk, wraw)
if from == "kept" {
b.kept = map[string][]json.RawMessage{"ca-trust": rs}
} else {
b.haves.stored = map[string]Have{"ca-trust": {Module: "ca-trust", FP: fp, Composed: dig, Phase: "running"}}
}
b.env.Recorded = func(context.Context, delivery.Composed) bool { return false }
stop := b.run()
defer stop()
eventually(t, "not applied", func() bool { return len(b.appliedList()) == 1 })
})
}
}
// The composition a want names is applied from the key's history when a newer one has replaced it under the same key:
// a want held at a fingerprint keeps naming the digest it was written with, and the controller writes every new
// composition at that fingerprint over the same key (ADR 0539 clause 1, the sweep's kept revisions; issue 1945, live on
// 2026-10-25: shanks' node-tools composition had digest 575a5c96, its held want named a36e63c8, and nothing was applied).
func TestTheRevisionAWantNamesIsAppliedFromHistory(t *testing.T) {
b := newBed(t)
fp := fpOf("a")
ck, oldRaw, oldDig := composition(t, "node-tools", fp, false, file("node-tools", "x", "old"))
_, newRaw, _ := composition(t, "node-tools", fp, false, file("node-tools", "x", "new"))
b.comps.before(ck, newRaw) // the key's latest value
wk, wraw := wantFor(t, "node-tools", fp, oldDig, "")
b.wants.before(wk, wraw)
b.env.Revisions = func(_ context.Context, key string) ([]Update, error) {
if key != ck {
return nil, nil
}
return []Update{{Key: ck, Value: oldRaw, Revision: 3}, {Key: ck, Value: newRaw, Revision: 7}}, nil
}
stop := b.run()
defer stop()
eventually(t, "the revision the want names was not applied", func() bool { return len(b.appliedList()) == 1 })
if h := b.haves.last("node-tools"); h.Composed != oldDig {
t.Fatalf("the have names %s, want %s", h.Composed, oldDig)
}
}
// 1945/AC18 (unit): a module that still runs from the kept declaration carries the kept declaration's filled paths of
// its files, so the filling writes them ("not named as filled" for claude-code and screen-lock, 2026-10-25).
func TestAModuleRunningFromTheKeptDeclarationKeepsItsFilledPaths(t *testing.T) {
b := newBed(t)
b.kept = map[string][]json.RawMessage{"claude-code": {file("claude-code", "settings", "")}}
b.env.KeptFilled = func() map[string][]string {
return map[string][]string{"claude-code": {"/tmp/r6/claude-code/settings"}}
}
a := New(b.env)
got := a.Everything()
if len(got) != 1 || got[0].Module != "claude-code" || strings.Join(got[0].Filled, ",") != "/tmp/r6/claude-code/settings" {
t.Fatalf("what runs from the kept declaration: %+v", got)
}
}
internal/apply/agent_home_guard.go
+1 -1
View File
@@ -14,7 +14,7 @@ import (
"fmt"
"path/filepath"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
// AgentHomeRefusedError is an archive or a written-into file the engine will not place below an agent's home.
internal/apply/agent_home_guard_test.go
+1 -1
View File
@@ -6,7 +6,7 @@ package apply
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
func TestNothingIsUnpackedOrWrittenIntoBelowAnAgentsHome(t *testing.T) {
internal/apply/alone.go
-67
View File
@@ -1,67 +0,0 @@
package apply
import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// WriteAlone writes one file a module's composition declares, through the apply's own file step, and records it as an
// apply would; nothing else of the module is applied, and nothing that reads the file is moved (novox/hq ADR 0547 §8).
//
// **For a file whose reader takes the change itself.** The bus's user list is the one it is for: the bus module's
// entrypoint watches the file and signals the server to reload in place, so the list is in force without a want of the
// bus, which nobody writes (ADR 0539 §15), and without the container being touched. A file that a service or a container
// must be moved to read is not written alone: it goes through the module's apply, which moves what reads it.
//
// The file is written only when its content differs from what is there (the file step's own rule), read back, and
// recorded under the module whose resource it is, so the module's next apply finds it as its own and writes nothing.
// The caller holds the apply's lock. log is said the path, the action and the digest, never the content.
func WriteAlone(ctx context.Context, sys system.System, file *declaration.File, known store.State, run Runner,
log func(string)) (Outcome, store.State, error) {
if log == nil {
log = func(string) {}
}
if file.Into != "" || file.CreateOnce || file.Sealed != "" || file.VaultSealed != nil || len(file.Secrets) > 0 {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is not a whole file of plain content, "+
"and only such a file is written alone; nothing was written", file.Path)}
}
if file.IsReference() {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is given by digest %s and was not "+
"resolved to its content; it is not written empty", file.Path, file.Digest)}
}
if held, isHeld := known.HeldAt(file.ID); isHeld {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is held as found for a module not "+
"yet taken (%s), so nothing was written", file.Path, held.Why)}
}
was, _ := known.Find(file.ID)
if was.ID == "" {
// Only over a file the mesh wrote: a file this node holds no record of is the module's apply's to take, with
// the original kept, never this write's. A record of the path under another id (the declaration before the
// switch named it so) is the one written to, so no second record of the file is left for an apply to remove.
rec, ok := known.At(string(declaration.TypeFile), file.Path)
if !ok {
return Outcome{}, known, &Error{Resource: file.ID, Err: fmt.Errorf("this node holds no record of %s; it "+
"is written by the module's apply first, never alone", file.Path)}
}
renamed := *file
renamed.ID = rec.ID
file, was = &renamed, rec
}
in := inputs{declares: map[string]string{file.Identity(): declaredDigest(file)}, known: &known,
groups: groupsWanted([]declaration.Resource{file}), handed: Handed{}, dirsBefore: map[string]bool{},
files: map[string]bool{file.Path: true}}
outcome, err := applyOne(ctx, sys, file, run, map[string]bool{}, in, was, nil, nil)
if err != nil {
log(fmt.Sprintf(" failed %s (%s), alone: %v", file.ID, file.Path, err))
return outcome, known, &Error{Resource: file.ID, Err: err}
}
recordKeeping(&known, file, outcome, was)
if moved(outcome) {
log(fmt.Sprintf(" %s %s (%s), alone: %s", outcome.Action, file.ID, file.Path, outcome.wrote))
}
return outcome, known, nil
}
internal/apply/alone_test.go
-90
View File
@@ -1,90 +0,0 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0547 §8: the bus node writes the bus's user list alone, with no want, through the apply's own
// file step: only when it differs, mode 0600, read back and recorded as the module's, and nothing else moves.
func TestAFileWrittenAloneMovesNothingElse(t *testing.T) {
const was, now = "user a\n", "user a\nuser b\n"
for _, c := range []struct {
name, fields, before string
recorded bool
recordedAs string // the id the record has, when not the composition's
action, refused string
after string
}{
{name: "no record of it", fields: `"content":%q,"mode":"0600"`, before: was, refused: "holds no record", after: was},
{name: "changed", fields: `"content":%q,"mode":"0600"`, before: was, recorded: true, action: "updated", after: now},
{name: "the same", fields: `"content":%q,"mode":"0600"`, before: now, recorded: true, action: "unchanged", after: now},
{name: "recorded under the id from before the switch", fields: `"content":%q,"mode":"0600"`, before: was,
recorded: true, recordedAs: "bus-users", action: "updated", after: now},
{name: "absent with a record", fields: `"content":%q,"mode":"0600"`, recorded: true, action: "created", after: now},
{name: "written into", fields: `"content":%q,"into":"block","owner-at-runtime":"managed"`, before: was,
recorded: true, refused: "only such a file", after: was},
{name: "sealed", fields: `"sealed":"c2VhbGVk"`, before: was, recorded: true,
refused: "only such a file", after: was},
} {
t.Run(c.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "accounts.conf")
if c.before != "" {
if err := os.WriteFile(path, []byte(c.before), 0o600); err != nil {
t.Fatal(err)
}
}
fields := strings.Replace(c.fields, "%q", fmt.Sprintf("%q", now), 1)
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"nats.bus-users","module":"nats",
"type":"file","path":%q,`+fields+`}]}`, path))
file := d.Resources[0].(*declaration.File)
known := store.State{}
id := file.ID
if c.recordedAs != "" {
id = c.recordedAs
}
if c.recorded {
known.Record(store.Applied{ID: id, Type: string(declaration.TypeFile), Module: "nats", Target: path,
Origin: store.OriginDeclared, Wrote: digestOf(c.before)})
}
calls := 0
counting := func(context.Context, string, ...string) (string, error) { calls++; return "", nil }
out, after, err := WriteAlone(context.Background(), archHost(t), file, known, counting, nil)
if c.refused != "" {
if err == nil || !strings.Contains(err.Error(), c.refused) {
t.Fatalf("not refused with %q: %v", c.refused, err)
}
} else if err != nil {
t.Fatal(err)
} else if out.Action != c.action {
t.Errorf("action %q, want %q", out.Action, c.action)
}
if got := readText(t, path); got != c.after {
t.Errorf("file holds %q, want %q", got, c.after)
}
if calls != 0 {
t.Errorf("%d commands were run: a file written alone moves nothing", calls)
}
if c.refused != "" {
return
}
if m := modeAt(t, path); m != 0o600 {
t.Errorf("mode %o", m)
}
rec, ok := after.Find(id)
if !ok || rec.Wrote != digestOf(now) || rec.Target != path {
t.Errorf("record %+v (found %v), want it to say %s was written", rec, ok, digestOf(now))
}
if n := len(after.Resources); n != 1 {
t.Errorf("%d records of one file", n)
}
})
}
}
internal/apply/apply.go
+129 -465
View File
@@ -15,18 +15,18 @@ import (
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"path/filepath"
"slices"
"sort"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/firewall"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
@@ -92,11 +92,6 @@ type Outcome struct {
// reads is, for a container, the digest of each file it was created reading, by path — so
// the next apply can say which one changed (novox/hq 04-ISSUES/103).
reads map[string]string
// busySince is, for a container left running because it said busy, when the first apply that would
// have replaced it ran (novox/hq ADR 0335): kept in its record, so the bound holds across passes and
// across a restart of the node-engine. busy is what the report says of it.
busySince *time.Time
busy *BusyWait
}
// Report is what an apply did, in the order it did it.
@@ -113,24 +108,6 @@ type Report struct {
// 224): a machine whose server is stopped at 03:31 because its collector is running is
// working, not broken, and the report is where that difference is said.
Windows []Window `json:"windows,omitempty"`
// Busy is every container this apply left running because it said busy, or replaced after it had
// waited (novox/hq ADR 0335): a machine whose walk waits for it says why, and how it ended.
Busy []BusyWait `json:"busy,omitempty"`
// MovedOut is every path a move-out directory moved into its kept folder (issue 1024, novox/hq ADR 0422).
MovedOut []MovedOut `json:"moved-out,omitempty"`
// Left is every socket, pipe or device a move-out directory holds and did not move (ADR 0422 §4).
Left []LeftInPlace `json:"left,omitempty"`
}
// Waiting says whether a container of this apply waits to be replaced because it is busy: what makes the
// node-engine ask again sooner than its reconcile (novox/hq ADR 0335 §2).
func (r Report) Waiting() bool {
for _, b := range r.Busy {
if b.State == BusyWaiting {
return true
}
}
return false
}
// Changed reports whether anything about the machine actually moved. An apply that changed
@@ -141,10 +118,7 @@ func (r Report) Changed() bool {
// whose account's manager is not running, which nothing moved either (novox/hq ADR 0177);
// held still is a container a maintenance window stopped, which the apply did not touch
// (novox/hq issue 224).
// waits busy is a container left running because it said busy (novox/hq ADR 0335), which nothing
// moved either.
if o.Action != "unchanged" && o.Action != "held" && o.Action != "waiting" && o.Action != heldStill &&
o.Action != waitsBusy {
if o.Action != "unchanged" && o.Action != "held" && o.Action != "waiting" && o.Action != heldStill {
return true
}
}
@@ -178,23 +152,6 @@ type Error struct {
// (novox/hq ADR 0136) or a directory the engine refused (issue 495). Named, because "everything
// was attempted" is then untrue, and a report that says so after skipping a module lies.
Modules []string
// Failures is every resource that failed, the first among them, each with its error (issue 1132, novox/hq ADR
// 0429 §4): Others counts the rest and names none, and a module's state on the bus names what failed in it.
Failures []Failure
}
// Failure is one resource that failed in an apply, and why.
type Failure struct {
Resource string
Err error
}
// ModuleResult is how one module's part of an apply ended (issue 1132).
type ModuleResult struct {
Failed bool
// Detail names each failing resource with its error, or why the module was not attempted.
Detail string
}
func (e *Error) Error() string {
@@ -299,60 +256,6 @@ func ApplyMindingWindows(
unseal Unseal,
keep Keep,
windows *Windows,
) (Report, store.State, error) {
return applyScoped(ctx, sys, d, nil, known, origin, run, log, unseal, keep, windows)
}
// Scope is what makes an apply one module's (novox/hq ADR 0429 §4, ADR 0539 §9): the module whose want changed, and
// the resources the node's other modules hold, so a path, a unit or a group another module still asks for is neither
// removed nor taken from it. Only that module's own records are orphans; a removed module is a Scope with an empty
// declaration.
type Scope struct {
// Module is the module applied. NodeModule (store.NodeModule) is the node's own composition.
Module string
// Others are the resources of every other module the node has a composition for.
Others []declaration.Resource
}
// owns reports whether a record belongs to the scope's module.
func (sc *Scope) owns(a store.Applied) bool {
return store.OwnerOf(a, declaration.ModuleGuess) == sc.Module
}
// isNode is whether the scope is the node's own composition, the one that carries identities, groups, libraries and
// the adoption.
func (sc *Scope) isNode() bool { return sc == nil || sc.Module == store.NodeModule }
// ApplyModule makes the machine match one module's declaration, and only that module's: everything else the node
// runs is left as it is. d holds the module's resources; scope names the module and what the others hold.
func ApplyModule(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
scope Scope,
known store.State,
origin string,
run Runner,
log func(string),
unseal Unseal,
keep Keep,
windows *Windows,
) (Report, store.State, error) {
return applyScoped(ctx, sys, d, &scope, known, origin, run, log, unseal, keep, windows)
}
func applyScoped(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
scope *Scope,
known store.State,
origin string,
run Runner,
log func(string),
unseal Unseal,
keep Keep,
windows *Windows,
) (report Report, _ store.State, _ error) {
// Said whichever way the apply ended, failure included: a window being open is a fact about the
// machine, not about this apply.
@@ -379,9 +282,7 @@ func applyScoped(
}
// What an unfinished apply found a unit as is kept only while its service is declared: one
// declared again later is read afresh, as any resource with no record is (novox/hq ADR 0118).
if scope == nil {
known.DropFoundUndeclared(declared, origin)
}
known.DropFoundUndeclared(declared, origin)
// Which units the mesh made, read before any removal can take a unit file's record away — so
// whichever order a module declared a unit and its file in, the unit is known for the mesh's
@@ -406,11 +307,7 @@ func applyScoped(
// it back would stop or disable a unit the declaration says is running, every container on it
// with it, only for the declared one to start it again in the same apply. The record is
// forgotten; the unit's found state is the remaining record's to give back.
everything := d.Resources
if scope != nil {
everything = append(append([]declaration.Resource{}, d.Resources...), scope.Others...)
}
heldUnits := unitsHeld(everything)
heldUnits := unitsHeld(d.Resources)
// **A whole file still declared at its path by another resource is handed to it, never removed
// first** (novox/hq ADR 0223). A file can change owners between modules — the machine's resolver
@@ -420,55 +317,20 @@ func applyScoped(
// and stayed so until the new resource's turn came, with every resource in between applied on a
// machine without it. The record going is kept until the one declaring the path now is recorded
// in its place: then it is forgotten, and what it kept of the original goes with the file.
heldFiles := filesHeld(everything)
heldFiles := filesHeld(d.Resources)
handedFiles := map[string]store.Applied{}
// **A group still asked for by another resource is not given back when one record of it goes**
// (novox/hq ADR 0252), on the unit's rule above: two modules may each want the account in one group,
// and the one going takes nothing the other still needs.
wanted := groupsWanted(everything)
wanted := groupsWanted(d.Resources)
// And what a record going hands to a resource that still asks for its group (novox/hq ADR 0252 §3).
handed := Handed{}
// **A resource another module's composition now holds is that module's, and is left** (issue 1945): a module that
// leaves the node's own composition for one of its own takes its resources with it, by their identity, and the node's
// own apply leaves each as it is, record and all, until that module's apply takes it. Removed, a node lost its
// resolver guard, its sudoers file and a running container between two applies.
heldByOthers := map[string]string{}
if scope != nil {
for _, r := range scope.Others {
heldByOthers[r.Identity()] = r.DeclaredModule()
}
}
removeOrphan := func(orphan store.Applied) error {
var action, detail string
var err error
if module, held := heldByOthers[orphan.ID]; held && !declared[orphan.ID] {
if module == "" {
module = "another module"
}
detail := "no longer declared here; the composition of " + module + " holds it, so it was left as it is"
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "kept", Detail: detail,
})
log(fmt.Sprintf(" kept %s (%s): %s", orphan.ID, orphan.Target, detail))
return nil
}
if by, held := heldFiles[fileKey(orphan)]; held && by != orphan.ID {
if scope != nil && !declared[by] {
// Another module declares the path: it is that module's, so this record is let go and the file is
// neither removed nor taken from it (ADR 0539 §9).
known.Forget(orphan.ID)
detail := "no longer declared; " + by + " of another module still holds the path, so it was left as it is"
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "forgotten", Detail: detail,
})
log(fmt.Sprintf(" forgotten %s (%s): %s", orphan.ID, orphan.Target, detail))
return nil
}
handedFiles[by] = orphan
return nil
}
@@ -576,13 +438,22 @@ func applyScoped(
}
handover := map[string][]store.Applied{}
for _, orphan := range known.Orphans(declared, origin) {
if scope != nil && !scope.owns(orphan) {
continue
}
if d.Adoption == nil && strings.HasPrefix(orphan.ID, declaration.AdoptionPrefix) {
protecting = append(protecting, orphan)
continue
}
// **A module the mesh left out is not a module the mesh removed** (novox/hq ADR 0163, rule
// 6): its resources are absent because a setting stored for it cannot compose, and the
// mesh said so by name. What the host wrote for it stays as it is, recorded, until the
// module is declared again or unassigned.
if module, left := d.LeftOutModuleOf(orphan.ID); left {
report.Outcomes = append(report.Outcomes, Outcome{
ID: orphan.ID, Type: orphan.Type, Target: orphan.Target,
Action: "unchanged", Detail: "kept: " + module + " was left out of this declaration by the mesh, not removed",
})
log(fmt.Sprintf(" kept %s (%s): %s was left out of this declaration by the mesh, not removed", orphan.ID, orphan.Target, module))
continue
}
if by, replaced := replacedBy[orphan.ID]; replaced {
handover[by] = append(handover[by], orphan)
continue
@@ -623,7 +494,12 @@ func applyScoped(
// what is assigned: a carried bundle's silence is not an unassignment.
if origin == store.OriginDeclared {
for _, h := range append([]store.Held{}, known.Held...) {
if declared[h.ID] || (scope != nil && h.Module != scope.Module) {
if declared[h.ID] {
continue
}
if slices.Contains(d.LeftOut, h.Module) {
// Left out, not unassigned (ADR 0163, rule 6): still held for the module, as the
// mesh asked.
continue
}
known.Release(h.ID)
@@ -664,7 +540,6 @@ func applyScoped(
in.refused = refusedPlaces(d.Resources)
in.agentHomes = rootNeverHomes(d)
in.dirsBefore = map[string]bool{}
in.files = declaredFiles(d.Resources)
for _, resource := range d.Resources {
in.declares[resource.Identity()] = declaredDigest(resource)
if dir, ok := resource.(*declaration.Directory); ok {
@@ -691,8 +566,6 @@ func applyScoped(
// is a different thing — one is "this machine could not do it", the other is "this was never
// a declaration", and they are fixed in different places.
var failures []*Error
// Which resources applied without error in this run, for the move-out pass after the loop (issue 1024).
appliedOK := map[string]bool{}
// Modules whose own step did not complete. What follows *within such a module* is not attempted;
// the rest of the machine is (novox/hq ADR 0136).
gated := map[string]bool{}
@@ -825,7 +698,7 @@ func applyScoped(
in.held = heldRoots(verdicts, d.Libraries)
// A tool runner no longer put in a group leaves it before anything is applied, so its unit, which names the
// change under restart-on, is restarted in this same apply to lose it (ADR 0328 §11, ADR 0334 §2, ADR 0252 §3).
if origin == store.OriginDeclared && scope.isNode() {
if origin == store.OriginDeclared {
for _, o := range leaveRunners(ctx, sys, d.Libraries, d.Identities, &known, run, wanted) {
if o.Action == "restored" {
changed[o.changes] = true
@@ -845,7 +718,6 @@ func applyScoped(
first := failures[0]
first.Done = report
first.Others = len(failures) - 1
first.Failures = failuresOf(failures)
first.Modules = gatedModules()
log(" kept " + guardPrefix + "*: the guard is not up, so what it replaces was left in force")
return report, known, first
@@ -857,7 +729,7 @@ func applyScoped(
// **A module whose step did not complete is skipped from there on** (novox/hq ADR 0136).
// Reported rather than passed over in silence: "not attempted" and "nothing to do" are
// different answers, and only one of them is somebody's to fix.
if module, ours := d.ModuleOfID(resource.Identity()); ours && gated[module] {
if module, ours := moduleOf(resource.Identity()); ours && gated[module] {
report.Outcomes = append(report.Outcomes, Outcome{
ID: resource.Identity(), Type: string(resource.Kind()), Target: resource.Target(),
Action: "skipped", Detail: gatedWhy[module],
@@ -887,16 +759,6 @@ func applyScoped(
}
}
// **A filled file is the provisioner client's to write** (novox/hq ADR 0349 §5, design 57 P7): one writer per
// file. The declaration says where it goes and how it is written, and only the write verb (WriteFilled)
// writes it, on the client's word. This apply — the controller's send and the five-minute re-apply alike —
// neither writes it nor records it, and nothing that reads it is moved for it; so only what the verb placed
// is recorded, and only that is removed as an orphan when the file leaves the declaration.
if f, isFile := resource.(*declaration.File); isFile && d.Fills(f.Path) {
report.Outcomes = append(report.Outcomes, leftToTheClient(f, &known, handedFiles, &report, log))
continue
}
// The private network takes over the tunnel it found, ahead of the service that replaces
// it (novox/hq ADR 0105): its configuration kept, its unit stopped and disabled, never
// flushed. A failure here fails the service too — the mesh's interface is not started on a
@@ -1019,7 +881,7 @@ func applyScoped(
in.refused[filepath.Clean(dir.Path)] = err
}
if dir, ok := resource.(*declaration.Directory); ok && in.refused[filepath.Clean(dir.Path)] != nil {
if module, ours := d.ModuleOfID(resource.Identity()); ours {
if module, ours := moduleOf(resource.Identity()); ours {
gated[module] = true
gatedWhy[module] = fmt.Sprintf("its directory %s was refused (novox/hq issue 495), so "+
"nothing more of this module was attempted", filepath.Clean(dir.Path))
@@ -1068,7 +930,7 @@ func applyScoped(
//
// An action still gates the machine: the bootstrap is a row of them, each making the
// next possible, and they belong to no module.
if module, ours := d.ModuleOfID(resource.Identity()); ours {
if module, ours := moduleOf(resource.Identity()); ours {
gated[module] = true
gatedWhy[module] = "a step this module declares did not complete"
log(fmt.Sprintf(" gated %s.*: a step it declares did not complete, so the rest "+
@@ -1078,7 +940,6 @@ func applyScoped(
failed.Stopped = true
failed.Done = report
failed.Others = len(failures) - 1
failed.Failures = failuresOf(failures)
failed.Modules = gatedModules()
return report, known, failed
}
@@ -1116,13 +977,33 @@ func applyScoped(
kept, keptMode, keptOwner = from.Kept, from.KeptMode, from.KeptOwner
}
// Only now. The record follows the fact, never leads it.
recordApplied(&known, origin, resource, outcome, kept, keptMode, keptOwner)
if scope != nil {
known.SetModule(resource.Identity(), scope.Module)
} else if module, ours := d.ModuleOfID(resource.Identity()); ours {
known.SetModule(resource.Identity(), module)
known.Record(store.Applied{
Origin: origin,
ID: resource.Identity(), Type: string(resource.Kind()),
Target: outcome.Target, AppliedAt: time.Now().UTC(),
Wrote: outcome.wrote,
Into: outcome.into,
Kept: kept,
KeptMode: keptMode,
KeptOwner: keptOwner,
Reads: outcome.reads,
Stateless: outcome.stateless,
Scope: outcome.scope,
User: outcome.user,
Found: outcome.found,
DeclaredUnit: outcome.declared,
Shell: outcome.shell,
Linger: outcome.linger,
Groups: outcome.groups,
Unpacked: outcome.unpacked,
AsFound: outcome.asFound,
Declared: outcome.asFoundDeclared,
FoundAt: outcome.asFoundAt,
Holds: holds(resource),
})
if outcome.found != nil {
known.DropFound(resource.Identity())
}
appliedOK[resource.Identity()] = true
// Its module has been taken, and what was held for it is now the mesh's. A file written
// into, or a service whose lifecycle is the machine's, replaced nothing that was found, so
// its outcome says what the apply did, not that a cutover happened; a hold from when it was
@@ -1165,13 +1046,10 @@ func applyScoped(
Action: "forgotten", Detail: detail})
log(fmt.Sprintf(" forgotten %s (%s): %s", from.ID, from.Target, detail))
}
if outcome.busy != nil {
report.Busy = append(report.Busy, *outcome.busy)
}
if outcome.Action == heldStill || outcome.Action == waitsBusy {
if outcome.Action == heldStill {
// Not changed: nothing about it moved, so nothing that restarts on it restarts.
log(fmt.Sprintf(" %s %s (%s): %s", outcome.Action, outcome.ID, outcome.Target, outcome.Detail))
} else if moved(outcome) {
} else if outcome.Action != "unchanged" {
changed[resource.Identity()] = true
// With the detail, when there is one: "updated app" says a container was replaced;
// which file made that happen is what somebody reading the log at the time needs
@@ -1194,15 +1072,6 @@ func applyScoped(
failures = append(failures, identityFailures...)
// A directory or archive that says move-out keeps only what it knows (issue 1024, novox/hq ADR 0422).
movedOut, leftIn, moveFailures := moveOutPass(d, known, appliedOK, log)
report.MovedOut = append(report.MovedOut, movedOut...)
report.Left = append(report.Left, leftIn...)
for _, f := range moveFailures {
f.Done = report
failures = append(failures, f)
}
// A replacement that did not apply — failed, skipped behind its module's step, held — leaves
// what it replaces running and recorded, said, for the next apply to hand over.
unhanded := make([]string, 0, len(handover))
@@ -1224,7 +1093,7 @@ func applyScoped(
}
// Only the mesh says what is assigned: a carried bundle's silence undeclares no identity and no folder.
if origin == store.OriginDeclared && scope.isNode() {
if origin == store.OriginDeclared {
// And an identity, then a library group, no longer declared goes, once what ran as it went above (ADR 0328
// §3, §4).
if tool, err := toolFor(sys); err == nil {
@@ -1243,7 +1112,7 @@ func applyScoped(
// the mesh's derived filter among it — applied cleanly (novox/hq ADR 0100), and on every
// converged apply, not once (ADR 0168). Skipped, it is said: a step that does nothing is never
// silent (issue 143).
if len(failures) == 0 && scope.isNode() {
if len(failures) == 0 {
did, err := retireFirewall(ctx, d, origin, &known, run, log)
if err != nil {
return report, known, &Error{Resource: "the firewall found on this machine", Err: err, Done: report}
@@ -1254,7 +1123,7 @@ func applyScoped(
return report, known, err
}
}
} else if rec := known.Firewall; len(failures) > 0 && scope.isNode() && origin == store.OriginDeclared && d.Adoption == nil && rec != nil &&
} else if rec := known.Firewall; origin == store.OriginDeclared && d.Adoption == nil && rec != nil &&
rec.Kind == string(firewall.UFW) && rec.WasActive && firewall.Active(ctx, run) {
report.Firewall = fmt.Sprintf("left in force: %d resource(s) failed, and the found firewall is retired only after a clean apply", len(failures))
log(" kept ufw in force: " + report.Firewall)
@@ -1267,54 +1136,12 @@ func applyScoped(
first := failures[0]
first.Done = report
first.Others = len(failures) - 1
first.Failures = failuresOf(failures)
first.Modules = gatedModules()
return report, known, first
}
return report, known, nil
}
// recordApplied records what applying a resource did, once it is done: the record follows the fact, never leads it.
// kept, keptMode and keptOwner are where the original of what it replaced was kept, and how it was found. What an
// earlier apply found its unit as goes with it.
func recordApplied(known *store.State, origin string, resource declaration.Resource, outcome Outcome,
kept, keptMode, keptOwner string) {
known.Record(store.Applied{
Origin: origin,
ID: resource.Identity(), Type: string(resource.Kind()),
Target: outcome.Target, AppliedAt: time.Now().UTC(),
Wrote: outcome.wrote,
Into: outcome.into,
Kept: kept,
KeptMode: keptMode,
KeptOwner: keptOwner,
Reads: outcome.reads,
BusySince: outcome.busySince,
Stateless: outcome.stateless,
Scope: outcome.scope,
User: outcome.user,
Found: outcome.found,
DeclaredUnit: outcome.declared,
Shell: outcome.shell,
Linger: outcome.linger,
Groups: outcome.groups,
Unpacked: outcome.unpacked,
AsFound: outcome.asFound,
Declared: outcome.asFoundDeclared,
FoundAt: outcome.asFoundAt,
Holds: holds(resource),
})
if outcome.found != nil {
known.DropFound(resource.Identity())
}
}
// moved reports whether applying a resource moved it, so what restarts or reloads on it does: anything but
// unchanged, and not a container held still or waiting for its busy one to say idle.
func moved(outcome Outcome) bool {
return outcome.Action != "unchanged" && outcome.Action != heldStill && outcome.Action != waitsBusy
}
// afterWhatTheyRead is the resources in declared order, except that **a service comes after every
// resource it names under restart-on or reload-on** (novox/hq issue 260).
//
@@ -1383,9 +1210,7 @@ func afterWhatTheyRead(resources []declaration.Resource) []declaration.Resource
const guardPrefix = declaration.AdoptionPrefix + "guard"
// Unseal opens a value the mesh sealed to this node. Nil when the node has no sealing key, which
// makes every sealed file an error rather than a silently skipped one. A value the vault sealed (a core
// credential, novox/hq ADR 0349 §16) is given as its box's JSON, which begins with `{`; one the controller sealed
// is base64, which never does (identity.SealingKey.Open).
// makes every sealed file an error rather than a silently skipped one.
type Unseal func(sealed string) ([]byte, error)
func applyOne(ctx context.Context, sys system.System, r declaration.Resource, run Runner,
@@ -1419,16 +1244,6 @@ func applyOne(ctx context.Context, sys system.System, r declaration.Resource, ru
"created once it is (novox/hq ADR 0328 §9)"
return out, nil
}
// Not created or replaced before a file it mounts is there: the runtime would make the missing source a
// directory, and the file could never be written there (novox/hq issue 1873). The write of the file applies
// what reads it (WriteFilled, readsChange), this container among them.
if src := mountsAFileNotThere(res, in.files); src != "" {
out := begin(r)
out.Action = "waiting"
out.Detail = "not created: it mounts " + src + ", a file the declaration declares that is not there " +
"yet; created once it is (novox/hq issue 1873)"
return out, nil
}
}
// Nothing below a home is touched through a link an account put there (novox/hq ADR 0266).
// And nothing is unpacked or written into below an agent's home (agent_home_guard.go).
@@ -1578,9 +1393,6 @@ func applyDirectory(r *declaration.Directory, previous store.Applied, wasBefore
}
if existed && wasBefore {
ours := previous.Target != "" && filepath.Clean(previous.Target) == filepath.Clean(r.Path) && !previous.AsFound
// In the mesh's own tree, a directory the engine's own account owns was made by the mesh, on the way to
// something below it or by an earlier apply that did not record it: it is the mesh's (novox/hq issue 1873).
ours = ours || madeInTheMeshsTree(r.Path, before)
if !ours {
owned, err := ownedBy(r.Path, r.Owner)
if err != nil {
@@ -1656,30 +1468,6 @@ func applyDirectory(r *declaration.Directory, previous store.Applied, wasBefore
return out, nil
}
// madeInTheMeshsTree is whether a directory lies in the mesh's own tree (meshDataRoot, below it, never the root
// itself) and belongs to the engine's own account: only that account makes a directory there, so it is the mesh's
// and not found (novox/hq issue 1873). Anywhere else, or owned by any other account, issue 339's rule stands.
func madeInTheMeshsTree(path string, found os.FileInfo) bool {
// By its text and by where it leads: a link in the tree pointing out of it leads outside, and what is there is
// not the tree's (review of issue 1873).
if !inTheMeshsTree(path) {
return false
}
uid, _, ok := ownerOf(found)
return ok && uid == os.Geteuid()
}
// inTheMeshsTree is whether a path lies strictly below the mesh's own tree, both as written and with every link in
// it followed (resolved), against the tree's root resolved the same way (novox/hq issue 1873).
func inTheMeshsTree(path string) bool {
root := filepath.Clean(meshDataRoot)
sep := string(filepath.Separator)
if !strings.HasPrefix(filepath.Clean(path), root+sep) {
return false
}
return strings.HasPrefix(resolved(filepath.Clean(path)), resolved(root)+sep)
}
// applyHandedOver gives a directory the operator handed to the mesh its declared mode and owner (novox/hq issue
// 356). **No link, checked again at the moment of use**: the hand-over refused a link when it was recorded, and a
// component of the path may have become one since — a parent swapped for a link to /etc would carry root's chmod
@@ -1729,19 +1517,6 @@ func applyHandedOver(r *declaration.Directory, out Outcome, mode os.FileMode) (O
// keepFound, when not nil, is where the original of a file this host has no record of is kept
// before it is written over (novox/hq ADR 0100): once, never overwritten, and named in the outcome.
func applyFile(r *declaration.File, previous store.Applied, unseal Unseal, keepFound Keep) (Outcome, error) {
cleared, err := clearRuntimesEmptyDirectory(r.Path)
if err != nil {
return begin(r), err
}
out, err := applyFileAt(r, previous, unseal, keepFound)
if cleared != "" {
out.Detail = joinDetail(cleared, out.Detail)
}
return out, err
}
// applyFileAt writes a declared file: applyFile's work once its path is cleared of what a runtime left there.
func applyFileAt(r *declaration.File, previous store.Applied, unseal Unseal, keepFound Keep) (Outcome, error) {
if r.Into == declaration.IntoBlock {
return applyBlock(r, previous)
}
@@ -1749,12 +1524,6 @@ func applyFileAt(r *declaration.File, previous store.Applied, unseal Unseal, kee
return applyInto(r, previous)
}
out := begin(r)
// A file given by its digest is content to be had, not content: written as it stands it would be an empty file
// (novox/hq ADR 0486 §1, issue 1695). ResolveFileReferences has put its content in before any apply.
if r.IsReference() {
return out, fmt.Errorf("%s is given by digest %s and was not resolved to its content before the apply; "+
"it is not written empty", r.Path, r.Digest)
}
// What actually goes on disk. For a sealed file the mesh never had this, and neither did
// whatever carried the declaration here.
@@ -1781,16 +1550,7 @@ func applyFileAt(r *declaration.File, previous store.Applied, unseal Unseal, kee
return out, fmt.Errorf(
"%s is sealed to this node and this node has no sealing key", r.Path)
}
sealed := r.Sealed
if r.VaultSealed != nil {
// The vault's box, handed to the opener as its JSON (Unseal).
raw, err := json.Marshal(r.VaultSealed)
if err != nil {
return out, err
}
sealed = string(raw)
}
opened, err := unseal(sealed)
opened, err := unseal(r.Sealed)
if err != nil {
return out, fmt.Errorf("cannot open %s: %w", r.Path, err)
}
@@ -1831,7 +1591,7 @@ func applyFileAt(r *declaration.File, previous store.Applied, unseal Unseal, kee
// A seed that is already there is left exactly as it is — whatever has grown in it since is
// not the mesh's to put back (novox/hq issue 035). What this host records is what it once
// wrote, so a later declaration that changes the seed is not mistaken for drift either.
if r.Seeded() && existed {
if r.CreateOnce && existed {
out.wrote = previous.Wrote
if out.wrote == "" {
out.wrote = digestOf(string(existing))
@@ -2464,18 +2224,12 @@ func ExecRunner(ctx context.Context, name string, args ...string) (string, error
if err != nil {
var exit *exec.ExitError
if errors.As(err, &exit) {
words := fmt.Sprintf("%s exited %d: %s", name, exit.ExitCode(), strings.TrimSpace(string(exit.Stderr)))
if exit.ExitCode() == -1 {
// Killed, not exited: say by what, and the context's end when that is what killed it (issue 1945:
// "docker exited -1:" said nothing while a bus container was half replaced).
words = fmt.Sprintf("%s was %s", name, exit.String())
if ctx.Err() != nil {
words += " because its context ended: " + ctx.Err().Error()
}
// The words as they always were, and the exit underneath them, so a caller asks the
// code (system.ExitCode) rather than matching text that this line is free to reword.
return string(out), &exited{
words: fmt.Sprintf("%s exited %d: %s", name, exit.ExitCode(), strings.TrimSpace(string(exit.Stderr))),
exit: exit,
}
// The words, and the exit underneath them, so a caller asks the code (system.ExitCode) rather than
// matching text that this line is free to reword.
return string(out), &exited{words: words, exit: exit}
}
return string(out), fmt.Errorf("%s: %w", name, err)
}
@@ -2580,9 +2334,6 @@ type inputs struct {
// held is the root of every library whose disk or mount is not there, with its name: no container that mounts
// it is created (novox/hq ADR 0328 §9).
held map[string]string
// files is the path of every file the declaration declares: a container mounting one that is not there yet waits
// for it (novox/hq issue 1873).
files map[string]bool
// handed is every group a record that went this apply handed to a resource still asking for it.
handed Handed
}
@@ -2750,11 +2501,6 @@ func containerSpecReading(r *declaration.Container, declares, reads map[string]s
if args := identityArgs(r); len(args) > 0 {
b.WriteString("identity " + strings.Join(args, " ") + "\n")
}
// And the supplementary groups it joins (novox/hq ADR 0480 clause 7): fixed when the container is created, so
// a changed set is a container recreated. Only when there are some, so no other container has its digest moved.
if args := groupArgs(r); len(args) > 0 {
b.WriteString("groups " + strings.Join(args, " ") + "\n")
}
// **What this container reads is part of what it is.**
//
// A container takes its environment and its mounted files once, at start, and never looks
@@ -2896,12 +2642,6 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
// (novox/hq 04-ISSUES/009). Recreating is how a container gets restart-on, which a service
// already has.
reasons := restartedBy(r.RestartOn, changed)
// **A replacement owed since a busy wait began is still owed** (novox/hq ADR 0335): a restart-on trigger
// counts as changed only in the pass it changed, so a container left running then would otherwise read
// unchanged at the next pass, never be replaced, and drop from the report as if it had been.
if owed := owedSinceBusy(r, previous); owed != "" {
reasons = append(reasons, owed)
}
// What this container was created reading, as last recorded. By its declared id first, and by
// its NAME when that id has no record: the bundle's `store` becomes the postgres module's
@@ -2952,8 +2692,6 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
// stop, so a container this apply found stopped by a window is one whose window it can see.
if win, held := in.windows.holding(r.Name, time.Now()); held {
out.reads = wasReading
// A replacement owed since a busy wait stays owed through the window (novox/hq ADR 0335).
out.busySince = previous.BusySince
out.Action = heldStill
out.Detail = fmt.Sprintf("held still by the maintenance window of %s since %s; left as it is, "+
"and converged by the first apply after the window closes (novox/hq issue 224)",
@@ -2964,132 +2702,16 @@ func applyContainer(ctx context.Context, r *declaration.Container, run Runner,
return out, nil
}
aside := ""
if !existed {
// A container an engine set aside and then stopped before it ran the replacement (it died between the two):
// the set-aside one of this apply, put back if the replacement fails and removed once it runs (issue 1945).
if _, err := containerState(ctx, r.Name+asideSuffix, run); err == nil {
aside = r.Name + asideSuffix
}
}
switch {
case existed && (before.Spec == want || legacy) && before.Running && len(reasons) == 0:
out.Action = "unchanged"
return out, nil
case existed:
// **A container that says it is busy is asked before it is replaced** (novox/hq ADR 0335 §2): left
// running while it says busy, within the bound, and replaced on the first pass it says idle. Only a
// running one: a stopped container holds no work in hand.
if r.Busy != nil && before.Running {
if waits := askBeforeReplacing(ctx, r, run, cri, previous, &out); waits {
out.reads = wasReading
return out, nil
}
}
// **The replacement's image is on the machine before the running container is removed**
// (novox/hq issue 1742). `docker run` pulls on its own, but only after the old container is
// gone: an image the registry does not hold then left the service down and every later apply
// failing. Pulled (or found present) first, an unobtainable image fails this resource loudly,
// naming the image, and the old container keeps running.
if err := ensureImage(ctx, cri, r.Image, run); err != nil {
return out, fmt.Errorf("not replacing container %s, which keeps running: %w", r.Name, err)
}
// **The running container is set aside, never removed, until its replacement runs** (issue 1945): stopped and
// renamed, so its name and ports are free, and started again under its own name when the replacement does not
// come up. Removed first, the bus's own container left a node without a bus when its replacement was never
// made (2026-10-25 at 11:23). A container set aside by an earlier apply that stopped midway goes first.
aside = r.Name + asideSuffix
_, _ = run(ctx, cri, "rm", "-f", aside)
if _, err := run(ctx, cri, "rename", r.Name, aside); err != nil {
return out, fmt.Errorf("replacing container %s: setting the running one aside: %w", r.Name, err)
}
if _, err := run(ctx, cri, "stop", aside); err != nil {
_, _ = run(ctx, cri, "rename", aside, r.Name)
return out, fmt.Errorf("replacing container %s: stopping the running one: %w", r.Name, err)
if _, err := run(ctx, cri, "rm", "-f", r.Name); err != nil {
return out, fmt.Errorf("replacing container %s: %w", r.Name, err)
}
}
// putBack starts the container set aside again under its own name, after a replacement that did not come up.
putBack := func(cause error) error {
if aside == "" {
return cause
}
_, _ = run(ctx, cri, "rm", "-f", r.Name)
if _, err := run(ctx, cri, "rename", aside, r.Name); err != nil {
return fmt.Errorf("%w; and the container it replaced could not be put back: %v", cause, err)
}
if _, err := run(ctx, cri, "start", r.Name); err != nil {
return fmt.Errorf("%w; and the container it replaced could not be started again: %v", cause, err)
}
return fmt.Errorf("%w; the container it replaced was started again", cause)
}
args := detachedRunArgs(r, want)
if _, err := run(ctx, cri, args...); err != nil {
return out, putBack(fmt.Errorf("starting container %s: %w", r.Name, err))
}
// Read back. `docker run --detach` returning an id says the container was created, not
// that it is still running — a container whose entrypoint exits immediately satisfies the
// command exactly as one that came up does.
after, err := containerState(ctx, r.Name, run)
if err != nil {
return out, putBack(fmt.Errorf("started container %s and it is not there: %w", r.Name, err))
}
if !after.Running {
return out, putBack(fmt.Errorf(
"container %s was started and is not running. It exited; ask the runtime for its "+
"logs", r.Name))
}
if after.Spec != want {
return out, putBack(fmt.Errorf("container %s is not the one that was declared after creating it", r.Name))
}
// Only now, with the replacement running, is the container set aside removed.
if aside != "" {
if _, err := run(ctx, cri, "rm", "-f", aside); err != nil {
out.Detail = fmt.Sprintf("the container it replaced, set aside as %s, could not be removed: %v", aside, err)
}
}
// The found networks a per-machine setting keeps for it (novox/hq ADR 0163, rule 4), joined
// once it runs: a runtime starts a container on one network, and the others are connected.
// Refused, not skipped, when one cannot be joined — a neighbour that was promised to keep
// reaching this container by name would silently not.
for _, n := range r.Networks {
if _, err := run(ctx, cri, "network", "connect", n, r.Name); err != nil {
return out, fmt.Errorf("container %s could not join the kept network %s: %w", r.Name, n, err)
}
}
out.Action = "created"
if existed {
out.Action = "updated"
switch {
case len(changedFiles) > 0:
out.Detail = "recreated: " + strings.Join(changedFiles, ", ") + " changed"
if len(reasons) > 0 {
out.Detail += "; and to pick up " + strings.Join(reasons, ", ")
}
case len(reasons) > 0:
out.Detail = "recreated to pick up " + strings.Join(reasons, ", ")
case r.RunAs != nil && before.Spec == specWithoutIdentity(r, in.declares, reads):
// It was created before it was told its module's identity (novox/hq ADR 0325 §3).
out.Detail = "recreated to run as its module's identity (novox/hq ADR 0325 §3)"
default:
out.Detail = "replaced; a container's configuration is fixed when it is created"
}
if out.busy != nil {
out.Detail += "; " + out.busy.Words()
}
}
return out, nil
}
// asideSuffix names a running container set aside while its replacement is started (issue 1945).
const asideSuffix = ".replaced"
// detachedRunArgs is the `docker run` that starts a service container and leaves it running.
func detachedRunArgs(r *declaration.Container, want string) []string {
args := []string{"run", "--detach", "--name", r.Name, "--restart", "unless-stopped"}
for _, file := range r.EnvFile {
args = append(args, "--env-file", file)
@@ -3123,7 +2745,6 @@ func detachedRunArgs(r *declaration.Container, want string) []string {
args = append(args, "--volume", v)
}
args = append(args, identityArgs(r)...)
args = append(args, groupArgs(r)...)
// **The one check the container carries is the one the module declared** (ADR 0240 rule 3): an
// exec command, or the image's own adopted by name, with the declared timing. Nothing else on the
// machine sets a container's check.
@@ -3139,7 +2760,55 @@ func detachedRunArgs(r *declaration.Container, want string) []string {
}
args = append(args, r.Image)
args = append(args, r.Args...)
return args
if _, err := run(ctx, cri, args...); err != nil {
return out, fmt.Errorf("starting container %s: %w", r.Name, err)
}
// Read back. `docker run --detach` returning an id says the container was created, not
// that it is still running — a container whose entrypoint exits immediately satisfies the
// command exactly as one that came up does.
after, err := containerState(ctx, r.Name, run)
if err != nil {
return out, fmt.Errorf("started container %s and it is not there: %w", r.Name, err)
}
if !after.Running {
return out, fmt.Errorf(
"container %s was started and is not running. It exited; ask the runtime for its "+
"logs", r.Name)
}
if after.Spec != want {
return out, fmt.Errorf("container %s is not the one that was declared after creating it", r.Name)
}
// The found networks a per-machine setting keeps for it (novox/hq ADR 0163, rule 4), joined
// once it runs: a runtime starts a container on one network, and the others are connected.
// Refused, not skipped, when one cannot be joined — a neighbour that was promised to keep
// reaching this container by name would silently not.
for _, n := range r.Networks {
if _, err := run(ctx, cri, "network", "connect", n, r.Name); err != nil {
return out, fmt.Errorf("container %s could not join the kept network %s: %w", r.Name, n, err)
}
}
out.Action = "created"
if existed {
out.Action = "updated"
switch {
case len(changedFiles) > 0:
out.Detail = "recreated: " + strings.Join(changedFiles, ", ") + " changed"
if len(reasons) > 0 {
out.Detail += "; and to pick up " + strings.Join(reasons, ", ")
}
case len(reasons) > 0:
out.Detail = "recreated to pick up " + strings.Join(reasons, ", ")
case r.RunAs != nil && before.Spec == specWithoutIdentity(r, in.declares, reads):
// It was created before it was told its module's identity (novox/hq ADR 0325 §3).
out.Detail = "recreated to run as its module's identity (novox/hq ADR 0325 §3)"
default:
out.Detail = "replaced; a container's configuration is fixed when it is created"
}
}
return out, nil
}
// runtimeCheckArgs is a declared check as the runtime takes it (novox/hq ADR 0240 rule 3, to-be 48 §3):
@@ -3232,7 +2901,6 @@ func foregroundRunArgs(r *declaration.Container, want string) []string {
args = append(args, "--volume", v)
}
args = append(args, identityArgs(r)...)
args = append(args, groupArgs(r)...)
for _, h := range r.Hosts {
args = append(args, "--add-host", h)
}
@@ -3241,16 +2909,6 @@ func foregroundRunArgs(r *declaration.Container, want string) []string {
return args
}
// groupArgs is the runtime's arguments that add a container's supplementary groups, one per number
// (novox/hq ADR 0480 clause 7, issue 1637). None for a container that declares none.
func groupArgs(r *declaration.Container) []string {
var args []string
for _, g := range r.Groups {
args = append(args, "--group-add", strconv.FormatInt(g, 10))
}
return args
}
// specWithoutIdentity is the spec r had before it was told who to run as.
func specWithoutIdentity(r *declaration.Container, declares, reads map[string]string) string {
plain := *r
@@ -3940,10 +3598,16 @@ func unitKey(scope, user, unit string) string {
// False for what the mesh declares in its own right: the foundation's resources carry no dot at all,
// and the adoption's are named for the mesh rather than for a module. Both belong to no module, and
// their gate is therefore the machine's.
//
// This is the guess from an id alone, for a caller with no declaration in hand; a caller with one asks
// Declaration.ModuleOfID, which reads the module the controller said (issue 1132).
func moduleOf(identity string) (string, bool) { return declaration.ModuleGuess(identity) }
func moduleOf(identity string) (string, bool) {
if strings.HasPrefix(identity, declaration.AdoptionPrefix) {
return "", false
}
at := strings.LastIndex(identity, ".")
if at <= 0 {
return "", false
}
return identity[:at], true
}
// handOver removes what a process replaces, once the process is running and still is a moment later
// (novox/hq issue 213, ADR 0184). A replacement that is not up keeps what it replaces in place
internal/apply/apply_test.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
internal/apply/archive.go
+1 -1
View File
@@ -13,7 +13,7 @@ import (
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/block.go
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"path/filepath"
"strings"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/busy.go
-153
View File
@@ -1,153 +0,0 @@
package apply
import (
"context"
"fmt"
"strings"
"time"
"unicode/utf8"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that says it is busy is replaced once it is idle, within a bound (novox/hq ADR 0335).
//
// **Asked, never waited on.** The apply asks the container once, and while it says busy leaves it running
// as it is and goes on with everything else of the declaration: the machine's lock is held for one ask,
// never for the wait. The wait is carried between passes in the container's record (store.Applied's
// BusySince), so the next pass — the node-engine asks again sooner than its reconcile while anything waits
// — asks again, and so does a node-engine started again in between: the bound runs from the first apply
// that would have replaced the container, whoever applied it.
// waitsBusy is the action of a container left running because it said busy (novox/hq ADR 0335 §2). Its own
// word rather than "unchanged": its declaration moved, and the machine is not yet what was declared.
const waitsBusy = "waits-busy"
// What a report says of a busy container (novox/hq ADR 0335 §2, §3).
const (
// BusyWaiting is a container left running because it said busy, waiting to be replaced.
BusyWaiting = "waiting"
// BusyReplacedIdle is a container replaced on the first pass it said idle, after it had waited.
BusyReplacedIdle = "replaced-when-idle"
// BusyReplacedAtBound is a container still busy at the bound, replaced anyway.
BusyReplacedAtBound = "replaced-at-bound"
)
// BusyWait is one container a busy command held, as a report says it.
type BusyWait struct {
// ID is the resource's id (`<module>.<resource>`), Container its runtime name.
ID string `json:"id"`
Container string `json:"container"`
// State is BusyWaiting, BusyReplacedIdle or BusyReplacedAtBound.
State string `json:"state"`
// Since is when the first apply that would have replaced it ran; Until when the bound replaces it.
Since time.Time `json:"since"`
Until time.Time `json:"until"`
// Said is the last output its busy command gave.
Said string `json:"said,omitempty"`
}
// Words is the wait in a sentence, for an outcome's detail and the console.
func (b BusyWait) Words() string {
since := b.Since.UTC().Format(time.RFC3339)
said := ""
if b.Said != "" {
said = "; its busy command last said: " + b.Said
}
switch b.State {
case BusyReplacedAtBound:
return fmt.Sprintf("replaced while still busy: it said busy since %s and the bound of %s ran out "+
"(novox/hq ADR 0335 §3)%s", since, declaration.BusyBound, said)
case BusyReplacedIdle:
return fmt.Sprintf("replaced once it said idle, after waiting since %s (novox/hq ADR 0335 §2)%s", since, said)
}
return fmt.Sprintf("busy: left running and replaced on the first pass it says idle, or at %s at the latest; "+
"waiting since %s (novox/hq ADR 0335 §2)%s", b.Until.UTC().Format(time.RFC3339), since, said)
}
// owedSinceBusy is the reason a container still owes the replacement it was left running for, or empty: its
// record keeps when its busy wait began, under the same name.
func owedSinceBusy(r *declaration.Container, previous store.Applied) string {
if previous.BusySince == nil || previous.Target != r.Name {
return ""
}
return "the replacement owed since its busy wait began at " + previous.BusySince.UTC().Format(time.RFC3339)
}
// busyClock is the clock the bound is read on. A test moves it.
var busyClock = time.Now
// busySaidMost is how much of a busy command's output a report carries: its last lines, enough to say what
// it held.
const busySaidMost = 400
// askBeforeReplacing asks a running container that declares busy whether it may be replaced now, and says
// in out what came of it. True when it waits: out is then the whole outcome of this pass. False when the
// caller replaces it — idle, or past the bound — with out carrying the wait it ended, if there was one.
func askBeforeReplacing(ctx context.Context, r *declaration.Container, run Runner, cri string,
previous store.Applied, out *Outcome) bool {
now := busyClock().UTC()
busy, said := askBusy(ctx, r, run, cri)
var since time.Time
waited := previous.BusySince != nil && previous.Target == r.Name
if waited {
since = previous.BusySince.UTC()
// A wait kept from a clock that has since gone back reads as starting now, never as later: the bound
// holds from no later than this pass.
if since.After(now) {
since = now
}
}
wait := BusyWait{ID: r.ID, Container: r.Name, Since: since, Said: said}
switch {
case !busy && !waited:
// Idle at the first ask: replaced at once, as a container that declares nothing is.
return false
case !busy:
wait.State, wait.Until = BusyReplacedIdle, since.Add(declaration.BusyBound)
out.busy = &wait
return false
}
if !waited {
since = now
wait.Since = now
}
wait.Until = since.Add(declaration.BusyBound)
if !now.Before(wait.Until) {
wait.State = BusyReplacedAtBound
out.busy = &wait
return false
}
wait.State = BusyWaiting
out.Action, out.Detail = waitsBusy, wait.Words()
out.busySince, out.busy = &since, &wait
return true
}
// askBusy runs a container's busy command inside it, by its shell, as an exec health check is run: busy is
// an exit of 0 and nothing else. An ask that could not be made, or did not answer within its timeout, is
// not busy: the bound is what keeps a container whose command never answers from being held.
func askBusy(ctx context.Context, r *declaration.Container, run Runner, cri string) (bool, string) {
ask, cancel := context.WithTimeout(ctx, r.Busy.Within())
defer cancel()
output, err := run(ask, cri, "exec", r.Name, "/bin/sh", "-c", r.Busy.Command)
said := strings.TrimSpace(output)
if err != nil && said == "" {
said = err.Error()
}
return err == nil, lastOf(said, busySaidMost)
}
// lastOf is the end of a text, at most n bytes, on one line.
func lastOf(s string, n int) string {
s = strings.Join(strings.Fields(s), " ")
if len(s) <= n {
return s
}
from := len(s) - n
for from < len(s) && !utf8.RuneStart(s[from]) {
from++
}
return "…" + s[from:]
}
internal/apply/busy_test.go
-434
View File
@@ -1,434 +0,0 @@
package apply
import (
"context"
"errors"
"path/filepath"
"slices"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that says it is busy is replaced once it is idle, within a bound (novox/hq ADR 0335). The
// machine of these tests runs two containers, a build agent that declares `busy` and a web server that
// does not; every call that replaces a container is counted.
type aBusyMachine struct {
mu sync.Mutex
// containers is each container's name and the spec it was created with; running whether it runs.
containers map[string]string
busy bool // what the agent's busy command answers
said string // what it prints
asked int
calls []string
}
func newBusyMachine() *aBusyMachine {
return &aBusyMachine{containers: map[string]string{}}
}
func (m *aBusyMachine) run(_ context.Context, _ string, args ...string) (string, error) {
args = asideShape(args)
m.mu.Lock()
defer m.mu.Unlock()
switch args[0] {
case "info", "image":
return "27.0\n", nil
case "container": // inspect
name := args[len(args)-1]
spec, there := m.containers[name]
if !there {
return "", errors.New("no such container")
}
return "true\t" + spec, nil
case "exec":
m.asked++
m.calls = append(m.calls, "ask "+args[1])
if strings.Join(args[2:], " ") != "/bin/sh -c mesh-builder busy" {
return "", errors.New("asked something else: " + strings.Join(args[2:], " "))
}
if m.busy {
return m.said + "\n", nil
}
return m.said + "\n", errors.New("exited 1")
case "rm":
name := args[len(args)-1]
m.calls = append(m.calls, "rm "+name)
delete(m.containers, name)
case "run":
name := args[slices.Index(args, "--name")+1]
m.calls = append(m.calls, "create "+name)
for i, a := range args {
if a == "--label" && strings.HasPrefix(args[i+1], specLabel+"=") {
m.containers[name] = strings.TrimPrefix(args[i+1], specLabel+"=")
}
}
}
return "", nil
}
func (m *aBusyMachine) mark() int {
m.mu.Lock()
defer m.mu.Unlock()
return len(m.calls)
}
// since is every call since mark that replaced a container, or asked one.
func (m *aBusyMachine) since(mark int) []string {
m.mu.Lock()
defer m.mu.Unlock()
return append([]string(nil), m.calls[mark:]...)
}
func (m *aBusyMachine) answer(busy bool, said string) {
m.mu.Lock()
defer m.mu.Unlock()
m.busy, m.said = busy, said
}
// twoContainers is the declaration at a version: v moves the agent's image tag through its env, w the web's.
func twoContainers(t *testing.T, agent, web string) *declaration.Declaration {
t.Helper()
return parse(t, `{"declaration":1,"resources":[
{"id":"build-agent.server","type":"container","name":"mesh-build-agent","image":"`+pinned+`",
"env":{"V":"`+agent+`"},"busy":{"command":"mesh-builder busy","timeout":"5s"}},
{"id":"web.server","type":"container","name":"web","image":"`+pinned+`","env":{"V":"`+web+`"}}
]}`)
}
// atClock moves the bound's clock for one test.
func atClock(t *testing.T, at *time.Time) {
t.Helper()
was := busyClock
busyClock = func() time.Time { return *at }
t.Cleanup(func() { busyClock = was })
}
func busyOf(r Report, id string) (BusyWait, bool) {
for _, b := range r.Busy {
if b.ID == id {
return b, true
}
}
return BusyWait{}, false
}
func applyBusy(t *testing.T, m *aBusyMachine, d *declaration.Declaration, known store.State) (Report, store.State) {
t.Helper()
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, m.run, nil, nil)
if err != nil {
t.Fatalf("the apply failed: %v", err)
}
return report, state
}
// The decision's first check: an apply that would replace a container whose busy command exits 0 leaves it
// running, applies the rest, and replaces it on the first pass the command exits non-zero.
func TestABusyContainerIsLeftRunningAndReplacedOnTheFirstPassItSaysIdle(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
// A merge moves both: the agent holds a build, the web server holds nothing.
m.answer(true, "building novox/mesh-controller (check of #280) since 00:58")
mark := m.mark()
report, state := applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm web", "create web"}) {
t.Fatalf("a busy agent was not left running while the rest applied: %v", got)
}
o := outcomeOf(report, "build-agent.server")
if o.Action != waitsBusy {
t.Fatalf("a busy container was reported %q, want %q: %+v", o.Action, waitsBusy, o)
}
for _, says := range []string{"busy", "first pass it says idle", "check of #280", "ADR 0335"} {
if !strings.Contains(o.Detail, says) {
t.Errorf("the outcome does not say %q: %s", says, o.Detail)
}
}
if (Report{Outcomes: []Outcome{o}}).Changed() {
t.Errorf("a container left running reads as a change: %+v", o)
}
b, said := busyOf(report, "build-agent.server")
if !said || b.State != BusyWaiting || !b.Since.Equal(now) || !b.Until.Equal(now.Add(15*time.Minute)) ||
b.Container != "mesh-build-agent" || !strings.Contains(b.Said, "check of #280") {
t.Fatalf("the report does not say the agent waits, since when and until when: %+v", report.Busy)
}
if !report.Waiting() {
t.Error("a report with a container waiting does not say so")
}
if rec, _ := state.Find("build-agent.server"); rec.BusySince == nil || !rec.BusySince.Equal(now) {
t.Fatalf("the wait is not kept in the container's record: %+v", rec)
}
// Asked again a minute later, still busy: still left running, the wait counted from the first apply.
now = now.Add(time.Minute)
mark = m.mark()
report, state = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent"}) {
t.Fatalf("a second pass while busy did more than ask: %v", got)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyWaiting || !b.Since.Equal(now.Add(-time.Minute)) {
t.Fatalf("the second pass does not keep the wait from the first apply: %+v", report.Busy)
}
// The build ends: the next pass replaces it, and says it waited.
now = now.Add(time.Minute)
m.answer(false, "idle")
mark = m.mark()
report, state = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("the agent was not replaced on the first pass it said idle: %v", got)
}
if o := outcomeOf(report, "build-agent.server"); o.Action != "updated" || !strings.Contains(o.Detail, "once it said idle") {
t.Errorf("the replacement does not say it waited: %+v", o)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyReplacedIdle {
t.Errorf("the report does not say the agent was replaced when idle: %+v", report.Busy)
}
if report.Waiting() {
t.Error("a report with nothing waiting says something waits")
}
if rec, _ := state.Find("build-agent.server"); rec.BusySince != nil {
t.Errorf("the wait is still kept after the replacement: %+v", rec)
}
// And the pass after that has nothing to say about it.
report, _ = applyBusy(t, m, twoContainers(t, "2", "2"), state)
if len(report.Busy) != 0 {
t.Errorf("a container replaced is still said: %+v", report.Busy)
}
}
// An idle container is replaced at once, as one that declares nothing is: asked once, nothing waits.
func TestAnIdleContainerIsReplacedAtOnce(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(false, "")
mark := m.mark()
report, _ := applyBusy(t, m, twoContainers(t, "2", "1"), state)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("an idle agent was not replaced at once: %v", got)
}
if len(report.Busy) != 0 {
t.Errorf("an idle container replaced at once is said as a wait: %+v", report.Busy)
}
// And a container whose declaration did not move is never asked.
mark = m.mark()
applyBusy(t, m, twoContainers(t, "2", "1"), state)
if got := m.since(mark); slices.Contains(got, "ask mesh-build-agent") {
t.Errorf("a container nothing replaces was asked: %v", got)
}
}
// Past the bound it is replaced anyway, the report carrying the command's last output — and the bound holds
// across a restart of the node-engine: the wait is read back from the state on disk.
func TestABusyContainerIsReplacedAtTheBoundEvenAfterARestart(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building novox/mesh-host (a check that never ends)")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
// The node-engine stops and starts again: what it knows is what it saved.
path := filepath.Join(t.TempDir(), "state.json")
if err := store.Save(path, state); err != nil {
t.Fatal(err)
}
reloaded, err := store.Load(path)
if err != nil {
t.Fatal(err)
}
now = now.Add(14 * time.Minute)
mark := m.mark()
report, reloaded := applyBusy(t, m, twoContainers(t, "2", "1"), reloaded)
if got := m.since(mark); slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("a container busy within the bound was replaced after a restart: %v", got)
}
if b, _ := busyOf(report, "build-agent.server"); b.State != BusyWaiting || !b.Since.Equal(now.Add(-14*time.Minute)) {
t.Fatalf("a restart started the wait again: %+v", report.Busy)
}
now = now.Add(time.Minute)
mark = m.mark()
report, _ = applyBusy(t, m, twoContainers(t, "2", "1"), reloaded)
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("a container still busy at the bound was not replaced: %v", got)
}
b, _ := busyOf(report, "build-agent.server")
if b.State != BusyReplacedAtBound || !strings.Contains(b.Said, "never ends") {
t.Fatalf("the report does not say it replaced a busy container, with its last output: %+v", report.Busy)
}
if o := outcomeOf(report, "build-agent.server"); !strings.Contains(o.Detail, "still busy") ||
!strings.Contains(o.Detail, "never ends") {
t.Errorf("the outcome does not say a busy container was replaced, with what it said: %s", o.Detail)
}
}
// While one container waits, a send that changes another module on the same machine is applied at once: the
// apply returns after one ask, and holds nothing for the wait.
func TestASendToAnotherModuleIsAppliedWhileOneWaits(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
// The next send moves only the web server.
mark := m.mark()
done := make(chan struct{})
var report Report
go func() {
defer close(done)
report, _ = applyBusy(t, m, twoContainers(t, "2", "2"), state)
}()
select {
case <-done:
case <-time.After(5 * time.Second):
t.Fatal("an apply while a container waits did not return")
}
got := m.since(mark)
if !slices.Contains(got, "create web") || slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("the other module's send was not applied at once beside the wait: %v", got)
}
if o := outcomeOf(report, "web.server"); o.Action != "updated" {
t.Errorf("the web server was reported %q: %+v", o.Action, o)
}
}
// A container's busy is held to its shape where it arrives.
func TestBusyIsReadAndHeldToItsShape(t *testing.T) {
d := twoContainers(t, "1", "1")
c := d.Resources[0].(*declaration.Container)
if c.Busy == nil || c.Busy.Command != "mesh-builder busy" || c.Busy.Within() != 5*time.Second {
t.Fatalf("busy was not read: %+v", c.Busy)
}
for name, raw := range map[string]string{
"no command": `"busy":{"command":" "}`,
"a long timeout": `"busy":{"command":"x","timeout":"5m"}`,
"not a duration": `"busy":{"command":"x","timeout":"soon"}`,
"an unknown key": `"busy":{"command":"x","every":"1m"}`,
"on a step": `"run-once":true,"busy":{"command":"x"}`,
} {
_, err := declaration.Parse([]byte(`{"declaration":1,"resources":[{"id":"a.b","type":"container",` +
`"name":"b","image":"` + pinned + `",` + raw + `}]}`))
if err == nil {
t.Errorf("%s: a busy of that shape was accepted", name)
}
}
}
// A replacement owed to a restart-on trigger is still owed after the wait: a trigger that is no file or
// directory (a service, an identity's or a library's change) is in no container's spec and counts as changed only
// in the pass it changed, so the container must not read unchanged at the next pass and drop from the report as if
// replaced (novox/hq ADR 0335).
func TestAReplacementOwedToARestartTriggerIsMadeOnceIdle(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
d := parse(t, `{"declaration":1,"resources":[
{"id":"build-agent.server","type":"container","name":"mesh-build-agent","image":"`+pinned+`",
"restart-on":["identity:build-agent"],"busy":{"command":"mesh-builder busy"}}
]}`)
r := d.Resources[0].(*declaration.Container)
in := inputs{declares: map[string]string{}}
m := newBusyMachine()
m.containers["mesh-build-agent"] = containerSpec(r, in)
ctx := context.Background()
// The trigger moved this pass, and the agent holds a build: left running.
m.answer(true, "building")
out, err := applyContainer(ctx, r, m.run, map[string]bool{"identity:build-agent": true}, in, store.Applied{})
if err != nil || out.Action != waitsBusy || out.busySince == nil {
t.Fatalf("a busy container was not left running for its trigger: %+v %v", out, err)
}
record := store.Applied{ID: r.ID, Target: r.Name, BusySince: out.busySince}
// The next pass: nothing triggers now, and the replacement is still owed.
now = now.Add(time.Minute)
m.answer(false, "idle")
mark := m.mark()
out, err = applyContainer(ctx, r, m.run, map[string]bool{}, in, record)
if err != nil {
t.Fatal(err)
}
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("the replacement owed to the trigger was dropped: %v (%+v)", got, out)
}
if out.busy == nil || out.busy.State != BusyReplacedIdle || out.busySince != nil {
t.Fatalf("the replacement does not say it was made once idle: %+v", out)
}
}
// A wait kept from a clock that later went back is never longer than the bound from now.
func TestAWaitFromAClockThatWentBackIsClampedToNow(t *testing.T) {
now := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
atClock(t, &now)
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "building")
_, state = applyBusy(t, m, twoContainers(t, "2", "1"), state)
now = now.Add(-time.Hour) // the clock went back an hour
report, _ := applyBusy(t, m, twoContainers(t, "2", "1"), state)
b, _ := busyOf(report, "build-agent.server")
if b.State != BusyWaiting || b.Since.After(now) || !b.Until.Equal(now.Add(15*time.Minute)) {
t.Fatalf("a wait from a later clock is not clamped to now: %+v (now %s)", b, now)
}
}
// An ask that fails — no shell in the image, a timeout, a paused container — is not busy: replaced at once.
func TestAnAskThatFailsIsIdle(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
failing := func(ctx context.Context, name string, args ...string) (string, error) {
if args[0] == "exec" {
m.mu.Lock()
m.calls = append(m.calls, "ask "+args[1])
m.mu.Unlock()
return "", errors.New("OCI runtime exec failed: exec: \"/bin/sh\": no such file or directory")
}
return m.run(ctx, name, args...)
}
mark := m.mark()
report, _, err := Apply(context.Background(), archHost(t), twoContainers(t, "2", "1"), state,
store.OriginDeclared, failing, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := m.since(mark); !slices.Equal(got, []string{"ask mesh-build-agent", "rm mesh-build-agent", "create mesh-build-agent"}) {
t.Fatalf("an ask that failed held the container: %v", got)
}
if len(report.Busy) != 0 {
t.Errorf("an ask that failed is said as a wait: %+v", report.Busy)
}
}
// A stopped container holds no work: it is replaced without being asked.
func TestAStoppedContainerIsNotAsked(t *testing.T) {
m := newBusyMachine()
_, state := applyBusy(t, m, twoContainers(t, "1", "1"), store.State{})
m.answer(true, "would say busy")
stopped := func(ctx context.Context, name string, args ...string) (string, error) {
if args[0] == "container" && args[len(args)-1] == "mesh-build-agent" {
out, err := m.run(ctx, name, args...)
return strings.Replace(out, "true\t", "false\t", 1), err
}
return m.run(ctx, name, args...)
}
mark := m.mark()
if _, _, err := Apply(context.Background(), archHost(t), twoContainers(t, "2", "1"), state,
store.OriginDeclared, stopped, nil, nil); err != nil && !strings.Contains(err.Error(), "not running") {
t.Fatal(err)
}
got := m.since(mark)
if slices.Contains(got, "ask mesh-build-agent") || !slices.Contains(got, "rm mesh-build-agent") {
t.Fatalf("a stopped container was asked, or not replaced: %v", got)
}
}
internal/apply/container_groups_test.go
-54
View File
@@ -1,54 +0,0 @@
package apply
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// Issue 1637 (novox/hq ADR 0480 clause 7): a container's supplementary groups reach the runtime as --group-add.
func groupsContainer(groups ...int64) *declaration.Container {
return &declaration.Container{ID: "n8n.app", Name: "n8n", Image: pinned, Groups: groups}
}
// 1637/AC1: both run commands carry one --group-add per group, and none without groups.
func TestAContainerIsAddedItsSupplementaryGroups(t *testing.T) {
with := groupsContainer(2000, 2001)
for name, args := range map[string][]string{
"detached": detachedRunArgs(with, "x"),
"foreground": foregroundRunArgs(with, "x"),
} {
got := " " + strings.Join(args, " ") + " "
if !strings.Contains(got, " --group-add 2000 ") || !strings.Contains(got, " --group-add 2001 ") {
t.Errorf("1637/AC1: the %s run does not add both groups: %s", name, got)
}
}
without := groupsContainer()
for name, args := range map[string][]string{
"detached": detachedRunArgs(without, "x"),
"foreground": foregroundRunArgs(without, "x"),
} {
if strings.Contains(strings.Join(args, " "), "--group-add") {
t.Errorf("1637/AC1: the %s run adds a group none was declared", name)
}
}
}
// 1637/AC2: a changed set of groups recreates the container; no groups leaves the spec as it was.
func TestAChangeOfGroupsRecreatesTheContainer(t *testing.T) {
none, a, b := groupsContainer(), groupsContainer(2000), groupsContainer(2000, 2001)
if containerSpec(none, inputs{}) == containerSpec(a, inputs{}) || containerSpec(a, inputs{}) == containerSpec(b, inputs{}) {
t.Error("1637/AC2: a change of groups does not change the container's spec")
}
same := groupsContainer(2000)
if containerSpec(a, inputs{}) != containerSpec(same, inputs{}) {
t.Error("1637/AC2: the same groups changed the spec")
}
plain := *none
plain.Groups = nil
if containerSpec(none, inputs{}) != containerSpec(&plain, inputs{}) {
t.Error("1637/AC2: a container with no groups has its spec moved")
}
}
internal/apply/declared_home_test.go
-158
View File
@@ -1,158 +0,0 @@
package apply
import (
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// A module that declares its own system account and a .ssh in that account's home is placed (novox/hq issue 1542):
// the account is not in the user database yet when the declaration is judged, and its home is under /var/lib.
func declaredHomeFixture(t *testing.T) (home string, user *declaration.User) {
t.Helper()
dir := t.TempDir()
passwd := filepath.Join(dir, "passwd")
if err := os.WriteFile(passwd, []byte("root:x:0:0::/root:/bin/sh\n"), 0o644); err != nil {
t.Fatal(err)
}
before := passwdFile
passwdFile = passwd
t.Cleanup(func() { passwdFile = before })
home = filepath.Join(dir, "lib", "mesh-build-go")
return home, &declaration.User{ID: "mesh-build-go.account", Type: declaration.TypeUser, Name: "mesh-build-go",
Shell: "/usr/bin/nologin", Home: home}
}
// 1542/AC1: the .ssh of a declared account, owned by it, is refused by nothing.
func TestIssue1542ADeclaredAccountsSSHIsPlaced(t *testing.T) {
home, user := declaredHomeFixture(t)
ssh := &declaration.Directory{ID: "mesh-build-go.ssh-dir", Type: declaration.TypeDirectory, Path: home + "/.ssh",
Mode: "0700", Owner: "mesh-build-go"}
refused := refusedPlaces([]declaration.Resource{user, ssh})
if len(refused) != 0 {
t.Fatalf("a declared account's own .ssh was refused: %v", refused)
}
if err := (inputs{refused: refused}).refused[filepath.Clean(ssh.Path)]; err != nil {
t.Fatalf("the apply-time check refuses it: %v", err)
}
if line := JudgeDirectories(&declaration.Declaration{Resources: []declaration.Resource{user, ssh}}); strings.Contains(line, "refused already by the placement guard: ") {
t.Fatalf("the read-only judgement refuses it: %s", line)
}
}
// 1542/AC2: another account's directory below the declared home is refused as below any home, and a .ssh below
// no home of the machine or the declaration is refused with today's words.
func TestIssue1542OnlyTheDeclaredAccountsOwnIsPlaced(t *testing.T) {
home, user := declaredHomeFixture(t)
other := &declaration.Directory{ID: "notes.ssh-dir", Type: declaration.TypeDirectory, Path: home + "/.ssh",
Mode: "0700", Owner: "intruder"}
refused := refusedPlaces([]declaration.Resource{user, other})
err := refused[filepath.Clean(other.Path)]
if err == nil || !strings.Contains(err.Error(), "below mesh-build-go's home and declared intruder's") {
t.Fatalf("another account's directory below a declared home: %v", err)
}
stray := &declaration.Directory{ID: "mesh-build-go.ssh-dir", Type: declaration.TypeDirectory,
Path: filepath.Join(filepath.Dir(home), "elsewhere", ".ssh"), Mode: "0700", Owner: "mesh-build-go"}
refused = refusedPlaces([]declaration.Resource{user, stray})
err = refused[filepath.Clean(stray.Path)]
if err == nil || !strings.Contains(err.Error(), "a .ssh directory is placed only below its account's home, as that account's") {
t.Fatalf("a .ssh below no home: %v", err)
}
// Not a .ssh: a root-owned directory below the declared home is placed exactly as on main.
data := &declaration.Directory{ID: "mesh-build-go.data", Type: declaration.TypeDirectory, Path: home + "/data",
Mode: "0755", Owner: "root"}
if refused := refusedPlaces([]declaration.Resource{user, data}); len(refused) != 0 {
t.Fatalf("a root-owned directory below a declared home, not a .ssh, was refused: %v", refused)
}
// And without the account declared, the same .ssh is refused as it was.
own := &declaration.Directory{ID: "mesh-build-go.ssh-dir", Type: declaration.TypeDirectory, Path: home + "/.ssh",
Mode: "0700", Owner: "mesh-build-go"}
if refusedPlaces([]declaration.Resource{own})[filepath.Clean(own.Path)] == nil {
t.Fatal("a .ssh below an account nobody makes was placed")
}
}
// 1542/AC3: a declared account is no home to anyone but the directory placement guard.
func TestIssue1542DeclaredAccountsAreNotHomesToTheOtherGuards(t *testing.T) {
home, user := declaredHomeFixture(t)
_ = refusedPlaces([]declaration.Resource{user})
if _, ok := accountsOfHomes()[home]; ok {
t.Fatal("a declared account became a home of the machine")
}
if homeAbove(home+"/.ssh") != "" {
t.Fatal("a declared account became a home for the links and the move-out")
}
}
// A declared account is added as a home and never takes one over (reviewer, issue 1542): the person's own .ssh
// stays refused to every other account, whatever a declaration says about a user.
func TestIssue1542ADeclaredAccountTakesNoExistingHome(t *testing.T) {
dir := t.TempDir()
passwd := filepath.Join(dir, "passwd")
body := "root:x:0:0::/root:/bin/sh\njochen:x:1000:1000::/home/jochen:/bin/sh\n" +
"svc:x:998:998::/var/lib/svc:/usr/bin/nologin\n"
if err := os.WriteFile(passwd, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
before := passwdFile
passwdFile = passwd
t.Cleanup(func() { passwdFile = before })
newHome := filepath.Join(dir, "lib", "mesh-build-go")
ssh := func(path, owner string) *declaration.Directory {
return &declaration.Directory{ID: "x.ssh", Type: declaration.TypeDirectory, Path: path, Mode: "0700", Owner: owner}
}
user := func(name, home string) *declaration.User {
return &declaration.User{ID: "x.user", Type: declaration.TypeUser, Name: name, Home: home}
}
// A declared user over a person's home, whoever it is named for.
for _, name := range []string{"evil", "postgres", "root"} {
d := ssh("/home/jochen/.ssh", name)
if refusedPlaces([]declaration.Resource{user(name, "/home/jochen"), d})[d.Path] == nil {
t.Errorf("%s, declared over jochen's home, was given jochen's .ssh", name)
}
}
// A declared home below a person's home.
d := ssh("/home/jochen/sub/.ssh", "evil")
if refusedPlaces([]declaration.Resource{user("evil", "/home/jochen/sub"), d})[d.Path] == nil {
t.Error("a home declared below jochen's home was counted")
}
// A name the machine has already, declared with another home: that is not its home.
d = ssh(newHome+"/e/.ssh", "svc")
if refusedPlaces([]declaration.Resource{user("svc", newHome+"/e"), d})[d.Path] == nil {
t.Error("an existing account's declared home was counted")
}
// A number is no name: chown -1 leaves the owner as it was.
for _, owner := range []string{"-1", "-1:0", "0", "1001"} {
d := ssh(newHome+"/.ssh", owner)
if refusedPlaces([]declaration.Resource{user("mesh-build-go", newHome), d})[d.Path] == nil {
t.Errorf("owner %q was placed below a declared home", owner)
}
}
// And the account that is new, by name, still is.
d = ssh(newHome+"/.ssh", "mesh-build-go")
if got := refusedPlaces([]declaration.Resource{user("mesh-build-go", newHome), d}); len(got) != 0 {
t.Errorf("a new account's own .ssh was refused: %v", got)
}
}
// The account the mesh made, as declared, is judged the same at the next apply (issue 1542): once it is in the
// user database with exactly the declared home, its .ssh is still placed.
func TestIssue1542TheAccountTheMeshMadeIsStillPlacedAtTheNextApply(t *testing.T) {
home, user := declaredHomeFixture(t)
ssh := &declaration.Directory{ID: "mesh-build-go.ssh-dir", Type: declaration.TypeDirectory, Path: home + "/.ssh",
Mode: "0700", Owner: "mesh-build-go"}
resources := []declaration.Resource{user, ssh}
if got := refusedPlaces(resources); len(got) != 0 {
t.Fatalf("first apply, the account not yet made: %v", got)
}
body := "root:x:0:0::/root:/bin/sh\nmesh-build-go:x:998:998::" + home + ":/usr/bin/nologin\n"
if err := os.WriteFile(passwdFile, []byte(body), 0o644); err != nil {
t.Fatal(err)
}
if got := refusedPlaces(resources); len(got) != 0 {
t.Fatalf("second apply, the account made as declared: %v", got)
}
}
internal/apply/exec_killed_test.go
-19
View File
@@ -1,19 +0,0 @@
package apply
import (
"context"
"strings"
"testing"
"time"
)
// A command killed because its context ended says so, and why: "docker exited -1:" said nothing while novox's bus
// container was half replaced (issue 1945, live on 2026-10-25 at 11:23).
func TestACommandKilledByItsContextSaysWhy(t *testing.T) {
ctx, cancel := context.WithCancel(context.Background())
go func() { time.Sleep(50 * time.Millisecond); cancel() }()
_, err := ExecRunner(ctx, "sleep", "5")
if err == nil || !strings.Contains(err.Error(), "killed") || !strings.Contains(err.Error(), "context canceled") {
t.Fatalf("a command its context killed said: %v", err)
}
}
internal/apply/filled.go
-235
View File
@@ -1,235 +0,0 @@
package apply
import (
"context"
"fmt"
"slices"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
// WriteFilled writes what the provisioner's client filled into one file the declaration names as filled (novox/hq
// ADR 0349 §4), and makes what reads the file take the change, exactly as an apply that changed it would.
//
// **The declared file says everything but the content.** Its path, mode, owner and who owns it at runtime are the
// declaration's; the client's request carries the content alone. So the file is applied as the declared one with
// that content in place of the declaration's: an owned file whole through applyFile, a seed only when absent, a
// managed file's keys through applyInto or its block through applyBlock — the same guards on where nothing is
// placed, the same Keep of an original found there, the same record. What the declaration seals or fills with
// `${secret:}` the client has already filled, so the content is written as it came; a file that carried a secret
// keeps the mode a secret is written with when the declaration gives none.
//
// **Then what reads it, in the apply's order** (afterWhatTheyRead): a service naming it under restart-on is
// restarted and under reload-on reloaded, a process or container naming it under restart-on is restarted, and a
// running container that reads it at creation — an env-file, a file mounted into it — is recreated, because what
// it read is recorded by digest and this write changed the digest. Each of those is applied and recorded as an
// apply does, and what moves goes on moving what names it. Nothing else of the declaration is touched.
//
// The caller holds the apply's lock: a write and an apply never write one file at once. log is said the path, the
// action and the digest, never the content.
//
// **It does not refresh the runtimes' data roots.** A filled write is one file, not an apply; the placement guard
// judges it with the roots the last apply read (runtimeDataRoots), falling back to the runtimes' configuration
// files when no apply has read them yet — the runtimes themselves are asked once, at an apply's start, not once
// per file (novox/hq issue 1138).
func WriteFilled(
ctx context.Context,
sys system.System,
d *declaration.Declaration,
known store.State,
path string,
content []byte,
run Runner,
log func(string),
keep Keep,
windows *Windows,
) (report Report, _ store.State, _ error) {
if log == nil {
log = func(string) {}
}
if !d.Fills(path) {
return report, known, &Error{Resource: path, Err: fmt.Errorf("this node's declaration does not name %s "+
"as a file the provisioner fills, so nothing was written", path)}
}
var declared []*declaration.File
for _, r := range d.Resources {
if f, ok := r.(*declaration.File); ok && f.Path == path {
declared = append(declared, f)
}
}
if len(declared) != 1 {
// The parser refuses a filled path no file declares; two at one path would need the request to say whose
// keys or block it carries, and it carries the content only.
return report, known, &Error{Resource: path, Err: fmt.Errorf("%d resources declare %s, and a filled file "+
"is written for exactly one, so nothing was written", len(declared), path)}
}
file := filledAs(declared[0], content)
if held, isHeld := known.HeldAt(file.ID); isHeld {
return report, known, &Error{Resource: file.ID, Err: fmt.Errorf("%s is held as found for a module not "+
"yet taken (%s), so nothing was written", path, held.Why)}
}
in := inputs{declares: map[string]string{}, known: &known, windows: windows, away: newStoreAway(windows, log),
groups: groupsWanted(d.Resources), handed: Handed{}}
in.refused = refusedPlaces(d.Resources)
in.agentHomes = rootNeverHomes(d)
in.dirsBefore = map[string]bool{}
in.files = declaredFiles(d.Resources)
// A container that reads the file and mounts a library whose disk or mount is not there is held, as a full apply
// holds it: not created over the missing disk (novox/hq ADR 0328 §9).
in.held = heldLibraries(ctx, d.Libraries)
for _, r := range d.Resources {
in.declares[r.Identity()] = declaredDigest(r)
}
changed := map[string]bool{}
// The file, as the apply loop applies it.
was, _ := known.Find(file.ID)
var keepFound Keep
if (was.ID == "" || was.Target != file.Path) && !known.Recorded(string(declaration.TypeFile), file.Path) {
keepFound = keep
}
outcome, err := applyOne(ctx, sys, file, run, changed, in, was, nil, keepFound)
if err != nil {
log(fmt.Sprintf(" failed %s (%s): %v", file.ID, file.Path, err))
return report, known, &Error{Resource: file.ID, Err: err, Done: report}
}
recordKeeping(&known, file, outcome, was)
report.Outcomes = append(report.Outcomes, outcome)
if !moved(outcome) {
return report, known, nil
}
changed[file.ID] = true
line := fmt.Sprintf(" %s %s (%s), filled: %s", outcome.Action, file.ID, file.Path, outcome.wrote)
if outcome.Detail != "" {
line += ": " + outcome.Detail
}
log(line)
// What reads it, in the apply's order, each moving what names it in turn.
var failures []*Error
for _, r := range afterWhatTheyRead(d.Resources) {
if r.Identity() == file.ID || !readsChange(r, path, changed) {
continue
}
if _, isHeld := known.HeldAt(r.Identity()); isHeld {
log(fmt.Sprintf(" held %s (%s): held as found for a module not yet taken, so it was not restarted "+
"for %s", r.Identity(), r.Target(), file.ID))
continue
}
was, _ := known.Find(r.Identity())
o, err := applyOne(ctx, sys, r, run, changed, in, was, nil, nil)
if err != nil {
if o.found != nil {
known.KeepFound(r.Identity(), store.OriginDeclared, *o.found)
}
failures = append(failures, &Error{Resource: r.Identity(), Err: err})
log(fmt.Sprintf(" failed %s (%s): %v", r.Identity(), o.Target, err))
continue
}
if o.Action == "waiting" {
if o.found != nil {
known.KeepFound(r.Identity(), store.OriginDeclared, *o.found)
}
report.Outcomes = append(report.Outcomes, o)
log(fmt.Sprintf(" waiting %s (%s): %s", o.ID, o.Target, o.Detail))
continue
}
recordKeeping(&known, r, o, was)
report.Outcomes = append(report.Outcomes, o)
if o.busy != nil {
report.Busy = append(report.Busy, *o.busy)
}
if moved(o) {
changed[r.Identity()] = true
line := fmt.Sprintf(" %s %s (%s)", o.Action, o.ID, o.Target)
if o.Detail != "" {
line += ": " + o.Detail
}
log(line)
} else if o.Action != "unchanged" {
log(fmt.Sprintf(" %s %s (%s): %s", o.Action, o.ID, o.Target, o.Detail))
}
}
if len(failures) > 0 {
first := failures[0]
first.Done = report
first.Others = len(failures) - 1
return report, known, first
}
return report, known, nil
}
// filledAs is the declared file with the filled content in place of the declaration's.
func filledAs(declared *declaration.File, content []byte) *declaration.File {
f := *declared
f.Content, f.Sealed, f.VaultSealed, f.Bytes, f.Secrets = string(content), "", nil, "", nil
if f.Mode == "" && (declared.Secret() || len(declared.Secrets) > 0) {
// Written as the declaration would have written it: a file carrying a credential is not world-readable.
f.Mode = "0600"
}
return &f
}
// readsChange reports whether applying r is what makes it take a change of what changed: a service's restart-on or
// reload-on, a process's or container's restart-on, or a running container that reads path at creation.
func readsChange(r declaration.Resource, path string, changed map[string]bool) bool {
switch res := r.(type) {
case *declaration.Service:
return len(restartedBy(res.RestartOn, changed)) > 0 || len(restartedBy(res.ReloadOn, changed)) > 0
case *declaration.Process:
return len(restartedBy(res.RestartOn, changed)) > 0
case *declaration.Container:
if len(restartedBy(res.RestartOn, changed)) > 0 {
return true
}
if res.RunOnce || res.Schedule != "" {
return false
}
if slices.Contains(res.EnvFile, path) {
return true
}
for _, v := range res.Volumes {
if mountSource(v) == path {
return true
}
}
}
return false
}
// recordKeeping records r as applied, carrying where the original it replaced was kept from its record when this
// write kept none — for as long as the resource is recorded at the same target, as the apply loop carries it.
func recordKeeping(known *store.State, r declaration.Resource, o Outcome, was store.Applied) {
kept, keptMode, keptOwner := o.kept, o.keptMode, o.keptOwner
if kept == "" && was.Target == o.Target {
kept, keptMode, keptOwner = was.Kept, was.KeptMode, was.KeptOwner
}
recordApplied(known, store.OriginDeclared, r, o, kept, keptMode, keptOwner)
}
// leftToTheClient is a filled file as an apply leaves it (novox/hq ADR 0349 §5, design 57 P7): untouched, and said
// so. A file the mesh wrote at the same path under another id, going in this apply, is handed to it as an apply hands
// a whole file (novox/hq ADR 0223): its record — what the mesh wrote there, and the original it kept — is the filled
// file's now, so the verb judges what is there against it, and the file is neither removed nor written.
func leftToTheClient(f *declaration.File, known *store.State, handedFiles map[string]store.Applied, report *Report,
log func(string)) Outcome {
if from, handed := handedFiles[f.ID]; handed {
if _, recorded := known.Find(f.ID); !recorded {
rec := from
rec.ID = f.ID
known.Record(rec)
}
delete(handedFiles, f.ID)
known.Forget(from.ID)
detail := "no longer declared; " + f.ID + " declares the file now, filled by the provisioner's client, so " +
"it was handed to it, not removed"
report.Outcomes = append(report.Outcomes, Outcome{ID: from.ID, Type: from.Type, Target: from.Target,
Action: "forgotten", Detail: detail})
log(fmt.Sprintf(" forgotten %s (%s): %s", from.ID, from.Target, detail))
}
return Outcome{ID: f.ID, Type: string(declaration.TypeFile), Target: f.Path, Action: "unchanged",
Detail: "filled by the provisioner's client, which alone has it written"}
}
internal/apply/filled_apply_test.go
-218
View File
@@ -1,218 +0,0 @@
package apply
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §5, one writer per file (design 57 P7, issue 673): a file the declaration names as
// filled is written only through the write verb (WriteFilled), on the provisioner client's word. An apply — the
// controller's send and the five-minute re-apply alike — never writes it, never records it, and removes it as an
// orphan only when the verb placed it.
// An apply leaves a filled file alone, whatever who owns it at runtime says: absent stays absent and is not
// recorded, and what the verb wrote stays as it wrote it through every re-apply.
func TestAnApplyNeverWritesAFilledFile(t *testing.T) {
for _, c := range []struct{ name, fields, content string }{
{"owned", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n"},
{"seed", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n"},
{"managed block", `"into":"block","content":"","owner-at-runtime":"managed"`, "10.42.0.1 ace\n"},
} {
t.Run(c.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
d := filledDecl(t, path, c.fields, "")
report, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); err == nil {
t.Fatalf("an apply wrote a filled file the client has not filled: %q", readText(t, path))
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("an apply recorded a filled file it did not write")
}
if o := outcomeOf(report, "m.conf"); o.Action != "unchanged" || !strings.Contains(o.Detail, "provisioner") {
t.Errorf("an apply did not say it left the file to the provisioner's client: %+v", o)
}
_, state, err = WriteFilled(context.Background(), archHost(t), d, state, path, []byte(c.content),
noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
placed, _ := state.Find("m.conf")
want, wantMode := readText(t, path), modeAt(t, path)
// The re-apply, twice over: nothing written, the verb's record kept as it wrote it.
for i := 0; i < 2; i++ {
var said []string
report, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared,
noServices, func(line string) { said = append(said, line) }, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != want || modeAt(t, path) != wantMode {
t.Fatalf("re-apply %d wrote the filled file again: %q, want %q", i+1, got, want)
}
if o := outcomeOf(report, "m.conf"); o.Action != "unchanged" {
t.Errorf("re-apply %d: %+v", i+1, o)
}
if rec, _ := state.Find("m.conf"); rec.Wrote != placed.Wrote || rec.Target != path {
t.Errorf("re-apply %d changed the verb's record: %+v, was %+v", i+1, rec, placed)
}
if len(said) != 0 {
t.Errorf("re-apply %d said something of an untouched file: %q", i+1, said)
}
}
})
}
}
// What reads a filled file is moved by the verb's write, once, never again by an apply: a service under
// restart-on is not restarted and a container reading it is not recreated by the re-apply.
func TestAnApplyMovesNothingForAFilledFile(t *testing.T) {
env := filepath.Join(t.TempDir(), "forge.env")
d := filledDecl(t, env, `"content":"","mode":"0600","owner-at-runtime":"owned"`,
`{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}`)
m := &machine{containers: map[string]*fakeContainer{}}
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, env, []byte("PORT=5432\n"),
m.run, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
m.asked = nil
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, m.run, nil,
nil); err != nil {
t.Fatal(err)
}
if m.removed("forge") || m.did("docker run") {
t.Errorf("the re-apply recreated the container reading the filled file: %v", m.asked)
}
if got := readText(t, env); got != "PORT=5432\n" {
t.Errorf("the re-apply wrote the filled file: %q", got)
}
conf := filepath.Join(t.TempDir(), "daemon.conf")
d = filledDecl(t, conf, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","restart-on":["m.conf"]}`)
var commands []string
_, state, err = WriteFilled(context.Background(), archHost(t), d, store.State{}, conf, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil)
if err != nil {
t.Fatal(err)
}
commands = nil
if _, _, err := Apply(context.Background(), archHost(t), d, state, store.OriginDeclared,
recordingServices(&commands), nil, nil); err != nil {
t.Fatal(err)
}
if n := count(commands, "systemctl stop m.service"); n != 0 {
t.Errorf("the re-apply restarted what reads the filled file: %v", commands)
}
}
// A filled file the verb never placed is not the mesh's to remove: one found there, or put there by hand, stays
// when the file leaves the declaration.
func TestAFilledFileTheVerbNeverPlacedIsNotRemoved(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
if err := os.WriteFile(path, []byte("the machine's own\n"), 0o644); err != nil {
t.Fatal(err)
}
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
gone := elsewhere(t)
report, state, err := Apply(context.Background(), archHost(t), gone, state, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != "the machine's own\n" {
t.Errorf("a filled file the verb never placed was touched when it left: %q", got)
}
if o := outcomeOf(report, "m.conf"); o.Action == "removed" {
t.Errorf("a filled file the verb never placed was removed: %+v", o)
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("a record of the filled file was left")
}
}
// What the verb placed is the mesh's: when the file leaves the declaration it is removed as an orphan, as a file
// an apply wrote is.
func TestAFilledFileTheVerbPlacedIsRemovedWhenItLeaves(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err := Apply(context.Background(), archHost(t), d, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
_, state, err = WriteFilled(context.Background(), archHost(t), d, state, path, []byte("port = 8080\n"),
noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
gone := elsewhere(t)
report, state, err := Apply(context.Background(), archHost(t), gone, state, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
if _, err := os.Stat(path); err == nil {
t.Errorf("a filled file the verb placed stayed when it left the declaration: %q", readText(t, path))
}
if o := outcomeOf(report, "m.conf"); o.Action != "removed" {
t.Errorf("not removed: %+v", o)
}
if _, ok := state.Find("m.conf"); ok {
t.Errorf("its record stayed")
}
}
// A file the mesh wrote the old way and that is now filled keeps its record for the verb to judge against, and is
// not written over by the apply that hands it to the client; a file at the same path recorded under another id
// is handed to the filled one, not removed.
func TestAFileThatMovesToTheClientIsHandedNotRewritten(t *testing.T) {
path := filepath.Join(t.TempDir(), "the-file")
old := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"m.old","type":"file","path":%q,`+
`"content":"port = 1\n"}]}`, path))
_, state, err := Apply(context.Background(), archHost(t), old, store.State{}, store.OriginDeclared, noServices,
nil, nil)
if err != nil {
t.Fatal(err)
}
was, _ := state.Find("m.old")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, state, err = Apply(context.Background(), archHost(t), d, state, store.OriginDeclared, noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := readText(t, path); got != "port = 1\n" {
t.Errorf("the file handed to the client was rewritten or removed: %q", got)
}
if _, ok := state.Find("m.old"); ok {
t.Errorf("the record going was kept beside the filled file's")
}
if rec, ok := state.Find("m.conf"); !ok || rec.Wrote != was.Wrote || rec.Target != path {
t.Errorf("the filled file did not take the record of what the mesh wrote there: %+v", rec)
}
}
// elsewhere is a declaration with nothing at the filled path: one unrelated file.
func elsewhere(t *testing.T) *declaration.Declaration {
t.Helper()
return parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"n.other","type":"file","path":%q,`+
`"content":"x\n"}]}`, filepath.Join(t.TempDir(), "other")))
}
internal/apply/filled_library_test.go
-57
View File
@@ -1,57 +0,0 @@
//go:build linux
package apply
import (
"context"
"fmt"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// A container that reads a filled file and mounts a library whose disk is not there is held by the write verb as a
// full apply holds it (novox/hq ADR 0328 §9, issue 668): never created or recreated over the missing disk.
func TestAWriteDoesNotRecreateAContainerHeldForItsLibrary(t *testing.T) {
// No folder is given the group: the library's root is absent, and nothing is made.
gid := 61000
libraries(t, nil, nil)
base := t.TempDir()
root := filepath.Join(base, "gone", "media")
env := filepath.Join(base, "radarr.env")
raw := fmt.Sprintf(`{"declaration":1,"provisioner_account":"mesh-node-provisioner","filled_files":[%q],
"groups":[{"name":"mesh-media","gid":%d}],"libraries":[%s],"resources":[
{"id":"radarr.env","type":"file","path":%q,"content":"","mode":"0600","owner-at-runtime":"owned"},
{"id":"radarr.app","type":"container","name":"radarr","image":%q,"env-file":[%q],"volumes":[%q]}]}`,
env, gid, libJSON("media", root, gid, true, false, "movies"), env, pinned, env,
filepath.Join(root, "movies")+":/movies")
d, err := declaration.Parse([]byte(raw))
if err != nil {
t.Fatal(err)
}
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
ran = append(ran, name+" "+strings.Join(args, " "))
return "", nil
}
var state store.State
for _, content := range []string{"PORT=1\n", "PORT=2\n"} {
var report Report
report, state, err = WriteFilled(context.Background(), archHost(t), d, state, env, []byte(content), run,
nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if o := outcomeOf(report, "radarr.app"); o.Action != "waiting" {
t.Errorf("after writing %q the container is %+v, not held for its library", content, o)
}
}
for _, line := range ran {
if strings.HasPrefix(line, "docker run") || strings.HasPrefix(line, "docker rm") {
t.Errorf("the container was created or removed over a missing disk: %v", ran)
}
}
}
internal/apply/filled_runtime_roots_test.go
-168
View File
@@ -1,168 +0,0 @@
package apply
import (
"context"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-host/internal/store"
)
// A filled write is one file, not an apply: it asks no container runtime, and judges where nothing is placed
// (issue 339) with the roots the last apply read, falling back to the runtimes' configuration files when no apply
// has read them yet (novox/hq issue 1138). These tests set the package's runtimeRoots and AskRuntimes, so none
// runs in parallel with another that reads or sets them, and each restores what it changed.
// setRuntimeRoots makes runtimeDataRoots answer roots, as if an apply had just read them (or, with nil, as if none
// ever had), restoring what a test before it left there once this one is done.
func setRuntimeRoots(t *testing.T, roots []string) {
t.Helper()
runtimeRootsMu.Lock()
before := runtimeRoots
runtimeRoots = roots
runtimeRootsMu.Unlock()
t.Cleanup(func() {
runtimeRootsMu.Lock()
runtimeRoots = before
runtimeRootsMu.Unlock()
})
}
// askNoRuntime makes AskRuntimes record the name of every runtime it is asked for, answering as one that is not
// running does (nothing, no error): were it consulted, that is what would say so. Restored once the test is done.
func askNoRuntime(t *testing.T, asked *[]string) {
t.Helper()
before := AskRuntimes
AskRuntimes = func(_ context.Context, name string, _ ...string) (string, error) {
*asked = append(*asked, name)
return "", nil
}
t.Cleanup(func() { AskRuntimes = before })
}
// aRuntimeConfig points the runtimes' configuration at a fresh, unmounted root below a temp directory, restoring
// the engine's real configuration once the test is done.
func aRuntimeConfig(t *testing.T) (dir, root string) {
t.Helper()
dir = t.TempDir()
root = filepath.Join(dir, "docker")
daemon := filepath.Join(dir, "daemon.json")
if err := os.WriteFile(daemon, []byte(`{"data-root":"`+root+`"}`), 0o644); err != nil {
t.Fatal(err)
}
before := runtimeConfigs
runtimeConfigs = runtimeFiles{daemonJSON: daemon, storageConf: filepath.Join(dir, "none.conf")}
t.Cleanup(func() { runtimeConfigs = before })
return dir, root
}
// refusedAsPlacement fails the test unless err is a PlacementRefusedError.
func refusedAsPlacement(t *testing.T, err error) {
t.Helper()
var refusal *PlacementRefusedError
if err == nil || !errors.As(err, &refusal) {
t.Fatalf("a filled file below a refused runtime root was not refused: %v", err)
}
}
// 1138/AC1: a filled write asks no runtime, and judges with the roots the last apply read.
func TestAFilledWriteAsksNoRuntimeAndUsesTheLastApplysRoots(t *testing.T) {
root := filepath.Join(t.TempDir(), "docker-root")
setRuntimeRoots(t, []string{root}) // as if an apply had just read it
var asked []string
askNoRuntime(t, &asked)
path := filepath.Join(root, "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
if len(asked) != 0 {
t.Errorf("a filled write asked the container runtimes: %v", asked)
}
}
// 1138/AC2: given a real apply that read a runtime data root, a filled write below it is refused as today, with
// no apply run again in between.
func TestAFilledWriteBelowARootARealApplyReadIsRefused(t *testing.T) {
dir, root := aRuntimeConfig(t)
t.Cleanup(func() {
runtimeRootsMu.Lock()
runtimeRoots = nil
runtimeRootsMu.Unlock()
})
// A full apply of something unrelated reads the runtimes' configuration once, at its start.
elsewhere := filepath.Join(dir, "elsewhere.conf")
if _, _, err := Apply(context.Background(), archHost(t), declare(t,
fmt.Sprintf(`{"id":"m.other","type":"file","path":%q,"content":"x\n"}`, elsewhere)),
store.State{}, store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
// A filled write below the runtime's root, with no apply run since, is still refused.
path := filepath.Join(root, "sub", "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
}
// 1138/AC3: with no apply having read the roots yet, a filled write judges by the runtimes' configuration files,
// asking no runtime.
func TestAFilledWriteWithNoApplyYetUsesTheRuntimesConfigFiles(t *testing.T) {
setRuntimeRoots(t, nil) // no apply has read the roots yet
_, root := aRuntimeConfig(t)
var asked []string
askNoRuntime(t, &asked)
path := filepath.Join(root, "sub", "app.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`, "")
_, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("x\n"), noServices,
nil, nil, nil)
refusedAsPlacement(t, err)
if len(asked) != 0 {
t.Errorf("a filled write with no apply yet asked the container runtimes: %v", asked)
}
}
// 1138/AC4: a full apply still asks each runtime once, at its start, whatever the number of files it writes.
func TestAFullApplyAsksEachRuntimeOnceRegardlessOfFileCount(t *testing.T) {
dir := t.TempDir()
var asked []string
before := AskRuntimes
AskRuntimes = func(_ context.Context, name string, _ ...string) (string, error) {
asked = append(asked, name)
return "", errors.New("not running in this test")
}
t.Cleanup(func() {
AskRuntimes = before
runtimeRootsMu.Lock()
runtimeRoots = nil
runtimeRootsMu.Unlock()
})
var resources []string
for i := 0; i < 3; i++ {
resources = append(resources, fmt.Sprintf(`{"id":"m.f%d","type":"file","path":%q,"content":"x\n"}`, i,
filepath.Join(dir, fmt.Sprintf("f%d.conf", i))))
}
if _, _, err := Apply(context.Background(), archHost(t), declare(t, strings.Join(resources, ",")),
store.State{}, store.OriginDeclared, noServices, nil, nil); err != nil {
t.Fatal(err)
}
if got := count(asked, "docker"); got != 1 {
t.Errorf("docker was asked %d times for one apply of 3 files; want 1: %v", got, asked)
}
if got := count(asked, "podman"); got != 1 {
t.Errorf("podman was asked %d times for one apply of 3 files; want 1: %v", got, asked)
}
}
internal/apply/filled_test.go
-271
View File
@@ -1,271 +0,0 @@
package apply
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// Defends novox/hq ADR 0349 §4 (issue 668): the provisioner's client has the node-engine write a file it filled,
// and the node-engine writes it as an apply of the declared file with that content would — through applyFile,
// applyInto or applyBlock by who owns it at runtime — and makes what reads it take the change as an apply does.
// filledDecl is a declaration naming path as filled, with the file's fields after id, type and path, and more
// resources after it.
func filledDecl(t *testing.T, path, fields, more string) *declaration.Declaration {
t.Helper()
if more != "" {
more = "," + more
}
return parseTrusted(t, fmt.Sprintf(`{"declaration":1,"provisioner_account":"mesh-node-provisioner",
"filled_files":[%q],"resources":[{"id":"m.conf","type":"file","path":%q,%s}%s]}`, path, path, fields, more))
}
func TestAWriteOfAPathNotFilledIsRefused(t *testing.T) {
dir := t.TempDir()
filled := filepath.Join(dir, "filled.conf")
other := filepath.Join(dir, "other.conf")
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"filled_files":[%q],"resources":[
{"id":"m.conf","type":"file","path":%q,"content":"x\n","owner-at-runtime":"owned"},
{"id":"m.other","type":"file","path":%q,"content":"y\n"}
]}`, filled, filled, other))
for _, path := range []string{other, filepath.Join(dir, "nothing.conf")} {
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("evil\n"),
noServices, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "does not name") {
t.Errorf("%s: a write of a path the declaration does not fill was not refused: %v", path, err)
}
if _, err := os.Stat(path); err == nil && path != other {
t.Errorf("%s was written", path)
}
if len(state.Resources) != 0 {
t.Errorf("a refused write recorded something: %+v", state.Resources)
}
}
if _, err := os.Stat(other); err == nil {
t.Errorf("%s was written", other)
}
t.Run("a filled path two resources declare", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "hosts")
if err := os.WriteFile(path, []byte(workstationHosts), 0o644); err != nil {
t.Fatal(err)
}
d := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"filled_files":[%q],"resources":[
{"id":"a.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.1 ace\n","owner-at-runtime":"managed"},
{"id":"b.hosts","type":"file","path":%q,"into":"block","content":"10.42.0.2 g14\n","owner-at-runtime":"managed"}
]}`, path, path, path))
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("6.6.6.6 evil\n"),
noServices, nil, nil, nil)
if err == nil || !strings.Contains(err.Error(), "2 resources declare") {
t.Fatalf("a path two resources declare was not refused: %v", err)
}
if got := readText(t, path); got != workstationHosts || len(state.Resources) != 0 {
t.Errorf("something was written or recorded: %q %+v", got, state.Resources)
}
})
}
// Each ownership is written by the write verb exactly as an apply of the declared file with the same content: the
// same action, the same bytes on disk, the same mode.
func TestAFilledFileIsWrittenAsItsOwnershipSays(t *testing.T) {
for _, c := range []struct {
name, fields, content, before, action string
}{
{"owned, absent", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n", "", "created"},
{"owned, over what is there", `"content":"","mode":"0640","owner-at-runtime":"owned"`, "port = 8080\n",
"port = 1\n", "updated"},
{"seed, absent", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n", "", "created"},
{"seed, present", `"content":"","mode":"0600","owner-at-runtime":"seed"`, "user default on\n",
"user default on\nuser app-one on\n", "kept"},
{"managed json", `"into":"json","content":"{}","owner-at-runtime":"managed"`,
`{"insecure-registries":["10.42.0.1:5000"]}`, machinesOwn, "updated"},
{"managed block", `"into":"block","content":"x\n","owner-at-runtime":"managed"`, "10.42.0.1 ace\n",
workstationHosts, "updated"},
} {
t.Run(c.name, func(t *testing.T) {
at := func() string {
path := filepath.Join(t.TempDir(), "the-file")
if c.before != "" {
if err := os.WriteFile(path, []byte(c.before), 0o644); err != nil {
t.Fatal(err)
}
}
return path
}
written := at()
report, state, err := WriteFilled(context.Background(), archHost(t), filledDecl(t, written, c.fields, ""),
store.State{}, written, []byte(c.content), noServices, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
applied := at()
withContent := strings.Replace(c.fields, fmt.Sprintf(`"content":%q`, contentOf(t, c.fields)),
fmt.Sprintf(`"content":%q`, c.content), 1)
// The same file declared the old way: an apply writes a filled file only on the client's word (P7).
plain := parseTrusted(t, fmt.Sprintf(`{"declaration":1,"resources":[{"id":"m.conf","type":"file",`+
`"path":%q,%s}]}`, applied, withContent))
want, _, err := Apply(context.Background(), archHost(t), plain, store.State{}, store.OriginDeclared,
noServices, nil, nil)
if err != nil {
t.Fatal(err)
}
if got := report.Outcomes[0].Action; got != c.action || got != want.Outcomes[0].Action {
t.Errorf("written %s; an apply of the same content %s; want %s", got, want.Outcomes[0].Action, c.action)
}
if got, apply := readText(t, written), readText(t, applied); got != apply {
t.Errorf("written, the file holds\n%q\nan apply of the same content leaves\n%q", got, apply)
}
if a, b := modeAt(t, written), modeAt(t, applied); a != b {
t.Errorf("written mode %o, an apply's %o", a, b)
}
if rec, ok := state.Find("m.conf"); !ok || rec.Target != written || rec.Origin != store.OriginDeclared {
t.Errorf("the write is not recorded as the file's: %+v", rec)
}
})
}
t.Run("a filled file that carried a secret is not world-readable", func(t *testing.T) {
path := filepath.Join(t.TempDir(), "token.env")
d := filledDecl(t, path, `"content":"TOKEN=${secret:token}\n","secrets":{"token":"c2VhbGVk"},"owner-at-runtime":"owned"`, "")
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("TOKEN=abc\n"),
noServices, nil, nil, nil); err != nil {
t.Fatal(err)
}
if m := modeAt(t, path); m != 0o600 {
t.Errorf("a filled file that carried a secret is mode %o", m)
}
})
}
func contentOf(t *testing.T, fields string) string {
t.Helper()
var holder struct {
Content string `json:"content"`
}
if err := json.Unmarshal([]byte("{"+fields+"}"), &holder); err != nil {
t.Fatal(err)
}
return holder.Content
}
func modeAt(t *testing.T, path string) os.FileMode {
t.Helper()
info, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
return info.Mode().Perm()
}
// A file that carried a secret keeps the mode a secret is written with: the client filled the value, and the file
// is no more readable for it.
func TestAWriteUnderReloadOnReloadsItsProgramOnce(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","reload-on":["m.conf"]},
{"id":"m.other","type":"service","unit":"other.service","state":"running"}`)
var commands []string
var said []string
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("level = 1\n"),
recordingServices(&commands), func(line string) { said = append(said, line) }, nil, nil)
if err != nil {
t.Fatal(err)
}
if n := count(commands, "systemctl reload m.service"); n != 1 {
t.Errorf("the program was reloaded %d times; commands were %v", n, commands)
}
joined := strings.Join(commands, "\n")
if strings.Contains(joined, "restart m.service") || strings.Contains(joined, "stop m.service") {
t.Errorf("the program was restarted, not reloaded: %v", commands)
}
if strings.Contains(joined, "other.service") {
t.Errorf("a service that does not read the file was touched: %v", commands)
}
if _, ok := state.Find("m.daemon"); !ok {
t.Errorf("the reloaded service was not recorded")
}
// The log says the path and the digest, never the content.
for _, line := range said {
if strings.Contains(line, "level = 1") {
t.Errorf("the content was said: %q", line)
}
}
if !strings.Contains(strings.Join(said, "\n"), digestOf("level = 1\n")) {
t.Errorf("the digest was not said: %q", said)
}
// The same content again moves nothing, and reloads nothing.
commands = nil
report, _, err := WriteFilled(context.Background(), archHost(t), d, state, path, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if report.Outcomes[0].Action != "unchanged" || count(commands, "reload") != 0 {
t.Errorf("an unchanged write reloaded: %v %+v", commands, report.Outcomes)
}
}
func TestAWriteUnderRestartOnRestartsItsProgram(t *testing.T) {
path := filepath.Join(t.TempDir(), "daemon.conf")
d := filledDecl(t, path, `"content":"","owner-at-runtime":"owned"`,
`{"id":"m.daemon","type":"service","unit":"m.service","state":"running","restart-on":["m.conf"]}`)
var commands []string
if _, _, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, path, []byte("level = 1\n"),
recordingServices(&commands), nil, nil, nil); err != nil {
t.Fatal(err)
}
// A restart is a stop and a start, once.
stops, starts := count(commands, "systemctl stop m.service"), count(commands, "systemctl start m.service")
if stops != 1 || starts != 1 {
t.Errorf("the program was stopped %d and started %d times; commands were %v", stops, starts, commands)
}
}
// A running container that reads the file at creation is recreated, as an apply recreates it for a changed file.
func TestAWriteOfAFileAContainerReadsRecreatesIt(t *testing.T) {
env := filepath.Join(t.TempDir(), "forge.env")
d := filledDecl(t, env, `"content":"","mode":"0600","owner-at-runtime":"owned"`,
`{"id":"forge.server","type":"container","name":"forge","image":"`+pinned+`","env-file":["`+env+`"]}`)
m := &machine{containers: map[string]*fakeContainer{}}
_, state, err := WriteFilled(context.Background(), archHost(t), d, store.State{}, env, []byte("PORT=5432\n"),
m.run, nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !m.did("docker run") {
t.Fatalf("the container was not created: %v", m.asked)
}
m.asked = nil
report, _, err := WriteFilled(context.Background(), archHost(t), d, state, env, []byte("PORT=5433\n"), m.run,
nil, nil, nil)
if err != nil {
t.Fatal(err)
}
if !m.removed("forge") || !m.did("docker run") {
t.Fatalf("the container kept what it read before the write: %v", m.asked)
}
if o := outcomeOf(report, "forge.server"); o.Action != "updated" || !strings.Contains(o.Detail, env) {
t.Errorf("the recreate did not say which file changed: %+v", o)
}
}
func count(lines []string, prefix string) int {
n := 0
for _, l := range lines {
if strings.Contains(l, prefix) {
n++
}
}
return n
}
// Two resources declaring one filled path — two modules' blocks in one file — are refused: the request carries the
// content only, so it cannot say whose block it is, and nothing is written.
internal/apply/groups.go
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"fmt"
"strings"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
internal/apply/health_check_test.go
+1 -1
View File
@@ -6,7 +6,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/hold.go
+1 -1
View File
@@ -13,7 +13,7 @@ import (
"syscall"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
internal/apply/hold_test.go
+2 -25
View File
@@ -9,7 +9,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
@@ -111,10 +111,7 @@ type fakeContainer struct {
// options. The host passes `--` before every unit (third review of mesh-catalog #147), which
// internal/system's own test holds; the fakes read the unit where they always did.
func oldShape(name string, args []string) []string {
if name != "systemctl" {
return asideShape(args)
}
if len(args) < 2 || args[len(args)-2] != "--" {
if name != "systemctl" || len(args) < 2 || args[len(args)-2] != "--" {
return args
}
v := 0
@@ -188,13 +185,6 @@ func (m *machine) run(_ context.Context, name string, args ...string) (string, e
return "", errors.New("no such image")
}
return "", nil
case "pull":
// A real runtime leaves the image present after a pull.
if m.images == nil {
m.images = map[string]string{}
}
m.images[args[len(args)-1]] = "2026-01-01T00:00:00Z"
return "", nil
case "network":
if len(args) > 1 && args[1] == "inspect" {
return strings.Join(m.members[args[len(args)-1]], ",") + ",\n", nil
@@ -1131,16 +1121,3 @@ func TestAPathTheMeshOnlySetAccessOnIsStillFound(t *testing.T) {
t.Errorf("it was re-moded to %o", info.Mode().Perm())
}
}
// asideShape reads the steps of a container set aside while its replacement starts (issue 1945) as the removal the
// tests were written against: setting it aside is its removal, and what is done to the set-aside container afterwards
// is no step of theirs.
func asideShape(args []string) []string {
if len(args) == 3 && args[0] == "rename" && args[2] == args[1]+asideSuffix {
return []string{"rm", "-f", args[1]}
}
if len(args) > 0 && strings.HasSuffix(args[len(args)-1], asideSuffix) && (args[0] == "rm" || args[0] == "stop") {
return []string{"aside"}
}
return args
}
internal/apply/homes_other.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"os"
"path/filepath"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
func chmodUnder(home, path string, mode os.FileMode) error {
internal/apply/identities.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
internal/apply/identities_test.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
internal/apply/into.go
+1 -1
View File
@@ -10,7 +10,7 @@ import (
"slices"
"sort"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/into_handover_test.go
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/into_test.go
+1 -1
View File
@@ -9,7 +9,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/judge_directories_test.go
+2 -5
View File
@@ -11,9 +11,7 @@ import (
// The read-only judgement of a node's directories (novox/hq issue 358): one line each — placed, would be refused
// naming the link and who could replace it, or not there yet (placed: an apply makes it) — and nothing made or
// changed. Who could replace the link is "every account", "the account X" or "the group X", by the account and the
// directories the test runs as: the checker's isolated step runs as the account nobody, which owns / there (issue 1919,
// R4g), so the test demands a who in any of the forms couldReplace and linkReplacer produce, not "every account" alone.
// changed.
func TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing(t *testing.T) {
base := t.TempDir()
plain := filepath.Join(base, "plain")
@@ -41,8 +39,7 @@ func TestJudgeDirectoriesSaysWhatTheWalkWouldRefuseAndChangesNothing(t *testing.
t.Errorf("a directory the walk reaches is not said placed: %q", said)
}
if !strings.Contains(lines[2], "would be refused") || !strings.Contains(lines[2], filepath.Join(base, "linked")) ||
!(strings.Contains(lines[2], "every account (") || strings.Contains(lines[2], "the account ") ||
strings.Contains(lines[2], "the group ")) {
!strings.Contains(lines[2], "every account") {
t.Errorf("the linked directory is not said refused naming the link and who: %q", lines[2])
}
if _, err := os.Stat(filepath.Join(base, "missing")); !os.IsNotExist(err) {
internal/apply/keep.go
+1 -1
View File
@@ -12,7 +12,7 @@ import (
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
internal/apply/containers_kept_test.go → internal/apply/left_out_test.go
+64 -4
View File
@@ -6,7 +6,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
@@ -20,9 +20,6 @@ func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
if name != "docker" {
return "", errors.New("not installed")
}
if len(args) > 0 && args[0] == "container" && strings.HasSuffix(args[len(args)-1], asideSuffix) {
return "", errors.New("no such container") // nothing was left set aside
}
ran = append(ran, strings.Join(args, " "))
switch args[0] {
case "info":
@@ -78,6 +75,69 @@ func TestAContainerJoinsTheNetworksItKeeps(t *testing.T) {
var specOfLast string
// A module the mesh left out of a declaration is not a module the mesh removed (novox/hq ADR 0163,
// rule 6): what the host wrote for it stays, recorded and said; what it holds for it stays held.
// A module simply absent is removed as it always was.
func TestALeftOutModuleIsNeitherRemovedNorForgotten(t *testing.T) {
var removed []string
gone := map[string]bool{}
run := func(_ context.Context, name string, args ...string) (string, error) {
args = oldShape(name, args)
if name != "docker" {
return "", nil
}
switch args[0] {
case "info":
return "29.0.0\n", nil
case "rm":
removed = append(removed, args[len(args)-1])
gone[args[len(args)-1]] = true
case "container":
if gone[args[len(args)-1]] {
return "", errors.New("no such container")
}
return "true\tspec", nil
}
return "", nil
}
known := store.State{
Resources: []store.Applied{
{ID: "web.server", Type: "container", Target: "web", Origin: store.OriginDeclared},
{ID: "old.server", Type: "container", Target: "old", Origin: store.OriginDeclared},
},
Held: []store.Held{{ID: "web.page", Module: "web", Kind: "file", Target: "/srv/web/index.html"}},
}
d := parse(t, `{"declaration":1,"left_out":["web"],"resources":[
{"id":"notes.conf","type":"file","path":"`+t.TempDir()+`/notes.conf","content":"x"}
]}`)
report, state, err := Apply(context.Background(), archHost(t), d, known, store.OriginDeclared, run, nil, nil)
if err != nil {
t.Fatal(err)
}
if len(removed) != 1 || removed[0] != "old" {
t.Fatalf("removed %v; only the module that is absent goes", removed)
}
if _, kept := state.At("container", "web"); !kept {
t.Fatal("the left-out module's record was forgotten")
}
if _, held := state.HeldAt("web.page"); !held {
t.Fatal("the left-out module's hold was released")
}
said := false
for _, o := range report.Outcomes {
if o.ID == "web.server" && o.Action == "unchanged" && strings.Contains(o.Detail, "web was left out of this declaration by the mesh") {
said = true
}
if o.ID == "web.server" && o.Action != "unchanged" {
t.Errorf("the left-out module's container was %s", o.Action)
}
}
if !said {
t.Fatalf("keeping the left-out module's container was not said: %+v", report.Outcomes)
}
}
// A container's capabilities reach the runtime and are part of its spec (novox/hq ADR 0170).
func TestACapabilityReachesTheRuntimeAndTheSpec(t *testing.T) {
var ran []string
run := func(_ context.Context, name string, args ...string) (string, error) {
internal/apply/library.go
+7 -13
View File
@@ -11,7 +11,7 @@ import (
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
"github.com/novox/mesh-host/internal/system"
)
@@ -318,17 +318,6 @@ func heldRoots(verdicts map[string]libraryVerdict, libs []declaration.Library) m
return out
}
// heldLibraries is the root of every library whose disk or mount is not there, with its name, judged now and read
// only: what a full apply holds its containers for (ADR 0328 §9), for one that judges without applying libraries.
func heldLibraries(ctx context.Context, libs []declaration.Library) map[string]string {
verdicts := map[string]libraryVerdict{}
for _, l := range libs {
rec, _ := libraryRecord(l.Library)
verdicts[l.Library] = assessLibrary(ctx, l, rec)
}
return heldRoots(verdicts, libs)
}
// mountsHeld is the library a container waits for: one whose root it mounts, or mounts below, is held.
func mountsHeld(c *declaration.Container, held map[string]string) string {
for _, v := range c.Volumes {
@@ -352,7 +341,12 @@ func WaitingOnLibrary(d *declaration.Declaration) map[string]bool {
if d == nil || len(d.Libraries) == 0 {
return out
}
held := heldLibraries(context.Background(), d.Libraries)
verdicts := map[string]libraryVerdict{}
for _, l := range d.Libraries {
rec, _ := libraryRecord(l.Library)
verdicts[l.Library] = assessLibrary(context.Background(), l, rec)
}
held := heldRoots(verdicts, d.Libraries)
for _, r := range d.Resources {
if c, ok := r.(*declaration.Container); ok && mountsHeld(c, held) != "" {
out[c.ID] = true
internal/apply/library_linux.go
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"fmt"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/library_test.go
+1 -27
View File
@@ -11,36 +11,18 @@ import (
"path/filepath"
"strconv"
"strings"
"syscall"
"testing"
"time"
"golang.org/x/sys/unix"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
// The engine's half of ADR 0328's checks (novox/hq ADR 0328, "How it is checked", issue 567). The engine runs as
// root; a test may not, and a non-root test gives its folders a group it is in itself.
// canSetGroups skips aloud a test that gives a folder a group when this step cannot. **A step that cannot set a folder's
// group judges nothing here** (issue 1919, R4g): the checker runs the tests in the build module's isolation, a user
// namespace that maps no group but the step's own, so a chown to any other group is refused (EINVAL or EPERM). A test that
// sets a group calls it first: the probe gives a file of the test's own that group, and when it is refused the test skips
// aloud; outside such an isolation it runs and is judged.
func canSetGroups(t *testing.T, gid int) {
t.Helper()
probe := filepath.Join(t.TempDir(), "probe")
if err := os.WriteFile(probe, nil, 0o600); err != nil {
t.Fatal(err)
}
if err := os.Chown(probe, -1, gid); errors.Is(err, syscall.EINVAL) || errors.Is(err, syscall.EPERM) {
t.Skipf("NOT JUDGED: this step cannot set a folder's group (a user namespace maps only its own account, as the "+
"checker's isolated step does): %v; issue 1919", err)
}
}
// testGID is a group the test may give its folders: any number as root, else one of its own added groups that is
// no root group's.
func testGID(t *testing.T) int {
@@ -139,7 +121,6 @@ func statOf(t *testing.T, path string) unix.Stat_t {
// those folders alone, never what is in them; owner, owner's and others' bits unchanged.
func TestALibraryIsKeptInItsGroup(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o755); err != nil {
@@ -189,7 +170,6 @@ func TestALibraryIsKeptInItsGroup(t *testing.T) {
// Row 9: a missing sub-path is made with the root's owner, the library's group, group rwx and setgid.
func TestAMissingSubPathIsMadeAsTheRootIs(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o751); err != nil {
@@ -246,7 +226,6 @@ func TestARootThatHoldsDataWaitsForTheOperatorsWord(t *testing.T) {
// (the act's ask, internal/link); an undeclared library refused; a walk over its bound goes on from its cursor.
func TestTheTakeActSetsTheGroupUnderTheRoot(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
base := t.TempDir()
root := filepath.Join(base, "media")
@@ -533,7 +512,6 @@ func TestNoLibraryGroupIsRoot(t *testing.T) {
// A library's filesystem that cannot be named this time is unknown, never a move; another filesystem at the root is.
func TestALibraryNotNamedIsUnknownAndAnotherFilesystemIsAMove(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
base := t.TempDir()
pool := filepath.Join(base, "pool")
root := filepath.Join(pool, "media")
@@ -672,7 +650,6 @@ func TestALibraryIsNeverTheMachinesOwnTree(t *testing.T) {
// and sample all in shape are kept; once taken, never asked again on that filesystem.
func TestARootInShapeWithDataBelowNotInTheGroupWaits(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
movies := filepath.Join(root, "movies")
@@ -722,7 +699,6 @@ func TestARootInShapeWithDataBelowNotInTheGroupWaits(t *testing.T) {
// file never keeps its setgid bit once regrouped.
func TestRootsLinkedFilesAreLeftAndSetgidIsNeverKept(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
f, err := os.CreateTemp(t.TempDir(), "file")
if err != nil {
t.Fatal(err)
@@ -753,7 +729,6 @@ func TestRootsLinkedFilesAreLeftAndSetgidIsNeverKept(t *testing.T) {
// waits for the operator's word as the root would, and nothing of it is changed; an empty one is kept.
func TestASubPathHoldingOldDataWaitsForTheOperatorsWord(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.Mkdir(root, 0o755); err != nil {
@@ -820,7 +795,6 @@ func TestAnApplyDoesNotWaitForATakeWalk(t *testing.T) {
// the root alone, says whether the data is the library's already (§7).
func TestTheSampleBeneathARootInShapeIsJudged(t *testing.T) {
gid := testGID(t)
canSetGroups(t, gid)
libraries(t, nil, nil)
root := filepath.Join(t.TempDir(), "media")
if err := os.MkdirAll(filepath.Join(root, "old"), 0o755); err != nil {
internal/apply/logging_test.go
+1 -1
View File
@@ -5,7 +5,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/maintenance_window_test.go
+1 -2
View File
@@ -11,7 +11,7 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
@@ -270,7 +270,6 @@ func newRegistryMachine(spec string) *aRegistryMachine {
}
func (m *aRegistryMachine) run(_ context.Context, _ string, args ...string) (string, error) {
args = asideShape(args)
m.mu.Lock()
switch args[0] {
case "info", "image":
internal/apply/mesh_dirs.go
+2 -8
View File
@@ -30,7 +30,7 @@ import (
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
// LinkInPathError is a mesh directory refused because a link in its path is one another account could replace.
@@ -94,14 +94,13 @@ func groupNamed(gid uint32) string {
func JudgeDirectories(d *declaration.Declaration) string {
var lines []string
var placed, refused, guarded, homed int
declared := declaredHomes(d.Resources)
for _, r := range d.Resources {
dir, ok := r.(*declaration.Directory)
if !ok {
continue
}
path := filepath.Clean(dir.Path)
if err := refusePathWith(path, placingDirectory, moduleOfID(dir.ID), dir.Owner, declared); err != nil {
if err := refusePath(path, placingDirectory, moduleOfID(dir.ID), dir.Owner); err != nil {
guarded++
lines = append(lines, path+": refused already by the placement guard: "+err.Error())
continue
@@ -124,8 +123,3 @@ func JudgeDirectories(d *declaration.Declaration) string {
placed+refused+guarded+homed, placed, refused, guarded, homed)
return strings.Join(append([]string{head}, lines...), "\n")
}
// meshDataRoot is the mesh's own tree, where a module placed "mesh" keeps its directory (<data root>/mesh/<module>):
// the engine's, 0755, so only the engine's account makes a directory in it (novox/hq issue 1873). A variable so a
// test can place it.
var meshDataRoot = "/var/lib/mesh"
internal/apply/mesh_dirs_apply_linux.go
+1 -1
View File
@@ -14,7 +14,7 @@ import (
"golang.org/x/sys/unix"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
// meshDirsByDescriptor says this system can open a directory component by component without following a link.
internal/apply/mesh_dirs_test.go
+1 -1
View File
@@ -8,7 +8,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/mesh_names_test.go
+1 -1
View File
@@ -3,7 +3,7 @@ package apply
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
)
// **A container's mesh names are part of what it is** (novox/hq 04-ISSUES/135).
internal/apply/mesh_tree_found_test.go
-100
View File
@@ -1,100 +0,0 @@
//go:build linux
package apply
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/store"
)
// novox/hq issue 1873: a module's directory in the mesh's own tree (<data root>/mesh/<module>, place "mesh") that is
// there before its declaration, owned by the engine's own account, was made by the mesh on the way to something below
// it: it is the mesh's, given its declared mode and owner, never used as found. Only the engine's account makes a
// directory in a tree that is the engine's (0755, root); on ace and novox /var/lib/mesh/postgres was there as
// root:root 0700 before postgres's data directory was applied, and the engine left it to root, so the postgres
// provider, running as the operator's account, could not reach its own files.
func TestADirectoryInTheMeshsOwnTreeMadeByTheEngineIsTheMeshs(t *testing.T) {
root := t.TempDir()
before := meshDataRoot
meshDataRoot = filepath.Join(root, "mesh")
t.Cleanup(func() { meshDataRoot = before })
made := filepath.Join(meshDataRoot, "postgres")
if err := os.MkdirAll(made, 0o700); err != nil {
t.Fatal(err)
}
if err := os.Chmod(made, 0o700); err != nil {
t.Fatal(err)
}
decl := `{"id":"postgres.data","type":"directory","path":"` + made + `","mode":"0750"}`
var said []string
_, state, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, func(line string) { said = append(said, line) }, nil)
if err != nil {
t.Fatal(err)
}
if strings.Contains(strings.Join(said, "\n"), "WARNING postgres.data") {
t.Errorf("a directory the engine's account made in the mesh's tree was used as found: %q", said)
}
if a, _ := state.Find("postgres.data"); a.AsFound {
t.Error("it is recorded as found")
}
if info, _ := os.Stat(made); info.Mode().Perm() != 0o750 {
t.Errorf("it was not given its declared mode: %04o", info.Mode().Perm())
}
// Outside the mesh's tree the rule of issue 339 stands: found, used as found.
other := filepath.Join(root, "var-lib", "notes")
if err := os.MkdirAll(other, 0o700); err != nil {
t.Fatal(err)
}
said = nil
decl = `{"id":"notes.data","type":"directory","path":"` + other + `","mode":"0750"}`
if _, _, err := Apply(context.Background(), archHost(t), declare(t, decl), store.State{}, store.OriginDeclared,
noServices, func(line string) { said = append(said, line) }, nil); err != nil {
t.Fatal(err)
}
if !strings.Contains(strings.Join(said, "\n"), "WARNING notes.data") {
t.Errorf("a directory found outside the mesh's tree was not used as found: %q", said)
}
}
// Review of issue 1873's fix: the tree is judged by where the path leads, not by its text. A root-only link inside the
// mesh's tree pointing out of it (to /var/lib/systemd, say) would otherwise make a directory outside the tree, owned by
// the engine's account, the mesh's to chown: issue 1873's fault again. It is used as found.
func TestALinkInTheMeshsTreeLeadingOutOfItIsNotTheTree(t *testing.T) {
rootAlone(t, "")
root := t.TempDir()
before := meshDataRoot
meshDataRoot = filepath.Join(root, "mesh")
t.Cleanup(func() { meshDataRoot = before })
outside := filepath.Join(root, "var-lib", "systemd")
if err := os.MkdirAll(filepath.Join(outside, "data"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.Chmod(filepath.Join(outside, "data"), 0o700); err != nil {
t.Fatal(err)
}
if err := os.MkdirAll(meshDataRoot, 0o755); err != nil {
t.Fatal(err)
}
if err := os.Symlink(outside, filepath.Join(meshDataRoot, "postgres")); err != nil {
t.Fatal(err)
}
r := &declaration.Directory{ID: "postgres.data", Path: filepath.Join(meshDataRoot, "postgres", "data"), Mode: "0750"}
out, err := applyDirectory(r, store.Applied{}, true)
if err != nil {
t.Fatal(err)
}
if !out.asFound {
t.Errorf("a directory reached through a link out of the mesh's tree was taken as the mesh's: %+v", out)
}
if got := permOf(t, filepath.Join(outside, "data")); got != 0o700 {
t.Errorf("the directory outside the tree was given mode %o (was 700)", got)
}
}
internal/apply/mode.go
+1 -1
View File
@@ -3,7 +3,7 @@ package apply
import (
"fmt"
"git.novox.be/novox/mesh-sdk/go/declaration"
"github.com/novox/mesh-host/internal/declaration"
"github.com/novox/mesh-host/internal/store"
)
internal/apply/module_results.go
-99
View File
@@ -1,99 +0,0 @@
package apply
import (
"errors"
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
// failuresOf is every failure of an apply as the error carries it: the first among them, then the rest, so a module
// whose resource failed second is still named (issue 1132, and issue 500, which counted the rest and named none).
func failuresOf(failures []*Error) []Failure {
out := make([]Failure, 0, len(failures))
for _, f := range failures {
out = append(out, Failure{Resource: f.Resource, Err: f.Err})
}
return out
}
// ModuleResults groups an apply's outcome per module the declaration names (novox/hq ADR 0429 §4, issue 1132): the
// module's resources all applied, or it failed and Detail says which resource failed with its error, which resources
// were skipped behind a step that did not complete, or that its resources were not attempted because a step of no
// module stopped the apply. A module is named by what the declaration says of each resource (ModuleOfID), so one
// whose id would guess another is still its own.
//
// A failure that is no resource's (an error that is not an Error, or a resource of no module) belongs to no module and
// changes none: the node says it in its report. The report is the outcome of an apply that did not fail; an apply that
// failed carries its own account in the Error, and that is read in its place.
func ModuleResults(d *declaration.Declaration, report Report, err error) map[string]ModuleResult {
if d == nil {
return nil
}
out := map[string]ModuleResult{}
attempted := map[string]bool{}
var failed *Error
errors.As(err, &failed)
outcomes := report.Outcomes
if failed != nil {
outcomes = append(append([]Outcome(nil), outcomes...), failed.Done.Outcomes...)
}
skipped := map[string]string{}
for _, o := range outcomes {
module, ok := d.ModuleOfID(o.ID)
if !ok {
continue
}
if o.Action == "skipped" {
skipped[module] = o.Detail
continue
}
attempted[o.ID] = true
}
var reasons = map[string][]string{}
for _, r := range d.Resources {
if module, ok := d.ModuleOfID(r.Identity()); ok {
out[module] = ModuleResult{}
if failed != nil && failed.Stopped && !attempted[r.Identity()] {
reasons[module] = append(reasons[module], fmt.Sprintf("%s was not attempted: %s stopped the apply", r.Identity(), failed.Resource))
}
}
}
if failed != nil {
for _, f := range failed.Failures {
if module, ok := d.ModuleOfID(f.Resource); ok {
if _, named := out[module]; named {
reasons[module] = append(reasons[module], fmt.Sprintf("%s failed: %v", f.Resource, f.Err))
}
}
}
if len(failed.Failures) == 0 {
if module, ok := d.ModuleOfID(failed.Resource); ok {
if _, named := out[module]; named {
reasons[module] = append(reasons[module], fmt.Sprintf("%s failed: %v", failed.Resource, failed.Err))
}
}
}
for _, module := range failed.Modules {
if _, named := out[module]; named {
why := skipped[module]
if why == "" {
why = "a step it declares did not complete"
}
reasons[module] = append(reasons[module], "gated: "+why)
}
}
}
for module, why := range skipped {
if _, named := out[module]; named && len(reasons[module]) == 0 {
reasons[module] = append(reasons[module], "gated: "+why)
}
}
for module, why := range reasons {
sort.Strings(why)
out[module] = ModuleResult{Failed: true, Detail: strings.Join(why, "; ")}
}
return out
}
internal/apply/module_results_test.go
-76
View File
@@ -1,76 +0,0 @@
package apply
import (
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/declaration"
)
func twoModules(t *testing.T) *declaration.Declaration {
t.Helper()
d, err := declaration.Parse([]byte(`{"declaration":1,"builds":{"a":"b-1","b":"b-2"},"resources":[
{"id":"a.dir","type":"directory","path":"/srv/a","module":"a"},
{"id":"b.dir","type":"directory","path":"/srv/b","module":"b"},
{"id":"b.conf","type":"file","path":"/srv/b/c","content":"x","module":"b"},
{"id":"c.dir","type":"directory","path":"/srv/c","module":"c"}]}`))
if err != nil {
t.Fatal(err)
}
return d
}
// 1132/AC3: every resource of module A applied and one of module B failed: A is a success, B failed with the failing
// resource and its error named; a module gated by another's failure is failed too.
func TestAnApplyIsGroupedPerModule(t *testing.T) {
d := twoModules(t)
done := Report{Outcomes: []Outcome{
{ID: "a.dir", Action: "created"},
{ID: "c.dir", Action: "skipped", Detail: "a step this module declares did not complete"},
}}
err := &Error{Resource: "b.dir", Err: errors.New("no space left"), Done: done, Others: 1,
Failures: []Failure{{Resource: "b.dir", Err: errors.New("no space left")}, {Resource: "b.conf", Err: errors.New("denied")}},
Modules: []string{"c"}}
got := ModuleResults(d, Report{}, err)
if len(got) != 3 {
t.Fatalf("modules %v, want a, b and c", got)
}
if got["a"].Failed || got["a"].Detail != "" {
t.Errorf("a: %+v, want a clean success", got["a"])
}
b := got["b"]
if !b.Failed || !strings.Contains(b.Detail, "b.dir") || !strings.Contains(b.Detail, "no space left") ||
!strings.Contains(b.Detail, "b.conf") || !strings.Contains(b.Detail, "denied") {
t.Errorf("b: %+v, want both failing resources and their errors", b)
}
c := got["c"]
if !c.Failed || !strings.Contains(c.Detail, "did not complete") {
t.Errorf("c: %+v, want it failed as gated, with why", c)
}
}
// 1132/AC3: an apply that did not fail is a success for every module the declaration names.
func TestACleanApplyIsASuccessForEveryModule(t *testing.T) {
got := ModuleResults(twoModules(t), Report{}, nil)
for _, m := range []string{"a", "b", "c"} {
if r, ok := got[m]; !ok || r.Failed {
t.Errorf("%s: %+v (%v), want a success", m, r, ok)
}
}
}
// 1132/AC3: a step that belongs to no module stops the apply there; a module with a resource not attempted is not a
// success.
func TestAModuleNotAttemptedIsNotASuccess(t *testing.T) {
d := twoModules(t)
err := &Error{Resource: "boot", Err: errors.New("boom"), Stopped: true, Failures: []Failure{{Resource: "boot", Err: errors.New("boom")}},
Done: Report{Outcomes: []Outcome{{ID: "a.dir", Action: "created"}}}}
got := ModuleResults(d, Report{}, err)
if got["a"].Failed {
t.Errorf("a was done: %+v", got["a"])
}
if !got["b"].Failed || !strings.Contains(got["b"].Detail, "boot") {
t.Errorf("b was not attempted: %+v", got["b"])
}
}

Some files were not shown because too many files have changed in this diff Show More