Compare commits

..
Author SHA1 Message Date
jschoubben 5b9908bed2 Hold a joining build for a person without superseding its delivery, join only on an engine that keeps own groups, and keep a held module's settings (hq ADR 0328 §8, ADR 0334; re-review of 96e15f86)
A delivery does not send a joining build held for its library or its engine: the module waits for a person,
the plan stays open saying so and judges its other modules, and sends it once nothing holds it. A held move is
no move for the gate. A module joins only on an engine at contract 7, and a held module is composed with the
settings it last ran with. The runner's verdicts are said for their modules and judged by whether a move
changes them. mesh-host pinned at #97 (7a41e120), its bundle now examples/foundation-first-node.lock.
2026-10-13 00:55:54 +02:00
jschoubben 1de9bbc448 Own directories of a joining module carry setgid, to an engine that takes it (hq ADR 0334 v4) 2026-10-13 00:31:54 +02:00
jschoubben 96e15f8693 Hold a joining module at the build it ran, own its directories by its own group, and take a library only when its act is done (hq ADR 0328 review, ADR 0334)
A held module is composed exactly as before it joined, at the build its machine runs, so its port, resources and
contributions stay. A joining module's directories belong to its user and its own group, and its tools' directories
are refused closed to that group; the runner joins it where the engine can say so. A take records the root and is
taken only when the engine states the act done. Libraries' paths and verdicts count modules pulled in by a
requirement, and a unit takes the identity.
2026-10-13 00:03:18 +02:00
jschoubben 7ea6d36eab Pin the node-engine at its ADR 0328 merge (#95), and speak its contract: a library's verdict is said for the modules that use it 2026-10-12 23:02:31 +02:00
jschoubben f634e33792 Libraries and the modules that join them, on the controller's side (hq ADR 0328) 2026-10-12 22:29:35 +02:00
jschoubben fa0a5f1c88 Carry over the ADR 0325 controller work as the base for ADR 0328 2026-10-12 21:57:29 +02:00
963 changed files with 22383 additions and 95480 deletions
+1 -3
View File
@@ -27,11 +27,9 @@ WORKDIR /src
COPY go.mod go.sum ./
COPY vendor/ vendor/
# `COPY . .` brings `.git` along, so Go is told not to stamp the commit: the version is said by -X, and a
# rebuild of an unchanged program is the same bytes (novox/hq ADR 0267 rule 6, issue 363).
COPY . .
ARG VERSION=development
RUN CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPROXY=off go build -trimpath -buildvcs=false \
RUN CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPROXY=off go build -trimpath \
-ldflags "-s -w -X main.version=${VERSION}" \
-o /mesh-controller ./cmd/mesh-controller
+3 -7
View File
@@ -95,7 +95,7 @@ PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build --build-arg GO_BASE=$(GO_BASE) -f modules/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -110,18 +110,14 @@ proxy-image:
# after formatting never ran through it. The script is the one list of steps; this target only gives it
# a database (MESH_TEST_POSTGRES, exported above).
#
# **Two levels** (issue 895): the unit level, `go test ./...` untagged, timed and held to a bound; and the integration
# level, `-tags integration`, the tests that start a bus, a store or another process. Each is said with its time.
#
# **Packages in parallel, under the race detector, each test on a bus of its own** (internal/testbus),
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# The unit level alone (untagged: no bus, no store, no other process), which a developer runs continuously. The tests
# that need a bus or the store carry `//go:build integration` and run in `check`, not here; without a database the
# store's tests would skip rather than fail anyway.
# Without a database the store's tests skip rather than fail, so this is the honest subset and not
# the gate.
test:
go test -timeout 15m ./...
-6
View File
@@ -1,6 +0,0 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
-29
View File
@@ -1,29 +0,0 @@
#!/bin/sh
# The unit tests of every Go module of its own below this repository's root, run from the root, for merge-check.sh
# and for the test that tries it on a tree (merge_check_nested_test.go). Its own file so that the test needs no
# particular merge-check.sh: the check that judges a change to merge-check.sh is main's.
set -eu
cgo="${cgo:-0}"
race="${race:-}"
# nested_modules vets and tests every Go module of its own below the root, `modules/<name>/go.mod` (novox/hq ADR 0420
# §5, issue 1025): `./...` stops at a nested go.mod, so a module that lives whole in this repository would have its
# unit tests run by nobody. None is a pass. Each runs in its own directory, outside the vendor directory this
# repository's own packages are built from; a nested module vendors what the forge holds (its own `vendor/`, so the
# check runs offline), as this repository's root module does (issue 1019). A failing module fails the check; the others are still run.
nested_modules() {
nested=0
for gomod in modules/*/go.mod; do
[ -e "$gomod" ] || continue
dir=$(dirname "$gomod")
echo "nested module $dir:"
if ! (cd "$dir" && GOFLAGS= CGO_ENABLED=0 go vet ./... &&
GOFLAGS= CGO_ENABLED="${cgo:-0}" go test ${race:-} -count=1 -timeout 30m ./...); then
echo "nested module $dir FAILED"
nested=1
fi
done
return $nested
}
nested_modules
+1 -1
View File
@@ -11,7 +11,7 @@ WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -buildvcs=false -ldflags '-s -w' -o /mesh-builder ./cmd/mesh-builder
RUN CGO_ENABLED=0 go build -trimpath -ldflags '-s -w' -o /mesh-builder ./cmd/mesh-builder
FROM ${ALPINE_BASE}
# git to clone what it is asked to build, and the docker client to build and push it. The daemon
-146
View File
@@ -1,146 +0,0 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// **The build agent says whether it is busy, and being asked drains it** (novox/hq ADR 0335 §1, §5). Its
// container declares `busy` as `mesh-builder busy`: the node-engine runs it inside the container before it
// replaces it. It exits 0 while a build is in hand and 1 when none is; and from the first ask the agent takes
// no new build request, so its answer ends once the build in hand ends — a replaced builder finishes the check
// it runs, instead of the build queue handing it out again from the start (issue 594).
//
// Two marks in the container's own temporary directory, as the ready mark (ready.go): a new container starts
// without them, and a restarted one removes them before it takes work again.
//
// - in hand: written by the agent when it claims a request, before it reads whether it is drained; removed
// when the build ends.
// - drained: written by the ask, before it reads whether a build is in hand.
//
// Each side writes its own mark before it reads the other's, so an ask racing a fetch sees the build in hand,
// or the agent sees the drain and hands the request back; never neither.
func inHandMark() string { return filepath.Join(os.TempDir(), "mesh-builder.in-hand") }
func drainMark() string { return filepath.Join(os.TempDir(), "mesh-builder.drained") }
// The drain lapses when no ask has come for drainLapse: the node-engine asks every 20 seconds for as long as it
// really waits, so an agent whose replacement was withdrawn — the declaration put back, the walk stopped —
// takes work again rather than staying drained for ever (review of ADR 0335). And an agent drained for longer
// than the node-engine's bound and the slack says it is not ready: the replacement it was drained for never
// came, which is a fault to see.
const (
drainLapse = 2 * time.Minute
drainTooLong = 15*time.Minute + 5*time.Minute
)
// drainedAt is when the drain began and when it was last asked, from its mark; false when there is no mark.
func drainedAt() (began, asked time.Time, ok bool) {
info, err := os.Stat(drainMark())
if err != nil {
return time.Time{}, time.Time{}, false
}
asked = info.ModTime()
began = asked
if raw, err := os.ReadFile(drainMark()); err == nil {
if t, err := time.Parse(time.RFC3339, strings.TrimSpace(string(raw))); err == nil && !t.After(asked) {
began = t
}
}
return began, asked, true
}
// drained says the agent was asked whether it is busy within drainLapse: it takes no new build request.
func drained() bool { return drainedSince(time.Now()) }
func drainedSince(now time.Time) bool {
_, asked, ok := drainedAt()
return ok && now.Sub(asked) < drainLapse
}
// drainedTooLong is the health check's second half: nil unless the agent has been drained, asked all along, for
// longer than its node-engine's bound and the slack.
func drainedTooLong(now time.Time) error {
began, asked, ok := drainedAt()
if !ok || now.Sub(asked) >= drainLapse || now.Sub(began) <= drainTooLong {
return nil
}
return fmt.Errorf("drained since %s and still asked: the replacement its node-engine waits for has not come "+
"within %s (novox/hq ADR 0335)", began.UTC().Format(time.RFC3339), drainTooLong)
}
// claim marks a request in hand, unless the agent is drained: then nothing is in hand, and the request goes
// back to the queue.
func claim() bool {
// Made, never written over: one build at a time, and a mark already there is a build in hand.
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
// Unmarked, an ask would say idle while this build runs: the build is not taken.
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
_, err = f.WriteString(time.Now().UTC().Format(time.RFC3339) + "\n")
if closeErr := f.Close(); err == nil {
err = closeErr
}
if err != nil {
_ = os.Remove(inHandMark())
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
if drained() {
_ = os.Remove(inHandMark())
fmt.Fprintln(os.Stderr, "asked whether busy, so this agent takes no new build: handed back")
return false
}
return true
}
// holding says what is in hand, for the ask to print: written once the request is read.
func holding(what string) {
_ = os.WriteFile(inHandMark(), []byte(time.Now().UTC().Format(time.RFC3339)+" "+what+"\n"), 0o644)
}
// released ends the build in hand.
func released() {
if err := os.Remove(inHandMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(os.Stderr, "cannot remove the mark of the build in hand (%s): %v\n", inHandMark(), err)
}
}
// undrained clears both marks at the start of a holder: a container started again is asked afresh.
func undrained() error {
for _, mark := range []string{drainMark(), inHandMark()} {
if err := os.Remove(mark); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove %s: %w", mark, err)
}
}
return nil
}
// busy is the ask: it drains the agent, then says whether a build is in hand. Nil — exit 0 — while one is.
func busy() (string, error) {
// The mark keeps when the drain began while it is asked within drainLapse, and is written again on every
// ask: its time is the last ask.
began := time.Now().UTC()
if was, _, ok := drainedAt(); ok && drained() {
began = was.UTC()
}
if err := os.WriteFile(drainMark(), []byte(began.Format(time.RFC3339)+"\n"), 0o644); err != nil {
// Not drained, it could take the next build: said, and answered busy only for what is in hand now.
fmt.Fprintf(os.Stderr, "cannot mark this agent drained (%s): %v\n", drainMark(), err)
}
held, err := os.ReadFile(inHandMark())
if err != nil {
return "", errors.New("idle: no build in hand, and this agent takes no new one")
}
since, what, _ := strings.Cut(strings.TrimSpace(string(held)), " ")
if what == "" {
what = "a build"
}
return fmt.Sprintf("busy: %s, in hand since %s; taking no new build", what, since), nil
}
-162
View File
@@ -1,162 +0,0 @@
package main
import (
"os"
"strings"
"sync"
"testing"
"time"
)
// `mesh-builder busy` answers 0 while a build is in hand and non-zero when idle, and after its first ask the
// agent takes no new build request (novox/hq ADR 0335 §5).
func TestBusyAnswersWhileABuildIsInHandAndDrainsTheAgent(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("an agent never asked is drained")
}
// A build is claimed and runs.
if !claim() {
t.Fatal("an agent never asked refused a build")
}
holding("novox/mesh-controller (check-280)")
said, err := busy()
if err != nil || !strings.Contains(said, "novox/mesh-controller (check-280)") {
t.Fatalf("an agent with a build in hand did not answer busy with what it holds: %q %v", said, err)
}
// From that ask on it takes nothing new: the fetch is not made, and one fetched is handed back.
if !drained() {
t.Fatal("the ask did not drain the agent")
}
if claim() {
t.Fatal("a drained agent took a new build")
}
// The build in hand is still in hand after the refused claim, and busy goes on saying so.
if _, err := busy(); err != nil {
t.Fatalf("a refused claim ended the build in hand: %v", err)
}
// It ends: idle, and still drained.
released()
if _, err := busy(); err == nil {
t.Fatal("an agent with nothing in hand answered busy")
}
if claim() {
t.Fatal("a drained agent took a build after its last one ended")
}
// Started again, it is asked afresh.
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() || !claim() {
t.Fatal("a holder started again is still drained")
}
released()
}
// An ask racing a fetch sees the build in hand, or the agent sees the drain and hands the request back: never
// neither, which would replace the container under a build just taken.
func TestAnAskRacingAFetchNeverMissesTheBuild(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
for i := 0; i < 200; i++ {
if err := undrained(); err != nil {
t.Fatal(err)
}
var claimed bool
var askErr error
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); claimed = claim() }()
go func() { defer wg.Done(); _, askErr = busy() }()
wg.Wait()
if claimed && askErr != nil {
t.Fatalf("round %d: the agent took a build and the ask said idle", i)
}
released()
}
}
// A drain no longer asked lapses: an agent whose replacement was withdrawn after the first ask (a put-back, a
// revert, a stopped walk) takes work again after drainLapse, rather than staying drained with its health green.
func TestADrainNoLongerAskedLapses(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if !drained() || claim() {
t.Fatal("an agent just asked takes work")
}
// The last ask was three minutes ago.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("a drain no longer asked for three minutes still holds")
}
if !claim() {
t.Fatal("an agent whose drain lapsed takes no work")
}
released()
}
// An agent drained, and asked all along, for longer than its node-engine's bound and the slack is not ready: the
// replacement it was drained for never came. One asked again after a lapse starts a new drain.
func TestAnAgentDrainedTooLongIsNotReady(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
taking("mesh-build-machine")
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
now := time.Now()
if err := drainedTooLong(now); err != nil {
t.Fatalf("an agent drained a moment ago is not ready: %v", err)
}
// The drain began 25 minutes ago, and the node-engine asked a moment ago.
began := now.Add(-25 * time.Minute).UTC().Format(time.RFC3339)
if err := os.WriteFile(drainMark(), []byte(began+"\n"), 0o644); err != nil {
t.Fatal(err)
}
err := drainedTooLong(time.Now())
if err == nil || !strings.Contains(err.Error(), "has not come") {
t.Fatalf("an agent drained for 25 minutes reads ready: %v", err)
}
// Each ask keeps when the drain began.
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if err := drainedTooLong(time.Now()); err == nil {
t.Fatal("an ask started the drain again")
}
// The asks stopped: the drain lapsed, and so did the fault.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if err := drainedTooLong(time.Now()); err != nil {
t.Fatalf("a lapsed drain still reads not ready: %v", err)
}
}
// An ask this build does not know exits non-zero: an older image asked `busy` reads idle, not busy for the
// node-engine's whole bound.
func TestAnUnknownCommandFails(t *testing.T) {
was := os.Args
t.Cleanup(func() { os.Args = was })
os.Args = []string{"mesh-builder", "some-later-word"}
if err := run(); err == nil {
t.Fatal("an unknown command exited 0")
}
os.Args = []string{"mesh-builder", "help"}
if err := run(); err != nil {
t.Fatalf("help failed: %v", err)
}
}
-38
View File
@@ -1,38 +0,0 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
)
// The build seat hands the check the judge cache the build agent declares (MESH_JUDGE_CACHE), never a path of its own.
func TestTheBuildSeatHandsTheCheckTheJudgeCacheItDeclares(t *testing.T) {
t.Setenv("MESH_JUDGE_CACHE", "/var/lib/mesh/build-agent/judge-cache")
if got := judgeCache(); got != "/var/lib/mesh/build-agent/judge-cache" {
t.Fatalf("the judge cache the manifest declares is read as %q", got)
}
t.Setenv("MESH_JUDGE_CACHE", "")
if got := judgeCache(); got != "" {
t.Fatalf("with no judge cache declared, %q is used", got)
}
}
// The siblings a check is asked to read reach the build seat's spec and, as read, the outcome (novox/hq ADR 0382).
func TestTheSiblingsAnAskNamesReachTheSpecAndTheOutcome(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main",
Siblings: []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", Dir: "mesh-controller"}},
SiblingsUnknown: "why"}})
if len(spec.Siblings) != 1 || spec.Siblings[0] != (builder.Sibling{Repository: "novox/mesh-controller", Number: 346,
Head: "a", Base: "main", Dir: "mesh-controller"}) || spec.SiblingsUnknown != "why" {
t.Fatalf("the spec reads %+v", spec)
}
out := siblingsOut([]builder.JudgedSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", BaseCommit: "b"}})
if len(out) != 1 || out[0].BaseCommit != "b" || out[0].Number != 346 {
t.Fatalf("the outcome reads %+v", out)
}
if plain := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "o", Repo: "r"}}); plain.Siblings != nil {
t.Fatalf("a request that names none has siblings %+v", plain.Siblings)
}
}
-31
View File
@@ -41,9 +41,6 @@ type holder struct {
say func(link.HolderState) error
// remove runs what a kill needs outside the build: the containers left behind.
remove builder.Runner
// scratch is where check builds are kept (novox/hq ADR 0350 §5): what the week's sweep lets go of. Nil where
// this machine publishes to no artifact store.
scratch builder.Scratch
mu sync.Mutex
paused bool
@@ -134,34 +131,6 @@ func (h *holder) stateWhilePaused(ctx context.Context, every time.Duration) {
}
}
// expireChecks lets go of every check build older than a week (novox/hq ADR 0350 §5), at start and every while
// after. Every holder sweeps the one store; what one let go of, another finds gone.
func (h *holder) expireChecks(ctx context.Context, every time.Duration, now func() time.Time) {
sweep := func() {
if h.scratch == nil {
return
}
n, err := h.scratch.Expire(ctx, builder.CheckBuildsKept, now())
switch {
case err != nil:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d let go of, and not all: %v\n", builder.CheckBuildsKept, n, err)
case n > 0:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d check(s)' builds let go of\n", builder.CheckBuildsKept, n)
}
}
sweep()
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
sweep()
}
}
}
// begin records the build this machine took, with the cancel that ends it.
func (h *holder) begin(request link.BuildRequest, cancel context.CancelFunc) *running {
r := &running{request: request, started: time.Now(), cancel: cancel, done: make(chan struct{})}
+67 -372
View File
@@ -18,16 +18,12 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"net/url"
"os"
"os/signal"
"sort"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
@@ -58,8 +54,6 @@ is dialled except the broker.
MESH_BINDING a file the mesh wrote saying where the artifact store is
MESH_PACKAGE_BINDING a file the mesh wrote saying where the package registry is
MESH_NPM_TOKEN_FILE a file the mesh sealed holding the token for it
MESH_GENESIS_PACKAGE_REGISTRY the document the installer sealed at genesis, read while no
binding is present (novox/hq ADR 0336)
MESH_NPM_REGISTRY a package registry URL, when the mesh has not said (a person, the bootstrap)
MESH_NPM_TOKEN the token for it, likewise
MESH_NPM_SCOPE the scope it answers for (default: @novox)
@@ -72,17 +66,6 @@ broker to take work from or a registry to publish into:
Without --registry the artifacts stay in this machine's container runtime, named by the
digest of their own configuration. The result is printed as JSON.
mesh-builder ready
exits 0 once the agent running in this container is bound to its build work and pulls it, and
1 before: its container's health check.
mesh-builder busy
exits 0 while the agent running in this container has a build in hand, and 1 when it has none;
from the first ask the agent takes no new build: its container's busy command, asked before the
container is replaced (novox/hq ADR 0335).
`
func run() error {
@@ -93,42 +76,12 @@ func run() error {
return nil
case "build":
return buildOnce(context.Background(), os.Args[2:])
case "ready":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work, and
// once it has been drained longer than its node-engine would wait to replace it (ADR 0335).
if err := ready(); err != nil {
return err
}
return drainedTooLong(time.Now())
case "busy":
// The container's busy command (novox/hq ADR 0335): drains the agent, and exits non-zero once
// nothing is in hand.
said, err := busy()
if err != nil {
return err
}
fmt.Println(said)
return nil
case "help", "-h", "--help":
default:
fmt.Print(usage)
return nil
default:
// **An ask this build does not know fails** (novox/hq ADR 0335): an image older than `busy`, asked
// it, must read as idle — exiting 0 would hold its container for the node-engine's whole bound.
fmt.Fprint(os.Stderr, usage)
return fmt.Errorf("%q is not a command this mesh-builder knows", os.Args[1])
}
}
if err := notReady(); err != nil {
return err
}
// **The builder's own version is part of every reuse key** (novox/hq ADR 0350 §1): a builder that builds
// differently never takes a build another one made.
builder.Version = builder.SelfVersion(version)
if err := undrained(); err != nil {
return err
}
credential, err := brokerFrom()
if err != nil {
return err
@@ -181,13 +134,7 @@ func run() error {
// **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder
// recreated mid-check left the check's store and bus running, and the ask went to another machine.
h.removeLeftBehind()
// Paused or drained, it takes nothing new (novox/hq ADR 0219, ADR 0335): a drained agent finishes the build
// in hand and then says idle, and its container is replaced between builds.
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{
Paused: func() bool { return h.Paused() || drained() },
Taking: func() { taking(seat) },
Claim: claim,
})
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused})
defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
@@ -211,13 +158,8 @@ func run() error {
fmt.Fprintf(os.Stderr, "building for the mesh, publishing to %s\n", registry)
publisher := builder.Registry{Address: registry, Run: builder.Command}
// Check builds are let go of after a week (novox/hq ADR 0350 §5).
h.scratch = publisher
go h.expireChecks(ctx, time.Hour, time.Now)
return machine.Take(ctx, func(ctx context.Context, work link.Build) {
defer released()
holding(work.Request().Repository + " (" + work.Request().ID + ")")
answer(ctx, publisher, on, workspace, work, h)
})
}
@@ -274,12 +216,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// role's own events under the build's id (novox/hq ADR 0157) — so whoever asked, and anybody
// watching, reads the same lines this container's log holds, live, and after the fact from the
// stream. Said first, before anything runs, so a build that hangs is one that visibly started.
// One line at a time: a check's builds for its merge say their lines while its tests say theirs (novox/hq
// issue 716).
var saying sync.Mutex
say := func(step, message string) {
saying.Lock()
defer saying.Unlock()
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
work.Say(step, message)
if mine != nil && step != "run" && step != "output" {
@@ -291,26 +228,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if err := work.Began(ctx); err != nil {
fmt.Fprintf(os.Stderr, "cannot say a build started: %v\n", err)
}
// **A running check watches its pull request's key** (novox/hq ADR 0406 §4, issue 974): when the head or the trunk
// commit in it moves, the check is for a state nobody wants any more, and ends itself — nothing is asked of the forge
// (issue 726). A group's check, a try and a build watch nothing.
var superseded atomic.Bool
if w, ok := work.(interface {
WatchPull(context.Context, func(link.PullKey)) (func(), error)
}); ok && request.Check != nil {
stopWatching, err := w.WatchPull(building, func(k link.PullKey) {
superseded.Store(true)
say("superseded", fmt.Sprintf("the pull request's key moved to %.12s on %.12s while this checked %.12s on %.12s: stopped",
k.Head, k.Trunk, request.Ref, request.Check.Trunk))
cancel()
})
if err != nil {
fmt.Fprintf(os.Stderr, "the key of %s/%s#%d could not be watched, so its check runs on if the key moves: %v\n",
request.Check.Owner, request.Check.Repo, request.Check.Number, err)
} else {
defer stopWatching()
}
}
result := link.BuildResult{
ID: request.ID, Repository: request.Repository, Path: request.Path,
@@ -325,11 +242,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
}
say("build", what)
npmrc, err := packagesFrom(on)
if err == nil {
say("packages", saysRegistry(npmrc))
}
registryHere, isRegistry := publisher.(builder.Registry)
npmrc, err := packagesFrom()
var built builder.Result
if request.Check != nil {
// **A pull request's merge check, not a build** (novox/hq to-be 45 §9): nothing is built,
@@ -339,37 +252,17 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if r, ok := publisher.(builder.Registry); ok {
registry = r.Address
}
spec := checkSpecOf(request)
// The judge cache the build agent declares as its own data, never in the workspace: only judge builds and
// mesh-lab's replays mount it, never a check's containers, so no pull request's script can write what a judge
// is built with or kept as, or what the replays run (novox/hq ADR 0358 §6, issue 709).
spec.JudgeCache = judgeCache()
// What the merge would move, built by the check for the walk to take (novox/hq ADR 0350 §1): into the
// scratch namespace. **Never with the package registry's credential**: this is code nobody has approved
// yet, and a recipe could read the credential it is handed (issue 462). A module that resolves packages
// at build time has no reuse key anyway, so nothing is lost by refusing it here.
if isRegistry && len(spec.Builds) > 0 {
spec.BuildWith = &builder.CheckBuilding{Publish: publisher, Scratch: registryHere, Held: request.Held,
Running: request.Check.Running, Seats: request.Seats, Named: namedOf(request.Check.Named)}
}
var v builder.CheckVerdict
v, err = builder.Check(building, builder.Command, spec, workspace, registry, forgeFrom(), say)
v, err = builder.Check(building, builder.Command, checkSpecOf(request), workspace, registry, forgeFrom(), say)
if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built),
Validator: v.Validator, Siblings: siblingsOut(v.Siblings), SiblingsUnknown: v.SiblingsUnknown, JudgeFrom: v.JudgeFrom}
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk})
}
// A branch tried before its pull request reads its outcome from this log, whole (novox/hq issue 686).
if request.Check.Try {
if line, err := json.Marshal(triedOf(v)); err == nil {
say(link.StepTried, string(line))
}
}
}
} else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a
@@ -377,22 +270,9 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// after a clone that then fails at npm ci.
// Every container it starts is labelled with its id, so a kill finds what outlived the
// docker client (ADR 0219).
if request.Reuse != nil && isRegistry && !request.DryRun {
// **A walk's build takes the check's build of the same key** (novox/hq ADR 0350 §2).
reuse := builder.Reuse{Scratch: registryHere}
for _, p := range request.Reuse.Pulls {
reuse.Pulls = append(reuse.Pulls, builder.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number})
}
// Only what a verdict names is taken (ADR 0350 §2).
reuse.Named = namedOf(request.Reuse.Builds)
built, err = builder.BuildReusing(building, builder.Labelled(builder.Command, request.ID), publisher, reuse,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
} else {
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
// What it copied into the store, whatever became of the build (novox/hq ADR 0257).
result.Mirrored = built.Mirrored
@@ -402,11 +282,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if mine != nil {
killed = h.returned(mine) && err != nil
}
if superseded.Load() && err != nil {
// Ended by its own key moving on (ADR 0406 §4): said as the check's failure, as a kill is.
result.Failed = link.SupersededByKey
say("failed", link.SupersededByKey)
} else if killed {
if killed {
// **Killed by hand is the outcome, whatever the build was doing** (novox/hq ADR 0219): the
// error it ended with is the kill's consequence, not a fault of the source.
result.Failed = link.KilledByHand
@@ -432,19 +308,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
}
result.Against = built.Against
result.SourceFingerprint = built.Source
// What the walk did instead of building, or why it built (novox/hq ADR 0350 §3, §4).
result.ReuseKey = built.ReuseKey
if built.Reused != nil {
f := built.Reused.From
result.Reused = built.Reused.Words()
result.ReusedFrom = &link.ReusedFrom{Pull: link.PullRef{Owner: f.Pull.Owner, Repo: f.Pull.Repo,
Number: f.Pull.Number}, Head: f.Head, Trunk: f.Trunk, Check: f.Check, Key: f.Key}
} else if request.Reuse != nil {
result.BuiltWhy = built.Why
if result.BuiltWhy == "" {
result.BuiltWhy = "it has no artifact to build"
}
}
result.Trunk, result.OnTrunk, result.Branches = built.Trunk, built.OnTrunk, built.Branches
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
@@ -452,11 +315,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
for _, s := range built.Sources {
result.Sources = append(result.Sources, link.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths})
}
// The list of predefined keys the manifest was judged against, for the controller to publish
// (novox/hq ADR 0348 §7).
if p := built.Predefined; p != nil {
result.Predefined = &link.PredefinedList{Keys: p.Keys, Generation: p.Generation}
}
say("built", built.Manifest.Module+" from "+short(built.Commit))
}
}
@@ -496,19 +354,10 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
spec := builder.CheckSpec{ID: request.ID, Repository: request.Repository, Ref: request.Ref,
Owner: c.Owner, Repo: c.Repo, Number: c.Number, Paths: c.Paths, Beside: map[string]builder.Beside{},
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, SiblingsUnknown: c.SiblingsUnknown}
for _, s := range c.Siblings {
spec.Siblings = append(spec.Siblings, builder.Sibling{Repository: s.Repository, Number: s.Number, Head: s.Head,
Base: s.Base, Dir: s.Dir})
}
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held)}
for dir, b := range c.Beside {
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref}
}
for _, b := range c.Builds {
spec.Builds = append(spec.Builds, builder.CheckBuildSpec{Module: b.Module, Repository: b.Repository, Path: b.Path,
Ref: b.Ref, Merged: b.Merged, Tier: b.Tier, Skip: b.Skip})
}
for _, m := range c.Members {
spec.Group = append(spec.Group, builder.GroupHead{Owner: m.Owner, Repo: m.Repo, Repository: m.Repository,
Ref: m.Ref, Paths: m.Paths, Base: m.Base})
@@ -516,68 +365,6 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
return spec
}
// siblingsOut is the siblings a check read as the outcome carries them (novox/hq ADR 0382 §4).
func siblingsOut(read []builder.JudgedSibling) []link.CheckSibling {
var out []link.CheckSibling
for _, s := range read {
out = append(out, link.CheckSibling{Repository: s.Repository, Number: s.Number, Head: s.Head, Base: s.Base,
BaseCommit: s.BaseCommit})
}
return out
}
// judgeCache is the judge cache the build agent's manifest declares (MESH_JUDGE_CACHE, novox/hq ADR 0358 §6): a
// directory mounted at the same path on the host, so a judge build's mounts name what the agent sees. Empty when the
// agent declares none, and every judge is then built cold under its check's root.
func judgeCache() string { return os.Getenv("MESH_JUDGE_CACHE") }
// triedOf is a try's outcome as its log line says it (novox/hq issue 686): the repository's own check's verdict,
// or the check's where that did not run — a head that does not merge — and the lines naming what failed.
func triedOf(v builder.CheckVerdict) link.Tried {
t := link.Tried{Verdict: v.Verdict, Summary: v.Summary, Took: v.Took.Round(time.Second).String(),
Head: v.Head, Trunk: v.Trunk}
if v.Repo != nil && (v.Gate == nil || v.Gate.Verdict == "pass") {
t.Verdict, t.Summary = v.Repo.Verdict, v.Repo.Summary
}
if v.Repo != nil && v.Repo.Failed != "" {
t.Failed = strings.Split(v.Repo.Failed, "\n")
}
if t.Verdict == "" {
t.Verdict = "error"
}
return t
}
// namedOf are the builds the seat's verdicts name, as the builder believes a scratch record by them (novox/hq ADR
// 0350 §2, issue 703): each module's key with every artifact's digest.
func namedOf(builds []link.CheckedBuild) []builder.NamedBuild {
var out []builder.NamedBuild
for _, b := range builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
out = append(out, named)
}
return out
}
// checkBuiltOf is a check's builds for its merge as the outcome carries them (novox/hq ADR 0350 §1).
func checkBuiltOf(built []builder.CheckBuilt) []link.CheckBuilt {
var out []link.CheckBuilt
for _, b := range built {
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String(),
Kept: b.Kept, Same: b.Same, SameAs: b.SameAs}
for _, m := range b.Made {
c.Made = append(c.Made, link.MadeArtifact{Name: m.Name, Kind: m.Kind, Reference: m.Reference})
}
out = append(out, c)
}
return out
}
// layerOf is one layer of a check as the outcome carries it.
func layerOf(l *builder.Layer) *link.CheckLayer {
if l == nil {
@@ -587,171 +374,79 @@ func layerOf(l *builder.Layer) *link.CheckLayer {
}
// packagesFrom is where a build resolves the mesh's own published packages — the SDK above all
// (novox/hq ADR 0076, issue 053) — on this machine.
// (novox/hq ADR 0076, issue 053).
//
// **The binding wins, then the genesis document** (novox/hq ADR 0336 §3, §4). The build agent's
// npm-package-registry requirement is optional: on genesis's node, before the gitea module is assigned,
// nothing provides it, and the installer sealed the forge it raised to this agent as one document
// instead. Once the requirement binds, the binding names the endpoint the way the artifact store's
// does and a sealed token file the credential, and the genesis document is left unread. A binding
// file named and absent is a requirement not bound here, never a fault: the environment names the
// path whether or not the mesh wrote it.
//
// The environment variables remain for a builder run by a person. With none of the three the result
// is disabled and says this machine, so a build that needs the registry is refused naming it (§6),
// and a build that needs none builds anyway.
func packagesFrom(machine string) (builder.Npmrc, error) {
// Preferably from the mesh: a package-registry binding names the endpoint the way the artifact
// store's binding does, and a sealed token file the credential the way the broker's does. The
// environment variables remain for a builder run by a person, and for the bootstrap, where there is
// no registry yet — there the result is disabled and a build that needs no mesh-published dependency
// builds anyway.
func packagesFrom() (builder.Npmrc, error) {
scope := strings.TrimSpace(os.Getenv("MESH_NPM_SCOPE"))
if scope == "" {
scope = "@novox"
}
none := builder.Npmrc{Machine: machine}
registry := strings.TrimSpace(os.Getenv("MESH_NPM_REGISTRY"))
var username string
if path := strings.TrimSpace(os.Getenv("MESH_PACKAGE_BINDING")); path != "" {
raw, err := os.ReadFile(path)
switch {
case errors.Is(err, os.ErrNotExist):
// Not bound on this machine: the genesis document, or nothing.
case err != nil:
return none, fmt.Errorf("cannot read what the mesh said about the package registry: %w", err)
default:
return fromBinding(path, raw, scope, machine)
}
}
if path := strings.TrimSpace(os.Getenv("MESH_GENESIS_PACKAGE_REGISTRY")); path != "" {
raw, err := os.ReadFile(path)
switch {
case errors.Is(err, os.ErrNotExist):
// Not this mesh's genesis node, or the installer has not sealed it yet.
case err != nil:
return none, fmt.Errorf("cannot read the genesis document for the package registry: %w", err)
default:
return fromGenesis(path, raw, scope, machine)
}
}
// A builder run by a person.
registry := strings.TrimSpace(os.Getenv("MESH_NPM_REGISTRY"))
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
username := strings.TrimSpace(os.Getenv("MESH_NPM_USER"))
if registry == "" && secret == "" {
return none, nil
}
n := builder.Npmrc{Scope: scope, Registry: registry, From: builder.FromEnvironment, Machine: machine}
if username != "" {
n.Username, n.Password = username, secret
} else {
n.Token = secret
}
return n, nil
}
// fromBinding composes the registry from the npm-package-registry binding the mesh wrote and the
// credential it sealed beside it. The mesh authenticates the ordinary way — a generated password the
// provider only applies (novox/hq ADR 0048) — so with a username the secret is a password (basic
// auth); without one it is a bearer token a provider minted.
func fromBinding(path string, raw []byte, scope, machine string) (builder.Npmrc, error) {
none := builder.Npmrc{Machine: machine}
var told struct {
From string `json:"from"`
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil {
return none, fmt.Errorf("%s is not a binding: %w", path, err)
}
if told.At == "" {
return none, fmt.Errorf(
"%s says the package registry is on %q and gives no address for it", path, told.From)
}
// Composed from what the provider serves, so nothing here knows gitea's URL shape from
// another registry's: it states its port, the path its registry answers on, and the scheme.
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
port, ok := told.Serves["port"]
if !ok {
return none, fmt.Errorf("%s says nothing about which port the package registry answers on", path)
}
npmPath, ok := told.Serves["npm-path"]
if !ok {
return none, fmt.Errorf("%s says nothing about the path the package registry answers on", path)
}
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if file := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); file != "" {
raw, err := os.ReadFile(file)
if err != nil {
return none, fmt.Errorf("cannot read this builder's package-registry credential: %w", err)
return builder.Npmrc{}, fmt.Errorf("cannot read what the mesh said about the package registry: %w", err)
}
var told struct {
From string `json:"from"`
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil {
return builder.Npmrc{}, fmt.Errorf("%s is not a binding: %w", path, err)
}
if told.At == "" {
return builder.Npmrc{}, fmt.Errorf(
"%s says the package registry is on %q and gives no address for it", path, told.From)
}
// Composed from what the provider serves, so nothing here knows gitea's URL shape from
// another registry's: it states its port, the path its registry answers on, and the scheme.
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
port, ok := told.Serves["port"]
if !ok {
return builder.Npmrc{}, fmt.Errorf("%s says nothing about which port the package registry answers on", path)
}
npmPath, ok := told.Serves["npm-path"]
if !ok {
return builder.Npmrc{}, fmt.Errorf("%s says nothing about the path the package registry answers on", path)
}
registry = fmt.Sprintf("%s://%s:%v%v", scheme, told.At, port, npmPath)
username = told.As
}
// The credential the mesh sealed to this machine. The mesh authenticates the ordinary way — a
// generated password the provider only applies (novox/hq ADR 0048) — so with a username this is
// a password (basic auth); without one it is a bearer token a provider minted.
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if path := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); path != "" {
raw, err := os.ReadFile(path)
if err != nil {
return builder.Npmrc{}, fmt.Errorf("cannot read this builder's package-registry credential: %w", err)
}
secret = strings.TrimSpace(string(raw))
}
n := builder.Npmrc{Scope: scope, Registry: fmt.Sprintf("%s://%s:%v%v", scheme, told.At, port, npmPath),
From: builder.FromBinding, Machine: machine}
if told.As != "" {
n.Username, n.Password = told.As, secret
} else {
n.Token = secret
if u := strings.TrimSpace(os.Getenv("MESH_NPM_USER")); u != "" {
username = u
}
return n, nil
}
// GenesisRegistry is the genesis document (novox/hq ADR 0336 §3): what the installer sealed to the
// build agent on its node at genesis's packages step, as the `genesis-package-registry` own-secret —
// the forge it raised, the port this node gave the package registry, its npm path, and the account it
// made with that account's password. mesh-host writes it; the field names are the two repositories'
// agreement.
type GenesisRegistry struct {
Scheme string `json:"scheme"`
At string `json:"at"`
Port int `json:"port"`
NpmPath string `json:"npm-path"`
As string `json:"as"`
Password string `json:"password"`
}
// fromGenesis composes the registry from the genesis document.
func fromGenesis(path string, raw []byte, scope, machine string) (builder.Npmrc, error) {
none := builder.Npmrc{Machine: machine}
var g GenesisRegistry
if err := json.Unmarshal(raw, &g); err != nil {
return none, fmt.Errorf("%s is not the genesis document for the package registry: %w", path, err)
if registry == "" && secret == "" {
return builder.Npmrc{}, nil
}
var missing []string
for name, value := range map[string]string{"at": g.At, "npm-path": g.NpmPath, "as": g.As, "password": g.Password} {
if strings.TrimSpace(value) == "" {
missing = append(missing, name)
}
if username != "" {
return builder.Npmrc{Scope: scope, Registry: registry, Username: username, Password: secret}, nil
}
if g.Port <= 0 {
missing = append(missing, "port")
}
if len(missing) > 0 {
sort.Strings(missing)
return none, fmt.Errorf("%s, the genesis document for the package registry, says nothing of %s",
path, strings.Join(missing, ", "))
}
scheme := g.Scheme
if scheme == "" {
scheme = "http"
}
return builder.Npmrc{Scope: scope, Registry: fmt.Sprintf("%s://%s:%d%s", scheme, g.At, g.Port, g.NpmPath),
Username: g.As, Password: g.Password, From: builder.FromGenesis, Machine: machine}, nil
}
// saysRegistry is the line each build's log carries about the package registry (novox/hq ADR 0336
// §4): which of the binding and the genesis document it resolved from, or that it has neither here.
func saysRegistry(n builder.Npmrc) string {
if n.Enabled() {
return "the package registry is resolved from " + n.From
}
machine := n.Machine
if machine == "" {
machine = "this machine"
}
return fmt.Sprintf("no package registry: %s is unbound on %s and no genesis document is here; "+
"a build needing it is refused", builder.RegistryRequirement, machine)
return builder.Npmrc{Scope: scope, Registry: registry, Token: secret}, nil
}
// forgeFrom is the git credential this builder may offer a clone, composed from the same binding
+1 -6
View File
@@ -84,15 +84,10 @@ func buildOnce(ctx context.Context, args []string) error {
}
fmt.Fprintln(os.Stderr)
machine := os.Getenv("MESH_NODE")
if machine == "" {
machine, _ = os.Hostname()
}
npmrc, err := packagesFrom(machine)
npmrc, err := packagesFrom()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, " [packages] %s\n", saysRegistry(npmrc))
built, buildErr := builder.Build(ctx, builder.Command, publisher, repository, *path, *ref, where, bases, npmrc,
forgeFrom(),
func(step, message string) { fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message) })
-102
View File
@@ -1,102 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/builder"
)
// The build agent's package registry (novox/hq ADR 0336, issue 121): its npm-package-registry
// requirement is optional, so the binding may be absent, and on genesis's node the installer seals
// the forge it raised as one genesis document instead.
// registryFiles names the three paths the build agent's environment always names, writing only those
// given.
func registryFiles(t *testing.T, binding, token, genesis string) {
t.Helper()
dir := t.TempDir()
for env, content := range map[string]string{
"MESH_PACKAGE_BINDING": binding,
"MESH_NPM_TOKEN_FILE": token,
"MESH_GENESIS_PACKAGE_REGISTRY": genesis,
} {
path := filepath.Join(dir, env)
t.Setenv(env, path)
if content != "" {
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
}
}
for _, env := range []string{"MESH_NPM_REGISTRY", "MESH_NPM_TOKEN", "MESH_NPM_USER", "MESH_NPM_SCOPE"} {
t.Setenv(env, "")
}
}
const (
aBinding = `{"from":"gitea","at":"anchor.mesh","as":"anchor-agent",
"serves":{"scheme":"http","port":3000,"npm-path":"/api/packages/novox/npm/"}}`
aGenesisDocument = `{"scheme":"http","at":"127.0.0.1","port":3100,"npm-path":"/api/packages/novox/npm/",
"as":"mesh-builder","password":"genesis-secret"}`
)
func TestUntilTheRequirementBindsTheGenesisDocumentIsRead(t *testing.T) {
registryFiles(t, "", "", aGenesisDocument)
n, err := packagesFrom("anchor")
if err != nil {
t.Fatalf("an absent binding beside a genesis document was refused: %v", err)
}
if n.From != builder.FromGenesis || n.Registry != "http://127.0.0.1:3100/api/packages/novox/npm/" ||
n.Username != "mesh-builder" || n.Password != "genesis-secret" || n.Scope != "@novox" {
t.Fatalf("the genesis document was not what the registry came from: %+v", n)
}
file, err := n.File()
if err != nil || !strings.Contains(file, "//127.0.0.1:3100/api/packages/novox/npm/:username=mesh-builder") {
t.Fatalf("the .npmrc is not written from the genesis document: %v\n%s", err, file)
}
if got := saysRegistry(n); !strings.Contains(got, "genesis document") {
t.Errorf("the log does not say the genesis document: %s", got)
}
}
func TestARealBindingWinsOverTheGenesisDocument(t *testing.T) {
registryFiles(t, aBinding, "bound-secret\n", aGenesisDocument)
n, err := packagesFrom("anchor")
if err != nil {
t.Fatal(err)
}
if n.From != builder.FromBinding || n.Registry != "http://anchor.mesh:3000/api/packages/novox/npm/" ||
n.Username != "anchor-agent" || n.Password != "bound-secret" {
t.Fatalf("the binding did not win over the genesis document: %+v", n)
}
if got := saysRegistry(n); !strings.Contains(got, "binding") {
t.Errorf("the log does not say the binding: %s", got)
}
}
func TestWithNeitherTheAgentHasNoRegistryAndSaysWhere(t *testing.T) {
registryFiles(t, "", "", "")
n, err := packagesFrom("workstation")
if err != nil {
t.Fatalf("an unbound optional requirement was a fault: %v", err)
}
if n.Enabled() || n.Machine != "workstation" {
t.Fatalf("a registry came from nowhere: %+v", n)
}
got := saysRegistry(n)
for _, want := range []string{"npm-package-registry", "unbound", "workstation"} {
if !strings.Contains(got, want) {
t.Errorf("the log does not say %q: %s", want, got)
}
}
}
func TestAGenesisDocumentMissingAFieldIsRefusedByName(t *testing.T) {
registryFiles(t, "", "", `{"at":"127.0.0.1","port":3000,"npm-path":"/api/packages/novox/npm/","as":"mesh-builder"}`)
if _, err := packagesFrom("anchor"); err == nil || !strings.Contains(err.Error(), "password") {
t.Fatalf("a genesis document with no password was taken: %v", err)
}
}
-44
View File
@@ -1,44 +0,0 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
)
// **The build agent says it is ready** (novox/hq issue 544). It listens on nothing, so no http or tcp
// check can see it work; it writes a mark once it is bound to its worker and pulls build work, and its
// container's health check runs `mesh-builder ready`, which passes only while the mark is there. Without
// it the node-engine could judge the agent only alive, and every walk that moved it waited out the whole
// grace on each machine.
//
// The mark lives in the container's own temporary directory, never in the workspace the machine keeps:
// a new container starts without it, and a restarted one removes it before it binds again.
func readyMark() string { return filepath.Join(os.TempDir(), "mesh-builder.ready") }
// notReady removes the mark: a holder starting is not taking work until it says so again.
func notReady() error {
if err := os.Remove(readyMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove the mark that this agent takes build work (%s): %w", readyMark(), err)
}
return nil
}
// taking writes the mark. A mark that cannot be written is said, and the agent goes on building: its
// check then says it is not ready, which is the truth the gate should hear.
func taking(seat string) {
if err := os.WriteFile(readyMark(), []byte(seat+"\n"), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "cannot mark that this agent takes build work (%s): %v\n", readyMark(), err)
return
}
fmt.Fprintf(os.Stderr, "taking build work from %s\n", seat)
}
// ready is the health check: nil when this container's agent is bound and pulling build work.
func ready() error {
if _, err := os.Stat(readyMark()); err != nil {
return fmt.Errorf("not taking build work yet: %s is not there", readyMark())
}
return nil
}
-32
View File
@@ -1,32 +0,0 @@
package main
import (
"os"
"testing"
)
// The health check passes only between the agent saying it takes work and its next start (novox/hq
// issue 544).
func TestTheAgentIsReadyOnlyOnceItTakesWork(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := ready(); err == nil {
t.Fatal("an agent that never took work is ready")
}
taking("mesh-build-machine")
if err := ready(); err != nil {
t.Fatalf("an agent taking work is not ready: %v", err)
}
// Started again, it is not ready until it binds again.
if err := notReady(); err != nil {
t.Fatal(err)
}
if err := ready(); err == nil {
t.Fatal("an agent starting again is ready before it takes work")
}
if err := notReady(); err != nil {
t.Fatalf("removing a mark that is not there failed: %v", err)
}
if _, err := os.Stat(readyMark()); err == nil {
t.Fatal("the mark is still there")
}
}
-1
View File
@@ -16,7 +16,6 @@ func TestBuilderDiagnosticsStayOffStdout(t *testing.T) {
"version)": true, // --version
`"stopping")`: true, // the loop.s shutdown line
"usage)": true, // --help text, for a human
"said)": true, // busy: the answer its node-engine reads (novox/hq ADR 0335)
}
for _, file := range []string{"once.go", "main.go"} {
src, err := os.ReadFile(file)
-31
View File
@@ -1,31 +0,0 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// A try's outcome, as the build seat says it in the build's log (novox/hq issue 686): the repository's own
// check's verdict, how long it took, and each line naming what failed.
func TestATrysOutcomeNamesWhatFailed(t *testing.T) {
tried := triedOf(builder.CheckVerdict{Verdict: "pass", Summary: "the change touches no module", Took: 303 * time.Second,
Head: "c0ffee", Trunk: "ca3f43c", Gate: &builder.Layer{Verdict: "pass"},
Repo: &builder.Layer{Verdict: "fail", Summary: "its merge-check.sh failed: TestX",
Failed: "--- FAIL: TestX (0.01s)\n x_test.go:12: got 1, want 2"}})
if tried.Verdict != "fail" || tried.Took != "5m3s" || len(tried.Failed) != 2 || tried.Trunk != "ca3f43c" ||
tried.Summary != "its merge-check.sh failed: TestX" {
t.Fatalf("a failed try: %+v", tried)
}
// A head that does not merge fails both layers: the check's own word.
conflict := triedOf(builder.CheckVerdict{Verdict: "fail", Summary: "does not merge", Gate: &builder.Layer{Verdict: "fail"},
Repo: &builder.Layer{Verdict: "fail", Summary: "does not merge"}})
if conflict.Verdict != "fail" || conflict.Summary != "does not merge" {
t.Fatalf("a try that does not merge: %+v", conflict)
}
// No verdict is an error, never a pass.
if none := triedOf(builder.CheckVerdict{}); none.Verdict != "error" {
t.Fatalf("a try with no verdict: %+v", none)
}
}
-145
View File
@@ -1,145 +0,0 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A provision whose credential only a person outside the mesh can give (novox/hq ADR 0337, issue 164, issue 165):
// given once for its provider, at the controller's terminal, and sealed in one act to every consumer that binds the
// provision from that provider now. The value is never read back, printed or kept.
// outsideOffer is the provider module's own name when `secret accept <node> <module> <name>` names a provision that
// module offers with a credential issued outside the mesh, and "" when it names anything else (an own secret, or a
// pair with --provider).
func outsideOffer(ctx context.Context, module, name, provider string) (string, error) {
if provider != "" {
return "", nil
}
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
if m, known := shelf[module]; known && m.IssuedOutsideOf(name) {
return module, nil
}
return "", nil
}
// offeredOutsideBy is a module that offers the provision with a credential issued outside the mesh, or "".
func offeredOutsideBy(ctx context.Context, provision string) (string, error) {
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
module, _ := catalogue.OffersIssuedOutside(shelf, provision)
return module, nil
}
// consumersFrom is every consumer that binds a provision from one provider now, found by resolving every machine as
// its next send would, and the machines that could not be resolved, with why.
func consumersFrom(ctx context.Context, open *stores, provider, providerModule, provision string) (
[]inventory.PairEnd, []string, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, nil, err
}
var out []inventory.PairEnd
var unresolved []string
for _, n := range nodes {
plan, _, err := planFor(ctx, open, n.Name)
if err != nil {
unresolved = append(unresolved, fmt.Sprintf("%s: %s", n.Name, oneLine(err.Error())))
continue
}
out = append(out, bindsFrom(plan, n.Name, provider, providerModule, provision)...)
}
return out, unresolved, nil
}
// bindsFrom is every consumer on one machine whose resolution answers the provision from that provider. Pure.
func bindsFrom(plan catalogue.Resolution, machine, provider, providerModule, provision string) []inventory.PairEnd {
var out []inventory.PairEnd
for _, need := range plan.Needs {
if need.ByRecord || need.Name != provision || need.From != provider {
continue
}
if need.Module != "" && need.Module != providerModule {
continue
}
out = append(out, inventory.PairEnd{Consumer: machine, ConsumerModule: need.For, Local: need.Local})
}
sort.Slice(out, func(i, j int) bool {
return out[i].ConsumerModule+"\x00"+out[i].Local < out[j].ConsumerModule+"\x00"+out[j].Local
})
return out
}
// acceptForProvision gives a provision's credential issued outside the mesh, once for its provider (ADR 0337 §3).
func acceptForProvision(ctx context.Context, provider, providerModule, provision, value string) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
consumers, unresolved, err := consumersFrom(ctx, open, provider, providerModule, provision)
if err != nil {
return err
}
accepted, err := open.inventory.AcceptSecretForProvision(ctx, provision, provider, providerModule, value, consumers)
value = ""
if err != nil {
return err
}
fmt.Print(acceptedSaid(provider, providerModule, provision, accepted, unresolved))
if len(accepted.Sealed) == 0 {
return nil
}
if err := sendTo(ctx, open, accepted.Machines); err != nil {
return fmt.Errorf("%s is sealed and could not be sent: %w; `push %s` sends it", provision, err,
strings.Join(accepted.Machines, "`, `push "))
}
fmt.Printf(" sent %s; each consumer starts again on it\n", strings.Join(accepted.Machines, ", "))
return nil
}
// acceptedSaid is what one accept for a provider did, in words that never carry the value. Pure.
func acceptedSaid(provider, providerModule, provision string, a inventory.ProvisionAccepted, unresolved []string) string {
var b strings.Builder
if len(a.Sealed) == 0 {
fmt.Fprintf(&b, "%s from %s on %s: no consumer binds it now, so nothing was sealed; a consumer that binds "+
"later waits for it, and this line given again seals it to every consumer then\n", provision,
providerModule, provider)
} else {
var names []string
for _, c := range a.Sealed {
names = append(names, fmt.Sprintf("%s on %s%s", c.ConsumerModule, c.Consumer, asLocal(c.Local)))
}
fmt.Fprintf(&b, "%s from %s on %s is given, sealed to %s and to %s\n", provision, providerModule, provider,
strings.Join(names, ", "), provider)
fmt.Fprintf(&b, " the mesh cannot read it back, never makes one of its own for it, and will not rotate it\n")
}
for _, c := range a.Unsealable {
fmt.Fprintf(&b, " not sealed to %s on %s: no sealing key on that machine; give it again once it has one\n",
c.ConsumerModule, c.Consumer)
}
for _, u := range unresolved {
fmt.Fprintf(&b, " not known whether a consumer on %s binds it, which did not resolve; give it again once it does\n", u)
}
return b.String()
}
+3 -11
View File
@@ -26,11 +26,8 @@ import (
//
// - **The serving controller** takes it before it does anything else — before it asserts the bus's
// objects, which are the controller's to write — waiting while another holds it, and renews it.
// Another holder seen, or the key's age passed with no renewal made, is the lease lost: the gate
// closes at once and the process exits, so its service manager restarts it as a candidate (serve,
// in push.go). A renewal the bus does not answer in time is not a loss (novox/hq issue 477): the
// lease is kept, no new act starts once half the key's age passed unrenewed, and the next renewal
// made opens the gate again.
// A renewal refused or failed is the lease lost: the gate closes at once and the process exits, so
// its service manager restarts it as a candidate (serve, in push.go).
// - **A command run at a shell** — `push` in the installer, the lab, a person repairing a mesh whose
// controller is down (issue 201) — acts **under the holder's epoch** when a controller holds the
// lease: it is the same mesh's word, composed and sent under the store's hold of each machine like
@@ -67,11 +64,6 @@ type actor struct {
// the highest epoch issued, and what was said of it; zero when it was not (S12).
reset time.Time
resetSaid string
// renewEvery and poll, when set, are how often the serving controller renews the lease it holds and
// looks again at one another holds, in place of the lease's own: a test's, so two controllers hand the
// lease over in a second rather than in a renewal of five (novox/hq issue 591). A renewal is given
// the lease's own RenewEvery to answer however often it is made, so a slow bus does not lose it.
renewEvery, poll time.Duration
}
// theLease is this process's standing to the lease.
@@ -231,7 +223,7 @@ func (a *actor) serveUnderTheLease(ctx context.Context, inv *inventory.Inventory
}
say := func(format string, args ...any) { fmt.Printf(format+"\n", args...) }
l, err := lease.Open(ctx, api, broker.LeaseBucket, lease.Options{Holder: holderOf(instance),
RenewEvery: a.renewEvery, RenewWithin: lease.RenewEvery, Poll: a.poll, Floor: inv.HighestEpoch, Say: say, Health: controllerHealth, Moved: func(was, floor uint64) {
Floor: inv.HighestEpoch, Say: say, Health: controllerHealth, Moved: func(was, floor uint64) {
a.mu.Lock()
defer a.mu.Unlock()
a.reset = time.Now()
+6 -11
View File
@@ -203,9 +203,8 @@ func seatDependenciesOnAssign(ctx context.Context, open *stores, node string, mo
if _, err := catalogue.AssignRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err
}
// Two modules declaring one path or unit, or one package present and absent, is refused before anything
// is recorded (novox/hq ADR 0210, ADR 0303, 04-ISSUES/235): kept, the node would not resolve until one came
// off again. Two modules declaring one package present is not a collision (ADR 0303, issue 628).
// Two modules declaring one package, path or unit is refused before anything is recorded
// (novox/hq ADR 0210, 04-ISSUES/235): kept, the node would not resolve until one came off again.
if err := catalogue.CollisionRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err
}
@@ -391,10 +390,10 @@ func blockedElsewhere(ctx context.Context, open *stores, except string) string {
}
// issueOnAssign gives a newly assigned module its bus credential, the way `module issue` does, and
// says what it did in one line. Nothing for a module that declares no broker secret and runs no
// provisioner client; nothing for one whose user is already minted (a credential is rotated on
// purpose, never by re-assigning); and when the bus cannot be reached from here, the line names the
// verb and the push that would refuse the module until it is run — never a silent placeholder (novox/hq issue 203).
// says what it did in one line. Nothing for a module that declares no broker secret; nothing for one
// whose user is already minted (a credential is rotated on purpose, never by re-assigning); and when
// the bus cannot be reached from here, the line names the verb and the push that would refuse the
// module until it is run — never a silent placeholder (novox/hq issue 203).
func issueOnAssign(ctx context.Context, open *stores, node, module string) string {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
@@ -406,10 +405,6 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return ""
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, reads := m.OwnSecrets["broker"]; !reads {
// A module running the provisioner client alone is minted the client's account (issue 917).
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
}
if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted {
return ""
}
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
@@ -1,228 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
)
func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aStore())
if _, err := assign(ctx, open, "anchor", "distribution"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5101}}); err != nil {
t.Fatal(err)
}
// A module the mesh built, recorded by digest and path, running on the other machine.
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "app", Commit: "abc",
Made: []inventory.Artifact{{Name: "server", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "app/server@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "app", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-app",
"image": catalogue.ArtifactStoreScheme + "app/server@" + aDigest}}})
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
// The runtime's trust is the runtime's module's to write (novox/hq ADR 0222): a stand-in for it
// asks where this machine reaches the store, as the docker module does.
register(t, open, catalogue.Manifest{Module: "runtime", Version: "1",
Resources: []map[string]any{{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json",
"into": "json", "content": `{"insecure-registries": ["${seat:mesh-artifact-store:reach}"]}` + "\n"}}})
if _, err := assign(ctx, open, "laptop", "runtime"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
if err != nil || !found || node != "anchor" || port != "5101" {
t.Fatalf("the store is found on %q:%q (%v, %v); the node put it on 5101", node, port, found, err)
}
var trust string
for _, r := range composed(t, open, "laptop").Resources {
if r["path"] == "/etc/docker/daemon.json" {
trust, _ = r["content"].(string)
}
if r["id"] == "app.server" && r["image"] != "anchor.internal:5101/app/server@"+aDigest {
t.Errorf("the image the mesh built is fetched as %v", r["image"])
}
}
if !strings.Contains(trust, "anchor.internal:5101") || strings.Contains(trust, ":5000") {
t.Fatalf("the runtime is told to trust %q; the node put the store on 5101", trust)
}
// And a replay to the catalogue says where the store is now.
announced, err := following{open: open}.Announceable(ctx)
if err != nil {
t.Fatal(err)
}
if len(announced) != 1 || announced[0].Made[0].Reference != "anchor.internal:5101/app/server@"+aDigest {
t.Fatalf("the replay announces %+v", announced)
}
}
func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "controller", Kind: catalogue.ArtifactBundle,
Reference: "https://registry.example/mesh-controller/controller.tar.gz", Digest: aDigest}})
if err != nil {
t.Fatal(err)
}
register(t, open, control)
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}, "image": "pg@" + aDigest}}})
register(t, open, catalogue.Manifest{Module: "lavinmq", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5671, From: catalogue.FromMesh},
{Port: 5672, From: catalogue.FromMesh}},
Guards: []int{15672},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}, "image": "mq@" + aDigest}}})
// The control plane's own bus user is the installer's, seeded at genesis before the controller
// runs (SeedBusUser); without it a push now refuses the credential nobody issued (issue 203).
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.mesh-controller",
Kind: inventory.BusController, Node: "anchor", Module: "mesh-controller"}, "bootstrap"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The store's module is registered and given its port, and NOT assigned: the state genesis
// leaves a given-port node in before the foundation is adopted as modules (04-ISSUES/085).
if err := open.inventory.SetSettings(ctx, "anchor", "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 6852}}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "lavinmq"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "lavinmq",
map[string]any{catalogue.PortsSetting: map[string]any{"5672": 5679}}); err != nil {
t.Fatal(err)
}
var env map[string]any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "mesh-controller.controller" {
env, _ = r["env"].(map[string]any)
}
}
if env == nil {
t.Fatal("the control plane's process is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
// Neither given nor assigned by the mesh: the manifest's own number is NOT the answer,
// because the sealed value beside it carries the port genesis wrote (finding F3).
"MESH_BROKER_ADDRESS_PORT": "",
"MESH_BROKER_MANAGEMENT_PORT": "",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node says %q", key, env[key], want)
}
}
}
func aLoneNode(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t)
licences.ForTest(t)
open, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(open.Close)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
}
}
record, err := open.inventory.AddNode(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
reported, _ := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}}})
var profile map[string]any
_ = json.Unmarshal(reported, &profile)
if err := open.inventory.RecordProfile(t.Context(), record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
return open
}
func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
open := aLoneNode(t)
ctx := t.Context()
register(t, open, aStore())
register(t, open, catalogue.Manifest{Module: "builder", Version: "1",
Requires: []string{catalogue.ArtifactStoreProvision},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-builder",
"image": "registry.example/mesh-builder@" + aDigest}}})
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "postgres", Commit: "abc",
Made: []inventory.Artifact{{Name: "runtime", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-postgres",
"image": catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}}})
for _, module := range []string{"distribution", "builder", "postgres"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5100}}); err != nil {
t.Fatal(err)
}
var image any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "postgres.runtime" {
image = r["image"]
}
}
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
+213
View File
@@ -2,10 +2,13 @@ package main
import (
"encoding/json"
"os"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
"github.com/novox/mesh-controller/internal/link"
)
@@ -83,12 +86,150 @@ func aStore() catalogue.Manifest {
// **The trust a machine writes for the store, and the address every built image is fetched
// through, say the port the node gave the store** — not the catalogue's number.
func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aStore())
if _, err := assign(ctx, open, "anchor", "distribution"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5101}}); err != nil {
t.Fatal(err)
}
// A module the mesh built, recorded by digest and path, running on the other machine.
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "app", Commit: "abc",
Made: []inventory.Artifact{{Name: "server", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "app/server@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "app", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-app",
"image": catalogue.ArtifactStoreScheme + "app/server@" + aDigest}}})
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
// The runtime's trust is the runtime's module's to write (novox/hq ADR 0222): a stand-in for it
// asks where this machine reaches the store, as the docker module does.
register(t, open, catalogue.Manifest{Module: "runtime", Version: "1",
Resources: []map[string]any{{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json",
"into": "json", "content": `{"insecure-registries": ["${seat:mesh-artifact-store:reach}"]}` + "\n"}}})
if _, err := assign(ctx, open, "laptop", "runtime"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
if err != nil || !found || node != "anchor" || port != "5101" {
t.Fatalf("the store is found on %q:%q (%v, %v); the node put it on 5101", node, port, found, err)
}
var trust string
for _, r := range composed(t, open, "laptop").Resources {
if r["path"] == "/etc/docker/daemon.json" {
trust, _ = r["content"].(string)
}
if r["id"] == "app.server" && r["image"] != "anchor.internal:5101/app/server@"+aDigest {
t.Errorf("the image the mesh built is fetched as %v", r["image"])
}
}
if !strings.Contains(trust, "anchor.internal:5101") || strings.Contains(trust, ":5000") {
t.Fatalf("the runtime is told to trust %q; the node put the store on 5101", trust)
}
// And a replay to the catalogue says where the store is now.
announced, err := following{open}.Announceable(ctx)
if err != nil {
t.Fatal(err)
}
if len(announced) != 1 || announced[0].Made[0].Reference != "anchor.internal:5101/app/server@"+aDigest {
t.Fatalf("the replay announces %+v", announced)
}
}
// **The control plane's own connections say the port the node gave the store and the broker.**
//
// Composed from the control plane's own manifest against a real inventory: the store's module is
// given 6852 on this node the way genesis or an operator gives it, and the control plane's
// process is told so beside the sealed connection genesis wrote.
func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "controller", Kind: catalogue.ArtifactBundle,
Reference: "https://registry.example/mesh-controller/controller.tar.gz", Digest: aDigest}})
if err != nil {
t.Fatal(err)
}
register(t, open, control)
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}, "image": "pg@" + aDigest}}})
register(t, open, catalogue.Manifest{Module: "lavinmq", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5671, From: catalogue.FromMesh},
{Port: 5672, From: catalogue.FromMesh}},
Guards: []int{15672},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}, "image": "mq@" + aDigest}}})
// The control plane's own bus user is the installer's, seeded at genesis before the controller
// runs (SeedBusUser); without it a push now refuses the credential nobody issued (issue 203).
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.mesh-controller",
Kind: inventory.BusController, Node: "anchor", Module: "mesh-controller"}, "bootstrap"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The store's module is registered and given its port, and NOT assigned: the state genesis
// leaves a given-port node in before the foundation is adopted as modules (04-ISSUES/085).
if err := open.inventory.SetSettings(ctx, "anchor", "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 6852}}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "lavinmq"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "lavinmq",
map[string]any{catalogue.PortsSetting: map[string]any{"5672": 5679}}); err != nil {
t.Fatal(err)
}
var env map[string]any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "mesh-controller.controller" {
env, _ = r["env"].(map[string]any)
}
}
if env == nil {
t.Fatal("the control plane's process is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
// Neither given nor assigned by the mesh: the manifest's own number is NOT the answer,
// because the sealed value beside it carries the port genesis wrote (finding F3).
"MESH_BROKER_ADDRESS_PORT": "",
"MESH_BROKER_MANAGEMENT_PORT": "",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node says %q", key, env[key], want)
}
}
}
// withSeatPorts is the control plane's manifest with the seat placeholders in its environment —
// added here until module.json carries them (the manifest lands one commit after the code that
@@ -133,7 +274,79 @@ func withSeatPorts(m catalogue.Manifest) catalogue.Manifest {
// aLoneNode is one capable machine with nothing placed on any network — the control-node during
// genesis, before the "network" step, which is after the store, the broker, the vault and the
// catalogue have each been built and pushed (finding F2).
func aLoneNode(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t)
licences.ForTest(t)
open, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(open.Close)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
}
}
record, err := open.inventory.AddNode(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
reported, _ := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}}})
var profile map[string]any
_ = json.Unmarshal(reported, &profile)
if err := open.inventory.RecordProfile(t.Context(), record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
return open
}
// **Before the network exists, the store's own node reaches it by loopback** — never refused,
// never handed the scheme: a genesis pushes the store, the broker, the vault and the catalogue to
// a node on no network, and builds the catalogue on a base it must be able to pull.
func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
open := aLoneNode(t)
ctx := t.Context()
register(t, open, aStore())
register(t, open, catalogue.Manifest{Module: "builder", Version: "1",
Requires: []string{catalogue.ArtifactStoreProvision},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-builder",
"image": "registry.example/mesh-builder@" + aDigest}}})
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "postgres", Commit: "abc",
Made: []inventory.Artifact{{Name: "runtime", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-postgres",
"image": catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}}})
for _, module := range []string{"distribution", "builder", "postgres"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5100}}); err != nil {
t.Fatal(err)
}
var image any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "postgres.runtime" {
image = r["image"]
}
}
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
+3 -18
View File
@@ -424,9 +424,8 @@ func comparisonOf(module string, c comparison, opts takeOptions) (preview string
}
// And its settings on this machine, composed against its definition (rule 1, rule 6).
for _, layer := range c.layers {
said := layer.Whole()
keys := make([]string, 0, len(said))
for k := range said {
keys := make([]string, 0, len(layer.Values))
for k := range layer.Values {
keys = append(keys, k)
}
sort.Strings(keys)
@@ -723,7 +722,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
return "", err
}
derived := derivedFilter{rules: rules, foundation: with.Foundation, mesh: with.Mesh,
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks, local: with.LocalNetworks}
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks}
filtering, err := inv.FilteringOf(ctx, node)
if err != nil {
return "", err
@@ -931,9 +930,6 @@ type derivedFilter struct {
// The filter constrains what arrives on them; everything arriving elsewhere is this machine's
// own guest and is not filtered.
outwardLinks []string
// local is the machine's local networks, which a port from `local-network` stays open to (novox/hq
// ADR 0340).
local []string
}
// closesOutside is what a narrowing from everywhere to the private network is called: it closes.
@@ -981,17 +977,6 @@ func (d derivedFilter) fate(r inventory.Reach) string {
if !onMesh {
return fmt.Sprintf("%s — declared by %s from the mesh only", closesOutside, by)
}
case catalogue.FromLocalNetwork:
// The private network and this machine's local networks (novox/hq ADR 0340): it closes to
// everything else outside, and says which local networks stay.
if !onMesh {
if len(d.local) == 0 {
return fmt.Sprintf("%s — declared by %s from the local network, and this machine "+
"reported none", closesOutside, by)
}
return fmt.Sprintf("WILL CLOSE to everything outside the private network and %s — declared "+
"by %s from the local network", strings.Join(d.local, ", "), by)
}
}
return fmt.Sprintf("declared by %s (from %s)", by, rule.From)
}
@@ -1,161 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-host/rootsearch"
)
func TestTheSelfCheckSaysAnAgentAccountThatCanBecomeRoot(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.NodeByName(ctx, n); err != nil {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAccount(ctx, n, "ops", ""); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
found = onlyMachine(found, "anchor")
if len(found) != 1 || found[0].Machine != "anchor" || found[0].Severity != conditions.Urgent ||
!strings.Contains(found[0].Said, "not judged") {
t.Fatalf("a named agent account with no verdict: %+v", found)
}
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if w.Headline == "" || w.Needs == "" || w.Resolved == "" {
t.Errorf("the condition has no plain words: %+v", w)
}
healthy := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: link.StateHealthy, Root: link.RootNever, Account: "agent"}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{healthy}}); err != nil {
t.Fatal(err)
}
if found, err = probeAgentAccounts(ctx, d); err != nil || len(onlyMachine(found, "anchor")) != 0 {
t.Fatalf("a judged agent account still fails: %+v %v", found, err)
}
if named, confined, why, err := agentConfined(ctx, inv, "anchor", time.Now()); err != nil || !named || !confined {
t.Fatalf("agentConfined on anchor: %v %v %q %v", named, confined, why, err)
}
if named, _, why, err := agentConfined(ctx, inv, "laptop", time.Now()); err != nil || named ||
!strings.Contains(why, "operator account") {
t.Fatalf("agentConfined on a machine naming none: %v %q %v", named, why, err)
}
}
func TestTheFactsCarryTheAgentAccountAsAPseudonym(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
if err := open.inventory.SetAccount(ctx, "anchor", "keeper", ""); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetAgentAccount(ctx, "anchor", "warden", ""); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
body, _ := f.Encode()
if strings.Contains(string(body), "warden") {
t.Error("the agent account's name is in the snapshot")
}
m, ok := f.Machine(snapshot.Pseudonym("machine", "anchor"))
if !ok || m.AgentAccount != snapshot.Pseudonym("account", "warden") || m.Account == m.AgentAccount {
t.Fatalf("the anchor's agent account reads as %q (operator %q)", m.AgentAccount, m.Account)
}
}
func TestASearchStillRunningAfterARestartIsNotUrgent(t *testing.T) {
if searchQuietFor != rootsearch.Quiet {
t.Fatalf("the quiet is %s and the node-engine's %s: they are one value", searchQuietFor, rootsearch.Quiet)
}
// A daily search that finishes within the quiet never leaves the account unjudged between two of them.
if rootsearch.FreshFor < rootsearch.Every+rootsearch.Quiet {
t.Fatalf("a complete search judges for %s, less than a day's search (%s) and the quiet (%s)",
rootsearch.FreshFor, rootsearch.Every, rootsearch.Quiet)
}
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.NodeByName(ctx, "anchor"); err != nil {
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
say := func(state, reason string) []conditions.Observation {
t.Helper()
// The engine's since is always now: it was just restarted.
v := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: state, Reason: reason, Root: link.RootNever,
Account: "agent", Since: time.Now()}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{v}}); err != nil {
t.Fatal(err)
}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
return onlyMachine(found, "anchor")
}
running := link.ReasonRootPending + ": the search for setuid programs, started at 19:21, has not finished yet"
healthy := func() {
t.Helper()
if found := say(link.StateHealthy, ""); len(found) != 0 {
t.Fatalf("a healthy verdict raised: %+v", found)
}
}
if found := say(link.StateUnknown, running); len(found) != 0 {
t.Fatalf("a search first seen now was raised: %+v", found)
}
if _, confined, why, _ := agentConfined(ctx, inv, "anchor", time.Now()); confined || !strings.Contains(why, "not judged") {
t.Fatalf("an account whose search runs was read as confined: %q", why)
}
// The engine restarted again and again, each statement's own since fresh: the controller's clock runs on.
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err)
}
healthy() // a complete verdict forgets when the waiting began …
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err) // … and this restart loop began past the bound
}
if found := say(link.StateUnknown, running); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search pending past the bound, by the controller's clock, was not urgent: %+v", found)
}
healthy()
incomplete := rootsearch.ReasonIncomplete + ": the search for setuid programs did not finish (last tried at " +
"19:23: timeout); it is tried again later"
if found := say(link.StateUnknown, incomplete); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search that did not finish was not urgent: %+v", found)
}
if found := say(link.StateUnhealthy, link.ReasonRoot+": in the group docker; "+running); len(found) != 1 ||
found[0].Severity != conditions.Urgent {
t.Fatalf("a way to root found while the search runs was not urgent: %+v", found)
}
}
+145
View File
@@ -1,12 +1,14 @@
package main
import (
"github.com/novox/mesh-host/rootsearch"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -63,6 +65,55 @@ func TestAnAgentAccountIsConfinedOnlyOnAHealthyVerdictJudgedForRoot(t *testing.T
// DA raises an urgent condition, with plain words, on a machine whose agent account is not judged unable to
// become root; a machine that names none is not its to judge.
func TestTheSelfCheckSaysAnAgentAccountThatCanBecomeRoot(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.NodeByName(ctx, n); err != nil {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAccount(ctx, n, "ops", ""); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
found = onlyMachine(found, "anchor")
if len(found) != 1 || found[0].Machine != "anchor" || found[0].Severity != conditions.Urgent ||
!strings.Contains(found[0].Said, "not judged") {
t.Fatalf("a named agent account with no verdict: %+v", found)
}
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if w.Headline == "" || w.Needs == "" || w.Resolved == "" {
t.Errorf("the condition has no plain words: %+v", w)
}
healthy := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: link.StateHealthy, Root: link.RootNever, Account: "agent"}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{healthy}}); err != nil {
t.Fatal(err)
}
if found, err = probeAgentAccounts(ctx, d); err != nil || len(onlyMachine(found, "anchor")) != 0 {
t.Fatalf("a judged agent account still fails: %+v %v", found, err)
}
if named, confined, why, err := agentConfined(ctx, inv, "anchor", time.Now()); err != nil || !named || !confined {
t.Fatalf("agentConfined on anchor: %v %v %q %v", named, confined, why, err)
}
if named, _, why, err := agentConfined(ctx, inv, "laptop", time.Now()); err != nil || named ||
!strings.Contains(why, "operator account") {
t.Fatalf("agentConfined on a machine naming none: %v %q %v", named, why, err)
}
}
func TestTheAgentRootWordsArePlain(t *testing.T) {
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
@@ -83,6 +134,28 @@ func onlyMachine(obs []conditions.Observation, machine string) []conditions.Obse
// The snapshot a merge check composes from carries the agent account as a pseudonym (novox/hq ADR 0266),
// so a change is judged against machines that name one, and the name never leaves.
func TestTheFactsCarryTheAgentAccountAsAPseudonym(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
if err := open.inventory.SetAccount(ctx, "anchor", "keeper", ""); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetAgentAccount(ctx, "anchor", "warden", ""); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
body, _ := f.Encode()
if strings.Contains(string(body), "warden") {
t.Error("the agent account's name is in the snapshot")
}
m, ok := f.Machine(snapshot.Pseudonym("machine", "anchor"))
if !ok || m.AgentAccount != snapshot.Pseudonym("account", "warden") || m.Account == m.AgentAccount {
t.Fatalf("the anchor's agent account reads as %q (operator %q)", m.AgentAccount, m.Account)
}
}
// No verb runs a `node` command that sets something: through the generic `command` verb, `node account`,
// `node agent-account` and every other `node` subcommand but list and show are refused, naming the terminal.
@@ -132,3 +205,75 @@ func TestTheNodeVerbOnlyShows(t *testing.T) {
// controller first saw it waiting — never from the engine's own "since", which a restart resets, so an engine
// restarted in a loop does not keep it quiet. The account is still not confined, and `node show` still says not
// judged; a search that failed, or a way to root found, is urgent at once.
func TestASearchStillRunningAfterARestartIsNotUrgent(t *testing.T) {
if searchQuietFor != rootsearch.Quiet {
t.Fatalf("the quiet is %s and the node-engine's %s: they are one value", searchQuietFor, rootsearch.Quiet)
}
// A daily search that finishes within the quiet never leaves the account unjudged between two of them.
if rootsearch.FreshFor < rootsearch.Every+rootsearch.Quiet {
t.Fatalf("a complete search judges for %s, less than a day's search (%s) and the quiet (%s)",
rootsearch.FreshFor, rootsearch.Every, rootsearch.Quiet)
}
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.NodeByName(ctx, "anchor"); err != nil {
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
say := func(state, reason string) []conditions.Observation {
t.Helper()
// The engine's since is always now: it was just restarted.
v := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: state, Reason: reason, Root: link.RootNever,
Account: "agent", Since: time.Now()}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{v}}); err != nil {
t.Fatal(err)
}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
return onlyMachine(found, "anchor")
}
running := link.ReasonRootPending + ": the search for setuid programs, started at 19:21, has not finished yet"
healthy := func() {
t.Helper()
if found := say(link.StateHealthy, ""); len(found) != 0 {
t.Fatalf("a healthy verdict raised: %+v", found)
}
}
if found := say(link.StateUnknown, running); len(found) != 0 {
t.Fatalf("a search first seen now was raised: %+v", found)
}
if _, confined, why, _ := agentConfined(ctx, inv, "anchor", time.Now()); confined || !strings.Contains(why, "not judged") {
t.Fatalf("an account whose search runs was read as confined: %q", why)
}
// The engine restarted again and again, each statement's own since fresh: the controller's clock runs on.
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err)
}
healthy() // a complete verdict forgets when the waiting began …
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err) // … and this restart loop began past the bound
}
if found := say(link.StateUnknown, running); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search pending past the bound, by the controller's clock, was not urgent: %+v", found)
}
healthy()
incomplete := rootsearch.ReasonIncomplete + ": the search for setuid programs did not finish (last tried at " +
"19:23: timeout); it is tried again later"
if found := say(link.StateUnknown, incomplete); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search that did not finish was not urgent: %+v", found)
}
if found := say(link.StateUnhealthy, link.ReasonRoot+": in the group docker; "+running); len(found) != 1 ||
found[0].Severity != conditions.Urgent {
t.Fatalf("a way to root found while the search runs was not urgent: %+v", found)
}
}
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
-46
View File
@@ -38,61 +38,15 @@ func watchApplyFailing(f *signalFacts) []conditions.Observation {
}
}
var out []conditions.Observation
said := map[string]bool{}
for _, d := range f.doing {
if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] {
continue
}
out = append(out, applyFailing(d, f.now))
said[d.Node] = true
}
// **A module's files the provisioner failed to place are a failed apply too** (novox/hq ADR 0362 §3): its client's
// newest word, judged by the same bound — the client says a failure once, and fills again on every change —
// on a machine whose node-engine applied what it was sent.
failed := map[string][]inventory.ProvReport{}
for _, r := range f.provFailed {
if r.Status != inventory.ProvFailed || f.now.Sub(r.At) <= applyEvery || !heard[r.Node] || said[r.Node] {
continue
}
failed[r.Node] = append(failed[r.Node], r)
}
for _, node := range sortedNames(failed) {
out = append(out, provisioningFailing(node, failed[node], f.now))
}
return out
}
// provisioningFailing is one machine's S21 condition for the modules whose files its provisioner client failed to
// place.
func provisioningFailing(m string, failed []inventory.ProvReport, now time.Time) conditions.Observation {
sort.Slice(failed, func(a, b int) bool { return failed[a].Module < failed[b].Module })
var names, words []string
since := failed[0].At
for _, r := range failed {
names = append(names, r.Module)
words = append(words, r.Module+": "+oneLine(orNoReason(r.Reason)))
if r.At.Before(since) {
since = r.At
}
}
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Kind: kindApplyFailing, Machine: m,
Severity: conditions.Warning, Said: strings.Join(words, "; "),
Resolved: m + " applies what it is sent again"}
o.Summary = fmt.Sprintf("%s's provisioner has failed to place the files of %s since %s (bound %s)", m,
strings.Join(names, ", "), since.UTC().Format("2006-01-02 15:04 MST"), ago(applyEvery))
o.Headline = m + " is not applying its update"
o.Explanation = fmt.Sprintf("The provisioner on %s has not been able to write the files of %s for %s. The "+
"details say why.", m, strings.Join(names, ", "), humanDuration(now.Sub(since)))
return o
}
func orNoReason(s string) string {
if s == "" {
return "no reason given"
}
return s
}
// applyFailing is one machine's S21 condition.
func applyFailing(d inventory.Doing, now time.Time) conditions.Observation {
m := d.Node
+23 -2
View File
@@ -117,10 +117,13 @@ type asked struct {
// Proposal is a settings layer proposed through a verb (proposals.go, novox/hq ADR 0277): about no
// condition, set on Approve by the serving controller itself, and left alone by the reconciling of conditions.
Proposal *settingsProposal `json:"proposal,omitempty"`
// Take is a library the operator is asked to let the mesh take (kept_libraries.go, novox/hq ADR 0328 §7): about no
// condition, taken on Approve by the serving controller itself, asked once per library.
Take *libraryTake `json:"take,omitempty"`
}
// ofACondition says an ask is one of a condition's: not a rehearsal, not a proposal.
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil }
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil && r.Take == nil }
// partKey is an ask's place among what is asked: its condition and its part.
func partKey(condition, part string) string { return condition + "#" + part }
@@ -192,6 +195,11 @@ type asker struct {
call func(ctx context.Context, a conditions.Action, args map[string]string) error
// setLayer sets a proposed settings layer on the operator's warrant (novox/hq ADR 0277); nil cannot.
setLayer setOnWarrant
// folders is every library waiting to be taken (novox/hq ADR 0328 §7), and take what takes one on the operator's
// warrant; nil asks about none. machines are the mesh's machine names, allowed in an ask's words.
folders func(ctx context.Context) ([]libraryTake, error)
take takeOnWarrant
machines func(ctx context.Context) ([]string, error)
// record writes the hand-act log.
record func(ctx context.Context, act link.HandAct) error
// routerRecord reads the router's record of an ask for a warrant missed; nil reads nothing.
@@ -554,6 +562,10 @@ func (a *asker) reconcile(ctx context.Context) error {
return err
}
}
// And every library waiting to be taken, once per library (novox/hq ADR 0328 §7).
if err := a.askFolders(ctx); err != nil {
a.logf("the operator could not be asked about the libraries waiting to be taken: %v", err)
}
why := "the router refused the ask"
if len(refusedWords) > 0 {
why += ": " + refusedWords[0]
@@ -818,7 +830,7 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
if err != nil {
return err
}
stillOpen := r.Rehearsal || r.Proposal != nil // a rehearsal and a proposal are about no condition
stillOpen := r.Rehearsal || r.Proposal != nil || r.Take != nil // a rehearsal, a proposal and a take are about no condition
for _, c := range open {
stillOpen = stillOpen || c.Key == r.Condition
}
@@ -876,6 +888,15 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
if acted == nil && outcome != "" && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case r.Take != nil && act.Verb == takeVerb:
// A library taken on the operator's warrant (novox/hq ADR 0328 §7): the act is composed again from the
// record and held to what the option bound before the engine is asked.
outcome, acted = a.decideTake(ctx, *r, act, w)
if acted == nil && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case r.Take != nil:
acted = fmt.Errorf("the take %s offers no act %s", r.ID, act.Verb)
case act.Arguments["silence"] != "":
acted = a.silence(ctx, act.Arguments["silence"], conditions.MaxSilence, byWords(w), why)
default:
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
+21
View File
@@ -572,6 +572,27 @@ func servingAsker(open *stores, conn *nats.Conn, js jetstream.JetStream, keeper
call: callAction(conn),
// A proposed settings layer is set by this controller itself on the warrant (novox/hq ADR 0277).
setLayer: setLayerIn(open),
// A shared folder waiting to be kept is asked about once, and taken on the warrant (novox/hq ADR 0325 §8).
folders: foldersIn(open),
take: takeFolderIn(open, func(ctx context.Context, act link.AccessActAsk) (link.HandOverAnswer, error) {
ident, err := open.Identity(ctx)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked of "+
"%s: %w", act.Node, err)
}
return link.AskTake(ctx, conn, ident, act, link.TakeWithin)
}),
machines: func(ctx context.Context) ([]string, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, err
}
names := make([]string, 0, len(nodes))
for _, n := range nodes {
names = append(names, n.Name)
}
return names, nil
},
record: func(ctx context.Context, act link.HandAct) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
@@ -1,182 +0,0 @@
//go:build integration
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/overlay"
)
func TestAModuleAssignedAndNotPushedIsAwaitingAPushNotUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop") // the machine was pushed before; then the module is assigned
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
// The read the rest of the mesh asks still refuses it, with the composer's typed error: nothing
// can be compared about a secret that does not exist (status), and nothing here string-matches.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
_, err = declarationWith(ctx, open, "laptop", plan, settings, gens, Reading)
var notMade *catalogue.NotMadeError
if !errors.As(err, &notMade) || notMade.Module != "keeper" || notMade.Name != "repository" {
t.Fatalf("a read composition did not say which secret is not made, typed: %v", err)
}
// Foreseen, it composes, names what the push makes, and made nothing.
foreseen, err := declarationWith(ctx, open, "laptop", plan, settings, gens, Foreseeing)
if err != nil || len(foreseen.foreseen) != 1 || foreseen.foreseen[0] != "keeper/repository" {
t.Fatalf("foreseen: %v %v", foreseen.foreseen, err)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "keeper", "repository"); err != nil || held {
t.Fatalf("asking ahead of the push made the secret (held %v, %v)", held, err)
}
// Within the bound: nothing at all — no urgent, no warning, nobody notified.
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a machine waiting for a push raised %+v", got)
}
// Past the bound: a warning, not urgent, saying what the push will make.
before := awaitingPushBound
awaitingPushBound = -time.Minute
t.Cleanup(func() { awaitingPushBound = before })
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.awaiting-push" || got[0].Severity != conditions.Warning ||
!strings.Contains(got[0].Summary, "keeper/repository") || !strings.Contains(got[0].Summary, "push laptop") {
t.Fatalf("a machine left un-pushed past the bound: %+v", got)
}
// Pushed: the secret is made and D1 says nothing.
pushedBy(t, open, "laptop")
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a pushed machine still raised %+v", got)
}
}
func TestASecretThePushCannotMakeIsStillUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aTalker())
if _, err := assign(ctx, open, "laptop", "talker"); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "module issue talker --node laptop") {
t.Fatalf("a push that will be refused was not said urgently: %+v", got)
}
// And a machine whose composition fails for anything else, with a secret waiting beside it, is
// uncomposable for that — the stand-in hides nothing.
register(t, open, aKeeper())
if _, err := assign(ctx, open, "anchor", "keeper"); err != nil {
t.Fatal(err)
}
one, two := rivals()
register(t, open, one)
register(t, open, two)
_, _ = assign(ctx, open, "anchor", "rival-one")
_, _ = assign(ctx, open, "anchor", "rival-two")
keys := map[string]conditions.Severity{}
for _, o := range d1(t, open) {
keys[o.Key()] = o.Severity
}
if keys["machine.anchor.uncomposable"] != conditions.Urgent {
t.Fatalf("a real failure beside a waiting secret: %v", keys)
}
}
func TestAGivenSecretUnderAnOldKeyIsNotForeseen(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "keeper", "repository", "given"); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || !strings.Contains(got[0].Summary, "issue it again") {
t.Fatalf("a given secret under an old key: %+v", got)
}
}
func TestAMachineAwaitsAPushSinceItsOldestAssignmentSinceTheLastSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
sent := time.Now()
since, err := open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.After(sent) {
t.Fatalf("nothing assigned since the send: waiting since %v (%v), the send was before %v", since, err, sent)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
since, err = open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.Before(sent.Add(-time.Second)) {
t.Fatalf("assigned after the send: waiting since %v (%v), not from the assignment", since, err)
}
}
func TestALastSendIsReadFromTheSendAndTheReport(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
sends, err := readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("a holder assigned and not pushed is expected to answer")
}
if !sends["laptop"].settled(overlay.Name, time.Now().Add(time.Hour)) {
t.Fatal("a module the machine was sent long ago is not expected to answer")
}
pushedBy(t, open, "laptop")
sends, err = readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("pushed a moment ago and not reported, a holder is already expected")
}
if !sends["laptop"].settled("keeper", time.Now().Add(reportGrace+time.Minute)) {
t.Fatal("pushed past the grace, a holder is not expected")
}
if _, ok := sends["laptop"].carried["keeper"]; !ok {
t.Fatalf("the send's builds do not carry keeper: %v", sends["laptop"].carried)
}
pushedAndApplied(t, open, "laptop")
if sends, err = readDeliveries(ctx, open.inventory); err != nil || !sends["laptop"].settled("keeper", time.Now()) {
t.Fatalf("pushed and reported applied, a holder is not expected to answer (%v)", err)
}
}
+166
View File
@@ -1,12 +1,15 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// Between `assign` and `push` a module's own secrets are not made yet: the push makes them. D1 composed
@@ -78,14 +81,140 @@ func d1(t *testing.T, open *stores) []conditions.Observation {
// **THE WINDOW, REPRODUCED**: assigned and not pushed, a module whose own secret the push makes is
// waiting, not uncomposable; past the bound it is a warning naming what the push makes; pushed, nothing.
func TestAModuleAssignedAndNotPushedIsAwaitingAPushNotUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop") // the machine was pushed before; then the module is assigned
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
// The read the rest of the mesh asks still refuses it, with the composer's typed error: nothing
// can be compared about a secret that does not exist (status), and nothing here string-matches.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
_, err = declarationWith(ctx, open, "laptop", plan, settings, gens, Reading)
var notMade *catalogue.NotMadeError
if !errors.As(err, &notMade) || notMade.Module != "keeper" || notMade.Name != "repository" {
t.Fatalf("a read composition did not say which secret is not made, typed: %v", err)
}
// Foreseen, it composes, names what the push makes, and made nothing.
foreseen, err := declarationWith(ctx, open, "laptop", plan, settings, gens, Foreseeing)
if err != nil || len(foreseen.foreseen) != 1 || foreseen.foreseen[0] != "keeper/repository" {
t.Fatalf("foreseen: %v %v", foreseen.foreseen, err)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "keeper", "repository"); err != nil || held {
t.Fatalf("asking ahead of the push made the secret (held %v, %v)", held, err)
}
// Within the bound: nothing at all — no urgent, no warning, nobody notified.
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a machine waiting for a push raised %+v", got)
}
// Past the bound: a warning, not urgent, saying what the push will make.
before := awaitingPushBound
awaitingPushBound = -time.Minute
t.Cleanup(func() { awaitingPushBound = before })
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.awaiting-push" || got[0].Severity != conditions.Warning ||
!strings.Contains(got[0].Summary, "keeper/repository") || !strings.Contains(got[0].Summary, "push laptop") {
t.Fatalf("a machine left un-pushed past the bound: %+v", got)
}
// Pushed: the secret is made and D1 says nothing.
pushedBy(t, open, "laptop")
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a pushed machine still raised %+v", got)
}
}
// **A REAL FAILURE IS STILL URGENT**: a secret the push would be refused on is not one it will make.
// A bus credential nobody issued (issue 203) fails the push, so D1 says it — urgent, in the push's words.
func TestASecretThePushCannotMakeIsStillUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aTalker())
if _, err := assign(ctx, open, "laptop", "talker"); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "module issue talker --node laptop") {
t.Fatalf("a push that will be refused was not said urgently: %+v", got)
}
// And a machine whose composition fails for anything else, with a secret waiting beside it, is
// uncomposable for that — the stand-in hides nothing.
register(t, open, aKeeper())
if _, err := assign(ctx, open, "anchor", "keeper"); err != nil {
t.Fatal(err)
}
one, two := rivals()
register(t, open, one)
register(t, open, two)
_, _ = assign(ctx, open, "anchor", "rival-one")
_, _ = assign(ctx, open, "anchor", "rival-two")
keys := map[string]conditions.Severity{}
for _, o := range d1(t, open) {
keys[o.Key()] = o.Severity
}
if keys["machine.anchor.uncomposable"] != conditions.Urgent {
t.Fatalf("a real failure beside a waiting secret: %v", keys)
}
}
// A given secret sealed to a key the machine no longer has is not the mesh's to make again: the push is
// refused on it, so D1 is too.
func TestAGivenSecretUnderAnOldKeyIsNotForeseen(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "keeper", "repository", "given"); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || !strings.Contains(got[0].Summary, "issue it again") {
t.Fatalf("a given secret under an old key: %+v", got)
}
}
// The bound is read from when the machine began waiting: the oldest assignment since its last send.
func TestAMachineAwaitsAPushSinceItsOldestAssignmentSinceTheLastSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
sent := time.Now()
since, err := open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.After(sent) {
t.Fatalf("nothing assigned since the send: waiting since %v (%v), the send was before %v", since, err, sent)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
since, err = open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.Before(sent.Add(-time.Second)) {
t.Fatalf("assigned after the send: waiting since %v (%v), not from the assignment", since, err)
}
}
// **D3 AND D13 WAIT FOR THE SEND**: a holder is expected to answer on a machine once the machine was sent
// it and had time to report — never between assign and push.
@@ -113,3 +242,40 @@ func TestAHolderIsExpectedOnlyOnceSentAndReported(t *testing.T) {
}
// And through the stores: assigned, not in the last send; pushed and reported, carried and settled.
func TestALastSendIsReadFromTheSendAndTheReport(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
sends, err := readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("a holder assigned and not pushed is expected to answer")
}
if !sends["laptop"].settled(overlay.Name, time.Now().Add(time.Hour)) {
t.Fatal("a module the machine was sent long ago is not expected to answer")
}
pushedBy(t, open, "laptop")
sends, err = readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("pushed a moment ago and not reported, a holder is already expected")
}
if !sends["laptop"].settled("keeper", time.Now().Add(reportGrace+time.Minute)) {
t.Fatal("pushed past the grace, a holder is not expected")
}
if _, ok := sends["laptop"].carried["keeper"]; !ok {
t.Fatalf("the send's builds do not carry keeper: %v", sends["laptop"].carried)
}
pushedAndApplied(t, open, "laptop")
if sends, err = readDeliveries(ctx, open.inventory); err != nil || !sends["laptop"].settled("keeper", time.Now()) {
t.Fatalf("pushed and reported applied, a holder is not expected to answer (%v)", err)
}
}
@@ -1,219 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// aBackupSeatOnTheBus is one machine's node-backup holder as a test makes it: it counts the `now` asks it hears and
// answers `backed-up` with the last night it is told (novox/hq ADR 0404).
type aBackupSeatOnTheBus struct {
mu sync.Mutex
nows []string
night map[string]any // module → lastNight
}
func (f *aBackupSeatOnTheBus) serve(t *testing.T, conn *nats.Conn, node string) {
t.Helper()
reply := func(m *nats.Msg, result any) {
body, _ := json.Marshal(map[string]any{"result": result, "error": "", "node": node})
_ = m.Respond(body)
}
now, err := conn.Subscribe(link.NodeSeatToolSubject(catalogue.BackupSeat, "now", node), func(m *nats.Msg) {
var a struct {
Module string `json:"module"`
}
_ = json.Unmarshal(m.Data, &a)
f.mu.Lock()
f.nows = append(f.nows, a.Module)
f.mu.Unlock()
reply(m, map[string]any{"started": a.Module})
})
if err != nil {
t.Fatal(err)
}
read, err := conn.Subscribe(link.NodeSeatToolSubject(catalogue.BackupSeat, "backed-up", node), func(m *nats.Msg) {
f.mu.Lock()
defer f.mu.Unlock()
var out []map[string]any
for module, n := range f.night {
out = append(out, map[string]any{"module": module, "lastNight": n})
}
reply(m, out)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = now.Unsubscribe(); _ = read.Unsubscribe() })
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
}
func (f *aBackupSeatOnTheBus) answers(module string, ok bool, words string) {
f.mu.Lock()
defer f.mu.Unlock()
f.night = map[string]any{module: map[string]any{"ok": ok, "at": time.Now().UTC().Add(time.Second), "error": words}}
}
func (f *aBackupSeatOnTheBus) asks() int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.nows)
}
// Acceptance test 3 of design 47, on the controller's side: a delivery plan that moves a module keeping data waits
// for its backups on both machines before it asks a build; a backup that fails holds it, with the machine, the
// module and the backup's words, and a condition says the same; the verb the delivery's release calls asks the
// backups again, and the plan goes on. One bus and one store for the scenarios.
func TestADeliveryThatMovesABackedUpModuleWaitsForItsBackupAndAFailedOneHoldsItAndIsSaid(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := asksRecorded(t)
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "keeper", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BackupSeat, Scope: catalogue.ScopeNode, Serves: []string{"backed-up", "now", "restore"}}}})
notes, err := catalogue.ParseManifest([]byte(`{"module":"notes","version":"1","data":{"own":[` +
`{"id":"files","path":"${dir:files}","class":"valuable","why":"people's notes"}]},` +
`"resources":[{"id":"files","type":"directory"}]}`))
if err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, notes, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/notes", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
if _, err := assign(ctx, open, node, "keeper"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, node, "notes"); err != nil {
t.Fatal(err)
}
}
seats := map[string]*aBackupSeatOnTheBus{"anchor": {}, "laptop": {}}
for node, s := range seats {
s.serve(t, js.Conn(), node)
}
// Short waits: the bound and the look are variables, so nothing here waits for the minutes of a real backup.
wasWithin, wasEvery := backupsWithin, backupsPollEvery
backupsWithin, backupsPollEvery = time.Minute, 10*time.Millisecond
t.Cleanup(func() { backupsWithin, backupsPollEvery = wasWithin, wasEvery })
newPlan := func(id string) {
t.Helper()
p := inventory.Plan{ID: id, Repository: "novox/mesh-catalog", Branch: "main", Commit: "c1", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"notes"}}, Modules: map[string]*inventory.PlanModule{"notes": {}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
}
read := func(id string) inventory.Plan {
t.Helper()
p, err := inv.PlanByID(ctx, id)
if err != nil {
t.Fatal(err)
}
return p
}
// 1. Both machines are asked; while one has not answered, no build is asked.
newPlan("plan-one")
advancePlans(ctx, open)
if seats["anchor"].asks() != 1 || seats["laptop"].asks() != 1 || len(*asked) != 0 {
t.Fatalf("asks anchor=%d laptop=%d builds=%v", seats["anchor"].asks(), seats["laptop"].asks(), *asked)
}
seats["anchor"].answers("notes", true, "")
advancePlans(ctx, open)
if len(*asked) != 0 || read("plan-one").Backups.State != inventory.BackupPending {
t.Fatalf("a tier was asked while a backup was pending: %v %+v", *asked, read("plan-one").Backups)
}
seats["laptop"].answers("notes", true, "")
advancePlans(ctx, open)
if len(*asked) != 1 || read("plan-one").Backups.State != inventory.BackupDone {
t.Fatalf("the tier was not asked once every backup was done: %v %+v", *asked, read("plan-one").Backups)
}
finish := read("plan-one")
finish.State = inventory.PlanDone
if err := inv.SavePlan(ctx, &finish); err != nil {
t.Fatal(err)
}
// 2. A backup that fails holds the plan: not failed, nothing asked, the words kept, a condition raised.
seats["anchor"].night, seats["laptop"].night = nil, nil
newPlan("plan-two")
advancePlans(ctx, open)
seats["anchor"].answers("notes", true, "")
seats["laptop"].answers("notes", false, "no space left on device")
advancePlans(ctx, open)
held := read("plan-two")
if held.State != inventory.PlanBuilding || held.Backups.State != inventory.BackupFailed || len(*asked) != 1 {
t.Fatalf("the plan was not held: %s %+v builds=%v", held.State, held.Backups, *asked)
}
var failed *inventory.PlanBackup
for i := range held.Backups.Asked {
if held.Backups.Asked[i].State == inventory.BackupFailed {
failed = &held.Backups.Asked[i]
}
}
if failed == nil || failed.Machine != "laptop" || failed.Module != "notes" || failed.Words != "no space left on device" ||
strings.Contains(held.Note, "tried again") {
t.Fatalf("the failed backup: %+v note=%q", failed, held.Note)
}
advancePlans(ctx, open) // not retried on the next look
if again := read("plan-two"); again.Backups.State != inventory.BackupFailed || len(*asked) != 1 ||
seats["laptop"].asks() != 2 {
t.Fatalf("a held plan was asked again by itself: %+v builds=%v asks=%d", again.Backups, *asked, seats["laptop"].asks())
}
said, err := probeBackupHeld(ctx, &doctor{open: open})
if err != nil || len(said) != 1 || said[0].Kind != kindBackupHeld || !strings.Contains(said[0].Said, "no space left on device") {
t.Fatalf("the condition: %+v %v", said, err)
}
// 3. The verb the release calls: the failed backup is asked again, and only it, and the plan goes on.
seats["laptop"].answers("notes", false, "no space left on device")
if err := deliveryCommand(ctx, []string{"backups-again", "plan-two", "--why", "a person released it", "--by",
"mesh-delivery for jochen"}); err != nil {
t.Fatal(err)
}
if err := deliveryCommand(ctx, []string{"backups-again", "plan-two", "--why", "again"}); err == nil {
t.Fatal("a plan with no failed backup was asked again")
}
seats["laptop"].night = nil
advancePlans(ctx, open)
if seats["laptop"].asks() != 3 || seats["anchor"].asks() != 2 {
t.Fatalf("asked again: laptop=%d anchor=%d", seats["laptop"].asks(), seats["anchor"].asks())
}
seats["laptop"].answers("notes", true, "")
advancePlans(ctx, open)
if got := read("plan-two"); got.Backups.State != inventory.BackupDone || len(*asked) != 2 {
t.Fatalf("the plan did not go on: %+v builds=%v", got.Backups, *asked)
}
if said, _ := probeBackupHeld(ctx, &doctor{open: open}); len(said) != 0 {
t.Fatalf("the condition stays: %+v", said)
}
}
+148 -356
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"os"
"regexp"
"slices"
"sort"
"strconv"
@@ -26,10 +27,9 @@ import (
// commit in it, with at most one commit per repository, the latest of its branch, which contains the earlier
// ones. The walk names every merge it answers.
//
// - One walk at a time beside a related one (ADR 0407 decision 4, issue 973). Merges that come while a walk
// runs join the next batch; a closed batch that shares nothing with an open walk is cut beside it. A walk
// that started is never superseded; one still waiting for its delivery's word is folded into the walk of a
// batch related to it, which answers every merge it answered — the only supersession left.
// - One walk at a time. Merges that come while a walk runs join the next batch. A walk that started is never
// superseded; one still waiting for its delivery's word is folded into the next batch's walk, which
// answers every merge it answered — the only supersession left.
// - A merge heard after a later merge of its branch was put in a walk is answered by that walk: a batch
// still assembling, an open walk that builds everything it moves, or a walk done that did, which then
// names it at once. Otherwise it joins the next batch.
@@ -39,9 +39,8 @@ import (
//
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, ADR 0306 decisions
// 2 and 3, issue 525): it joins a batch it is related to, and one that is related to none and whose delivery
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Such a batch is cut
// at once, beside any open walk it shares nothing with, and queued behind the first one it is related to
// (cutUnrelated).
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Walks stay one at
// a time: such a batch is cut when no walk is open, and queued behind the one that is otherwise.
//
// A merge on the controller's own path (a module whose walk waits for nobody's word) never shares a batch with
// one that waits for mesh-delivery's: each kind has a batch of its own, an own-path batch is cut first and folds
@@ -62,17 +61,16 @@ var (
// mergeWindowFileVar names the file the controller's window settings are written to.
const mergeWindowFileVar = "MESH_MERGE_WINDOW_FILE"
// batchCutEvery is how often the controller looks whether a batch's window closed.
const batchCutEvery = 5 * time.Second
// batchLateAfter is how long past its maximum a batch may still be assembling, with no walk open, before
// the controller is said to have failed to cut it (ADR 0276 decision 9).
const batchLateAfter = time.Minute
// mergeGrace is how long before a batch's first merge a merge may have been made and still be told apart from one
// made before the controller began keeping merges (owedLate).
const mergeGrace = 10 * time.Minute
// edgeChangeOrder is the order the rules of a plan draw among its modules (novox/hq ADR 0419 clauses 3 and 4, delivery_edges.go)
// made an edge of its tiers: the module after depends on the module before, and is gated on it.
const edgeChangeOrder = "change-order"
// edgeGroupOrder is a delivery group's order made an edge of a walk's tiers (ADR 0276 decision 5): the
// modules of the member after depend on the modules of the member before, and are gated on them.
const edgeGroupOrder = "group-order"
// mergeWindowOf is the window and its maximum, from the controller's settings; a value unreadable is its
// default, and said.
@@ -127,7 +125,8 @@ func durationSetting(v any) (time.Duration, bool) {
// hearMerge puts a merge the forge announced into the batch that answers it, at the moment it was heard,
// and cuts what is due.
func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.Time) error {
// One merge acted on at a time, whoever hands it over (novox/hq issue 266).
// One merge acted on at a time, whoever hands it over: the bus, or the catch-up that reads back
// what the bus did not hand over (novox/hq issue 266).
actingOnMerges.Lock()
defer actingOnMerges.Unlock()
@@ -176,12 +175,12 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
repository, m.Base, m.Commit, already)
return nil
}
fmt.Print(nothingReadsIt(repository, m))
fmt.Printf("%s merged into %s (%.8s); nothing the mesh holds reads it\n", repository, m.Base, m.Commit)
return nil
}
// **Its delivery group's word, before the plans are held** (ADR 0306 decision 2): a slow answer holds up no walk.
own := ownPath(touches)
waits, holding, err := groupWaitHolding(ctx, inv, m, deliverySeatHeld(entries))
waits, err := groupWait(ctx, inv, m, deliverySeatHeld(entries), own)
if err != nil {
return notNow(err)
}
@@ -237,7 +236,6 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
if added, err := inv.AddMerge(ctx, heard); err != nil || !added {
return notNow(err)
}
holdFor(&batch, m.Head, waits, holding, own, now)
if err := keepBatch(ctx, inv, &batch, now, ""); err != nil {
return notNow(err)
}
@@ -246,46 +244,9 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
return cutBatchesHeld(ctx, f.open, now)
}
// holdFor keeps in a batch that a merge of a group joined whether that group still has a member to merge (ADR 0418
// clause 3): held while it has, the window running again from the moment the last hold is cleared. A group with a
// merge of the controller's own path makes its batch an own-path one: such a plan waits for nobody's word, and is
// started by the controller when the group's batch closes (clause 6).
func holdFor(batch *inventory.Plan, head, why string, holding, own bool, now time.Time) {
if batch.Delivery == nil {
batch.Delivery = &inventory.PlanDelivery{}
}
if batch.Delivery.Batch == nil {
batch.Delivery.Batch = &inventory.PlanBatch{Own: own}
}
w := batch.Delivery.Batch
w.Own = w.Own || own
if head == "" {
return
}
if holding {
if w.Holds == nil {
w.Holds = map[string]string{}
}
w.Holds[head] = why
return
}
if _, held := w.Holds[head]; held {
holds := map[string]string{}
for h, y := range w.Holds {
if h != head {
holds[h] = y
}
}
w.Holds = holds
if len(holds) == 0 {
w.Holds, w.HeldAt = nil, now
}
}
}
// owedLate says a merge that reads as history is still the record's to answer (review of this change): a cut
// marks every module it moved seen at that moment, so a merge of that branch
// handed over late — made before the cut, heard after it — reads as history and was dropped. One that touches
// marks every module it moved seen at that moment, so a merge of that branch the bus lost and the catch-up
// hands over later — made before the cut, heard after it — reads as history and was dropped. One that touches
// a module the mesh holds, of a branch whose merges the controller keeps, made since it began keeping them,
// is kept; one made before it began is old news, as before.
func owedLate(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, merged time.Time,
@@ -379,23 +340,14 @@ func ownPath(touches []inventory.Entry) bool {
// names why): a new batch, which waits for the window, beside the ones it shares nothing with; its words say
// why.
// - none, and no member of its group is still to merge: a new batch whose window is closed at once. Cut at
// once means only that: with a walk open that it is related to the batch is queued behind it as any closed
// batch is (ADR 0276 decision 4, ADR 0407 decision 4); beside one it shares nothing with it is cut.
// once means only that: walks stay one at a time (ADR 0276 decision 4), so with a walk open the batch is
// queued behind it as any closed batch is.
func batchToJoin(ctx context.Context, inv *inventory.Inventory, heard inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, now time.Time, own bool, waits string) (inventory.Plan, string, error) {
edges, err := inv.Dependencies(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
// **A group is one batch, whatever kinds its members are** (novox/hq ADR 0418 clause 6): a merge joins the batch
// of the other kind that holds a merge from its head.
if group, ok, err := batchOfGroup(ctx, inv, announcedOf(heard).Head, heard.Repository); err != nil {
return inventory.Plan{}, "", err
} else if ok && group.OwnPath() != own && group.Delivery != nil && group.Delivery.Batch != nil {
group.Delivery.Batch.CutAtOnce = false
return group, "its group " + announcedOf(heard).Head + " has a merge in " + group.ID + ", a batch of the other kind: " +
"a group is one batch", nil
}
joins, whys, _, err := relatedBatches(ctx, inv, []inventory.BatchedMerge{heard}, own, "", entries, read, edges)
if err != nil {
return inventory.Plan{}, "", err
@@ -454,9 +406,7 @@ func relatedBatches(ctx context.Context, inv *inventory.Inventory, merges []inve
}
carry := newFootprint()
if !own {
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch folds those it is related to
// (cutUnrelated). Every one is counted here, which only makes a merge join a batch more readily: it waits
// for the window, never walks beside what it shares something with (issue 973).
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch may fold every one.
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, nil, nil, err
@@ -594,11 +544,6 @@ func newBatch(ctx context.Context, inv *inventory.Inventory, first inventory.Bat
// keepBatch writes a batch as its merges and the clock say it now: the commits it holds, the merges they
// answer, its window, and assembling or queued behind a walk. Saved only when that changed.
func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind string) error {
return keepBatchBecause(ctx, inv, b, now, behind, "")
}
// keepBatchBecause is keepBatch for a batch queued behind a walk it is related to, which its words say (issue 973).
func keepBatchBecause(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind, why string) error {
merges, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return err
@@ -619,42 +564,29 @@ func keepBatchBecause(ctx context.Context, inv *inventory.Inventory, b *inventor
if b.Delivery == nil {
b.Delivery = &inventory.PlanDelivery{}
}
anchor := b.Created
if was := b.Delivery.Batch; was != nil && was.HeldAt.After(anchor) {
anchor = was.HeldAt // the maximum counts from the last hold's end (ADR 0418 clause 3)
}
closes, latest := windowOf(merges, anchor, window, atMost)
closes, latest := windowOf(merges, b.Created, window, atMost)
b.Delivery.Merges = named
var closed, cutAtOnce bool
var withheld []string
var waits string
var holds map[string]string
var heldAt time.Time
if was := b.Delivery.Batch; was != nil {
holds, heldAt = was.Holds, was.HeldAt
closed, withheld = was.Closed, was.Withheld // a stop's, which no look takes back (novox/hq ADR 0299)
cutAtOnce = was.CutAtOnce // the merge's, until another joins it (ADR 0304 decision 1)
waits = was.Waits // its group's, as said when it was made (ADR 0306 decision 2)
}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, BehindWhy: why, Own: b.OwnPath(),
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits, Holds: holds, HeldAt: heldAt}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, Own: b.OwnPath(),
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits}
b.State = inventory.PlanAssembling
if behind != "" && windowClosed(*b, now) {
b.State = inventory.PlanQueued
} else {
b.Delivery.Batch.Behind, b.Delivery.Batch.BehindWhy = "", ""
b.Delivery.Batch.Behind = ""
}
b.Note = batchWords(*b, now)
if unNoted(*b) == before {
return nil
}
if err := inv.SavePlan(ctx, b); err != nil {
return err
}
// The window's clock follows the record, only when the record moved: a merge made the quiet key again, a
// batch that opened made its most (ADR 0407 decision 1).
armWindow(ctx, *b, now)
return nil
return inv.SavePlan(ctx, b)
}
// windowOf is when a batch's window closes, unless another merge comes, and when at the latest: the last
@@ -673,20 +605,13 @@ func windowOf(merges []inventory.BatchedMerge, created time.Time, window, atMost
}
// windowClosed says a batch's window is closed: closed by a stop, closed at once for a merge that shares
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached. **A batch holding a group
// with a member still to merge is closed by none of the last three** (novox/hq ADR 0418 clause 3).
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached.
func windowClosed(b inventory.Plan, now time.Time) bool {
if b.Delivery == nil || b.Delivery.Batch == nil {
return true
}
w := b.Delivery.Batch
if w.Closed {
return true
}
if len(w.Holds) > 0 {
return false // a group's last member is still to merge: neither the window nor its maximum closes the batch
}
return w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
return w.Closed || w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
}
// carriedOf is a set of merges as a walk carries them: the latest merge of each repository's branch, and
@@ -763,7 +688,7 @@ func spelledOf(m inventory.BatchedMerge) string {
func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge {
k := announcedOf(m)
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves,
Head: k.HeadSHA, Merged: m.Merged, Heard: m.Heard}
Merged: m.Merged, Heard: m.Heard}
}
// laterMerge says a was merged after b: by the forge's merge time, then by when each was heard.
@@ -814,10 +739,31 @@ func nameMerges(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan
return inv.SavePlan(ctx, p)
}
// batchCutter cuts what is due on a timer, until the context ends: a window closes with no merge to say so.
func batchCutter(ctx context.Context, open *stores) {
tick := time.NewTicker(batchCutEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
release, err := open.inventory.HoldPlans(ctx, false)
if err != nil {
continue // whoever holds the plans moves them, and cuts at the end
}
if err := cutBatchesHeld(ctx, open, time.Now().UTC()); err != nil {
fmt.Printf("batches: %v\n", err)
}
release()
}
}
// cutBatchesHeld takes the step that is due, for a caller holding the plans: after a failed walk, the search
// for the merge that brought the failure; with no walk open, the next merge of that search walked alone; else
// each closed batch cut into a walk with the waiting walks it is related to folded in, unless it is related to a
// walk open, behind which it is said queued (cutUnrelated). While the search runs everything waits.
// for the merge that brought the failure; with no walk open, the next merge of that search walked alone, or a
// closed batch cut into one walk with the waiting walks folded in; with a walk open, a closed batch said
// queued behind it.
func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
inv := open.inventory
if err := searchAfterFailures(ctx, inv); err != nil {
@@ -834,7 +780,8 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
if err != nil {
return err
}
var started, waiting []inventory.Plan
var started *inventory.Plan
var waiting []inventory.Plan
for i := range plans {
p := plans[i]
if p.Release != nil {
@@ -842,19 +789,14 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
}
if p.Waiting() {
waiting = append(waiting, p)
} else {
started = append(started, p)
} else if started == nil {
started = &plans[i]
}
}
batches, err := inv.Batches(ctx)
if err != nil {
return err
}
// **A hold is looked at again at every cut** (ADR 0418 clause 3): a group whose last member merged or closed
// since is released, and its window runs again from now.
if batches, err = releaseHolds(ctx, inv, batches, now); err != nil {
return err
}
keepAll := func(behind string) error {
for i := range batches {
if err := keepBatch(ctx, inv, &batches[i], now, behind); err != nil {
@@ -863,151 +805,46 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
}
return nil
}
if started != nil {
// **One walk at a time** (ADR 0276 decision 4): every batch waits behind it, whatever its window says.
return keepAll(started.ID)
}
if len(alone) > 0 {
// A failed walk's search walks its merges alone, one at a time: everything waits for it (kept as it was,
// issue 973).
if len(started) > 0 {
return keepAll(started[0].ID)
}
if len(waiting) > 0 {
// The waiting walk is the one open; the search walks after it, and the batches wait behind both.
return keepAll(waiting[0].ID)
}
return walkAlone(ctx, open, alone[0], now)
}
return cutUnrelated(ctx, open, batches, started, waiting, now)
}
// releaseHolds asks mesh-delivery again about each group a batch holds, and clears the holds of those with no member
// left to merge. A group mesh-delivery cannot be asked about stays held: the safe error is to wait. It answers the
// batches as kept.
func releaseHolds(ctx context.Context, inv *inventory.Inventory, batches []inventory.Plan, now time.Time) ([]inventory.Plan, error) {
var entries []inventory.Entry
for i := range batches {
b := &batches[i]
if b.Delivery == nil || b.Delivery.Batch == nil || len(b.Delivery.Batch.Holds) == 0 {
continue
}
if entries == nil {
var err error
if entries, err = inv.Catalogued(ctx); err != nil {
return nil, err
}
}
changed := false
for head := range b.Delivery.Batch.Holds {
open, isGroup, _, err := openMembersOf(ctx, inv, deliverySeatHeld(entries), head, "")
if err != nil || (isGroup && len(open) > 0) {
continue
}
holdFor(b, head, "", false, false, now)
changed = true
fmt.Printf("%s: group %s has no member left to merge; its window runs again from now\n", b.ID, head)
}
if changed {
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return nil, err
}
}
}
return batches, nil
}
// cutUnrelated cuts the closed batches that share nothing with an open walk, each into a walk beside the open
// ones, and says the others queued behind the first walk they are related to (novox/hq ADR 0407 decision 4,
// issue 973). A batch cut in this look counts as open for the batches after it.
//
// - An own-path batch goes first and folds no waiting walk; a catalogue batch folds the waiting walks it is
// related to, and those it shares nothing with keep waiting.
// - **The controller's own path is related to everything** (decided for issue 973): a walk or batch that
// replaces the controller, the node-engine, the tool runner, the bus or mesh-delivery never runs beside
// another, and nothing runs beside it.
// - A batch whose window is still open is assembling; one related to an open walk is queued behind it, and
// ADR 0276 decision 4 stands for it: nothing supersedes a walk that started.
func cutUnrelated(ctx context.Context, open *stores, batches, started, waiting []inventory.Plan, now time.Time) error {
inv := open.inventory
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
var entries []inventory.Entry
var read map[string][]inventory.ReadRepository
var edges []inventory.Edge
judged := false
openWalks, err := openWalksOf(ctx, inv, started)
if err != nil {
if err := keepAll(""); err != nil {
return err
}
// The oldest batch whose window closed is cut, an own-path one before a catalogue one (it goes first, and
// the catalogue batch queues behind it rather than being cut and overtaken). Several of a kind may be open
// since a merge that shares nothing is cut at once (ADR 0304 decision 1): walks stay one at a time, so one
// is cut per look, and once its walk started the others are queued behind it.
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
for i := range batches {
b := &batches[i]
if !windowClosed(*b, now) {
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return err
}
continue
}
if !judged {
var err error
if entries, err = inv.Catalogued(ctx); err != nil {
if b.OwnPath() {
// A walk on the controller's own path folds no walk waiting for its word (decided during the build,
// 2026-10-10): that walk keeps waiting beside it, and starts once this one ended (advanceOnce).
waiting = nil
}
if err := cutBatch(ctx, open, b, waiting, now); err != nil {
return err
}
// The batches left wait behind the walk just cut when it started; the next look does the same.
if b.Open() && !b.Waiting() {
if batches, err = inv.Batches(ctx); err != nil {
return err
}
if read, err = readForPlanning(ctx, inv); err != nil {
return err
}
if edges, err = inv.Dependencies(ctx); err != nil {
return err
}
judged = true
}
held, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return err
}
fp := footprintAsCut(held, entries, read, edges)
var fold, keep []inventory.Plan
if !b.OwnPath() {
for _, w := range waiting {
ww, err := openWalkOf(ctx, inv, w)
if err != nil {
return err
}
if ok, _ := related(fp, ww.fp, edges); ok {
fold = append(fold, w)
fp = unionFootprint(fp, ww.fp)
} else {
keep = append(keep, w)
}
}
}
if behind, why := relatedOpen(fp, b.OwnPath(), openWalks, edges); behind != "" {
if err := keepBatchBecause(ctx, inv, b, now, behind, why); err != nil {
return err
}
continue
}
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return err
}
if err := cutBatch(ctx, open, b, fold, now); err != nil {
return err
}
waiting = keep
if c := windowClockOf(); c != nil {
c.Forget(ctx, b.ID) // a batch cut has no window left to close
}
cut, err := inv.PlanByID(ctx, b.ID)
if err != nil {
return err
}
if !cut.Open() {
continue
}
w, err := openWalkOf(ctx, inv, cut)
if err != nil {
return err
}
if cut.Waiting() {
waiting = append(waiting, cut)
} else {
openWalks = append(openWalks, w)
return keepAll(b.ID)
}
return nil
}
return nil
}
@@ -1102,16 +939,11 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
commits, named := carriedOf(merges)
var moved []string
var members []batchMember
unchanged := map[string]string{}
for _, m := range combinedMerges(merges) {
// **News, whatever was seen since** (review of this change): each merge was judged when it was heard, and
// a cut, a fold or a failed walk since marks its modules seen — so a merge walked alone after a failed
// walk, or folded, would read as history and move nothing. When it was made is not asked again here.
m.MergedAt = ""
// **Unchanged, said** (novox/hq issue 611): read before the merge is acted on, from the same records.
for n, why := range unchangedOfMerge(m, entries, read) {
unchanged[n] = why
}
names, err := movesOfMerge(ctx, inv, m, entries, read)
if err != nil {
return err
@@ -1139,13 +971,9 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
}
}
sort.Strings(also)
ordered, err := changeOrderEdges(ctx, merges, members, entries, moved, edges)
if err != nil {
return err
}
ordered := groupOrderEdges(members, entries, read, edges)
var withheld []string
plan := planOfMoves(moved, append(append([]inventory.Edge{}, edges...), ordered...))
placedBy(&plan, ordered)
// **Withheld, a stopped merge's modules** (novox/hq ADR 0299): a module is built at its branch, which holds the
// merge a stop left out of the walk these merges were walked again from — built here only as a dependent, it
// would deliver that merge. One a merge of this batch moves is built: that merge carries the stopped change,
@@ -1167,22 +995,12 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
plan.Delivery.Merges = named
plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone
plan.Delivery.Withheld = withheld
plan.Delivery.Order = ordered
// **Left where it is, a module whose program did not change** (novox/hq issue 721): its check found it the same as
// the build the mesh holds.
for n, why := range leaveSameAsRunning(ctx, inv, &plan, moved, append(append([]inventory.Edge{}, edges...), ordered...)) {
unchanged[n] = why
}
plan.Delivery.Unchanged = unchangedLeft(unchanged, plan.Modules)
// **Its moments and its class** (novox/hq ADR 0282 decision 6): measured, never acted on.
plan.Times = walkTimesAtCut(*batch, plan, entries, now)
if len(moved) == 0 || len(plan.Modules) == 0 {
if len(moved) == 0 {
plan.State = inventory.PlanDone
plan.Tiers = [][]string{}
plan.Note = "moved nothing the mesh holds: nothing to walk"
if len(plan.Delivery.Unchanged) > 0 {
plan.Note = "moved nothing the mesh holds: nothing to walk; " + unchangedSaid(plan.Delivery.Unchanged)
}
if err := inv.SavePlan(ctx, &plan); err != nil {
return err
}
@@ -1199,14 +1017,7 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
}
var tiers []string
for i, t := range plan.Tiers {
var names []string
for _, n := range t {
if m := plan.Modules[n]; m != nil && m.Placed != "" {
n += " " + m.Placed
}
names = append(names, n)
}
tiers = append(tiers, fmt.Sprintf("%d: %s", i, strings.Join(names, ", ")))
tiers = append(tiers, fmt.Sprintf("%d: %s", i, strings.Join(t, ", ")))
}
var answers []string
for _, n := range named {
@@ -1224,7 +1035,7 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil
}
// Written and its first tier asked as one act on the plans (novox/hq issue 213), under the caller's hold.
if err := askTier(ctx, inv, &plan, plan.Tiers[plan.Tier]); err != nil {
if err := askTier(ctx, inv, &plan); err != nil {
return err
}
if err := inv.SavePlan(ctx, &plan); err != nil {
@@ -1234,48 +1045,6 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil
}
// placedBy writes on each module of a plan the edges of its order that put it after another module of the plan, with the
// rule that drew each (novox/hq ADR 0419 clause 5).
func placedBy(plan *inventory.Plan, ordered []inventory.Edge) {
for _, e := range ordered {
m, in := plan.Modules[e.From]
if _, before := plan.Modules[e.To]; !in || !before {
continue
}
said := "after " + e.To + " (" + e.Why + ")"
if m.Placed == "" {
m.Placed = said
} else if !strings.Contains(m.Placed, said) {
m.Placed += "; " + said
}
}
}
// unchangedSaid names the modules a walk says matched what runs, for its note.
func unchangedSaid(unchanged map[string]string) string {
names := make([]string, 0, len(unchanged))
for n := range unchanged {
names = append(names, n)
}
sort.Strings(names)
return "unchanged, the build the mesh holds is theirs: " + strings.Join(names, ", ")
}
// unchangedLeft is what a walk says matched what runs: every module a merge left unchanged that the walk does not
// build anyway, as another merge's module or a dependent (novox/hq issue 611). Nil when none is.
func unchangedLeft(unchanged map[string]string, builds map[string]*inventory.PlanModule) map[string]string {
out := map[string]string{}
for n, why := range unchanged {
if _, built := builds[n]; !built {
out[n] = why
}
}
if len(out) == 0 {
return nil
}
return out
}
// withhold takes from a plan every module named that its merges do not move, and the tiers left empty: the
// modules taken.
func withhold(plan *inventory.Plan, names, moved []string) []string {
@@ -1439,46 +1208,73 @@ type batchMember struct {
moved []string
}
// changeOrderEdges is the order of a batch's moved modules as edges of its plan's tiers (novox/hq ADR 0419): drawn
// per module from the module graph and from what the merges say of the manifests they touch (delivery_edges.go),
// whatever repository the merges are in and whether or not they form a delivery group. No `after:` line is read.
// A cycle is said, never lost: the plan puts the modules left in its last tier, and the composed check fails the
// group before it merges.
func changeOrderEdges(ctx context.Context, merges []inventory.BatchedMerge, members []batchMember,
entries []inventory.Entry, moved []string, edges []inventory.Edge) ([]inventory.Edge, error) {
// What each repository's branch held before the batch: the first parent of its earliest merge in it.
earliest := map[string]inventory.BatchedMerge{}
for _, k := range merges {
key := strings.ToLower(k.Repository) + "\x00" + k.Branch
if e, ok := earliest[key]; !ok || laterMerge(e, k) {
earliest[key] = k
}
}
var sources []changeSource
// afterLines are the `after: <repository>` lines of a pull request's description, as mesh-delivery reads them.
var afterLines = regexp.MustCompile(`(?im)^\s*after:\s*([A-Za-z0-9_.\-/]+)\s*$`)
// groupOrderEdges is the order of every delivery group among a batch's merges as edges of its walk's tiers
// (ADR 0276 decision 5, ADR 0249 unchanged): merges of two repositories or more from one head branch name are
// a group; each pair its order rules give puts the modules of the member after in a later tier than the
// member before's. A pair in a contradiction orders nothing: its group was refused before it merged.
func groupOrderEdges(members []batchMember, entries []inventory.Entry, read map[string][]inventory.ReadRepository,
edges []inventory.Edge) []inventory.Edge {
byHead := map[string][]batchMember{}
for _, m := range members {
s := changeSource{Owner: m.merge.Owner, Repo: m.merge.Repo, New: m.merge.Commit,
Paths: m.merge.Paths, Truncated: m.merge.PathsTruncated, Removed: m.merge.Removed}
if e, ok := earliest[strings.ToLower(m.merge.Owner+"/"+m.merge.Repo)+"\x00"+m.merge.Base]; ok {
s.Since = e.Commit
if m.merge.Head != "" {
byHead[m.merge.Head] = append(byHead[m.merge.Head], m)
}
if s.Truncated {
s.Paths, s.Truncated = truncatedPaths(entries, s.Owner, s.Repo, moved), false
}
var out []inventory.Edge
heads := make([]string, 0, len(byHead))
for h := range byHead {
heads = append(heads, h)
}
sort.Strings(heads)
for _, h := range heads {
group := byHead[h]
repos := map[string]bool{}
for _, m := range group {
repos[strings.ToLower(m.merge.Owner+"/"+m.merge.Repo)] = true
}
if len(repos) < 2 {
continue
}
var om []orderMember
moved := map[string][]string{}
for _, m := range group {
repository := m.merge.Owner + "/" + m.merge.Repo
id := repository + "@" + m.merge.Commit
var after []string
for _, l := range afterLines.FindAllStringSubmatch(m.merge.Body, -1) {
after = append(after, strings.TrimSuffix(l[1], ".git"))
}
om = append(om, orderMember{ID: id, Repository: repository, Base: m.merge.Base, Head: m.merge.Commit,
Number: m.merge.Number, Paths: m.merge.Paths, PathsTruncated: m.merge.PathsTruncated,
Removed: m.merge.Removed, ModuleDirs: m.merge.ModuleDirs, ModuleDirsSaid: m.merge.ModuleDirsSaid,
CloneURL: m.merge.CloneURL, After: after})
moved[id] = m.moved
}
order := orderOf(om, reachOfMembers(om, entries, read, edges), edges)
contradicted := map[string]bool{}
for _, c := range order.Contradictions {
for _, id := range c.Members {
contradicted[id] = true
}
}
for _, p := range order.Pairs {
if contradicted[p.Before] && contradicted[p.After] {
continue
}
for _, before := range moved[p.Before] {
for _, after := range moved[p.After] {
if before != after {
out = append(out, inventory.Edge{From: after, To: before, Kind: edgeGroupOrder})
}
}
}
fmt.Printf(" group %s: %s before %s (%s) inside the walk\n", h, p.Before, p.After, p.Why)
}
sources = append(sources, s)
}
changes, err := changesOf(ctx, sources)
if err != nil {
return nil, fmt.Errorf("what the merges say of their manifests cannot be read, so the order of their modules "+
"cannot be drawn: %w", err)
}
order := orderEdges(moved, changes, edges)
for _, e := range order.Edges {
fmt.Printf(" order: %s\n", edgeSaid(e))
}
if said := order.CycleSaid(); said != "" {
fmt.Printf(" %s\n", said)
}
return order.Edges, nil
return out
}
// searchAfterFailures starts the search a failed walk leaves (ADR 0276 decision 3): every merge it answered
@@ -1561,11 +1357,7 @@ func batchWords(b inventory.Plan, now time.Time) string {
grouped += " (on the controller's own path)"
}
if b.State == inventory.PlanQueued {
because := ""
if w.BehindWhy != "" {
because = " (" + w.BehindWhy + ")"
}
return fmt.Sprintf("queued behind %s%s; grouped: %s; plan not yet calculated", w.Behind, because, grouped)
return fmt.Sprintf("queued behind %s; grouped: %s; plan not yet calculated", w.Behind, grouped)
}
if w.CutAtOnce {
return fmt.Sprintf("cut at once, sharing nothing with a batch assembling; grouped: %s; plan not yet calculated",
@@ -1,698 +0,0 @@
//go:build integration
package main
import (
"bytes"
"io"
"os"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func windowed(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []struct{ module, repository, path string }{
{"app", "novox/mesh-catalog", "modules/app"},
{"notes", "novox/mesh-catalog", "modules/notes"},
{"one", "novox/one", ""},
{"two", "novox/two", ""},
{"three", "novox/three", ""},
} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1"},
inventory.Source{Repository: m.repository, Seat: "git", Path: m.path, Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
}
return open
}
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
if err := (following{open: open}).hearMerge(t.Context(), m, now); err != nil {
t.Fatal(err)
}
}
func cutAt(t *testing.T, open *stores, now time.Time) {
t.Helper()
if err := cutBatchesHeld(t.Context(), open, now); err != nil {
t.Fatal(err)
}
}
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, claude, t0.Add(time.Second))
hear(t, open, dunst, t0.Add(18*time.Second))
if ws, bs := walks(t, open); len(ws) != 0 || len(bs) != 1 {
t.Fatalf("within the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
cutAt(t, open, t0.Add(18*time.Second+89*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window closed: %+v", ws)
}
cutAt(t, open, t0.Add(18*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if w.Commit != dunst.Commit || len(w.Commits) != 1 || w.Commits[0].Commit != dunst.Commit {
t.Fatalf("the walk is at %s %+v, not the later commit", w.Commit, w.Commits)
}
for _, m := range []string{"app", "notes"} {
if _, in := w.Modules[m]; !in {
t.Fatalf("the walk does not build %s, which one of its merges moved: %v", m, w.Modules)
}
}
want := []inventory.PlanMerge{{Repository: "novox/mesh-catalog", Commit: claude.Commit, Carried: dunst.Commit},
{Repository: "novox/mesh-catalog", Commit: dunst.Commit}}
if w.Delivery == nil || !slices.EqualFunc(w.Delivery.Merges, want, sameMerge) {
t.Fatalf("the walk answers %+v, want %+v", w.Delivery, want)
}
if len(*asked) != 2 || (*asked)[0][2] != "main" || (*asked)[1][2] != "main" {
t.Fatalf("the walk did not ask its modules at the branch, which holds the commit it carries: %v", *asked)
}
// Heard again, as the bus may hand it over twice: never a second merge, nor a second walk.
hear(t, open, claude, t0.Add(5*time.Minute))
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 {
t.Fatalf("a merge heard twice made %d walk(s) and %d batch(es)", len(ws), len(bs))
}
}
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if len(w.Commits) != 3 || len(w.Delivery.Merges) != 3 {
t.Fatalf("the walk carries %+v and answers %+v", w.Commits, w.Delivery.Merges)
}
for _, r := range []string{"one", "two", "three"} {
if w.CommitOf("novox/"+r) != "c-"+r {
t.Fatalf("the walk carries %s at %q", r, w.CommitOf("novox/"+r))
}
if _, in := w.Modules[r]; !in {
t.Fatalf("the walk does not build %s: %v", r, w.Modules)
}
}
// Each module's ask is recorded at its own repository's commit.
for _, r := range []string{"one", "two", "three"} {
q, found, err := open.inventory.BuildRequestByID(t.Context(), w.Modules[r].Build)
if err != nil || !found || q.Commit != "c-"+r {
t.Fatalf("%s's build was not recorded at its own commit: %+v %v %v", r, q, found, err)
}
}
}
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
}
cutAt(t, open, t0.Add(170*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || len(ws[0].Commits) != 2 {
t.Fatalf("the window did not close 90 seconds after its last merge: %+v", ws)
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
}
break
}
}
ws, _ := walks(t, busy)
if len(ws) != 1 {
t.Fatalf("ten minutes of merges a minute apart were never cut: %d walk(s)", len(ws))
}
}
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
claude.Number, claude.Title = 174, "claude-code: an agent proposes a section"
dunst := catalogueMerge("48bda475dunst", "notes", now.Add(-2*time.Second))
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
for _, want := range []string{"assembling: ", " s left (at the latest ", "grouped: novox/mesh-catalog@48bda475 " +
"(answers 553b7191), novox/one@a6bc0931; plan not yet calculated"} {
if !strings.Contains(first, want) {
t.Fatalf("plans' first line %q does not say %q", first, want)
}
}
// Under it, one line per repository: the pull request, what it answers, what it moves.
if len(lines) < 3 || strings.TrimSpace(lines[1]) != "mesh-catalog #175 dunst: the font the operator chose (answers "+
"#174 claude-code: an agent proposes a section) · app, notes" ||
strings.TrimSpace(lines[2]) != "one a6bc0931 · one" {
t.Fatalf("the grouped list reads %q", lines[1:4])
}
if len(said) == 0 || said[len(said)-1].State != inventory.PlanAssembling || said[len(said)-1].Delivery.Batch == nil ||
len(said[len(said)-1].Delivery.Merges) != 3 {
t.Fatalf("the batch was not said as assembling with its merges: %+v", said)
}
}
func captured(t *testing.T, run func() error) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
runErr := run()
os.Stdout = was
_ = w.Close()
var b bytes.Buffer
_, _ = io.Copy(&b, r)
if runErr != nil {
t.Fatal(runErr)
}
return b.String()
}
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first batch was not cut: %+v", ws)
}
first := ws[0]
hear(t, open, repoMerge("one", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(5*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
t.Fatalf("the merge during the walk: %d walk(s), batches %+v", len(ws), bs)
}
if got, _ := open.inventory.PlanByID(t.Context(), first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the next merge", got.State)
}
if !strings.HasPrefix(batchWords(bs[0], t0.Add(5*time.Minute)), "queued behind "+first.ID) {
t.Fatalf("a queued batch reads %q", batchWords(bs[0], t0.Add(5*time.Minute)))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(6*time.Minute))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut when the walk ended: %+v %+v", ws, bs)
}
}
func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
waiting := ws[0]
hear(t, open, repoMerge("one", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
ws, bs := walks(t, open)
if len(bs) != 0 || len(ws) != 2 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
folded, err := open.inventory.PlanByID(ctx, waiting.ID)
if err != nil {
t.Fatal(err)
}
newer := ws[0]
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
}
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c2" {
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
}
if _, in := newer.Modules["one"]; !in {
t.Fatalf("what the folded walk was to build is not built: %v", newer.Modules)
}
}
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, later, t0.Add(18*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
// Moves only notes, which the done walk built from the branch after it: answered there, at once.
hear(t, open, catalogueMerge("553b7191early", "notes", t0), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if got.State != inventory.PlanDone || len(got.Delivery.Merges) != 2 ||
!sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: "553b7191early",
Carried: later.Commit}) {
t.Fatalf("the late merge is not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// Moves app, which that walk never built: the next batch walks it.
hear(t, open, catalogueMerge("1111aaaaapp", "app", t0.Add(time.Second)), t0.Add(16*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "1111aaaaapp" {
t.Fatalf("a late merge moving what the walk never built did not join the next batch: %+v", bs)
}
}
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
oldest := catalogueMerge("aaaa0001", "app", t0.Add(-3*time.Minute))
middle := catalogueMerge("bbbb0002", "app", t0.Add(-3*time.Minute+10*time.Second))
newest := catalogueMerge("cccc0003", "notes", t0.Add(-3*time.Minute+20*time.Second))
for i, m := range []link.SourceMoved{oldest, middle, newest} {
hear(t, open, m, t0.Add(time.Duration(i)*10*time.Second+time.Second))
}
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
failed := ws[0]
failed.State, failed.Note = inventory.PlanFailed, "notes failed to build in tier 0"
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, failed.ID)
if got.State != inventory.PlanFailed || len(got.Delivery.Merges) != 3 {
t.Fatalf("the failed walk's record changed: %s %+v", got.State, got.Delivery.Merges)
}
ws, _ = walks(t, open)
alone := ws[0]
if alone.ID == failed.ID || alone.Commit != middle.Commit || len(alone.Delivery.Merges) != 1 ||
!sameMerge(alone.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: middle.Commit}) {
t.Fatalf("the newest earlier merge was not walked alone on its own commit: %+v", alone)
}
if _, in := alone.Modules["app"]; !in || (*asked)[len(*asked)-1] != [3]string{"novox/mesh-catalog", "modules/app",
middle.Commit} {
t.Fatalf("the merge walked alone does not build app at its own commit: %v, asked %v", alone.Modules, *asked)
}
// Retried, the failed walk would name merges the search answers now.
if _, err := retryPlan(ctx, open, failed.ID); err == nil || !strings.Contains(err.Error(), "walked alone") {
t.Fatalf("a failed walk whose merges are searched was retried: %v", err)
}
alone.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &alone); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(4*time.Minute))
ws, bs := walks(t, open)
if ws[0].ID != alone.ID || len(bs) != 0 {
t.Fatalf("the search went on after a merge was delivered: %+v", ws[0])
}
done, _ := open.inventory.PlanByID(ctx, alone.ID)
if len(done.Delivery.Merges) != 2 || !sameMerge(done.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog",
Commit: oldest.Commit, Carried: middle.Commit}) {
t.Fatalf("the oldest merge is not answered by the walk that delivered the one after it: %+v", done.Delivery.Merges)
}
// A stopped walk starts no search: a person ended it.
hear(t, open, catalogueMerge("dddd0004", "app", t0.Add(5*time.Minute)), t0.Add(5*time.Minute))
hear(t, open, catalogueMerge("eeee0005", "notes", t0.Add(5*time.Minute+time.Second)), t0.Add(5*time.Minute+time.Second))
cutAt(t, open, t0.Add(8*time.Minute))
ws, _ = walks(t, open)
if _, err := stopWalk(ctx, open.inventory, ws[0].ID, "mesh-delivery for jochen", "not now"); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(9*time.Minute))
if again, _ := walks(t, open); again[0].ID != ws[0].ID {
t.Fatalf("a stopped walk's earlier merge was walked alone: %+v", again[0])
}
}
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1"},
inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
host := repoMerge("mesh-host", "h1", t0)
ctl := repoMerge("mesh-controller", "k1", t0.Add(time.Second))
host.Head, ctl.Head = "feat/together", "feat/together"
if !grouped {
ctl.Head = "feat/alone"
}
hear(t, open, ctl, t0.Add(2*time.Second))
hear(t, open, host, t0.Add(3*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walks", len(ws))
}
tiers := ws[0].Tiers
if grouped {
if len(tiers) != 2 || !slices.Equal(tiers[0], []string{"mesh-host"}) ||
!slices.Equal(tiers[1], []string{"mesh-controller"}) {
t.Fatalf("the group's order is not the walk's tiers: %v", tiers)
}
} else if len(tiers) != 1 {
t.Fatalf("two merges of no group were ordered: %v", tiers)
}
}
}
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
}
cutAt(t, again, t0.Add(120*time.Second))
ws, bs := walks(t, again)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller did not cut the batch at its window: %+v %+v", ws, bs)
}
}
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
hear(t, open, c2, t0)
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0.Add(-60*time.Second)), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the late merge was not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// A walk of two that built only two: a late merge of one is not its to name.
hear(t, open, repoMerge("two", "d2", t0.Add(16*time.Minute)), t0.Add(16*time.Minute))
cutAt(t, open, t0.Add(18*time.Minute))
ws, _ = walks(t, open)
w = ws[0]
delete(w.Modules, "two")
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("two", "d1", t0.Add(15*time.Minute)), t0.Add(20*time.Minute))
if got, _ := open.inventory.PlanByID(ctx, w.ID); len(got.Delivery.Merges) != 1 {
t.Fatalf("a walk that never built what the late merge moves named it: %+v", got.Delivery.Merges)
}
if _, bs := walks(t, open); len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "d1" {
t.Fatalf("the late merge did not join the next batch: %+v", bs)
}
}
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(2*time.Minute))
started, _ := walks(t, open)
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/one", Branch: "main", Commit: "d1", Created: t0,
State: inventory.PlanBuilding, Tiers: [][]string{{"one"}}, Modules: map[string]*inventory.PlanModule{"one": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if _, err := letGo(ctx, open.inventory, waiting.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatalf("the word was refused beside %s: %v", started[0].ID, err)
}
before := len(*asked)
advanceHeld(ctx, open)
got, _ := open.inventory.PlanByID(ctx, waiting.ID)
if len(*asked) != before || !strings.Contains(got.Note, "starts once "+started[0].ID) {
t.Fatalf("a walk let go beside a started one asked (%d → %d) or does not say it waits: %q", before, len(*asked), got.Note)
}
// Read as a wait, an hour on: its note on the line, never LATE, and no tier of it late for S3.
later := time.Now().Add(time.Hour)
if line := planLine(got, later); !strings.Contains(line, "starts once "+started[0].ID) || strings.Contains(line, "LATE") {
t.Fatalf("a deferred walk reads %q", line)
}
facts, _, err := gatherPlans(ctx, open.inventory, later, nil)
if err != nil {
t.Fatal(err)
}
for _, f := range facts {
if f.id == got.ID {
t.Fatalf("a deferred walk is watched as a tier running late: %+v", f)
}
}
done := started[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) != before+1 {
t.Fatalf("the walk did not start once the started one ended: asked %v", *asked)
}
}
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("aaaa0001", "app", t0), t0)
hear(t, open, repoMerge("mesh-controller", "k1", t0.Add(10*time.Second)), t0.Add(10*time.Second))
hear(t, open, catalogueMerge("bbbb0002", "notes", t0.Add(20*time.Second)), t0.Add(20*time.Second))
_, bs := walks(t, open)
var ownBatch, catalogueBatch inventory.Plan
for _, b := range bs {
if b.OwnPath() {
ownBatch = b
} else {
catalogueBatch = b
}
}
if len(bs) != 2 || ownBatch.ID == "" || catalogueBatch.ID == "" || len(catalogueBatch.Delivery.Merges) != 2 ||
len(ownBatch.Delivery.Merges) != 1 {
t.Fatalf("a controller merge and two catalogue merges in one window: %+v", bs)
}
if !strings.Contains(batchWords(ownBatch, t0.Add(30*time.Second)), "own path") {
t.Fatalf("the own-path batch does not say so: %q", batchWords(ownBatch, t0.Add(30*time.Second)))
}
// Both windows closed, the controller's batch is cut first and starts; the catalogue batch queues behind it,
// is cut when the controller's walk ended, and waits for its word with both merges.
cutAt(t, open, t0.Add(2*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the controller's batch was not cut first into a started walk: %+v", ws)
}
ownWalk := ws[0]
for _, m := range []string{"app", "notes"} {
if _, in := ownWalk.Modules[m]; in {
t.Fatalf("the controller's walk builds %s, a catalogue module: it skipped its turn", m)
}
}
if len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != ownWalk.ID || bs[0].OwnPath() {
t.Fatalf("the catalogue batch does not queue behind the controller's walk: %+v", bs)
}
ownWalk.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &ownWalk); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
ws, bs = walks(t, open)
if len(bs) != 0 || ws[0].ID != catalogueBatch.ID || !ws[0].Waiting() || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the catalogue batch was not cut into a waiting walk once the controller's ended: %+v %+v", ws, bs)
}
for _, m := range []string{"app", "notes"} {
if _, in := ws[0].Modules[m]; !in {
t.Fatalf("the catalogue walk does not build %s: %v", m, ws[0].Modules)
}
}
for _, a := range *asked {
if a[1] == "modules/app" || a[1] == "modules/notes" {
t.Fatalf("a catalogue module was asked without the word: %v", *asked)
}
}
// A catalogue walk waiting for its word when an own-path batch is cut keeps waiting beside the own-path
// walk, is not folded into it, and its word is taken meanwhile: it starts once the own-path walk ended.
catalogueWalk := ws[0]
hear(t, open, repoMerge("mesh-controller", "k2", t0.Add(4*time.Minute)), t0.Add(4*time.Minute))
cutAt(t, open, t0.Add(6*time.Minute))
ws, _ = walks(t, open)
kept, _ := open.inventory.PlanByID(ctx, catalogueWalk.ID)
if !kept.Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k2" || ws[0].Waiting() {
t.Fatalf("the waiting catalogue walk was not kept waiting beside the controller's walk: %s %+v", kept.State, ws)
}
if _, in := ws[0].Modules["app"]; in {
t.Fatalf("the controller's walk folded the waiting catalogue walk in: %v", ws[0].Modules)
}
if _, err := letGo(ctx, open.inventory, catalogueWalk.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatal(err)
}
before := len(*asked)
advanceHeld(ctx, open)
if len(*asked) != before {
t.Fatalf("the catalogue walk started beside the controller's: asked %v", (*asked)[before:])
}
own2 := ws[0]
own2.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &own2); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) < before+1 || (*asked)[before][1] != "modules/app" {
t.Fatalf("the catalogue walk did not start once the controller's ended: %v", (*asked)[before:])
}
}
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
module string
claims []catalogue.Claim
}{
{"nats", nil},
{"mesh-delivery", []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1", Claims: m.claims},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/" + m.module, Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
mixed := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "m1xed", MergedAt: t0.Format(time.RFC3339Nano),
Paths: []string{"modules/nats/module.json", "modules/app/module.json"}, ModuleDirs: []string{"modules/nats", "modules/app"},
ModuleDirsSaid: true}
hear(t, open, mixed, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() {
t.Fatalf("the mixed merge's walk waited, or was not cut: %+v", ws)
}
done := ws[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("ear1ier", "app", t0.Add(-30*time.Second)), t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, done.ID)
_, bs := walks(t, open)
if len(got.Delivery.Merges) != 1 || len(bs) != 1 || bs[0].OwnPath() || bs[0].Delivery.Merges[0].Commit != "ear1ier" {
t.Fatalf("the late catalogue merge was answered by the own-path walk (%+v) rather than the next catalogue batch (%+v)",
got.Delivery.Merges, bs)
}
}
+719 -46
View File
@@ -1,9 +1,10 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"os"
"slices"
"strings"
@@ -18,19 +19,42 @@ import (
// The suite's tests of a merge, but for the merge window's own (this file's), plan the merge as it is heard: a
// window of nothing closes at once, so a merge with no walk open is cut into its walk at once, as every merge
// was planned before novox/hq ADR 0276. The window's tests set it through the controller's settings.
func init() {
mergeWindowDefault = 0
// No forge in a test: no manifest is found at any commit, so a merge draws no edge of the manifests' change
// (delivery_edges.go). A test that wants some replaces these (forgeHolding).
manifestAt = func(context.Context, string, string, string, string) (catalogue.Manifest, bool, error) {
return catalogue.Manifest{}, false, nil
}
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
}
func init() { mergeWindowDefault = 0 }
// windowed is a mesh whose controller's settings give a merge window of 90 seconds and at most 10 minutes, as its
// settings file says them, and
// modules built from the catalogue (app, notes) and from three repositories of their own.
func windowed(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []struct{ module, repository, path string }{
{"app", "novox/mesh-catalog", "modules/app"},
{"notes", "novox/mesh-catalog", "modules/notes"},
{"one", "novox/one", ""},
{"two", "novox/two", ""},
{"three", "novox/three", ""},
} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1"},
inventory.Source{Repository: m.repository, Seat: "git", Path: m.path, Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
}
return open
}
// t0 is the moment a test's clock starts: merges are dated by it, and heard after it.
var t0 = time.Now().UTC().Truncate(time.Second)
@@ -59,16 +83,15 @@ func noGroups(t *testing.T) {
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupsWaiting stubs mesh-delivery's word on groups for a test of the merge window's own timing: each head named
// is one whose group mesh-delivery cannot say, so a merge from it waits for the window (ADR 0306 decision 2) and
// nothing holds the batch open; every other head names no group. A group with a member still to merge, which holds
// its batch open however long it takes (ADR 0418 clause 3), is groupStates'.
// groupsWaiting stubs mesh-delivery's word on groups for a test: each head named is a group with a member of another
// repository whose pull request is open (novox/elsewhere, ready), so a merge from it waits for the window (ADR 0306
// decision 2); every other head names no group.
func groupsWaiting(t *testing.T, heads ...string) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, head string) ([]groupMember, bool, error) {
if slices.Contains(heads, head) {
return nil, false, errors.New("the test's mesh-delivery does not say")
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
return nil, false, nil
}
@@ -76,37 +99,12 @@ func groupsWaiting(t *testing.T, heads ...string) {
}
// everyHeadWaits stubs mesh-delivery's word on groups for a test whose merges each wait for the window as they did
// before novox/hq ADR 0304: mesh-delivery cannot say any head's group.
// before novox/hq ADR 0304: every head is a group with a member of another repository still to merge.
func everyHeadWaits(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return nil, false, errors.New("the test's mesh-delivery does not say")
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupStates stubs mesh-delivery's word on one group: its members and their states, which a test changes as the
// members merge.
func groupStates(t *testing.T, head string, states map[string]string) {
t.Helper()
groupStatesFrom(t, head, states, true)
}
// groupStatesFrom is groupStates for a mesh-delivery that does (composed) or does not yet say the group's gate is the
// composed check's: only the first holds the group's batch open for its last member.
func groupStatesFrom(t *testing.T, head string, states map[string]string, composed bool) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, asked string) ([]groupMember, bool, error) {
if asked != head {
return nil, false, nil
}
var members []groupMember
for repository, state := range states {
members = append(members, groupMember{Repository: repository, State: state, Composed: composed})
}
return members, true, nil
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
@@ -126,8 +124,20 @@ func lone(m link.SourceMoved) link.SourceMoved {
}
// hear is a merge heard at a moment.
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
if err := (following{open: open}).hearMerge(t.Context(), m, now); err != nil {
t.Fatal(err)
}
}
// cutAt is the cutter looking at a moment.
func cutAt(t *testing.T, open *stores, now time.Time) {
t.Helper()
if err := cutBatchesHeld(t.Context(), open, now); err != nil {
t.Fatal(err)
}
}
// walks is every plan record that is a walk, newest first, and the batches not yet cut.
func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
@@ -148,34 +158,426 @@ func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
// The replay of issue 362: two catalogue merges 18 seconds apart are one batch, cut once into one walk at the
// later commit, which names both merges — the earlier carried by the later.
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, claude, t0.Add(time.Second))
hear(t, open, dunst, t0.Add(18*time.Second))
if ws, bs := walks(t, open); len(ws) != 0 || len(bs) != 1 {
t.Fatalf("within the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
cutAt(t, open, t0.Add(18*time.Second+89*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window closed: %+v", ws)
}
cutAt(t, open, t0.Add(18*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if w.Commit != dunst.Commit || len(w.Commits) != 1 || w.Commits[0].Commit != dunst.Commit {
t.Fatalf("the walk is at %s %+v, not the later commit", w.Commit, w.Commits)
}
for _, m := range []string{"app", "notes"} {
if _, in := w.Modules[m]; !in {
t.Fatalf("the walk does not build %s, which one of its merges moved: %v", m, w.Modules)
}
}
want := []inventory.PlanMerge{{Repository: "novox/mesh-catalog", Commit: claude.Commit, Carried: dunst.Commit},
{Repository: "novox/mesh-catalog", Commit: dunst.Commit}}
if w.Delivery == nil || !slices.EqualFunc(w.Delivery.Merges, want, sameMerge) {
t.Fatalf("the walk answers %+v, want %+v", w.Delivery, want)
}
if len(*asked) != 2 || (*asked)[0][2] != "main" || (*asked)[1][2] != "main" {
t.Fatalf("the walk did not ask its modules at the branch, which holds the commit it carries: %v", *asked)
}
// Heard again, as the bus may hand it over twice: never a second merge, nor a second walk.
hear(t, open, claude, t0.Add(5*time.Minute))
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 {
t.Fatalf("a merge heard twice made %d walk(s) and %d batch(es)", len(ws), len(bs))
}
}
// Merges of three repositories in one window are one walk, one commit for each.
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if len(w.Commits) != 3 || len(w.Delivery.Merges) != 3 {
t.Fatalf("the walk carries %+v and answers %+v", w.Commits, w.Delivery.Merges)
}
for _, r := range []string{"one", "two", "three"} {
if w.CommitOf("novox/"+r) != "c-"+r {
t.Fatalf("the walk carries %s at %q", r, w.CommitOf("novox/"+r))
}
if _, in := w.Modules[r]; !in {
t.Fatalf("the walk does not build %s: %v", r, w.Modules)
}
}
// Each module's ask is recorded at its own repository's commit.
for _, r := range []string{"one", "two", "three"} {
q, found, err := open.inventory.BuildRequestByID(t.Context(), w.Modules[r].Build)
if err != nil || !found || q.Commit != "c-"+r {
t.Fatalf("%s's build was not recorded at its own commit: %+v %v %v", r, q, found, err)
}
}
}
// Each merge restarts the window: a merge at second 80 keeps the batch open until second 170; the window
// closes at most ten minutes after its first merge however busy.
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
}
cutAt(t, open, t0.Add(170*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || len(ws[0].Commits) != 2 {
t.Fatalf("the window did not close 90 seconds after its last merge: %+v", ws)
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
}
break
}
}
ws, _ := walks(t, busy)
if len(ws) != 1 {
t.Fatalf("ten minutes of merges a minute apart were never cut: %d walk(s)", len(ws))
}
}
// `plans` lists the batch being assembled first, in the operator's words: how long is left, when at the latest,
// what is grouped, and that the plan is not yet calculated. It says so on the bus as plan-moved, too.
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
claude.Number, claude.Title = 174, "claude-code: an agent proposes a section"
dunst := catalogueMerge("48bda475dunst", "notes", now.Add(-2*time.Second))
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
for _, want := range []string{"assembling: ", " s left (at the latest ", "grouped: novox/mesh-catalog@48bda475 " +
"(answers 553b7191), novox/one@a6bc0931; plan not yet calculated"} {
if !strings.Contains(first, want) {
t.Fatalf("plans' first line %q does not say %q", first, want)
}
}
// Under it, one line per repository: the pull request, what it answers, what it moves.
if len(lines) < 3 || strings.TrimSpace(lines[1]) != "mesh-catalog #175 dunst: the font the operator chose (answers "+
"#174 claude-code: an agent proposes a section) · app, notes" ||
strings.TrimSpace(lines[2]) != "one a6bc0931 · one" {
t.Fatalf("the grouped list reads %q", lines[1:4])
}
if len(said) == 0 || said[len(said)-1].State != inventory.PlanAssembling || said[len(said)-1].Delivery.Batch == nil ||
len(said[len(said)-1].Delivery.Merges) != 3 {
t.Fatalf("the batch was not said as assembling with its merges: %+v", said)
}
}
// captured is what a command printed.
func captured(t *testing.T, run func() error) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
runErr := run()
os.Stdout = was
_ = w.Close()
var b bytes.Buffer
_, _ = io.Copy(&b, r)
if runErr != nil {
t.Fatal(runErr)
}
return b.String()
}
// A merge heard during a walk it is related to (the same repository, issue 973) joins the next batch, which is
// cut when the walk ends; the walk that started is never superseded (ADR 0276 decision 4, ADR 0407 decision 4).
// One walk at a time: a merge heard while a walk runs joins the next batch, which is cut when the walk ends;
// the walk that started is never superseded.
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first batch was not cut: %+v", ws)
}
first := ws[0]
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(5*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
t.Fatalf("the merge during the walk: %d walk(s), batches %+v", len(ws), bs)
}
if got, _ := open.inventory.PlanByID(t.Context(), first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the next merge", got.State)
}
if !strings.HasPrefix(batchWords(bs[0], t0.Add(5*time.Minute)), "queued behind "+first.ID) {
t.Fatalf("a queued batch reads %q", batchWords(bs[0], t0.Add(5*time.Minute)))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(6*time.Minute))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut when the walk ended: %+v %+v", ws, bs)
}
}
// A walk waiting for its delivery's word is folded into the next batch's walk, which answers its merges; it
// names the walk that took it over.
func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
waiting := ws[0]
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
ws, bs := walks(t, open)
if len(bs) != 0 || len(ws) != 2 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
folded, err := open.inventory.PlanByID(ctx, waiting.ID)
if err != nil {
t.Fatal(err)
}
newer := ws[0]
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
}
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c1" || newer.CommitOf("novox/two") != "c2" {
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
}
if _, in := newer.Modules["one"]; !in {
t.Fatalf("what the folded walk was to build is not built: %v", newer.Modules)
}
}
// A merge heard after a later merge of its branch was walked is answered by that walk when it builds all it
// moves — named at once on a walk done — and joins the next batch when it does not.
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, later, t0.Add(18*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
// Moves only notes, which the done walk built from the branch after it: answered there, at once.
hear(t, open, catalogueMerge("553b7191early", "notes", t0), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if got.State != inventory.PlanDone || len(got.Delivery.Merges) != 2 ||
!sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: "553b7191early",
Carried: later.Commit}) {
t.Fatalf("the late merge is not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// Moves app, which that walk never built: the next batch walks it.
hear(t, open, catalogueMerge("1111aaaaapp", "app", t0.Add(time.Second)), t0.Add(16*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "1111aaaaapp" {
t.Fatalf("a late merge moving what the walk never built did not join the next batch: %+v", bs)
}
}
// A failed walk marks nothing delivered: its earlier merges are walked alone, on their own commit, the newest
// first; the first delivered answers the older ones, and the search stops.
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
oldest := catalogueMerge("aaaa0001", "app", t0.Add(-3*time.Minute))
middle := catalogueMerge("bbbb0002", "app", t0.Add(-3*time.Minute+10*time.Second))
newest := catalogueMerge("cccc0003", "notes", t0.Add(-3*time.Minute+20*time.Second))
for i, m := range []link.SourceMoved{oldest, middle, newest} {
hear(t, open, m, t0.Add(time.Duration(i)*10*time.Second+time.Second))
}
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
failed := ws[0]
failed.State, failed.Note = inventory.PlanFailed, "notes failed to build in tier 0"
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, failed.ID)
if got.State != inventory.PlanFailed || len(got.Delivery.Merges) != 3 {
t.Fatalf("the failed walk's record changed: %s %+v", got.State, got.Delivery.Merges)
}
ws, _ = walks(t, open)
alone := ws[0]
if alone.ID == failed.ID || alone.Commit != middle.Commit || len(alone.Delivery.Merges) != 1 ||
!sameMerge(alone.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: middle.Commit}) {
t.Fatalf("the newest earlier merge was not walked alone on its own commit: %+v", alone)
}
if _, in := alone.Modules["app"]; !in || (*asked)[len(*asked)-1] != [3]string{"novox/mesh-catalog", "modules/app",
middle.Commit} {
t.Fatalf("the merge walked alone does not build app at its own commit: %v, asked %v", alone.Modules, *asked)
}
// Retried, the failed walk would name merges the search answers now.
if _, err := retryPlan(ctx, open, failed.ID); err == nil || !strings.Contains(err.Error(), "walked alone") {
t.Fatalf("a failed walk whose merges are searched was retried: %v", err)
}
alone.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &alone); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(4*time.Minute))
ws, bs := walks(t, open)
if ws[0].ID != alone.ID || len(bs) != 0 {
t.Fatalf("the search went on after a merge was delivered: %+v", ws[0])
}
done, _ := open.inventory.PlanByID(ctx, alone.ID)
if len(done.Delivery.Merges) != 2 || !sameMerge(done.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog",
Commit: oldest.Commit, Carried: middle.Commit}) {
t.Fatalf("the oldest merge is not answered by the walk that delivered the one after it: %+v", done.Delivery.Merges)
}
// A stopped walk starts no search: a person ended it.
hear(t, open, catalogueMerge("dddd0004", "app", t0.Add(5*time.Minute)), t0.Add(5*time.Minute))
hear(t, open, catalogueMerge("eeee0005", "notes", t0.Add(5*time.Minute+time.Second)), t0.Add(5*time.Minute+time.Second))
cutAt(t, open, t0.Add(8*time.Minute))
ws, _ = walks(t, open)
if _, err := stopWalk(ctx, open.inventory, ws[0].ID, "mesh-delivery for jochen", "not now"); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(9*time.Minute))
if again, _ := walks(t, open); again[0].ID != ws[0].ID {
t.Fatalf("a stopped walk's earlier merge was walked alone: %+v", again[0])
}
}
// A delivery group's order holds inside the walk (ADR 0249 unchanged): merges of two repositories from one head
// branch name, the node-engine's before the controller's, are tiered so; without the group, one tier.
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1"},
inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
host := repoMerge("mesh-host", "h1", t0)
ctl := repoMerge("mesh-controller", "k1", t0.Add(time.Second))
host.Head, ctl.Head = "feat/together", "feat/together"
if !grouped {
ctl.Head = "feat/alone"
}
hear(t, open, ctl, t0.Add(2*time.Second))
hear(t, open, host, t0.Add(3*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walks", len(ws))
}
tiers := ws[0].Tiers
if grouped {
if len(tiers) != 2 || !slices.Equal(tiers[0], []string{"mesh-host"}) ||
!slices.Equal(tiers[1], []string{"mesh-controller"}) {
t.Fatalf("the group's order is not the walk's tiers: %v", tiers)
}
} else if len(tiers) != 1 {
t.Fatalf("two merges of no group were ordered: %v", tiers)
}
}
}
// A restarted controller resumes the window where it stood: the batch, its merges and their times are in the
// store, a merge handed over again is not doubled, and the batch is cut when its window closes.
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
}
cutAt(t, again, t0.Add(120*time.Second))
ws, bs := walks(t, again)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller did not cut the batch at its window: %+v %+v", ws, bs)
}
}
// The window's settings reach the controller in its settings file, read at every look; one that says no
// duration, or no file, is the default.
@@ -220,9 +622,94 @@ func TestAWaitingWalkNamesTheMergesItAnswers(t *testing.T) {
// A merge heard after a later merge of its branch was cut is named by that walk however its modules read since
// the cut, in a repository of one module (review of this change); a walk that never built what it moves names
// nothing, and the merge joins the next batch.
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
hear(t, open, c2, t0)
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0.Add(-60*time.Second)), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the late merge was not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// A related walk waits for the delivery's word too: a waiting walk let go beside a started one it is related to
// (the same repository, issue 973) takes the word and starts once that one ended, asking nothing before.
// A walk of two that built only two: a late merge of one is not its to name.
hear(t, open, repoMerge("two", "d2", t0.Add(16*time.Minute)), t0.Add(16*time.Minute))
cutAt(t, open, t0.Add(18*time.Minute))
ws, _ = walks(t, open)
w = ws[0]
delete(w.Modules, "two")
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("two", "d1", t0.Add(15*time.Minute)), t0.Add(20*time.Minute))
if got, _ := open.inventory.PlanByID(ctx, w.ID); len(got.Delivery.Merges) != 1 {
t.Fatalf("a walk that never built what the late merge moves named it: %+v", got.Delivery.Merges)
}
if _, bs := walks(t, open); len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "d1" {
t.Fatalf("the late merge did not join the next batch: %+v", bs)
}
}
// One walk at a time holds against the delivery's word too: a waiting walk let go beside a started one takes the
// word and starts once that one ended, asking nothing before.
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(2*time.Minute))
started, _ := walks(t, open)
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/two", Branch: "main", Commit: "d1", Created: t0,
State: inventory.PlanBuilding, Tiers: [][]string{{"two"}}, Modules: map[string]*inventory.PlanModule{"two": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if _, err := letGo(ctx, open.inventory, waiting.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatalf("the word was refused beside %s: %v", started[0].ID, err)
}
before := len(*asked)
advanceHeld(ctx, open)
got, _ := open.inventory.PlanByID(ctx, waiting.ID)
if len(*asked) != before || !strings.Contains(got.Note, "starts once "+started[0].ID) {
t.Fatalf("a walk let go beside a started one asked (%d → %d) or does not say it waits: %q", before, len(*asked), got.Note)
}
// Read as a wait, an hour on: its note on the line, never LATE, and no tier of it late for S3.
later := time.Now().Add(time.Hour)
if line := planLine(got, later); !strings.Contains(line, "starts once "+started[0].ID) || strings.Contains(line, "LATE") {
t.Fatalf("a deferred walk reads %q", line)
}
facts, _, err := gatherPlans(ctx, open.inventory, later, nil)
if err != nil {
t.Fatal(err)
}
for _, f := range facts {
if f.id == got.ID {
t.Fatalf("a deferred walk is watched as a tier running late: %+v", f)
}
}
done := started[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) != before+1 {
t.Fatalf("the walk did not start once the started one ended: asked %v", *asked)
}
}
// A file a merge of the batch removed and a later one brought back is not removed; one removed last is.
func TestARemovedFileIsTheLastMergesWord(t *testing.T) {
@@ -242,10 +729,153 @@ func TestARemovedFileIsTheLastMergesWord(t *testing.T) {
}
}
// The catch-up hands over what the bus lost after its branch's later merges were cut, and the record keeps it:
// an earlier merge is named by the walk that carried it; a merge made before a cut and heard after it, which
// no later merge carries, joins the next batch — neither reads as history and is dropped (review of this change).
func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
base := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
hear(t, open, repoMerge("one", "c2", base.Add(10*time.Second)), base.Add(11*time.Second))
cutAt(t, open, base.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
lost := []link.AnnouncedMerge{
{SourceMoved: repoMerge("one", "c1", base), At: base.Add(time.Second)},
{SourceMoved: repoMerge("one", "c3lost", base.Add(20*time.Second)), At: base.Add(21 * time.Second)},
}
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return nil, nil, err
}
read, err := readForPlanning(ctx, open.inventory)
return entries, read, err
}
if err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announcedList(lost), open.inventory}, catalogued,
(following{open}).SourceMoved, t.Logf); err != nil {
t.Fatal(err)
}
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the earlier merge the catch-up handed over is not named by the walk that carried it: %+v",
got.Delivery.Merges)
}
if _, kept, err := open.inventory.MergeOf(ctx, "novox/one", "c3lost"); err != nil || !kept {
t.Fatalf("the merge made before the cut and heard after it was dropped: %v %v", kept, err)
}
}
// A merge on the controller's own path never shares a batch with one that waits for mesh-delivery's word
// (decided during the build, 2026-10-10): the two are batches of their own, cut one after the other; a walk on
// the own path folds no waiting catalogue walk but batches its merges again behind it; the catalogue's walk
// still waits for the word, so no catalogue delivery skips its turn.
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("aaaa0001", "app", t0), t0)
hear(t, open, repoMerge("mesh-controller", "k1", t0.Add(10*time.Second)), t0.Add(10*time.Second))
hear(t, open, catalogueMerge("bbbb0002", "notes", t0.Add(20*time.Second)), t0.Add(20*time.Second))
_, bs := walks(t, open)
var ownBatch, catalogueBatch inventory.Plan
for _, b := range bs {
if b.OwnPath() {
ownBatch = b
} else {
catalogueBatch = b
}
}
if len(bs) != 2 || ownBatch.ID == "" || catalogueBatch.ID == "" || len(catalogueBatch.Delivery.Merges) != 2 ||
len(ownBatch.Delivery.Merges) != 1 {
t.Fatalf("a controller merge and two catalogue merges in one window: %+v", bs)
}
if !strings.Contains(batchWords(ownBatch, t0.Add(30*time.Second)), "own path") {
t.Fatalf("the own-path batch does not say so: %q", batchWords(ownBatch, t0.Add(30*time.Second)))
}
// Both windows closed, the controller's batch is cut first and starts; the catalogue batch queues behind it,
// is cut when the controller's walk ended, and waits for its word with both merges.
cutAt(t, open, t0.Add(2*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the controller's batch was not cut first into a started walk: %+v", ws)
}
ownWalk := ws[0]
for _, m := range []string{"app", "notes"} {
if _, in := ownWalk.Modules[m]; in {
t.Fatalf("the controller's walk builds %s, a catalogue module: it skipped its turn", m)
}
}
if len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != ownWalk.ID || bs[0].OwnPath() {
t.Fatalf("the catalogue batch does not queue behind the controller's walk: %+v", bs)
}
ownWalk.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &ownWalk); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
ws, bs = walks(t, open)
if len(bs) != 0 || ws[0].ID != catalogueBatch.ID || !ws[0].Waiting() || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the catalogue batch was not cut into a waiting walk once the controller's ended: %+v %+v", ws, bs)
}
for _, m := range []string{"app", "notes"} {
if _, in := ws[0].Modules[m]; !in {
t.Fatalf("the catalogue walk does not build %s: %v", m, ws[0].Modules)
}
}
for _, a := range *asked {
if a[1] == "modules/app" || a[1] == "modules/notes" {
t.Fatalf("a catalogue module was asked without the word: %v", *asked)
}
}
// A catalogue walk waiting for its word when an own-path batch is cut keeps waiting beside the own-path
// walk, is not folded into it, and its word is taken meanwhile: it starts once the own-path walk ended.
catalogueWalk := ws[0]
hear(t, open, repoMerge("mesh-controller", "k2", t0.Add(4*time.Minute)), t0.Add(4*time.Minute))
cutAt(t, open, t0.Add(6*time.Minute))
ws, _ = walks(t, open)
kept, _ := open.inventory.PlanByID(ctx, catalogueWalk.ID)
if !kept.Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k2" || ws[0].Waiting() {
t.Fatalf("the waiting catalogue walk was not kept waiting beside the controller's walk: %s %+v", kept.State, ws)
}
if _, in := ws[0].Modules["app"]; in {
t.Fatalf("the controller's walk folded the waiting catalogue walk in: %v", ws[0].Modules)
}
if _, err := letGo(ctx, open.inventory, catalogueWalk.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatal(err)
}
before := len(*asked)
advanceHeld(ctx, open)
if len(*asked) != before {
t.Fatalf("the catalogue walk started beside the controller's: asked %v", (*asked)[before:])
}
own2 := ws[0]
own2.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &own2); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) < before+1 || (*asked)[before][1] != "modules/app" {
t.Fatalf("the catalogue walk did not start once the controller's ended: %v", (*asked)[before:])
}
}
// `plans` names a walk by what it moves (asked by the operator, 2026-10-10): the repository's pull request and
// title, the modules it moves and the machines running them, then the state words; a record naming no pull
@@ -291,3 +921,46 @@ func sameMerge(a, b inventory.PlanMerge) bool {
// A catalogue merge heard after a later merge of its branch was walked without the word (a merge that touched
// the bus too, on the controller's own path) is not answered by that walk while the delivery seat has a holder:
// its delivery would skip its turn. It joins the next catalogue batch.
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
module string
claims []catalogue.Claim
}{
{"nats", nil},
{"mesh-delivery", []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1", Claims: m.claims},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/" + m.module, Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
mixed := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "m1xed", MergedAt: t0.Format(time.RFC3339Nano),
Paths: []string{"modules/nats/module.json", "modules/app/module.json"}, ModuleDirs: []string{"modules/nats", "modules/app"},
ModuleDirsSaid: true}
hear(t, open, mixed, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() {
t.Fatalf("the mixed merge's walk waited, or was not cut: %+v", ws)
}
done := ws[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("ear1ier", "app", t0.Add(-30*time.Second)), t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, done.ID)
_, bs := walks(t, open)
if len(got.Delivery.Merges) != 1 || len(bs) != 1 || bs[0].OwnPath() || bs[0].Delivery.Merges[0].Commit != "ear1ier" {
t.Fatalf("the late catalogue merge was answered by the own-path walk (%+v) rather than the next catalogue batch (%+v)",
got.Delivery.Merges, bs)
}
}
+2 -48
View File
@@ -3,8 +3,6 @@ package main
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
@@ -19,9 +17,6 @@ import (
const (
kindBatchNotCut = "batch-not-cut"
kindBatchBehindWalk = "batch-behind-walk"
// kindGroupNotMerged is raised for a group whose last member has not merged by the merge window's maximum after
// its first (novox/hq ADR 0418 clause 3).
kindGroupNotMerged = "group-not-merged"
)
// batchFacts is one batch not yet cut, as the watchdogs read it.
@@ -33,11 +28,6 @@ type batchFacts struct {
// tiers.
behind string
walkBound time.Duration
// holds are the groups the batch is held for, by head, with why; created is when the batch was made, the first
// merge of the group heard; atMostWindow is the maximum a window may take (the controller's setting).
holds map[string]string
created time.Time
atMostWindow time.Duration
}
// gatherBatches is every batch not yet cut, with the bound of the walk it waits behind.
@@ -52,10 +42,8 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
}
var out []batchFacts
for _, b := range batches {
_, atMostWindow := mergeWindowOf()
f := batchFacts{id: b.ID, state: b.State, grouped: groupedWords(b), created: b.Created, atMostWindow: atMostWindow}
f := batchFacts{id: b.ID, state: b.State, grouped: groupedWords(b)}
if w := b.Delivery; w != nil && w.Batch != nil {
f.holds = w.Batch.Holds
f.atMost, f.closed, f.behind = w.Batch.AtMost, w.Batch.ClosesAt, w.Batch.Behind
if f.atMost.Before(f.closed) {
f.closed = f.atMost
@@ -79,8 +67,7 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
func watchBatchesNotCut(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, b := range f.batches {
if b.state != inventory.PlanAssembling || b.atMost.IsZero() || f.now.Sub(b.atMost) <= batchLateAfter ||
len(b.holds) > 0 { // a batch held for a group waits on a member, which has its own condition
if b.state != inventory.PlanAssembling || b.atMost.IsZero() || f.now.Sub(b.atMost) <= batchLateAfter {
continue
}
late := f.now.Sub(b.atMost)
@@ -118,36 +105,3 @@ func watchBatchesBehind(f *signalFacts) []conditions.Observation {
}
return out
}
// watchGroupsNotMerged is the condition of a batch held for a group whose last member has not merged by the merge
// window's maximum after the first member merged, naming the member and why (novox/hq ADR 0418 clause 3): nothing of
// the group is delivered while it waits.
func watchGroupsNotMerged(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, b := range f.batches {
if len(b.holds) == 0 || b.created.IsZero() || f.now.Sub(b.created) <= b.atMostWindow {
continue
}
heads := make([]string, 0, len(b.holds))
for h := range b.holds {
heads = append(heads, h)
}
sort.Strings(heads)
var why []string
for _, h := range heads {
why = append(why, b.holds[h])
}
waited := f.now.Sub(b.created)
out = append(out, conditions.Observation{Scope: conditions.ScopePlan, ID: b.id,
Kind: kindGroupNotMerged, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s; its first member merged %s ago, past the merge window's maximum (%s), and nothing "+
"of the group is delivered while it waits (batch %s)", strings.Join(why, "; "), ago(waited), ago(b.atMostWindow), b.id),
Said: "a group waits for a member that has not merged",
Headline: "A change waits for the rest of its group to merge",
Explanation: fmt.Sprintf("The changes of a group are delivered together, once the last one is merged. "+
"One has not merged in %s. Nothing is lost; the merged ones are kept and delivered with it.",
humanDuration(waited)),
Resolved: "The group's last change merged"})
}
return out
}
-108
View File
@@ -1,108 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// An unrelated closed batch is cut into a walk beside an open one, and that walk asks its first tier at once
// (issue 973, novox/hq ADR 0407 decision 4): a hotfix is never held by an unrelated delivery.
func TestAnUnrelatedBatchWalksBesideAnOpenWalk(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first walk is not open: %+v", ws)
}
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("two", "c2", heard)), heard)
advanceHeld(ctx, open)
ws, bs := walks(t, open)
if len(ws) != 2 || len(bs) != 0 {
t.Fatalf("%d walk(s) and %d batch(es); want the unrelated batch cut beside the open walk", len(ws), len(bs))
}
for _, w := range ws {
if !w.Open() || w.Waiting() || deferred(w) || !askedAny(w) {
t.Errorf("%s (%s) is not walking: open %v, deferred %v, asked %v, note %q", w.ID, w.Named(), w.Open(),
deferred(w), askedAny(w), w.Note)
}
}
}
// A closed batch related to an open walk is queued behind it with the reason, while one that shares nothing is
// cut beside it; the walk that started is never superseded (ADR 0276 decision 4, issue 973).
func TestARelatedBatchWaitsBehindTheOpenWalk(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
first := func() inventory.Plan { ws, _ := walks(t, open); return ws[len(ws)-1] }()
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("one", "c2", heard)), heard)
hear(t, open, lone(repoMerge("three", "c3", heard.Add(time.Second))), heard.Add(time.Second))
ws, bs := walks(t, open)
if len(ws) != 2 || len(bs) != 1 {
t.Fatalf("%d walk(s) and %d batch(es); want the unrelated merge walking and the related one queued", len(ws), len(bs))
}
b := bs[0]
words := batchWords(b, heard)
if b.State != inventory.PlanQueued || b.Delivery.Batch.Behind != first.ID || b.CommitOf("novox/one") != "c2" ||
!strings.Contains(words, "queued behind "+first.ID) || !strings.Contains(words, "both change novox/one") {
t.Fatalf("the related batch is %s, behind %q, reading %q", b.State, b.Delivery.Batch.Behind, words)
}
if got, _ := open.inventory.PlanByID(ctx, first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the related merge", got.State)
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(2*time.Second))
if _, bs := walks(t, open); len(bs) != 0 {
t.Fatalf("the queued batch was not cut once the walk it waited for ended: %+v", bs)
}
}
// A batch on the controller's own path never walks beside another walk, and nothing walks beside it, whatever
// the two share (issue 973): it replaces the controller, so it goes alone.
func TestAnOwnPathBatchNeverWalksBeside(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
first := ws[0]
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("mesh-controller", "k1", heard)), heard)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || !bs[0].OwnPath() || bs[0].State != inventory.PlanQueued ||
bs[0].Delivery.Batch.Behind != first.ID || !strings.Contains(batchWords(bs[0], heard), "own path") {
t.Fatalf("an own-path batch beside an open walk: %d walk(s), batches %+v", len(ws), bs)
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(time.Second))
ws, _ = walks(t, open)
if len(ws) != 2 || !ws[0].Open() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the own-path batch was not cut once the walk ended: %+v", ws)
}
own := ws[0]
// Unrelated to the own-path walk by every footprint, a merge still waits for it.
later := heard.Add(time.Minute)
hear(t, open, lone(repoMerge("two", "c2", later)), later)
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != own.ID {
t.Fatalf("a merge beside the own-path walk: %d walk(s), batches %+v", len(ws), bs)
}
}
@@ -1,102 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
func TestTheIncidentAConsumerStaysBesideItsDataUntilAPersonPinsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
for _, m := range storeManifests() {
register(t, open, m)
}
assignAll := func(pairs ...[2]string) {
for _, a := range pairs {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
}
// The anchor's store and resolver hold the mesh's seats, on record; the laptop runs its own of each.
assignAll([2]string{"anchor", "store"}, [2]string{"anchor", "resolver"})
for _, seat := range [][2]string{{"mesh-store", "store"}, {"mesh-dns-resolver", "resolver"}} {
if err := inv.HoldSeat(ctx, seat[0], catalogue.ScopeMesh, "anchor", seat[1]); err != nil {
t.Fatal(err)
}
}
assignAll([2]string{"laptop", "store"}, [2]string{"laptop", "resolver"}, [2]string{"laptop", "network"},
[2]string{"laptop", "board"})
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "laptop" || n.Module != "store" || !n.KeepsData {
t.Fatalf("the consumer was bound to %s/%s (keeps data: %v); its data is beside it", n.From, n.Module, n.KeepsData)
}
if n := need(t, plan, "network", "wildcard-resolution"); n.From != "anchor" || n.KeepsData {
t.Fatalf("the resolver was bound to %s (keeps data: %v); its seat is held on anchor (issue 258)", n.From, n.KeepsData)
}
// Sent, and recorded: only the binding to data.
bindings := boundToData(plan, nil)
if len(bindings) != 1 || bindings[0].Provider != (catalogue.Chosen{Node: "laptop", Module: "store"}) {
t.Fatalf("recorded %+v", bindings)
}
if err := inv.RecordBindings(ctx, "laptop", bindings); err != nil {
t.Fatal(err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a mesh bound where it was sent: %+v, %v", found, err)
}
// The laptop's store taken away: refused, not moved to the anchor's empty one.
if err := inv.Unassign(ctx, "laptop", "store"); err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) ||
!strings.Contains(err.Error(), "board on laptop is bound to laptop/store") ||
!strings.Contains(err.Error(), "pin laptop postgres-database anchor store") {
t.Fatalf("the consumer's store went and it was answered elsewhere: %v", err)
}
// A person pins the anchor's: it moves, said before it is sent, and the record keeps where it was.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "anchor", "store"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "anchor" {
t.Fatalf("pinned to the anchor and bound to %s", n.From)
}
found, err := probeBindings(ctx, &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || found[0].Kind != kindBindingMoving || found[0].Severity != conditions.Warning ||
!strings.Contains(found[0].Summary, "board's postgres-database moves from laptop/store to anchor/store") {
t.Fatalf("a pinned move is not said before it is sent: %+v", found)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
all, err := inv.Bindings(ctx)
if err != nil || len(all) != 1 || all[0].MovedFrom != "laptop/store" {
t.Fatalf("%+v, %v", all, err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a move sent is still said: %+v, %v", found, err)
}
}
+90
View File
@@ -39,6 +39,96 @@ func need(t *testing.T, plan catalogue.Resolution, consumer, provision string) c
// The incident through the stores: the laptop runs its own store and a consumer of it, the anchor's
// store holds the mesh's seat. The consumer stays beside its data, the resolver follows its seat, the
// binding is recorded as sent, and a pin — only a pin — moves it, said before the push that carries it.
func TestTheIncidentAConsumerStaysBesideItsDataUntilAPersonPinsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
for _, m := range storeManifests() {
register(t, open, m)
}
assignAll := func(pairs ...[2]string) {
for _, a := range pairs {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
}
// The anchor's store and resolver hold the mesh's seats, on record; the laptop runs its own of each.
assignAll([2]string{"anchor", "store"}, [2]string{"anchor", "resolver"})
for _, seat := range [][2]string{{"mesh-store", "store"}, {"mesh-dns-resolver", "resolver"}} {
if err := inv.HoldSeat(ctx, seat[0], catalogue.ScopeMesh, "anchor", seat[1]); err != nil {
t.Fatal(err)
}
}
assignAll([2]string{"laptop", "store"}, [2]string{"laptop", "resolver"}, [2]string{"laptop", "network"},
[2]string{"laptop", "board"})
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "laptop" || n.Module != "store" || !n.KeepsData {
t.Fatalf("the consumer was bound to %s/%s (keeps data: %v); its data is beside it", n.From, n.Module, n.KeepsData)
}
if n := need(t, plan, "network", "wildcard-resolution"); n.From != "anchor" || n.KeepsData {
t.Fatalf("the resolver was bound to %s (keeps data: %v); its seat is held on anchor (issue 258)", n.From, n.KeepsData)
}
// Sent, and recorded: only the binding to data.
bindings := boundToData(plan, nil)
if len(bindings) != 1 || bindings[0].Provider != (catalogue.Chosen{Node: "laptop", Module: "store"}) {
t.Fatalf("recorded %+v", bindings)
}
if err := inv.RecordBindings(ctx, "laptop", bindings); err != nil {
t.Fatal(err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a mesh bound where it was sent: %+v, %v", found, err)
}
// The laptop's store taken away: refused, not moved to the anchor's empty one.
if err := inv.Unassign(ctx, "laptop", "store"); err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) ||
!strings.Contains(err.Error(), "board on laptop is bound to laptop/store") ||
!strings.Contains(err.Error(), "pin laptop postgres-database anchor store") {
t.Fatalf("the consumer's store went and it was answered elsewhere: %v", err)
}
// A person pins the anchor's: it moves, said before it is sent, and the record keeps where it was.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "anchor", "store"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "anchor" {
t.Fatalf("pinned to the anchor and bound to %s", n.From)
}
found, err := probeBindings(ctx, &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || found[0].Kind != kindBindingMoving || found[0].Severity != conditions.Warning ||
!strings.Contains(found[0].Summary, "board's postgres-database moves from laptop/store to anchor/store") {
t.Fatalf("a pinned move is not said before it is sent: %+v", found)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
all, err := inv.Bindings(ctx)
if err != nil || len(all) != 1 || all[0].MovedFrom != "laptop/store" {
t.Fatalf("%+v, %v", all, err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a move sent is still said: %+v, %v", found, err)
}
}
// What one machine's resolution says against its record.
func TestBindingFindingsSayAKeptMoveAndAMoveNothingAskedFor(t *testing.T) {
+3 -84
View File
@@ -15,7 +15,6 @@ import (
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -183,10 +182,8 @@ func buildFrom(result link.BuildResult) inventory.Build {
kept.Manifest = recordedManifest(result.Manifest, announced)
// The module name comes from the manifest, which only exists when the build got that far.
if len(kept.Manifest) > 0 {
if list, err := judgedWith(result); err == nil {
if m, err := catalogue.ParseManifestWith(kept.Manifest, list); err == nil {
kept.Module = m.Module
}
if m, err := catalogue.ParseManifest(kept.Manifest); err == nil {
kept.Module = m.Module
}
}
return kept
@@ -486,7 +483,6 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
Seats: seatBases(ctx),
DryRun: dryRun,
}
request.Reuse = reuseAsked(ctx, dryRun)
fmt.Printf("asked for %s", source)
if source.Seat != "" {
fmt.Printf(" (%s)", repository)
@@ -605,11 +601,6 @@ func saysWhenThePolicyActs(ctx context.Context, inv *inventory.Inventory, module
// written with the same values.
func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResult) (
catalogue.Manifest, inventory.Build, error) {
// The list of predefined keys as the store keeps it now: another controller process may have kept a newer one.
if err := loadPredefined(ctx, inv); err != nil {
return catalogue.Manifest{}, inventory.Build{}, fmt.Errorf("the list of predefined keys kept cannot be "+
"read: %w", err)
}
kept := buildFrom(result)
if err := inv.RecordBuild(ctx, kept); err != nil {
return catalogue.Manifest{}, kept, err
@@ -620,15 +611,7 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return catalogue.Manifest{}, kept, fmt.Errorf("%s could not build %s:\n%s",
result.On, result.Repository, result.Failed)
}
// Judged first against the list the build was judged against: the catalogue's at the commit built (novox/hq
// ADR 0348 §7), so a module naming a key its own commit adds is read; and again below against the list the
// mesh publishes, before it is registered.
list, err := judgedWith(result)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
result.Repository, short(result.Commit), err)
}
manifest, err := catalogue.ParseManifestWith(kept.Manifest, list)
manifest, err := catalogue.ParseManifest(kept.Manifest)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s and what came back is not a manifest: %w",
result.On, result.Repository, err)
@@ -686,12 +669,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
manifest.Module, short(result.Commit), err)
}
// **Never a build in the scratch namespace** (novox/hq ADR 0239 §9, ADR 0350 §2): what a check builds is
// registered only as the walk's copy in the module's own namespace.
if scratch := inScratch(result.Made); scratch != "" {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %s is in the scratch "+
"namespace, which only a walk's copy leaves (ADR 0350)", result.On, manifest.Module, short(result.Commit), scratch)
}
// **A build that failed its gate is never registered again** (novox/hq ADR 0236): an outcome heard
// twice, or replayed, would otherwise make the build a rollback put back what the module is again,
// and the next push would send it.
@@ -702,23 +679,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
"back: it is recorded and not registered again — a newer build is", result.On, manifest.Module,
short(result.Commit))
}
// **The newest trunk build's list of predefined keys is published** (novox/hq ADR 0348 §7): a build of the
// catalogue that may be registered keeps its list when its commit is newer than the one kept. Then the module
// is judged against the list the mesh publishes, as every manifest read again is: one naming a key that list
// does not hold, or of another type, is refused here and never enters the catalogue.
if carried, err := carriedList(result); err != nil {
return manifest, kept, err
} else if carried != nil {
if err := publishPredefined(ctx, inv, result, carried); err != nil {
fmt.Printf("%s: the list of predefined keys it carried is not published: %v\n", result.ID, err)
}
}
if manifest.NamesPredefined() {
if _, err := catalogue.ParseManifest(kept.Manifest); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: judged against the "+
"list of predefined keys the mesh publishes, %w", result.On, manifest.Module, short(result.Commit), err)
}
}
// **A build whose source is unchanged is never a move** (novox/hq issue 280): a rebuild made from
// what the build the mesh stands on was made from registers that build's artifacts at the new
// commit, so no machine is sent a new digest for a source nobody changed — an image is not
@@ -754,16 +714,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, nil
}
// inScratch is the first artifact a build made into the scratch namespace; empty when none is.
func inScratch(made []link.MadeArtifact) string {
for _, m := range made {
if path, kept := catalogue.InArtifactStore(catalogue.Recorded(m.Reference)); kept && strings.HasPrefix(path, builder.ScratchPrefix) {
return m.Reference
}
}
return ""
}
// errOffTheTrunk is a build of a commit off its repository's trunk, which is never published.
var errOffTheTrunk = errors.New("the commit is not on its repository's trunk: a commit off the trunk is checked, " +
"never published (ADR 0238) — merge it, and the merge builds it")
@@ -852,9 +802,6 @@ type answers struct {
sources map[string]inventory.Source
// waiting is every machine not running what the mesh would send it.
waiting []inventory.Machine
// recordedWaits is, per machine, every move of a recorded module that waits there for a person (novox/hq ADR
// 0377 §6): the machine is behind for these, and `push <node>` sends them.
recordedWaits map[string][]string
// reported is every machine's last word beside when it was last sent a declaration — the
// pair that answers "has it caught up", which waiting alone cannot (the sent digest is
// recorded at send, not at apply).
@@ -934,34 +881,6 @@ type answers struct {
//
// Routed through the artifact store as the network reaches it now (novox/hq 04-ISSUES/102): a
// base is recorded by digest and path, and a build machine needs something it can pull.
// runningBy is, keyed "<module>/<artifact>", each artifact of the build the mesh has registered for its module, as a
// builder reaches the artifact store (novox/hq issue 721): what a check compares its builds with. Empty when it cannot
// be read, said, and then nothing is said unchanged.
func runningBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
defer open.Close()
running, err := open.inventory.RegisteredHeld(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
address, err := whereABuilderReachesTheStore(ctx, open.inventory)
if err != nil || address == "" {
return running
}
routed := make(map[string]string, len(running))
for key, reference := range running {
routed[key] = catalogue.Rerouted(reference, address)
}
return routed
}
func heldBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
@@ -1,131 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func TestABuildHeardIsRecordedAndRegistered(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "shop", "version": "3"})
m, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, link.BuildResult{
ID: "b-1", Repository: "http://forge.internal:20000/novox/shop.git", Path: "modules/shop",
Ref: "main", On: "anchor", Commit: "abcdef0123", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/shop"},
}))
if err != nil {
t.Fatal(err)
}
if m.Module != "shop" {
t.Fatalf("registered %q", m.Module)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if _, held := shelf["shop"]; !held {
t.Fatal("the module a heard build produced is not in the catalogue")
}
src, err := open.inventory.SourceOf(ctx, "shop")
if err != nil || src.Seat != "git" || src.Repository != "novox/shop" || src.BuiltFrom != "abcdef0123" {
t.Fatalf("the source is the seat and the path, never the URL: %+v %v", src, err)
}
builds, err := open.inventory.Builds(ctx, "shop", 5)
if err != nil || len(builds) != 1 || builds[0].ID != "b-1" {
t.Fatalf("the build is not recorded once: %v %v", builds, err)
}
named, _ := json.Marshal(map[string]any{"module": "idp", "version": "1", "resources": []any{
map[string]any{"id": "server", "type": "container", "image": "x@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}}}})
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{
ID: "b-2", Repository: "/r", On: "anchor", Commit: "0123456789", Manifest: named})
if err == nil || !strings.Contains(err.Error(), "does not register it") {
t.Fatalf("a definition naming an installation was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["idp"].Module != "" {
t.Fatal("the refused module was registered anyway")
}
if builds, _ := open.inventory.Builds(ctx, "idp", 5); len(builds) != 1 {
t.Fatalf("the refused build was not recorded: %v", builds)
}
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{ID: "b-3", Repository: "/r", On: "anchor", Failed: "no compiler"})
if err == nil || !strings.Contains(err.Error(), "no compiler") {
t.Fatalf("a failure is said in the builder's words: %v", err)
}
}
func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "unifi", "version": "1"})
result := func(id, ref, commit string) link.BuildResult {
return link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/unifi", Ref: ref, On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result("b-1", "main", "1111111aaaa"))); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, result("b-2", "9c97a8a", "9c97a8a1d2c3")); err != nil {
t.Fatal(err)
}
src, err := open.inventory.SourceOf(ctx, "unifi")
if err != nil {
t.Fatal(err)
}
if src.Ref != "main" || src.BuiltFrom != "9c97a8a1d2c3" {
t.Errorf("after a build at a commit the module follows %q, built from %q; want main, 9c97a8a1d2c3", src.Ref, src.BuiltFrom)
}
// One new to the catalogue, first built at a commit, follows the default branch.
other, _ := json.Marshal(map[string]any{"module": "letta", "version": "1"})
r := result("b-3", "deadbeef", "deadbeefcafe")
r.Manifest, r.Path = other, "modules/letta"
if _, _, err := takeIn(ctx, open.inventory, r); err != nil {
t.Fatal(err)
}
if src, _ := open.inventory.SourceOf(ctx, "letta"); src.Ref != "" {
t.Errorf("a module first built at a commit follows %q, want the default branch", src.Ref)
}
}
func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
result := func(asked time.Time, image string) link.BuildResult {
manifest, _ := json.Marshal(map[string]any{"module": "postgres", "version": image})
return link.BuildResult{ID: link.NewBuildID(asked), Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result(newer, "4bcd5f73"))); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
if !errors.Is(err, inventory.ErrSuperseded) {
t.Fatalf("the older request's outcome was taken in as current: %v", err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "4bcd5f73" {
t.Errorf("postgres is %q; want the newer request's 4bcd5f73", got)
}
if builds, _ := open.inventory.Builds(ctx, "postgres", 5); len(builds) != 2 {
t.Errorf("the late build was not recorded: %v", builds)
}
}
+1 -20
View File
@@ -435,20 +435,6 @@ func mayRegisterFrom(ctx context.Context, inv *inventory.Inventory, module strin
module, sourceWords(built.Repository, built.Seat))
}
}
// **Or the move the operator approved** (novox/hq ADR 0420 §7, issue 1025): a known module built from another
// repository by a merge, whose pull request mesh-delivery merged on the operator's Approve of that head.
// A verb alone, or a merge with no such warrant, is refused as before.
movedWhy := ""
if why != "" && !terminal && !isNew {
accepted, err := approvedMove(ctx, inv, forms, buildID, built)
switch {
case err == nil:
movedWhy, why = accepted, ""
case !errors.Is(err, errNotAMerge):
fmt.Printf("%s: %s is built from another repository, and no approved move covers this build: %v\n",
buildID, module, err)
}
}
if why != "" && !terminal {
return refuse(why)
}
@@ -485,8 +471,7 @@ func mayRegisterFrom(ctx context.Context, inv *inventory.Inventory, module strin
}
// The same repository by the forge's id, not only its name: one deleted and made again is another.
recorded := map[string]int64{}
// A moved module is registered from the repository the operator approved, not the one it was registered from.
if !isNew && movedWhy == "" {
if !isNew {
id, err := inv.SourceIdentity(ctx, module)
if err != nil {
return 0, err
@@ -507,10 +492,6 @@ func mayRegisterFrom(ctx context.Context, inv *inventory.Inventory, module strin
owner, name, m, facts.ID, m, id))
}
}
if movedWhy != "" {
fmt.Printf("%s: %s moved from %s to %s/%s — registered at the operator's Approve: %s\n", buildID, module,
sourceWords(was.Repository, was.Seat), owner, name, movedWhy)
}
return facts.ID, nil
}
@@ -1,353 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func theCatalogue(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
for _, b := range []link.BuildResult{
onTrunk("build-sudo", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "1"}),
onTrunk("build-host", "novox/mesh-host", "git", "", map[string]any{"module": "mesh-host", "version": "1"}),
} {
keptAsked(t, open.inventory, b.ID, b.Source.Repository, b.Path, true)
if _, _, err := takeIn(ctx, open.inventory, b); err != nil {
t.Fatal(err)
}
}
return open
}
func TestABuildFromAnAgentsRepositoryIsNotRegisteredAsAModuleTheMeshHolds(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
for _, c := range []struct {
name, repository, path, module string
}{
{"its own repository naming sudo", "agent/sudo", "modules/sudo", "sudo"},
{"a fork of the catalogue", "agent/mesh-catalog", "modules/sudo", "sudo"},
{"a fork of the node-engine", "agent/mesh-host", "", "mesh-host"},
} {
t.Run(c.name, func(t *testing.T) {
id := "build-" + strings.ReplaceAll(c.repository, "/", "-")
keptAsked(t, open.inventory, id, c.repository, c.path, false) // through the build verb
evil := onTrunk(id, c.repository, "git", c.path, map[string]any{"module": c.module, "version": "evil"})
_, _, err := takeIn(ctx, open.inventory, evil)
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a build of %s was taken in as %s: %v", c.repository, c.module, err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf[c.module].Version; got != "1" {
t.Fatalf("%s is now %q, from %s", c.module, got, c.repository)
}
if _, found, _ := open.inventory.BuildByID(ctx, id); !found {
t.Errorf("the refused build %s is not recorded", id)
}
})
}
}
func TestANewModuleFromARepositoryTheCatalogueDoesNotBuildFromIsNotRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-new", "agent/tools", "", false)
_, _, err := takeIn(ctx, open.inventory, onTrunk("build-new", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from an agent's repository was taken in: %v", err)
}
// And one asked of nobody here — an outcome on the bus no request was kept for — the same.
_, _, err = takeIn(ctx, open.inventory, onTrunk("build-unasked", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome nobody asked for was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["agent-tools"].Module != "" {
t.Fatal("the refused module is in the catalogue")
}
}
func TestAtTheTerminalAnotherRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-moved", "novox/sudo", "", true)
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-moved", "novox/sudo", "git", "",
map[string]any{"module": "sudo", "version": "2"})); err != nil {
t.Fatalf("a move the operator asked for at the terminal was refused: %v", err)
}
if src, _ := open.inventory.SourceOf(ctx, "sudo"); src.Repository != "novox/sudo" {
t.Fatalf("sudo is built from %q", src.Repository)
}
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-external",
Repository: "http://forge.internal:20000/someone/app.git", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-external", "someone/app", "", "",
map[string]any{"module": "app", "version": "1"})); err != nil {
t.Fatalf("a new module the operator asked for at the terminal was refused: %v", err)
}
}
func TestADeliveryFromTheRegisteredRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
rebuilt := onTrunk("build-plan", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: rebuilt.ID,
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/sudo", For: "plan"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, rebuilt); err != nil {
t.Fatalf("a plan's build of the module's own repository was refused: %v", err)
}
added := onTrunk("build-merge", "novox/mesh-catalog", "git", "modules/zram", map[string]any{"module": "zram", "version": "1"})
if _, _, err := takeIn(ctx, open.inventory, added); err != nil {
t.Fatalf("a module a merge added to the catalogue repository was refused: %v", err)
}
shelf, _ := open.inventory.Catalogue(ctx)
if shelf["sudo"].Version != "2" || shelf["zram"].Module == "" {
t.Fatalf("not registered: sudo %q, zram %q", shelf["sudo"].Version, shelf["zram"].Module)
}
}
func TestAVerbAsksNoBuildOfARepositoryTheCatalogueDoesNotBuildFrom(t *testing.T) {
theCatalogue(t)
ctx := t.Context()
t.Setenv(verbVar, "build")
t.Setenv(link.CallerVar, "node-tools.anchor, through the mesh-controller seat")
err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "http://forge.internal:20000/agent/sudo.git")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("a verb's build of an agent's repository was asked: %v", err)
}
if err := verbMayAsk(ctx, buildSource{Repository: "novox/mesh-catalog", Seat: "git"},
"http://forge.internal:20000/novox/mesh-catalog.git"); err != nil {
t.Fatalf("a verb's build of the catalogue repository was refused: %v", err)
}
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
if err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, ""); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
func TestARollbackNeverPutsBackABuildFromAnotherRepository(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
fork := onTrunk("build-1791500000000000000", "agent/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "evil"})
fork.Commit = "c0ffee0123456789" // the commit sudo was registered at
keptAsked(t, inv, fork.ID, "agent/mesh-catalog", "modules/sudo", false)
if _, _, err := takeIn(ctx, inv, fork); !errors.Is(err, errNotItsSource) {
t.Fatalf("the fork's build was taken in: %v", err)
}
// Asked after the registered build, whenever the test runs: an id naming a fixed moment read as older
// than the registered build once the clock passed it (2026-10-10 02:40 UTC), and the test failed on main.
failed := onTrunk(fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano()), "novox/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "2"})
failed.Commit = "badbadbad0123456"
if _, _, err := takeIn(ctx, inv, failed); err != nil {
t.Fatal(err)
}
record, _, err := inv.BuildByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
previous, found, err := inv.PreviousBuild(ctx, "sudo", "c0ffee0123456789", record)
if err != nil {
t.Fatal(err)
}
if found && previous.ID == fork.ID {
t.Fatalf("a rollback would put back the fork's build %s", previous.ID)
}
if !found || previous.ID != "build-sudo" {
t.Fatalf("a rollback puts back %q (found %v), want the registered build-sudo", previous.ID, found)
}
}
func TestATrunkTheForgeDoesNotProtectRegistersNothing(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/unguarded", forgeFacts{ID: 7, Why: "the rule main lets a person push to main directly"})
t.Cleanup(func() { theForge.Delete("novox/unguarded") })
first := onTrunk("build-unguarded-1", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/unguarded", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatalf("at the terminal: %v", err)
}
again := onTrunk("build-unguarded-2", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "2"})
if _, _, err := takeIn(ctx, open.inventory, again); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "push to main directly") {
t.Fatalf("a rebuild from an unprotected trunk was taken in: %v", err)
}
theForge.Store("novox/unguarded", errors.New("nothing serves gitea.gitea_api"))
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-unguarded-3", "novox/unguarded", "git", "",
map[string]any{"module": "unguarded", "version": "3"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a forge that could not say was read as a protected trunk: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["unguarded"].Version != "1" {
t.Fatalf("unguarded is %q", shelf["unguarded"].Version)
}
}
func TestARepositoryMadeAgainUnderItsNameIsNotTheModulesSource(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/remade", forgeFacts{ID: 100, Guarded: true})
t.Cleanup(func() { theForge.Delete("novox/remade") })
first := onTrunk("build-remade-1", "novox/remade", "git", "", map[string]any{"module": "remade", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/remade", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "remade"); id != 100 {
t.Fatalf("the forge's id was not recorded: %d", id)
}
theForge.Store("novox/remade", forgeFacts{ID: 101, Guarded: true}) // deleted, and made again by an agent
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-2", "novox/remade", "git", "",
map[string]any{"module": "remade", "version": "evil"})); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "made again") {
t.Fatalf("a repository made again under the name was taken in: %v", err)
}
// And a new module from it, beside the one registered from the first, the same.
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-3", "novox/remade", "git", "modules/other",
map[string]any{"module": "other", "version": "1"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from a repository made again was taken in: %v", err)
}
}
func TestATerminalRequestCoversOnlyWhatItAsked(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-asked", Repository: "novox/app",
Seat: "git", Path: "modules/app", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
other := onTrunk("build-asked", "agent/sudo", "git", "modules/app", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, other); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another repository under a terminal request's id was taken in: %v", err)
}
elsewhere := onTrunk("build-asked", "novox/app", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, elsewhere); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another path under a terminal request's id was taken in: %v", err)
}
}
func TestTheServingControllerIsNeverTheTerminal(t *testing.T) {
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
t.Setenv(servedVar, "")
if !startedAtTheTerminal() {
t.Fatal("a process started by hand is not the terminal")
}
markServed()
if startedAtTheTerminal() {
t.Fatal("the serving controller reads as the terminal")
}
child := exec.Command(os.Args[0], "-test.run=^$")
child.Env = os.Environ()
if !slices.Contains(child.Env, servedVar+"=1") {
t.Fatal("what the serving controller starts does not carry its mark")
}
}
func TestAnUnknownIdentityKeepsTheRecordedOne(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 100); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 0); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "sudo"); id != 100 {
t.Fatalf("the recorded id became %d", id)
}
}
func TestAModuleMovesOnTheOperatorsApproveOfTheMergeThatMovedIt(t *testing.T) {
const commit = "c0ffee0123456789"
for _, c := range []struct {
what string
change func(*anApprovedMerge)
unkept bool
request string // who asked: "merge", or "build" through a verb
moved bool
}{
{what: "the operator's Approve of the merged head", request: "merge", moved: true},
{what: "no merge mesh-delivery announced", unkept: true, request: "merge"},
{what: "a warrant for other heads", request: "merge", change: func(a *anApprovedMerge) {
a.merge.Heads = map[string]string{"novox/mesh-host@aaaaaaaaaaaa": "e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5"}
}},
{what: "a Decline", request: "merge", change: func(a *anApprovedMerge) {
a.warrant.Option, a.warrant.Label, a.merge.Option = "decline", "Decline", "decline"
}},
{what: "a person who is not the operator", request: "merge", change: func(a *anApprovedMerge) { a.warrant.By.Who = "agent" }},
{what: "a build asked through a verb", request: "build"},
} {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
a := approvedMerge(t)
a.merge.Owner, a.merge.Repo, a.merge.Number, a.merge.HeadSHA = "novox", "mesh-host", 7, "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1"
if c.change != nil {
c.change(&a)
}
wasKept, wasAsk := keptDeliveryMerge, routerAskOf
keptDeliveryMerge = func(_ context.Context, owner, repo string, number int, head string) (deliveryMerge, bool, error) {
return a.merge, !c.unkept && owner == "novox" && repo == "mesh-host" && number == 7 && head == a.merge.HeadSHA, nil
}
routerAskOf = func(_ context.Context, _ *inventory.Inventory, id string) (*asks.Ask, *asks.Warrant, error) {
if id != a.merge.AskID {
t.Errorf("%s: the router was asked for %q", c.what, id)
}
return &a.ask, &a.warrant, nil
}
t.Cleanup(func() { keptDeliveryMerge, routerAskOf = wasKept, wasAsk })
event, _ := json.Marshal(map[string]any{"owner": "novox", "repo": "mesh-host", "base": "main", "number": 7,
"head_sha": a.merge.HeadSHA, "merge_commit_sha": commit})
if _, err := inv.AddMerge(ctx, inventory.BatchedMerge{Repository: "novox/mesh-host", Branch: "main", Commit: commit,
Merged: time.Now(), Heard: time.Now(), Event: event}); err != nil {
t.Fatal(err)
}
id := fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano())
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: id, Repository: "novox/mesh-host", Seat: "git",
Path: "modules/sudo", Ref: "main", Commit: commit, For: c.request}); err != nil {
t.Fatal(err)
}
moved := onTrunk(id, "novox/mesh-host", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
moved.Commit = commit
_, _, err := takeIn(ctx, inv, moved)
src, _ := inv.SourceOf(ctx, "sudo")
switch {
case c.moved && err != nil:
t.Errorf("%s: the move was refused: %v", c.what, err)
case c.moved && src.Repository != "novox/mesh-host":
t.Errorf("%s: sudo is built from %q", c.what, src.Repository)
case !c.moved && !errors.Is(err, errNotItsSource):
t.Errorf("%s: the move was not refused as ADR 0266 has it: %v", c.what, err)
case !c.moved && src.Repository != "novox/mesh-catalog":
t.Errorf("%s: a refused move changed sudo's source to %q", c.what, src.Repository)
}
}
}
+262 -6
View File
@@ -3,10 +3,16 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"hash/fnv"
"os"
"os/exec"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -39,14 +45,141 @@ func keptAsked(t *testing.T, inv *inventory.Inventory, id, repository, path stri
}
// theCatalogue is a mesh whose sudo is built from the catalogue repository and whose node-engine from its own.
func theCatalogue(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
for _, b := range []link.BuildResult{
onTrunk("build-sudo", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "1"}),
onTrunk("build-host", "novox/mesh-host", "git", "", map[string]any{"module": "mesh-host", "version": "1"}),
} {
keptAsked(t, open.inventory, b.ID, b.Source.Repository, b.Path, true)
if _, _, err := takeIn(ctx, open.inventory, b); err != nil {
t.Fatal(err)
}
}
return open
}
func TestABuildFromAnAgentsRepositoryIsNotRegisteredAsAModuleTheMeshHolds(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
for _, c := range []struct {
name, repository, path, module string
}{
{"its own repository naming sudo", "agent/sudo", "modules/sudo", "sudo"},
{"a fork of the catalogue", "agent/mesh-catalog", "modules/sudo", "sudo"},
{"a fork of the node-engine", "agent/mesh-host", "", "mesh-host"},
} {
t.Run(c.name, func(t *testing.T) {
id := "build-" + strings.ReplaceAll(c.repository, "/", "-")
keptAsked(t, open.inventory, id, c.repository, c.path, false) // through the build verb
evil := onTrunk(id, c.repository, "git", c.path, map[string]any{"module": c.module, "version": "evil"})
_, _, err := takeIn(ctx, open.inventory, evil)
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a build of %s was taken in as %s: %v", c.repository, c.module, err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf[c.module].Version; got != "1" {
t.Fatalf("%s is now %q, from %s", c.module, got, c.repository)
}
if _, found, _ := open.inventory.BuildByID(ctx, id); !found {
t.Errorf("the refused build %s is not recorded", id)
}
})
}
}
func TestANewModuleFromARepositoryTheCatalogueDoesNotBuildFromIsNotRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-new", "agent/tools", "", false)
_, _, err := takeIn(ctx, open.inventory, onTrunk("build-new", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from an agent's repository was taken in: %v", err)
}
// And one asked of nobody here — an outcome on the bus no request was kept for — the same.
_, _, err = takeIn(ctx, open.inventory, onTrunk("build-unasked", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome nobody asked for was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["agent-tools"].Module != "" {
t.Fatal("the refused module is in the catalogue")
}
}
// The operator at the terminal may still move a module, or add one from a new repository.
func TestAtTheTerminalAnotherRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-moved", "novox/sudo", "", true)
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-moved", "novox/sudo", "git", "",
map[string]any{"module": "sudo", "version": "2"})); err != nil {
t.Fatalf("a move the operator asked for at the terminal was refused: %v", err)
}
if src, _ := open.inventory.SourceOf(ctx, "sudo"); src.Repository != "novox/sudo" {
t.Fatalf("sudo is built from %q", src.Repository)
}
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-external",
Repository: "http://forge.internal:20000/someone/app.git", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-external", "someone/app", "", "",
map[string]any{"module": "app", "version": "1"})); err != nil {
t.Fatalf("a new module the operator asked for at the terminal was refused: %v", err)
}
}
// The delivery's flow is untouched: a merge's rebuild of a module from its own repository, and a merge adding
// a module to a repository the catalogue builds from (novox/hq issue 300), are registered with no terminal.
func TestADeliveryFromTheRegisteredRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
rebuilt := onTrunk("build-plan", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: rebuilt.ID,
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/sudo", For: "plan"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, rebuilt); err != nil {
t.Fatalf("a plan's build of the module's own repository was refused: %v", err)
}
added := onTrunk("build-merge", "novox/mesh-catalog", "git", "modules/zram", map[string]any{"module": "zram", "version": "1"})
if _, _, err := takeIn(ctx, open.inventory, added); err != nil {
t.Fatalf("a module a merge added to the catalogue repository was refused: %v", err)
}
shelf, _ := open.inventory.Catalogue(ctx)
if shelf["sudo"].Version != "2" || shelf["zram"].Module == "" {
t.Fatalf("not registered: sudo %q, zram %q", shelf["sudo"].Version, shelf["zram"].Module)
}
}
// Through a verb, a build of a repository the catalogue builds nothing from is not even asked: the build node
// would run what the agent wrote.
func TestAVerbAsksNoBuildOfARepositoryTheCatalogueDoesNotBuildFrom(t *testing.T) {
theCatalogue(t)
ctx := t.Context()
t.Setenv(verbVar, "build")
t.Setenv(link.CallerVar, "node-tools.anchor, through the mesh-controller seat")
err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "http://forge.internal:20000/agent/sudo.git")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("a verb's build of an agent's repository was asked: %v", err)
}
if err := verbMayAsk(ctx, buildSource{Repository: "novox/mesh-catalog", Seat: "git"},
"http://forge.internal:20000/novox/mesh-catalog.git"); err != nil {
t.Fatalf("a verb's build of the catalogue repository was refused: %v", err)
}
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
if err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, ""); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
// asTheOperator keeps a build as asked at the controller's terminal, as the operator's first build of a module
// from a repository the catalogue does not yet build from is (novox/hq ADR 0266), and hands it back.
@@ -66,6 +199,40 @@ func asTheOperator(t *testing.T, inv *inventory.Inventory, b link.BuildResult) l
// A rollback puts back only a build of the module's own repository: an agent's build of the module's name,
// recorded and refused, at the very commit the machine ran before (a fork carries it), is never registered by
// the back door of a failed gate.
func TestARollbackNeverPutsBackABuildFromAnotherRepository(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
fork := onTrunk("build-1791500000000000000", "agent/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "evil"})
fork.Commit = "c0ffee0123456789" // the commit sudo was registered at
keptAsked(t, inv, fork.ID, "agent/mesh-catalog", "modules/sudo", false)
if _, _, err := takeIn(ctx, inv, fork); !errors.Is(err, errNotItsSource) {
t.Fatalf("the fork's build was taken in: %v", err)
}
// Asked after the registered build, whenever the test runs: an id naming a fixed moment read as older
// than the registered build once the clock passed it (2026-10-10 02:40 UTC), and the test failed on main.
failed := onTrunk(fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano()), "novox/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "2"})
failed.Commit = "badbadbad0123456"
if _, _, err := takeIn(ctx, inv, failed); err != nil {
t.Fatal(err)
}
record, _, err := inv.BuildByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
previous, found, err := inv.PreviousBuild(ctx, "sudo", "c0ffee0123456789", record)
if err != nil {
t.Fatal(err)
}
if found && previous.ID == fork.ID {
t.Fatalf("a rollback would put back the fork's build %s", previous.ID)
}
if !found || previous.ID != "build-sudo" {
t.Fatalf("a rollback puts back %q (found %v), want the registered build-sudo", previous.ID, found)
}
}
// theForge is what the forge says in a test, by owner/name: a repository not named here is protected as a
// trunk must be, with an id of its own.
@@ -89,15 +256,97 @@ func init() {
// A module's trunk the forge does not protect — direct pushes, no required status — or a forge that cannot say,
// registers nothing through a verb: the trunk rule means nothing on a branch anyone pushes to.
func TestATrunkTheForgeDoesNotProtectRegistersNothing(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/unguarded", forgeFacts{ID: 7, Why: "the rule main lets a person push to main directly"})
t.Cleanup(func() { theForge.Delete("novox/unguarded") })
first := onTrunk("build-unguarded-1", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/unguarded", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatalf("at the terminal: %v", err)
}
again := onTrunk("build-unguarded-2", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "2"})
if _, _, err := takeIn(ctx, open.inventory, again); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "push to main directly") {
t.Fatalf("a rebuild from an unprotected trunk was taken in: %v", err)
}
theForge.Store("novox/unguarded", errors.New("nothing serves gitea.gitea_api"))
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-unguarded-3", "novox/unguarded", "git", "",
map[string]any{"module": "unguarded", "version": "3"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a forge that could not say was read as a protected trunk: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["unguarded"].Version != "1" {
t.Fatalf("unguarded is %q", shelf["unguarded"].Version)
}
}
// A repository deleted and made again under the module's repository's name is another repository: the forge's
// id, recorded at registration, tells them apart.
func TestARepositoryMadeAgainUnderItsNameIsNotTheModulesSource(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/remade", forgeFacts{ID: 100, Guarded: true})
t.Cleanup(func() { theForge.Delete("novox/remade") })
first := onTrunk("build-remade-1", "novox/remade", "git", "", map[string]any{"module": "remade", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/remade", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "remade"); id != 100 {
t.Fatalf("the forge's id was not recorded: %d", id)
}
theForge.Store("novox/remade", forgeFacts{ID: 101, Guarded: true}) // deleted, and made again by an agent
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-2", "novox/remade", "git", "",
map[string]any{"module": "remade", "version": "evil"})); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "made again") {
t.Fatalf("a repository made again under the name was taken in: %v", err)
}
// And a new module from it, beside the one registered from the first, the same.
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-3", "novox/remade", "git", "modules/other",
map[string]any{"module": "other", "version": "1"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from a repository made again was taken in: %v", err)
}
}
// The terminal's mark is the operator's for the repository and path they asked: an outcome of another, under that
// build's id, is not theirs.
func TestATerminalRequestCoversOnlyWhatItAsked(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-asked", Repository: "novox/app",
Seat: "git", Path: "modules/app", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
other := onTrunk("build-asked", "agent/sudo", "git", "modules/app", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, other); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another repository under a terminal request's id was taken in: %v", err)
}
elsewhere := onTrunk("build-asked", "novox/app", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, elsewhere); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another path under a terminal request's id was taken in: %v", err)
}
}
// The serving controller, and everything it starts, is never the terminal: a build asked in it reads as asked
// through the mesh even when no verb and no caller is named.
func TestTheServingControllerIsNeverTheTerminal(t *testing.T) {
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
t.Setenv(servedVar, "")
if !startedAtTheTerminal() {
t.Fatal("a process started by hand is not the terminal")
}
markServed()
if startedAtTheTerminal() {
t.Fatal("the serving controller reads as the terminal")
}
child := exec.Command(os.Args[0], "-test.run=^$")
child.Env = os.Environ()
if !slices.Contains(child.Env, servedVar+"=1") {
t.Fatal("what the serving controller starts does not carry its mark")
}
}
// The forge applies one rule to a branch: the rule named for it, else the first glob that covers it. A stronger
// rule later in the list is not what guards the branch, and is not read as if it were (the confirmation review).
@@ -120,9 +369,16 @@ func TestTheRuleJudgedIsTheOneTheForgeApplies(t *testing.T) {
}
// An id the forge could not give keeps the id already recorded.
// **The operator's Approve of a reviewed group registers a moved module from its new repository** (novox/hq ADR 0420 §7,
// issue 1025). sudo is built from the catalogue; a merge into the repository of node-engine's module, announced by
// mesh-delivery with the ask it merged on, brings its build from there. Accepted with the operator's Approve of the
// merged head; refused with no kept merge, a warrant for other heads, a Decline, a person who is not the operator, and
// through a verb.
func TestAnUnknownIdentityKeepsTheRecordedOne(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 100); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 0); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "sudo"); id != 100 {
t.Fatalf("the recorded id became %d", id)
}
}
+118
View File
@@ -1,9 +1,13 @@
package main
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -11,12 +15,126 @@ import (
// the module registered with its source as the seat and path when the request said so — never the
// URL. A definition naming an installation is recorded and not registered; a failure is recorded
// and said.
func TestABuildHeardIsRecordedAndRegistered(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "shop", "version": "3"})
m, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, link.BuildResult{
ID: "b-1", Repository: "http://forge.internal:20000/novox/shop.git", Path: "modules/shop",
Ref: "main", On: "anchor", Commit: "abcdef0123", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/shop"},
}))
if err != nil {
t.Fatal(err)
}
if m.Module != "shop" {
t.Fatalf("registered %q", m.Module)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if _, held := shelf["shop"]; !held {
t.Fatal("the module a heard build produced is not in the catalogue")
}
src, err := open.inventory.SourceOf(ctx, "shop")
if err != nil || src.Seat != "git" || src.Repository != "novox/shop" || src.BuiltFrom != "abcdef0123" {
t.Fatalf("the source is the seat and the path, never the URL: %+v %v", src, err)
}
builds, err := open.inventory.Builds(ctx, "shop", 5)
if err != nil || len(builds) != 1 || builds[0].ID != "b-1" {
t.Fatalf("the build is not recorded once: %v %v", builds, err)
}
named, _ := json.Marshal(map[string]any{"module": "idp", "version": "1", "resources": []any{
map[string]any{"id": "server", "type": "container", "image": "x@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}}}})
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{
ID: "b-2", Repository: "/r", On: "anchor", Commit: "0123456789", Manifest: named})
if err == nil || !strings.Contains(err.Error(), "does not register it") {
t.Fatalf("a definition naming an installation was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["idp"].Module != "" {
t.Fatal("the refused module was registered anyway")
}
if builds, _ := open.inventory.Builds(ctx, "idp", 5); len(builds) != 1 {
t.Fatalf("the refused build was not recorded: %v", builds)
}
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{ID: "b-3", Repository: "/r", On: "anchor", Failed: "no compiler"})
if err == nil || !strings.Contains(err.Error(), "no compiler") {
t.Fatalf("a failure is said in the builder's words: %v", err)
}
}
// novox/hq 04-ISSUES/215: a build asked at a commit is recorded as built from that commit, and the
// module keeps following the branch it followed — a new one, the default branch.
func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "unifi", "version": "1"})
result := func(id, ref, commit string) link.BuildResult {
return link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/unifi", Ref: ref, On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result("b-1", "main", "1111111aaaa"))); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, result("b-2", "9c97a8a", "9c97a8a1d2c3")); err != nil {
t.Fatal(err)
}
src, err := open.inventory.SourceOf(ctx, "unifi")
if err != nil {
t.Fatal(err)
}
if src.Ref != "main" || src.BuiltFrom != "9c97a8a1d2c3" {
t.Errorf("after a build at a commit the module follows %q, built from %q; want main, 9c97a8a1d2c3", src.Ref, src.BuiltFrom)
}
// One new to the catalogue, first built at a commit, follows the default branch.
other, _ := json.Marshal(map[string]any{"module": "letta", "version": "1"})
r := result("b-3", "deadbeef", "deadbeefcafe")
r.Manifest, r.Path = other, "modules/letta"
if _, _, err := takeIn(ctx, open.inventory, r); err != nil {
t.Fatal(err)
}
if src, _ := open.inventory.SourceOf(ctx, "letta"); src.Ref != "" {
t.Errorf("a module first built at a commit follows %q, want the default branch", src.Ref)
}
}
// novox/hq 04-ISSUES/219: an older request heard after a newer one is recorded and not registered,
// so a push sends what the newer request built.
func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
result := func(asked time.Time, image string) link.BuildResult {
manifest, _ := json.Marshal(map[string]any{"module": "postgres", "version": image})
return link.BuildResult{ID: link.NewBuildID(asked), Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result(newer, "4bcd5f73"))); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
if !errors.Is(err, inventory.ErrSuperseded) {
t.Fatalf("the older request's outcome was taken in as current: %v", err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "4bcd5f73" {
t.Errorf("postgres is %q; want the newer request's 4bcd5f73", got)
}
if builds, _ := open.inventory.Builds(ctx, "postgres", 5); len(builds) != 2 {
t.Errorf("the late build was not recorded: %v", builds)
}
}
func TestABuildIDSaysWhenItWasAsked(t *testing.T) {
at := time.Date(2026, 10, 3, 21, 51, 57, 392539762, time.UTC)
@@ -1,92 +0,0 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **Every user's slow consumer is S9's, not the controller's alone** (novox/hq ADR 0357, issue 270). On
// 2026-10-07 the bus closed two clients on the laptop as slow consumers and nothing raised a condition:
// the controller heard only its own connection. Said by the bus module and read back, a user the bus
// closes is one condition naming the module and its machine, raised at the first look and cleared an hour
// after the last closure.
func TestAUserTheBusClosedAsASlowConsumerIsOneConditionNamingItsModuleAndMachine(t *testing.T) {
closed := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
now := closed.Add(20 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
advisories := link.BusClosures([]link.BusClosure{
{User: "laptop.claude-code", CID: 10, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Hour),
Closed: closed.Add(-10 * time.Minute)},
{User: "laptop.claude-code", CID: 44, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Minute),
Closed: closed},
{User: "anchor.desk", CID: 7, Reason: "Maximum Message Payload Exceeded", Opened: closed.Add(-time.Hour),
Closed: closed},
}, []string{link.AdvisorySlowConsumer, link.AdvisorySlowConsumer, link.AdvisoryConnectionClosed}, closed.Add(-time.Hour))
look := func() {
t.Helper()
f := &signalFacts{now: now, host: "anchor", advisories: advisories}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
}
look()
slow, found, err := conditions.ReadOne(t.Context(), store, "bus.laptop.claude-code.slow-consumer")
if err != nil || !found {
t.Fatalf("no slow-consumer condition for the user: %v %v", found, err)
}
if slow.Subject.Machine != "laptop" || slow.Observations != 2 || !slow.LastObserved.Equal(closed) {
t.Fatalf("the slow consumer is said on %q, %d closure(s), last %s: %+v", slow.Subject.Machine, slow.Observations,
slow.LastObserved, slow)
}
if !strings.Contains(slow.Summary, "claude-code on laptop") || strings.Contains(slow.Summary, "laptop.claude-code") {
t.Fatalf("the summary does not name the module in words, or carries the dotted user: %q", slow.Summary)
}
if !strings.Contains(slow.Headline, "Claude-code on laptop") {
t.Fatalf("the headline is %q", slow.Headline)
}
cut, found, err := conditions.ReadOne(t.Context(), store, "bus.anchor.desk.connection-closed")
if err != nil || !found || cut.Subject.Machine != "anchor" || !strings.Contains(cut.Summary, "maximum message payload exceeded") {
t.Fatalf("the user cut off for breaking the bus's rules: %v %v %+v", found, err, cut)
}
// An hour after the newest closure without another, both clear.
now = closed.Add(time.Hour + time.Minute)
look()
for _, key := range []string{"bus.laptop.claude-code.slow-consumer", "bus.anchor.desk.connection-closed"} {
if _, found, _ := k.Get(t.Context(), key); found {
t.Fatalf("%s is still open an hour after the last closure", key)
}
}
}
// **A read of the bus module's events that fails does not blind S9** (review of #326): the controller's own
// signals are still said, and the failed read is a condition of its own.
func TestABusModuleReadThatFailsKeepsTheControllersOwnSignals(t *testing.T) {
at := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
f := &signalFacts{now: at.Add(time.Minute), host: "anchor",
advisories: []link.Advisory{{Kind: link.AdvisorySlowConsumer, ID: "controller", Said: "the controller fell behind",
First: at, Last: at, Count: 1}},
busFaultsErr: errors.New("nats: timeout")}
keys := map[string]bool{}
for _, o := range watchAdvisories(f) {
keys[o.Key()] = true
}
if !keys["bus.controller.slow-consumer"] || !keys["bus.bus-module.bus-faults-unread"] {
t.Fatalf("observed %v: the controller's own slow consumer and the unread events, both", keys)
}
for _, row := range signalsTable {
if row.Row == "S9" && row.needs(f) != nil {
t.Fatal("the S9 row went blind for a read of the bus module's events")
}
}
}
@@ -1,70 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
func TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds(t *testing.T) {
now := time.Date(2026, 10, 8, 18, 30, 0, 0, time.UTC)
b := aBusOnAnchor()
walk := func(id, repo, note string, updated time.Time) inventory.Plan {
return inventory.Plan{ID: id, Repository: repo, Commit: "c0ffee001122", State: inventory.PlanRolling, Tier: 0,
Tiers: [][]string{{"mesh-host"}}, Modules: map[string]*inventory.PlanModule{"mesh-host": {}}, Note: note,
Updated: updated}
}
for _, c := range []struct {
name string
bus busPending
plans []inventory.Plan
first map[string]time.Time
want []string
since time.Time
}{
{"a walk refused for the bus", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-27*time.Minute))}, nil,
[]string{"plan-1"}, now.Add(-27 * time.Minute)},
{"refused earlier than its last save, as this controller saw it", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-5*time.Minute))},
map[string]time.Time{"plan-1": now.Add(-28 * time.Minute)}, []string{"plan-1"}, now.Add(-28 * time.Minute)},
{"a walk refused for another reason", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", "tier 0: the build seat is paused — tried again", now)}, nil,
nil, time.Time{}},
{"refused for an older bus build than the one held now", func() busPending { o := b; o.to = "99999999cccc"; return o }(),
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"the bus already runs the build held: its step started", func() busPending {
o := aBusOnAnchor()
o.from = map[string]string{"anchor": o.to}
return o
}(), []inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"a walk waiting for its delivery's word asks no send", b, func() []inventory.Plan {
p := walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)
p.Delivery = &inventory.PlanDelivery{Awaits: "mesh-delivery"}
return []inventory.Plan{p}
}(), nil, nil, time.Time{}},
} {
t.Run(c.name, func(t *testing.T) {
got := busWaitsOf(c.plans, c.bus, func(id string) time.Time { return c.first[id] })
var ids []string
for _, w := range got.waits {
ids = append(ids, w.plan)
}
if strings.Join(ids, ",") != strings.Join(c.want, ",") {
t.Fatalf("held for the bus: %v, want %v", ids, c.want)
}
if len(c.want) > 0 {
if !got.waits[0].since.Equal(c.since) {
t.Errorf("waiting since %s, want %s", got.waits[0].since, c.since)
}
if strings.Join(got.machines, ",") != "anchor" || got.to != b.to || got.module != "nats" {
t.Errorf("behind %+v", got)
}
}
})
}
}
+60
View File
@@ -7,6 +7,7 @@ import (
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 336: a send held because it would replace the bus outside its planned step waits for a
@@ -26,6 +27,65 @@ func refusedNote(b busPending, tier int) string {
return "tier " + string(rune('0'+tier)) + ": " + errBusWaits.Error() + ": " + held + " — tried again"
}
func TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds(t *testing.T) {
now := time.Date(2026, 10, 8, 18, 30, 0, 0, time.UTC)
b := aBusOnAnchor()
walk := func(id, repo, note string, updated time.Time) inventory.Plan {
return inventory.Plan{ID: id, Repository: repo, Commit: "c0ffee001122", State: inventory.PlanRolling, Tier: 0,
Tiers: [][]string{{"mesh-host"}}, Modules: map[string]*inventory.PlanModule{"mesh-host": {}}, Note: note,
Updated: updated}
}
for _, c := range []struct {
name string
bus busPending
plans []inventory.Plan
first map[string]time.Time
want []string
since time.Time
}{
{"a walk refused for the bus", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-27*time.Minute))}, nil,
[]string{"plan-1"}, now.Add(-27 * time.Minute)},
{"refused earlier than its last save, as this controller saw it", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-5*time.Minute))},
map[string]time.Time{"plan-1": now.Add(-28 * time.Minute)}, []string{"plan-1"}, now.Add(-28 * time.Minute)},
{"a walk refused for another reason", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", "tier 0: the build seat is paused — tried again", now)}, nil,
nil, time.Time{}},
{"refused for an older bus build than the one held now", func() busPending { o := b; o.to = "99999999cccc"; return o }(),
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"the bus already runs the build held: its step started", func() busPending {
o := aBusOnAnchor()
o.from = map[string]string{"anchor": o.to}
return o
}(), []inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"a walk waiting for its delivery's word asks no send", b, func() []inventory.Plan {
p := walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)
p.Delivery = &inventory.PlanDelivery{Awaits: "mesh-delivery"}
return []inventory.Plan{p}
}(), nil, nil, time.Time{}},
} {
t.Run(c.name, func(t *testing.T) {
got := busWaitsOf(c.plans, c.bus, func(id string) time.Time { return c.first[id] })
var ids []string
for _, w := range got.waits {
ids = append(ids, w.plan)
}
if strings.Join(ids, ",") != strings.Join(c.want, ",") {
t.Fatalf("held for the bus: %v, want %v", ids, c.want)
}
if len(c.want) > 0 {
if !got.waits[0].since.Equal(c.since) {
t.Errorf("waiting since %s, want %s", got.waits[0].since, c.since)
}
if strings.Join(got.machines, ",") != "anchor" || got.to != b.to || got.module != "nats" {
t.Errorf("behind %+v", got)
}
}
})
}
}
// **Said at once, not as lateness, and cleared when the step starts**: the walk held only for the bus raises
// the bus's condition on the first tick, before any tier bound, naming what waits, the bus build from and to,
// since when and `bus upgrade`, for the operator; S3 says nothing of it even past its bound; and the
+5 -42
View File
@@ -49,12 +49,10 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus
// by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28).
consumers, records, err := moduleConsumersAndRecords(ctx, inv)
consumers, err := moduleConsumers(ctx, inv)
if err != nil {
return nil, err
}
// And a grant of a restricted stream that composes nothing, said on every raise and send (novox/hq ADR 0376 §2).
sayWithheld(records, " ")
// And the work queues of seats that name their caller or their kind, with each holder's worker
// (novox/hq ADR 0259 §3): an ask queues until the router takes it, a channel's work until that kind
// takes it.
@@ -62,19 +60,9 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err != nil {
return nil, err
}
// And every module's own stream (novox/hq ADR 0363), from the catalogue, before the consumers on it.
modStreams, err := inv.DeclaredStreams(ctx)
if err != nil {
return nil, err
}
// Every one tried, and every failure named: one module's consumer the bus refuses is no reason
// the modules after it in the list hear nothing (novox/hq issue 208, where this runs on each send).
var failed []error
for _, s := range broker.ModuleStreams(modStreams) {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the module stream %s: %w", s.Name, err))
}
}
for _, s := range trafficStreams {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the work queue %s: %w", s.Name, err))
@@ -85,30 +73,11 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
failed = append(failed, fmt.Errorf("the worker %s on %s: %w", c.Name, c.Stream, err))
}
}
restricted := map[string]bool{}
for _, s := range modStreams {
restricted[s.Name] = s.Restricted
}
for _, c := range consumers {
if err := broker.EnsureHeard(r, c.Consumer, restricted[c.Consumer.Stream]); err != nil {
if err := r.EnsureConsumer(c.Consumer); err != nil {
failed = append(failed, fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err))
}
}
// **And a consumer no longer granted loses what it reads through** (novox/hq ADR 0363 §6): every consumer
// on a module's stream the composition no longer asks for is removed, on this very send.
if p, prunes := r.(broker.Pruner); prunes && len(modStreams) > 0 {
var wanted []broker.Consumer
for _, c := range consumers {
wanted = append(wanted, c.Consumer)
}
removed, err := broker.PruneHeard(p, modStreams, wanted)
if err != nil {
failed = append(failed, err)
}
for _, gone := range removed {
fmt.Printf(" removed the consumer %s: its module is no longer granted that stream (novox/hq ADR 0363)\n", gone)
}
}
if len(failed) > 0 {
return nil, errors.Join(failed...)
}
@@ -167,21 +136,15 @@ func unassertedObservation(err error) conditions.Observation {
// moduleConsumers is every module's durable consumer, from the records the user list is composed from.
func moduleConsumers(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, error) {
consumers, _, err := moduleConsumersAndRecords(ctx, inv)
return consumers, err
}
// moduleConsumersAndRecords is moduleConsumers with the records they were composed from.
func moduleConsumersAndRecords(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, broker.Records, error) {
records, err := inv.BusRecords(ctx)
if err != nil {
return nil, broker.Records{}, err
return nil, err
}
users, err := broker.Users(records)
if err != nil {
return nil, broker.Records{}, err
return nil, err
}
return broker.ConsumersOf(users), records, nil
return broker.ConsumersOf(users), nil
}
// seatTrafficObjects is the work queues and workers of seats that name their caller or their kind, from
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
-103
View File
@@ -1,103 +0,0 @@
package main
import (
"fmt"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A first-node gate waits for a busy container (novox/hq ADR 0335 §4).
//
// When a machine reports, on what the gate sent it, that a container of a judged module was left running
// because it said busy, the judging has not started: the build is not yet what runs there. The walk says it
// waits for a busy container on that machine, counts no pass and no bound, and starts the judging — its
// Since — at the report that no longer lists the container waiting. The wait is the node-engine's, bounded
// at 15 minutes; the controller's own bound (busyGateBound past the engine's) is for an engine that stopped
// reporting while it waited, so no walk waits for ever. An engine older than the field never lists one, and
// its gate starts as before.
// busyGateSlack is how long past the engine's bound the controller still waits for the report that says the
// container replaced: a pass of the engine (its asks every 20 seconds) and the report's way here.
var busyGateSlack = 5 * time.Minute
// busyEngineBound is the node-engine's bound on a wait (novox/hq ADR 0335 §3), counted again here on this
// controller's clock from when it first read the wait.
const busyEngineBound = 15 * time.Minute
// busyGateEnds is when the gate stops waiting: the engine's own word on its bound, or this controller's count of
// it from when it first read the wait, whichever is sooner, and the slack — so neither a clock ahead on the
// machine nor a wrong word from it holds a walk longer.
func busyGateEnds(b *inventory.GateBusy) time.Time {
until := b.Seen.Add(busyEngineBound)
if !b.Until.IsZero() && b.Until.Before(until) {
until = b.Until
}
return until.Add(busyGateSlack)
}
// busyWait decides, before a judging, whether the gate waits for a busy container on a machine it judges:
// true while it waits, the gate saying so. False when nothing waits, or no longer: a wait that ended moves the
// gate's Since to its end. Pure but for the gate it is handed.
func busyWait(g *inventory.PlanGate, pairs []judged, f gateFacts, now time.Time) bool {
if g.Busy != nil && g.Busy.Ended != nil {
return false
}
for _, j := range pairs {
b, waits := busyOn(j, f)
if !waits {
continue
}
if g.Busy == nil {
g.Busy = &inventory.GateBusy{Machine: j.node, Module: j.module, Container: b.Container, Since: b.Since,
Seen: now}
}
if g.Busy.Seen.IsZero() {
g.Busy.Seen = now
}
g.Busy.Until, g.Busy.Said = b.Until, b.Said
if now.After(busyGateEnds(g.Busy)) {
// The engine said it would have replaced it by now: the judging starts, and finds the module not
// yet healthy while the machine still lists it waiting (busyStill), until the gate's own bound.
end := now
g.Busy.Ended, g.Busy.PastBound = &end, true
g.Since, g.Passes, g.LastPass, g.Healthy, g.HealthyAt, g.Applied = &end, 0, nil, nil, nil, nil
return false
}
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, g.Busy.Words(), nil
g.Read(now, false, g.Busy.Words())
return true
}
if g.Busy == nil {
return false
}
// Replaced: the judging starts at the report that said so.
end := now
if r, said := f.reports[g.Busy.Machine]; said && r.At != nil && r.At.After(g.Busy.Since) && r.At.Before(now) {
end = r.At.UTC()
}
g.Busy.Ended = &end
g.Since, g.Passes, g.LastPass, g.Last, g.Healthy, g.HealthyAt, g.Applied = &end, 0, nil, "", nil, nil, nil
return false
}
// busyOn is the container of a judged module its machine last reported, on what the gate sent it, as left
// running because it said busy.
func busyOn(j judged, f gateFacts) (inventory.BusyWait, bool) {
r, said := f.reports[j.node]
if !said || r.At == nil || !f.reportedOn(j.node, r) {
return inventory.BusyWait{}, false
}
return inventory.WaitingFor(r.Busy, j.module)
}
// busyStill is a judged module whose container its machine still lists waiting once the gate stopped waiting
// for it: not yet what was sent, so not healthy, whatever its health says of the container still running.
func busyStill(j judged, f gateFacts) (string, bool) {
b, waits := busyOn(j, f)
if !waits {
return "", false
}
return fmt.Sprintf("%s still runs its busy container %s, past its node-engine's bound of %s",
j.node, b.Container, b.Until.UTC().Format(time.RFC3339)), true
}
-148
View File
@@ -1,148 +0,0 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A walk whose module's container waits as busy on its first machine says so, and starts that machine's
// first-machine gate only once the machine reports the container replaced (novox/hq ADR 0335 §4).
func TestAGateWaitsForABusyContainerAndStartsWhenItIsReplaced(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent,
Sent: map[string]inventory.SentDeclaration{"novox": {Digest: "d1", Sequence: 7}}}
pairs := []judged{{module: "build-agent", node: "novox"}, {module: "mesh-controller", node: "novox"}}
reported := sent.Add(20 * time.Second)
waiting := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: reported, Until: reported.Add(15 * time.Minute),
Said: "busy: novox/mesh-controller (check of #280), in hand since 00:58:00Z"}}
facts := func(at time.Time, busy []inventory.BusyWait) gateFacts {
// As judgeMoves hands it: the report held against the gate's own send.
return gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"novox": {Node: "novox",
Outcome: inventory.OutcomeApplied, At: &at, Declared: "d1", ReportedSequence: 7, Busy: busy}}}
}
now := reported.Add(time.Minute)
if !busyWait(g, pairs, facts(reported, waiting), now) {
t.Fatal("a gate whose module's container waits as busy judged it")
}
if !g.Since.Equal(sent) || g.Busy == nil || g.Busy.Machine != "novox" || g.Busy.Ended != nil {
t.Fatalf("the gate does not keep the wait: %+v %+v", g.Since, g.Busy)
}
for _, said := range []string{gateLine(g), g.Last} {
if !strings.Contains(said, "waits for a busy container on novox") {
t.Errorf("the walk does not say it waits for a busy container there: %s", said)
}
}
if !strings.Contains(g.Last, "check of #280") {
t.Errorf("the wait does not say what the container holds: %s", g.Last)
}
// Ten minutes on, still busy, within the engine's bound: still waiting, nothing judged, no bound run.
now = reported.Add(10 * time.Minute)
if !busyWait(g, pairs, facts(reported.Add(9*time.Minute), waiting), now) || g.Passes != 0 {
t.Fatalf("the gate stopped waiting within the bound: %+v", g)
}
// The machine reports it replaced: the judging starts at that report.
replacedAt := reported.Add(12 * time.Minute)
now = replacedAt.Add(5 * time.Second)
replaced := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: "replaced-when-idle", Since: reported, Until: reported.Add(15 * time.Minute)}}
if busyWait(g, pairs, facts(replacedAt, replaced), now) {
t.Fatal("the gate still waits after the container was replaced")
}
if g.Since == nil || !g.Since.Equal(replacedAt) || g.Busy.Ended == nil || !g.Busy.Ended.Equal(replacedAt) {
t.Fatalf("the gate does not start at the report that said it replaced: since %v, %+v", g.Since, g.Busy)
}
if strings.Contains(gateLine(g), "waits for a busy container") {
t.Errorf("a gate judging still says it waits: %s", gateLine(g))
}
// And a later report listing it waiting again does not hold a gate that started.
if busyWait(g, pairs, facts(now, waiting), now.Add(time.Minute)) {
t.Error("a gate that started waits again")
}
}
// An engine that stops reporting while it waits holds no walk for ever: past the engine's bound and the slack,
// the judging starts, and finds the module not yet healthy while its container is still listed waiting.
func TestAGateWaitsNoLongerThanTheEnginesBound(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent}
pairs := []judged{{module: "build-agent", node: "novox"}}
waiting := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: sent, Until: sent.Add(15 * time.Minute)}}
f := gateFacts{reports: map[string]inventory.Reported{"novox": {Node: "novox", Outcome: inventory.OutcomeApplied,
At: &sent, Current: true, Busy: waiting}}}
now := sent.Add(15*time.Minute + busyGateSlack + time.Second)
if busyWait(g, pairs, f, now) {
t.Fatal("a gate waited past the engine's bound and the slack")
}
if !g.Busy.PastBound || !g.Since.Equal(now) {
t.Fatalf("the gate does not start, said past the bound: %+v %v", g.Busy, g.Since)
}
if why, still := busyStill(pairs[0], f); !still || !strings.Contains(why, "past its node-engine's bound") {
t.Fatalf("a module still running its busy container reads as what was sent: %q", why)
}
}
// A report from an engine older than the field lists nothing: the gate starts at the send, as before; and a
// report not about the gate's send says nothing of it.
func TestAGateOfAnEngineThatNeverWaitsStartsAsBefore(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"ace"}, Since: &sent,
Sent: map[string]inventory.SentDeclaration{"ace": {Digest: "d2", Sequence: 9}}}
pairs := []judged{{module: "build-agent", node: "ace"}}
at := sent.Add(time.Minute)
older := gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"ace": {Node: "ace", Outcome: inventory.OutcomeApplied,
At: &at, Declared: "d2", ReportedSequence: 9}}}
if busyWait(g, pairs, older, at) || g.Busy != nil || !g.Since.Equal(sent) {
t.Fatalf("a gate on an engine that never waits did not start at the send: %+v %v", g.Busy, g.Since)
}
elsewhere := gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"ace": {Node: "ace", Outcome: inventory.OutcomeApplied,
At: &at, Declared: "d1", ReportedSequence: 8, Busy: []inventory.BusyWait{{ID: "build-agent.server",
State: inventory.BusyWaiting, Until: at.Add(time.Hour)}}}}}
if busyWait(g, pairs, elsewhere, at) {
t.Fatal("a wait reported about an earlier declaration held the gate")
}
}
// The wait is bounded on this controller's clock: an engine whose clock runs an hour ahead, saying it waits until
// then, holds the gate no longer than 15 minutes and the slack from when the controller first read the wait.
func TestAGateWaitIsBoundedOnTheControllersClock(t *testing.T) {
seen := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &seen}
pairs := []judged{{module: "build-agent", node: "novox"}}
ahead := seen.Add(time.Hour)
f := gateFacts{reports: map[string]inventory.Reported{"novox": {Node: "novox", Outcome: inventory.OutcomeApplied,
At: &seen, Current: true, Busy: []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: ahead, Until: ahead.Add(15 * time.Minute)}}}}}
if !busyWait(g, pairs, f, seen) || !g.Busy.Seen.Equal(seen) {
t.Fatalf("the gate did not start waiting, seen now: %+v", g.Busy)
}
if !busyWait(g, pairs, f, seen.Add(15*time.Minute+busyGateSlack-time.Second)) {
t.Fatal("the gate stopped waiting within the bound on its own clock")
}
if busyWait(g, pairs, f, seen.Add(15*time.Minute+busyGateSlack+time.Second)) || !g.Busy.PastBound {
t.Fatalf("an engine's later word held the gate past the controller's bound: %+v", g.Busy)
}
}
// `plan` says, for a machine whose node-engine is older than the field, each container whose busy is not sent.
func TestAPlanSaysABusyAnOlderEngineIsNotSent(t *testing.T) {
modules := []catalogue.Manifest{{Module: "build-agent", Resources: []map[string]any{
{"id": "server", "type": "container", "busy": map[string]any{"command": "mesh-builder busy"}},
{"id": "agent-env", "type": "file"}}}}
lines := busyNotSent(modules, false)
if len(lines) != 1 || !strings.Contains(lines[0], "build-agent.server declares busy") ||
!strings.Contains(lines[0], "replaced at once") {
t.Fatalf("the plan does not say the busy not sent: %v", lines)
}
if lines := busyNotSent(modules, true); len(lines) != 0 {
t.Fatalf("a busy sent is said as not sent: %v", lines)
}
}
@@ -1,36 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"strings"
"testing"
)
func TestCallsIsServedAndSaysWhatItKeeps(t *testing.T) {
handlers, behind, err := seatToolHandlers()
if err != nil || len(behind) != 0 {
t.Fatalf("%v %v", behind, err)
}
calls, ok := handlers["calls"]
if !ok {
t.Fatal("calls is not served")
}
if _, err := calls(context.Background(), json.RawMessage(`{"node":"anchor"}`)); err == nil ||
!strings.Contains(err.Error(), `"node"`) {
t.Errorf("calls took an argument it does not declare: %v", err)
}
if _, err := calls(context.Background(), json.RawMessage(`{"call":"call-0-0"}`)); err == nil ||
!strings.Contains(err.Error(), "not across a restart") {
t.Errorf("an unknown call was not said plainly: %v", err)
}
got, err := calls(context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if _, listed := got.(map[string]any)["calls"]; !listed {
t.Errorf("calls answered %v", got)
}
}
+28
View File
@@ -1,6 +1,9 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
@@ -49,3 +52,28 @@ func TestAPushAnswersBeforeItSends(t *testing.T) {
}
// `calls` is served, takes a call's id and nothing else, and says plainly when it holds no such call.
func TestCallsIsServedAndSaysWhatItKeeps(t *testing.T) {
handlers, behind, err := seatToolHandlers()
if err != nil || len(behind) != 0 {
t.Fatalf("%v %v", behind, err)
}
calls, ok := handlers["calls"]
if !ok {
t.Fatal("calls is not served")
}
if _, err := calls(context.Background(), json.RawMessage(`{"node":"anchor"}`)); err == nil ||
!strings.Contains(err.Error(), `"node"`) {
t.Errorf("calls took an argument it does not declare: %v", err)
}
if _, err := calls(context.Background(), json.RawMessage(`{"call":"call-0-0"}`)); err == nil ||
!strings.Contains(err.Error(), "not across a restart") {
t.Errorf("an unknown call was not said plainly: %v", err)
}
got, err := calls(context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if _, listed := got.(map[string]any)["calls"]; !listed {
t.Errorf("calls answered %v", got)
}
}
-5
View File
@@ -73,11 +73,6 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
"(issue 280)", busUpgradeNeeded, name, busUpgradeVerb, strings.Join(e.On, ", ")))
case name == "nats" || catalogue.ProvidesBus(e.Manifest):
p.Steps = append(p.Steps, "a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send")
case waits && len(e.On) > 0 && (u.From == catalogue.FromModule || u.From == catalogue.FromData):
// Judged per machine by the walk, against the build each was last sent (novox/hq ADR 0377).
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s) — unless on a machine "+
"its move recreates none of its containers and moves none of its data, which the walk judges and then sends",
name, orNone(u.From)))
case waits && len(e.On) > 0:
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s)", name,
orNone(u.From)))
+9 -61
View File
@@ -11,7 +11,6 @@ import (
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// moduleCheck judges manifests where they are written, with no mesh (novox/hq ADR 0037, issue 148).
@@ -48,24 +47,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
shelf := catalogue.Shelf{}
faulted := map[string]bool{}
failed := 0
// The list of predefined keys at the root of the repository above each manifest (novox/hq ADR 0348 §7), read
// once per repository: a manifest naming a key is judged against the list its own tree holds.
lists := map[string]catalogue.PredefinedList{}
listOf := func(path string) (catalogue.PredefinedList, error) {
root := catalogue.RepositoryRoot(filepath.Dir(path))
if root == "" {
return nil, nil
}
if list, read := lists[root]; read {
return list, nil
}
list, err := catalogue.ReadPredefined(root)
if err != nil {
return nil, err
}
lists[root] = list
return list, nil
}
for _, path := range paths {
raw, err := os.ReadFile(path)
if err != nil {
@@ -73,18 +54,7 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed++
continue
}
list, err := listOf(path)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
continue
}
m, err := catalogue.ParseManifestWith(raw, list)
if err == nil {
// Each template's text read from the tree beside the manifest, and judged as inline content is
// (novox/hq ADR 0349 §2): a missing template is refused by name.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
@@ -105,30 +75,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed += len(wrong)
faulted[m.Module] = true
}
// A module that declares a setting names the keys of its mergeable files (novox/hq ADR 0326 §3,
// issue 574): judged where a definition is written, as at registration, never where a stored
// build is read again.
if wrong := catalogue.WrittenSettingsProblems(m); len(wrong) > 0 {
for _, p := range wrong {
fmt.Fprintf(out, "%s: %s\n", path, p)
}
failed += len(wrong)
faulted[m.Module] = true
}
// A module lives whole in one repository (novox/hq ADR 0420 §1, issue 1019): when the check is told the
// repository it runs for (MESH_CHECK_REPOSITORY, set by the build's check), an artifact building from
// another is refused, as the merge gate refuses it. A run by hand names no repository and judges none.
if repo := os.Getenv("MESH_CHECK_REPOSITORY"); repo != "" && m.Build != nil {
for _, a := range m.Build.Artifacts {
if a.Context != nil && a.Context.Repository != "" && !sameRepository(a.Context.Repository,
link.SourceMoved{Owner: ownerOf(repo), Repo: repoOf(repo)}) {
fmt.Fprintf(out, "%s: %s builds artifact %s from %s: a module lives whole in one repository "+
"(novox/hq ADR 0420 §1)\n", path, m.Module, a.Name, a.Context.Repository)
failed++
faulted[m.Module] = true
}
}
}
if named := catalogue.InstallationProblems(m); len(named) > 0 {
for _, p := range named {
fmt.Fprintf(out, "%s: %s\n", path, p)
@@ -206,6 +152,14 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
}
// **A module that uses several libraries is warned** (novox/hq ADR 0328 §4): an image that drops added groups when
// it changes user reaches only the first. Printed, never counted.
for _, name := range names {
for _, w := range catalogue.LibraryWarnings(shelf[name]) {
fmt.Fprintf(out, "warning: %s\n", w)
}
}
// **A file that asks for a setting and does not say whether it is trusted counts as trusted** (novox/hq issue
// 339): listed and counted, never refused, so an author can opt out a file nothing trusts.
unsaid := 0
@@ -259,12 +213,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
checks = append(checks, fmt.Sprintf("%v by %s", r["id"], catalogue.HealthWords(h)))
}
}
// And its own runtime's, where it runs as its own account (novox/hq ADR 0366).
if own, has := catalogue.OwnRuntimeHealth(m); has {
if h, _, _ := catalogue.ReadHealth(own); h.Kind != "" {
checks = append(checks, fmt.Sprintf("%v by %s", own["id"], catalogue.HealthWords(h)))
}
}
if len(checks) > 0 {
fmt.Fprintf(out, ", ready: %s", strings.Join(checks, "; "))
}
-87
View File
@@ -1,87 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"strconv"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
)
// A pull request's check builds go when it closes (novox/hq ADR 0350 §5): the build seat keeps them in the artifact
// store's scratch namespace for the walk of its merge to take. **Let go of where the moment is heard**: the
// controller ends the walk answering a merge — after that walk took what it reuses — and the delivery's owner, which
// hears the forge say a pull request closed unmerged, asks the `forget-checks` verb; both let go through the store,
// as the controller's own collection does. The build seat's holders let go of anything a week old on their own,
// so a word lost here leaves a week's builds, never more.
// checkBuildsIn is the scratch namespace as the controller reaches the artifact store.
func checkBuildsIn(ctx context.Context, inv *inventory.Inventory) (builder.Scratch, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, errors.New("the artifact store is not on the mesh's network")
}
return builder.Registry{Address: address}, nil
}
// forgetChecks lets go of a pull request's check builds and answers what it did.
var forgetChecks = func(ctx context.Context, inv *inventory.Inventory, pull builder.PullRef) (string, error) {
scratch, err := checkBuildsIn(ctx, inv)
if err != nil {
return "", err
}
n, err := scratch.Forget(ctx, pull)
if err != nil {
return "", fmt.Errorf("%s: %d check(s)' builds let go of, and not all: %w", pull, n, err)
}
return fmt.Sprintf("%s: %d check(s)' builds let go of", pull, n), nil
}
// forgetChecksOfWalk lets go, on the save that ends a walk, of the check builds of every pull request whose merge
// it answers: the walk took what it reuses. Said, never fatal.
func forgetChecksOfWalk(ctx context.Context, inv *inventory.Inventory, p inventory.Plan) {
if p.WalkSaid != inventory.WalkEnded || p.Delivery == nil {
return
}
for _, pull := range reuseAskOf(&p).Pulls {
said, err := forgetChecks(ctx, inv, builder.PullRef{Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number})
if err != nil {
fmt.Fprintf(os.Stderr, "%s: its check builds stay until the week's sweep: %v\n", p.ID, err)
continue
}
fmt.Printf("%s: %s\n", p.ID, said)
}
}
// forgetChecksCommand is `forget-checks <owner> <repo> <number>`: the verb the delivery's owner asks when the forge
// says a pull request closed unmerged (novox/hq ADR 0350 §5).
func forgetChecksCommand(ctx context.Context, args []string) error {
if len(args) != 3 {
return errors.New("forget-checks <owner> <repo> <number>")
}
number, err := strconv.Atoi(args[2])
if err != nil || number <= 0 {
return fmt.Errorf("%q is not a pull request's number", args[2])
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
said, err := forgetChecks(ctx, open.inventory, builder.PullRef{Owner: args[0], Repo: args[1], Number: number})
if err != nil {
return err
}
fmt.Println(said)
return nil
}
@@ -1,102 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func TestOnlyAWalksAskTakesAChecksBuildAndADryRunNever(t *testing.T) {
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}, {Repository: "http://forge.invalid/novox/web.git", Number: 3},
{Repository: "novox/shop", Number: 7}, {Repository: "novox/shop"}}}}
asked := reuseAskOf(p)
if want := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}, {Owner: "novox", Repo: "web", Number: 3}}; !reflect.DeepEqual(asked.Pulls, want) {
t.Fatalf("the walk names %+v", asked.Pulls)
}
walk := withReuse(t.Context(), asked)
if reuseAsked(walk, false) != asked {
t.Fatal("a walk's ask does not carry its leave")
}
if reuseAsked(walk, true) != nil {
t.Fatal("a dry run carries a leave to take a check's build")
}
if reuseAsked(t.Context(), false) != nil {
t.Fatal("an ask that is not a walk's carries a leave")
}
}
func TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
gateEvery = 0
digest := "sha256:" + strings.Repeat("c", 64)
reused := func(id, reference string) link.BuildResult {
manifest := mustJSON(t, catalogue.Manifest{Module: "app", Version: "c3", Resources: []map[string]any{
{"id": "svc", "type": "container", "name": "app", "image": reference}}})
return link.BuildResult{ID: id, Repository: "novox/mesh-catalog", Path: "modules/app", Ref: "main", On: "anchor",
Commit: "c3", Manifest: manifest, Trunk: "main", OnTrunk: true, Branches: []string{"main"},
Made: []link.MadeArtifact{{Name: "server", Kind: "image", Reference: reference}},
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"},
Reused: "reused from the check of novox/mesh-catalog#9 at head abcdefabcdef (merged into trunk 123456123456): server " + digest,
ReusedFrom: &link.ReusedFrom{Pull: link.PullRef{Owner: "novox", Repo: "mesh-catalog", Number: 9},
Head: "abcdefabcdef", Trunk: "123456123456", Key: "r1:" + strings.Repeat("1", 64)}}
}
if _, _, err := takeIn(ctx, inv, reused(link.NewBuildID(time.Now()), "store.internal:5000/scratch/app/server@"+digest)); err == nil ||
!strings.Contains(err.Error(), "scratch") {
t.Fatalf("a build in the scratch namespace was registered: %v", err)
}
id := link.NewBuildID(time.Now().Add(time.Second))
result := reused(id, "store.internal:5000/app/server@"+digest)
if _, _, err := takeIn(ctx, inv, result); err != nil {
t.Fatal(err)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil || current["app"].Commit != "c3" {
t.Fatalf("the reused build is not what app is registered at: %+v %v", current["app"], err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(shelf["app"].Resources)
if !strings.Contains(string(raw), "app/server@"+digest) {
t.Fatalf("app is registered as %s, not by the check's digest in its own namespace", raw)
}
// The walk asked it: the walk keeps what its build did, and its gate judges the registered build.
asked := time.Now().UTC()
p := g.plan(t)
p.Commit = "c3"
p.Modules["app"] = &inventory.PlanModule{State: "asked", AskedAt: &asked, Build: id}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
planBuiltSaying(ctx, g.open, "app", "c3", "", asked, id, result.MadeWords())
p = g.plan(t)
if m := p.Modules["app"]; m.State != "built" || !strings.HasPrefix(m.Made, "reused from the check of novox/mesh-catalog#9") {
t.Fatalf("the walk keeps %+v", m)
}
for i := 0; i < 6; i++ {
advancePlans(ctx, g.open)
}
p = g.plan(t)
gate := p.Modules["app"].Gate
if gate == nil || gate.To != "c3" || gate.Verdict != inventory.GatePassed {
t.Fatalf("the gate judged %+v, not the reused build", gate)
}
if sent, _, _ := inv.SentBuilds(ctx, "anchor"); sent["app"] != "c3" {
t.Fatalf("the first machine was sent %v, not the reused build", sent)
}
}
-193
View File
@@ -1,193 +0,0 @@
package main
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// One build per change (novox/hq ADR 0350), as the controller asks it and takes it in.
// The check of a change is asked to build what a merge of it would move, in the walk's order: a module of the
// repository checked from the merge, one standing on it from its own repository as the walk builds it, and one
// that reads the repository through a context not at all. A change that moves no module builds nothing.
func TestACheckIsAskedToBuildWhatTheMergeWouldMove(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "app"}, Source: inventory.Source{Repository: "novox/shop", Seat: "git", Path: "modules/app"}},
{Manifest: catalogue.Manifest{Module: "web"}, Source: inventory.Source{Repository: "novox/web", Seat: "git", Ref: "main"}},
{Manifest: catalogue.Manifest{Module: "docs"}, Source: inventory.Source{Repository: "novox/docs", Seat: "git"}},
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Base: "main", Commit: "abc"}
scope := checkScope{Reach: mergeReach{Packaging: entries[2:], Plan: inventory.Plan{Tiers: [][]string{{"app", "docs"}, {"web"}}}}}
clone := func(s inventory.Source) (string, error) { return "http://forge.invalid/" + s.Repository + ".git", nil }
got := checkBuildsOf(p, scope, entries, clone)
want := []link.CheckBuild{
{Module: "app", Path: "modules/app", Merged: true, Tier: 0},
{Module: "docs", Tier: 0, Skip: "its build reads novox/shop through a context, which a check cannot hand the merge"},
{Module: "web", Repository: "http://forge.invalid/novox/web.git", Ref: "main", Tier: 1},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("the check is asked to build\n%+v\nwant\n%+v", got, want)
}
if got := checkBuildsOf(p, checkScope{}, entries, clone); got != nil {
t.Fatalf("a change that moves no module asks for builds: %+v", got)
}
}
// A walk's ask carries its leave to take a check's build, naming the pull requests of the merges it answers; a dry
// run never does, and neither does an ask that is not a walk's.
// A walk ended lets go of its merges' check builds — only once it ended, after it took what it reuses.
func TestAWalkEndedLetsGoOfItsMergesCheckBuilds(t *testing.T) {
var forgot []builder.PullRef
was := forgetChecks
forgetChecks = func(_ context.Context, _ *inventory.Inventory, pull builder.PullRef) (string, error) {
forgot = append(forgot, pull)
return "", nil
}
t.Cleanup(func() { forgetChecks = was })
p := inventory.Plan{ID: "plan-1", Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}}}}
p.WalkSaid = inventory.WalkStarted
forgetChecksOfWalk(t.Context(), nil, p)
p.WalkSaid = ""
forgetChecksOfWalk(t.Context(), nil, p)
if len(forgot) != 0 {
t.Fatalf("a walk not ended let go of %v, which it may still take", forgot)
}
p.WalkSaid = inventory.WalkEnded
forgetChecksOfWalk(t.Context(), nil, p)
if want := []builder.PullRef{{Owner: "novox", Repo: "shop", Number: 7}}; !reflect.DeepEqual(forgot, want) {
t.Fatalf("the walk ended let go of %v", forgot)
}
}
// A build in the scratch namespace is never registered: only the walk's copy in the module's own namespace is.
func TestABuildInTheScratchNamespaceIsNeverRegistered(t *testing.T) {
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: "store.internal:5000/app/server@sha256:aa"}}); got != "" {
t.Fatalf("a module's own build reads as a scratch build: %s", got)
}
scratch := "store.internal:5000/scratch/app/server@sha256:aa"
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: scratch}}); got != scratch {
t.Fatalf("a scratch build was not found: %q", got)
}
if got := inScratch([]link.MadeArtifact{{Name: "look", Reference: "http://store.internal:5000/v2/scratch/app/look/blobs/sha256:bb"}}); got == "" {
t.Fatal("a scratch archive was not found")
}
}
// The walk says per module what it did: "reused from …" or "built: <why>" — and a build seat from before says
// neither, which is said.
func TestAWalksBuildSaysWhetherItReusedOrBuilt(t *testing.T) {
if got := (link.BuildResult{Reused: "reused from the check of novox/shop#7"}).MadeWords(); got != "reused from the check of novox/shop#7" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{BuiltWhy: "no check built it"}).MadeWords(); got != "built: no check built it" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{}).MadeWords(); !strings.Contains(got, "predates ADR 0350") {
t.Fatalf("said %q", got)
}
}
// A walk's module the check's build stood in for is registered by the digest the check built — the copy in the
// module's namespace — and the walk keeps "reused from …" for it; its first-node gate judges that registered
// build as it judges one the walk built. A scratch reference is refused. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
// A walk's leave names the builds its own pull requests' verdicts made at the head each merge merged — never an
// earlier head's — read once per walk, and none when they cannot be read: the builder takes nothing a verdict does
// not name (novox/hq ADR 0350 §2, issue 703).
func TestAWalksLeaveNamesTheBuildsItsPullRequestsVerdictsMade(t *testing.T) {
was := checkedBuildsOf
forgetWalkVerdicts()
t.Cleanup(func() { checkedBuildsOf = was; forgetWalkVerdicts() })
var asked [][]link.PullRef
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
atHead := link.CheckedBuild{Pull: pull, Check: "build-2", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:2"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{atHead, earlier}, nil
}
pulls := []link.PullRef{pull}
heads := map[link.PullRef]string{pull: "bbbbbbbbbbbb0123456789"}
for range 3 {
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, heads); !reflect.DeepEqual(got.Builds,
[]link.CheckedBuild{atHead}) {
t.Fatalf("the leave names %+v, not only the merged head's build", got.Builds)
}
}
if !reflect.DeepEqual(asked, [][]link.PullRef{pulls}) {
t.Fatalf("the verdicts were read %d times for one walk: %v", len(asked), asked)
}
// A merge whose head was not said takes nothing.
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, nil); got.Builds != nil {
t.Fatalf("a merge of no said head names %+v", got.Builds)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := reuseWithVerdicts(t.Context(), "plan-2", &link.ReuseAsk{Pulls: pulls}, heads); got.Builds != nil {
t.Fatalf("verdicts that could not be read named %+v", got.Builds)
}
}
// A check is handed only the builds earlier checks of its own pull request at its own head made: another pull
// request's build, or an earlier head's, is never kept (novox/hq issue 703). None when they cannot be read.
func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
was := checkedBuildsOf
t.Cleanup(func() { checkedBuildsOf = was })
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
mine := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-0", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
another := link.CheckedBuild{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 8}, Check: "build-9",
Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
var asked [][]link.PullRef
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{mine, earlier, another}, nil
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Commit: "bbbbbbbbbbbb0123456789"}
builds := []link.CheckBuild{{Module: "app"}}
if got := namedForTheCheck(t.Context(), p, builds); !reflect.DeepEqual(got, []link.CheckedBuild{mine}) ||
!reflect.DeepEqual(asked, [][]link.PullRef{{pull}}) {
t.Fatalf("the check is handed %+v, read for %v", got, asked)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := namedForTheCheck(t.Context(), p, builds); got != nil {
t.Fatalf("verdicts that could not be read named %+v", got)
}
}
func forgetWalkVerdicts() {
walkVerdicts.Lock()
defer walkVerdicts.Unlock()
walkVerdicts.read = map[string]verdictsRead{}
}
// A walk's merged heads are the forge's word for each merge; a merge announced without its head, or a pull request
// merged at two heads in one walk, has none, and its walk takes no check's build (novox/hq issue 703).
func TestAWalksMergedHeadsAreTheForgesWord(t *testing.T) {
merge := func(number int, head string) inventory.PlanMerge {
return inventory.PlanMerge{Repository: "novox/shop", Number: number, Head: head}
}
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
merge(7, "aaaaaaaaaaaa1"), merge(8, ""), merge(9, "bbbbbbbbbbbb"), merge(9, "cccccccccccc")}}}
heads := mergedHeadsOf(p)
shop := func(n int) link.PullRef { return link.PullRef{Owner: "novox", Repo: "shop", Number: n} }
if heads[shop(7)] != "aaaaaaaaaaaa1" || heads[shop(8)] != "" || heads[shop(9)] != "" {
t.Fatalf("merged heads %v", heads)
}
}
-165
View File
@@ -1,165 +0,0 @@
package main
import (
"context"
"fmt"
"regexp"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382, issue 867).
//
// A mesh-host change and a mesh-controller change that need each other could not both pass their own merge checks:
// each was read beside the other's repository at the commit the mesh runs, which holds neither change. So the
// controller asks mesh-delivery's `groups` once for every pull request's check it asks (§1), and names, in the ask,
// the group's other open pull requests into a base of the same name, in a repository the build seat reads beside the
// check — mesh-lab never — each with its whole head. The build seat reads those repositories at the sibling's merge (§2).
// A pull request for which mesh-delivery answered and named no sibling is asked exactly as before (§7).
// wholeCommit is a commit as the forge says a head: forty hexadecimal characters.
var wholeCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
// besideRepositories are the repositories (lower case, owner/repo) a check reads beside its own, by the directory each
// is found under — the core's, as besideOf clones them, and mesh-sdk beside the controller; never mesh-lab, which is
// always read at its trunk.
func besideRepositories(entries []inventory.Entry) map[string]string {
read := map[string]string{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
read[nameOf(e.Source.Repository)] = dir
if dir == "mesh-controller" && e.Source.Seat != "" {
read[nameOf(siblingOf(e.Source.Repository, "mesh-sdk"))] = "mesh-sdk"
}
}
delete(read, "")
return read
}
// nameOf is a recorded repository as owner/repo in lower case, empty when it names no owner.
func nameOf(repository string) string {
owner, repo := sourceOwner(repository), sourceName(repository)
if owner == "" || repo == "" {
return ""
}
return strings.ToLower(owner + "/" + repo)
}
// siblingsFor is the pull request's siblings, as mesh-delivery's groups say them (ADR 0382 §1): the other members of
// the group its head branch names that are open, unmerged and into a base of its own base's name, in a repository the
// check reads beside it (beside: the directories the ask already names). The second answer is why they could not be
// learned, empty when they could: the group's members carry no head or base — a mesh-delivery from before the rule.
func siblingsFor(p link.PullUpdated, entries []inventory.Entry, beside map[string]link.CheckedOut,
lines []groupsLine) ([]link.CheckSibling, string) {
own := strings.ToLower(p.Owner + "/" + p.Repo)
var group *groupsLine
if p.Head == "" {
// A recheck (mesh-delivery's `recheck` through `delivery check`) carries no head branch name: the group is the one
// whose members hold this very pull request, by repository and number.
learnable := false
for i := range lines {
for _, m := range lines[i].Details {
learnable = true
if p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
group = &lines[i]
}
}
}
if group == nil {
for _, l := range lines {
if len(l.Details) == 0 && len(l.Members) > 0 && !learnable {
return nil, "mesh-delivery named the groups' members without their pull requests (a mesh-delivery from before novox/hq ADR 0382): " +
"the group of a pull request announced without its head branch cannot be found"
}
}
return nil, "" // a lone pull request
}
} else {
for i := range lines {
if lines[i].ID == p.Head {
group = &lines[i]
}
}
if group == nil {
return nil, ""
}
}
if len(group.Details) == 0 && len(group.Members) > 0 {
return nil, "mesh-delivery named the group's members without their heads and bases (a mesh-delivery from before novox/hq ADR 0382)"
}
read := besideRepositories(entries)
var siblings []link.CheckSibling
for _, m := range group.Details {
repository := strings.ToLower(m.Repository)
if repository == own {
continue
}
if !m.Open {
continue
}
if m.Head == "" || m.Base == "" {
return nil, "mesh-delivery named " + m.ID + " without its head or base (a mesh-delivery from before novox/hq ADR 0382)"
}
dir, readBeside := read[repository]
if m.Base != p.Base || !readBeside || dir == "mesh-lab" {
continue
}
if _, there := beside[dir]; !there {
continue
}
head := strings.ToLower(m.Head)
if !wholeCommit.MatchString(head) {
return nil, fmt.Sprintf("mesh-delivery named %s with a head that is not a whole commit (%q)", m.ID, m.Head)
}
siblings = append(siblings, link.CheckSibling{Repository: m.Repository, Number: m.Number, Head: head, Base: m.Base, Dir: dir})
}
sort.Slice(siblings, func(i, j int) bool { return siblings[i].Dir < siblings[j].Dir })
return siblings, ""
}
// withSiblings asks mesh-delivery's `groups` once and says in the request the pull request's siblings, or why they
// could not be learned (ADR 0382 §1): a check never waits on mesh-delivery past groupAskWithin and never fails for it,
// so a fix to mesh-delivery is still checked without it.
func withSiblings(ctx context.Context, request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry) {
lines, err := groupsAsked(ctx, deliverySeatHeld(entries))
siblingsFrom(request, p, entries, lines, err)
}
// siblingsFrom is withSiblings from mesh-delivery's answer already had.
func siblingsFrom(request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry, lines []groupsLine, err error) {
if request.Check == nil {
return
}
if err != nil {
request.Check.SiblingsUnknown = fmt.Sprintf("mesh-delivery did not say the group's pull requests (%v): "+
"the check reads no siblings", err)
return
}
request.Check.Siblings, request.Check.SiblingsUnknown = siblingsFor(p, entries, request.Check.Beside, lines)
}
// composedMember says the pull request is a member of a delivery group whose gate is the composed check's (novox/hq
// ADR 0403 clauses 1 and 7): the group has two members or more and mesh-delivery says so on its line. The group is
// found as siblingsFor finds it, by its head branch name, or, for a recheck that carries none, by the pull request's
// repository and number — not by siblings, which name only what the check reads beside it.
func composedMember(p link.PullUpdated, lines []groupsLine) bool {
own := strings.ToLower(p.Owner + "/" + p.Repo)
for _, l := range lines {
found := p.Head != "" && l.ID == p.Head
for _, m := range l.Details {
if p.Head == "" && p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
found = true
}
}
if found {
return l.ComposedGate && max(len(l.Details), len(l.Members)) >= 2
}
}
return false
}
-248
View File
@@ -1,248 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382 §1, §7, issue 867): the
// controller asks mesh-delivery's `groups` once for every check it asks, and names in the ask the group's other open
// pull requests into a base of the same name, in a repository the build seat reads beside it — mesh-lab never.
const (
headHost = "1111111111111111111111111111111111111111"
headController = "2222222222222222222222222222222222222222"
headSDK = "3333333333333333333333333333333333333333"
headLab = "4444444444444444444444444444444444444444"
headHq = "5555555555555555555555555555555555555555"
)
func theCoreOnTheGitSeat() []inventory.Entry {
onSeat := func(module, repository string) inventory.Entry {
e := fromRepo(module, repository, "")
e.Source.Seat = "git"
return e
}
nats := onSeat("nats", "novox/mesh-catalog")
nats.Source.Path = "modules/nats"
held := catalogue.Manifest{Module: "mesh-delivery", Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}}
return []inventory.Entry{onSeat("mesh-controller", "novox/mesh-controller"), onSeat("mesh-host", "novox/mesh-host"),
onSeat("node-tools", "novox/mesh-tools"), nats, {Manifest: held, On: []string{"anchor"}}}
}
func besideTheCore() map[string]link.CheckedOut {
return map[string]link.CheckedOut{"mesh-controller": {Repository: "c"}, "mesh-host": {Repository: "h"},
"mesh-tools": {Repository: "t"}, "mesh-catalog": {Repository: "k"}, "mesh-sdk": {Repository: "s"}, "mesh-lab": {Repository: "l"}}
}
func member(repository string, number int, head, base string, open bool) memberLine {
return memberLine{ID: repository + "@" + head[:12], Repository: repository, State: "checked", Number: number, Head: head,
Base: base, Open: open}
}
func group(id string, members ...memberLine) groupsLine {
line := groupsLine{ID: id, Details: members}
for _, m := range members {
line.Members = append(line.Members, m.ID+" ("+m.State+")")
}
return line
}
func TestASiblingIsTheGroupsOtherOpenPullRequestIntoABaseOfTheSameName(t *testing.T) {
entries := theCoreOnTheGitSeat()
pull := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
for _, c := range []struct {
what string
lines []groupsLine
want []string
unknown string
}{
{"the other member, open, same base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
[]string{"mesh-controller=novox/mesh-controller#346@" + headController + " onto main"}, ""},
{"a member into a base of another name is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "release", true))},
nil, ""},
{"a member not open (merged or closed) is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", false))},
nil, ""},
{"mesh-lab is never one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-lab", 5, headLab, "main", true))},
nil, ""},
{"a repository the check reads nothing of is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 9, headHq, "main", true))},
nil, ""},
{"mesh-sdk, read beside the controller, is one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-sdk", 7, headSDK, "main", true))},
[]string{"mesh-sdk=novox/mesh-sdk#7@" + headSDK + " onto main"}, ""},
{"another group is none of its business", []groupsLine{group("feat/y",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
nil, ""},
{"a group of its own alone has none", []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))},
nil, ""},
{"an older mesh-delivery names no heads", []groupsLine{{ID: "feat/x",
Members: []string{"novox/mesh-host@111111111111 (checked)", "novox/mesh-controller@222222222222 (checked)"}}},
nil, "without their heads and bases"},
{"a member without its base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "", true))},
nil, "without its head or base"},
{"a head that is no commit", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController[:12]+"zzzzzzzzzzzzzzzzzzzzzzzzzzzz", "main", true))},
nil, "not a whole commit"},
} {
got, unknown := siblingsFor(pull, entries, besideTheCore(), c.lines)
var said []string
for _, s := range got {
said = append(said, s.Dir+"="+s.Repository+"#"+itoa(s.Number)+"@"+s.Head+" onto "+s.Base)
}
if strings.Join(said, ";") != strings.Join(c.want, ";") {
t.Errorf("%s: siblings %v, wanted %v", c.what, said, c.want)
}
if (c.unknown == "") != (unknown == "") || !strings.Contains(unknown, c.unknown) {
t.Errorf("%s: could not learn %q, wanted %q", c.what, unknown, c.unknown)
}
}
// A pull request into a release branch has siblings into the release branch of the same name (ADR 0358 §1).
release := pull
release.Base = "feat/57"
lines := []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "feat/57", true),
member("novox/mesh-controller", 346, headController, "feat/57", true), member("novox/mesh-tools", 8, headLab, "main", true))}
if got, _ := siblingsFor(release, entries, besideTheCore(), lines); len(got) != 1 || got[0].Dir != "mesh-controller" {
t.Errorf("into a release branch: %+v", got)
}
// A recheck announces the pull request without its head branch: its group is the one holding it (repository, number).
pull.Head = ""
got, unknown := siblingsFor(pull, entries, besideTheCore(), lines)
if len(got) != 1 || got[0].Dir != "mesh-tools" || unknown != "" {
t.Errorf("a recheck of a group member: %+v %q", got, unknown)
}
// A pull request in no group, rechecked, is alone: no sibling, and nothing said.
pull.Number = 999
if got, unknown := siblingsFor(pull, entries, besideTheCore(), lines); got != nil || unknown != "" {
t.Errorf("a lone pull request: %+v %q", got, unknown)
}
if got, unknown := siblingsFor(pull, entries, besideTheCore(), nil); got != nil || unknown != "" {
t.Errorf("no groups at all: %+v %q", got, unknown)
}
// A mesh-delivery that names members without their pull requests cannot find the group of a recheck: it says so.
older := []groupsLine{{ID: "feat/x", Members: []string{"novox/mesh-host@111111111111 (checked)"}}}
pull.Number = 114
if got, unknown := siblingsFor(pull, entries, besideTheCore(), older); got != nil || unknown == "" {
t.Errorf("an older mesh-delivery: %+v %q", got, unknown)
}
}
func itoa(n int) string { b, _ := json.Marshal(n); return string(b) }
// Every check asks mesh-delivery's `groups` once; the ask names the sibling's repository, head and base.
func TestEveryCheckAsksMeshDeliveryOnceAndNamesItsSiblings(t *testing.T) {
entries := theCoreOnTheGitSeat()
asked := 0
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
asked++
if !held {
t.Error("the seat is held, yet it was asked as not held")
}
return []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true),
member("novox/mesh-controller", 346, headController, "main", true))}, nil
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if asked != 1 {
t.Fatalf("mesh-delivery was asked %d times for one check", asked)
}
want := link.CheckSibling{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", Dir: "mesh-controller"}
if got := request.Check.Siblings; len(got) != 1 || got[0] != want || request.Check.SiblingsUnknown != "" {
t.Fatalf("the ask names siblings %+v, unknown %q", got, request.Check.SiblingsUnknown)
}
body, _ := json.Marshal(request.Check)
if !strings.Contains(string(body), `"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"`+headController+
`","base":"main","dir":"mesh-controller"}]`) {
t.Fatalf("on the wire: %s", body)
}
}
// When mesh-delivery does not answer in time (or is not held, or answered without the fields), the check reads no
// sibling and its request says it could not learn them, so a fix to mesh-delivery is still checked without it.
func TestWhenMeshDeliveryDoesNotAnswerTheCheckReadsNoSiblingAndSaysSo(t *testing.T) {
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, c := range []struct {
what string
err error
held bool
}{{"it did not answer", context.DeadlineExceeded, true}, {"nothing serves it", errors.New("nothing serves mesh-delivery.groups"), true},
{"no holder on record", errors.New("no holder of mesh-delivery is on record to say it"), false}} {
entries := theCoreOnTheGitSeat()
if !c.held {
entries = entries[:4]
}
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
if held != c.held {
t.Errorf("%s: held %v", c.what, held)
}
return nil, c.err
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if len(request.Check.Siblings) != 0 || !strings.Contains(request.Check.SiblingsUnknown, c.err.Error()) ||
!strings.Contains(request.Check.SiblingsUnknown, "reads no siblings") {
t.Errorf("%s: siblings %+v, unknown %q", c.what, request.Check.Siblings, request.Check.SiblingsUnknown)
}
}
}
// **A pull request for which mesh-delivery answered and named no sibling is asked exactly as before** (ADR 0382 §7):
// byte for byte the request it was without the rule.
func TestAPullRequestWithNoSiblingIsAskedByteForByteAsBefore(t *testing.T) {
entries := theCoreOnTheGitSeat()
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, answer := range [][]groupsLine{nil, {group("feat/other", member("novox/mesh-controller", 346, headController, "main", true))},
{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 3, headHq, "main", true))}} {
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, nil }
build := func() link.BuildRequest {
return link.BuildRequest{ID: "build-1", Repository: "r", Ref: headHost, Check: &link.CheckRequest{Owner: "novox",
Repo: "mesh-host", Number: 114, Base: "main", Beside: besideTheCore(), Modules: []string{"mesh-host"}, Judge: "validator"}}
}
before, _ := json.Marshal(build())
after := build()
withSiblings(t.Context(), &after, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
got, _ := json.Marshal(after)
if string(before) != string(got) {
t.Fatalf("the request changed:\nbefore %s\nafter %s", before, got)
}
}
}
// What the build seat's verdict says of its siblings reaches the forge's holder in the controller's `checked`.
func TestTheVerdictCarriesTheSiblingsAndWhereTheJudgeCameFrom(t *testing.T) {
read := []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", BaseCommit: headHost}}
c := checkedOf(link.BuildResult{ID: "b", Ref: headHost, Checked: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114},
Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok", Siblings: read, JudgeFrom: "sibling novox/mesh-controller#346 merge abc",
Validator: "the change's own validator"}})
if len(c.Siblings) != 1 || c.Siblings[0] != read[0] || c.JudgeFrom == "" || c.Validator == "" {
t.Fatalf("the verdict says %+v", c)
}
body, _ := json.Marshal(c)
for _, want := range []string{`"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"` + headController +
`","base":"main","base_commit":"` + headHost + `"}]`, `"judge_from":"sibling novox/mesh-controller#346 merge abc"`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the verdict lacks %s:\n%s", want, body)
}
}
unknown := checkedOf(link.BuildResult{ID: "b", Check: &link.CheckOutcome{Verdict: "pass", SiblingsUnknown: "mesh-delivery did not answer"}})
if body, _ := json.Marshal(unknown); !strings.Contains(string(body), `"siblings_unknown":"mesh-delivery did not answer"`) ||
strings.Contains(string(body), `"siblings":`) {
t.Fatalf("the verdict that could not learn them says %s", body)
}
}
-37
View File
@@ -122,40 +122,3 @@ func TestASeatAnotherModuleDeclaresIsANoteOverSomeManifests(t *testing.T) {
t.Fatalf("the note was not said:\n%s", out.String())
}
}
// **A module lives whole in one repository** (novox/hq ADR 0420 §1, issue 1019): `module check`, told the
// repository it runs for, refuses an artifact whose build context names another; run by hand, it does not know.
func TestModuleCheckRefusesAnArtifactContextNamingAnotherRepository(t *testing.T) {
dir := t.TempDir()
write := func(name, context string) string {
path := filepath.Join(dir, name+".json")
raw := `{"module":"` + name + `","version":"1","build":{"artifacts":[{"name":"server","kind":"image","from":"Dockerfile"` + context + `}]}}`
if err := os.WriteFile(path, []byte(raw), 0o600); err != nil {
t.Fatal(err)
}
return path
}
other := write("other", `,"context":{"seat":"git","repository":"novox/mesh-controller","ref":"main"}`)
own := write("own", `,"context":{"seat":"git","repository":"novox/mesh-catalog","ref":"main"}`)
none := write("none", ``)
t.Setenv("MESH_CHECK_REPOSITORY", "novox/mesh-catalog")
var out bytes.Buffer
if err := moduleCheck([]string{other}, &out); err == nil {
t.Fatalf("a context naming another repository passed:\n%s", out.String())
}
if !strings.Contains(out.String(), "other builds artifact server from novox/mesh-controller: a module lives whole in one repository (novox/hq ADR 0420 §1)") {
t.Errorf("the refusal is not in the words of the gate:\n%s", out.String())
}
for _, path := range []string{own, none} {
out.Reset()
if err := moduleCheck([]string{path}, &out); err != nil {
t.Errorf("%s was refused: %v\n%s", path, err, out.String())
}
}
t.Setenv("MESH_CHECK_REPOSITORY", "")
out.Reset()
if err := moduleCheck([]string{other}, &out); err != nil {
t.Errorf("a local run, with no repository named, refused it: %v\n%s", err, out.String())
}
}
+14 -193
View File
@@ -154,24 +154,6 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
return err
}
scope := pullScope(p, entries, read, edges)
// **A cycle of edges fails the check, naming every module and edge** (novox/hq ADR 0419 clause 6), a lone pull
// request's as a group's composed check is: no order exists for what it changes.
alone := orderMember{ID: p.Owner + "/" + p.Repo + "@" + twelve(p.Commit), Repository: p.Owner + "/" + p.Repo, Base: p.Base,
Head: p.Commit, Number: p.Number, Paths: p.Paths, PathsTruncated: p.PathsTruncated, Removed: p.Removed, BaseSHA: p.BaseSHA}
if failing, cyclic, err := cycleVerdict(ctx, "", []orderMember{alone},
map[string]orderReach{alone.ID: {Moved: scope.Modules}}, edges, entries); err != nil {
return err
} else if cyclic {
failing.Commit, failing.Trunk = p.Commit, twelve(p.BaseSHA)
fmt.Printf("%s/%s#%d (%.8s): %s\n", p.Owner, p.Repo, p.Number, p.Commit, failing.Summary)
sayChecked(ctx, failing)
return nil
}
// **A member of a group whose gate is the composed check's runs its repository's check alone** (novox/hq ADR 0403
// clause 1): mesh-delivery is asked once, and says so on the group's line. Told nothing, the pull request is
// checked alone, gate included (clause 6).
lines, groupsErr := groupsAsked(ctx, deliverySeatHeld(entries))
composed := groupsErr == nil && composedMember(p, lines)
// The change plan of the commit at hand (ADR 0238): what a merge of it would build and send, posted
// with the verdict whatever the verdict is.
plan := changePlanOf(p.Owner+"/"+p.Repo, p.Base, p.Commit, scope.Reach, entries, func(module string) (inventory.Upgrade, bool) {
@@ -180,10 +162,6 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
})
fmt.Print(planText(plan))
direct := directVerdict(p, plan)
if composed {
scope.Modules, scope.Dependents, scope.New, scope.Manifests = nil, nil, nil, nil
composedGate(&direct)
}
switch {
case !scope.gated() && !scope.Mesh:
fmt.Printf("%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
@@ -201,9 +179,6 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
return err
}
request.Check.Plan = &plan
request.Check.Composed = composed
// **The group's other open pull requests are read beside it** (novox/hq ADR 0382 §1).
siblingsFrom(&request, p, entries, lines, groupsErr)
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
@@ -213,13 +188,7 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
if err := ask.Ask(ctx, request); err != nil {
return err
}
// **A newer request replaces the one still waiting, and a newer key stops the one running** (novox/hq ADR 0406 §3, §4,
// issue 974): the check queue keeps one request a pull request, and a running check watches its key. Nothing is asked
// of the forge here (issue 726).
what := "its own merge-check.sh alone: " + noModuleTouched
if composed {
what = "its own merge-check.sh alone: the gate is its group's composed check's"
}
if scope.gated() {
what = fmt.Sprintf("the gate over %s (a merge would build %d module(s) in %d tier(s))",
strings.Join(append(append([]string{}, scope.Modules...), prefixedAll("new:", scope.New)...), ", "),
@@ -230,20 +199,6 @@ func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
return nil
}
// composedWords is what a group member's own verdict says of its gate layer (novox/hq ADR 0403 clause 1).
const composedWords = "judged by its group's composed check"
// gateComposed is a gate layer's verdict when no gate was run for the member alone: never a pass. Mesh-delivery reads it
// as the composed check's to judge, and the forge's holder sets no mesh/merge-gate from it.
const gateComposed = "composed"
// composedGate says a group member's own verdict has no gate of its own: its gate layer, and the verdict that is
// the gate's, read "composed" in place of a pass.
func composedGate(c *link.Checked) {
c.Verdict, c.Summary = gateComposed, composedWords
c.Gate = &link.CheckLayer{Verdict: gateComposed, Summary: composedWords}
}
// directVerdict is the verdict of a change no build judges — it touches no module, and either the repository is not
// the mesh's or the head holds no merge-check.sh. **It names the trunk commit it holds for** (novox/hq ADR 0323 §3):
// the base branch's head the forge announced with the head, so the merge tool can weigh it against the trunk's head
@@ -293,29 +248,7 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
if err != nil {
return link.BuildRequest{}, err
}
beside, err := besideOf(entries, func(module string) string { return current[module].Commit }, clone)
if err != nil {
return link.BuildRequest{}, err
}
builds := checkBuildsOf(p, scope, entries, clone)
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
Ref: p.Commit,
Held: heldBy(ctx),
Seats: seatBases(ctx),
Source: sourceOnSeat(source),
Check: &link.CheckRequest{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base, Trunk: p.BaseSHA, Ready: p.Ready,
Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents, New: scope.New,
Manifests: scope.Manifests, Judge: scope.Judge, Builds: builds, Named: namedForTheCheck(ctx, p, builds),
Running: runningBy(ctx)},
}, nil
}
// besideOf is what a check reads beside its own repository, by the directory it is found under: each core repository
// by the module the mesh builds from it, at besideRefs. running is the commit the mesh runs of a module.
func besideOf(entries []inventory.Entry, running func(module string) string,
clone func(inventory.Source) (string, error)) (map[string]link.CheckedOut, error) {
// Beside it, at what the mesh runs: each core repository by the module the mesh builds from it.
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
@@ -324,12 +257,11 @@ func besideOf(entries []inventory.Entry, running func(module string) string,
}
url, err := clone(e.Source)
if err != nil {
return nil, err
return link.BuildRequest{}, err
}
refs := besideRefs(dir, running(e.Manifest.Module))
refs := besideRefs(dir, current[e.Manifest.Module].Commit)
beside[dir] = link.CheckedOut{Repository: url, Ref: refs[dir]}
if dir == "mesh-controller" {
// The controller's main stays its main: it judges only a check the running controller predates.
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
@@ -341,98 +273,17 @@ func besideOf(entries []inventory.Entry, running func(module string) string,
}
}
}
return beside, nil
}
// trunkOf is the branch a module's source follows: main when it records none, or records a commit.
func trunkOf(s inventory.Source) string {
if t := strings.TrimSpace(followedBranch(s.Ref)); t != "" {
return t
}
return "main"
}
// sourceName is the name of a recorded repository, a path on the git seat or a URL: novox/mesh-host → mesh-host.
func sourceName(repository string) string {
parts := strings.Split(strings.Trim(strings.TrimSuffix(repository, ".git"), "/"), "/")
if len(parts) < 2 {
return ""
}
return parts[len(parts)-1]
}
// namedForTheCheck are the builds earlier checks of the same pull request at the same head named, read once for the
// check (novox/hq issue 703): asked again because only the trunk moved, the check keeps what they built of a key
// still the same. Another pull request's build, or one of an earlier head, is never kept. None when they cannot be
// read, and the check builds, as before.
func namedForTheCheck(ctx context.Context, p link.PullUpdated, builds []link.CheckBuild) []link.CheckedBuild {
if len(builds) == 0 || p.Commit == "" {
return nil
}
pull := link.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number}
read, err := checkedBuildsOf(ctx, []link.PullRef{pull})
if err != nil {
fmt.Printf(" the earlier checks' builds of %s/%s#%d could not be read, so the check builds: %v\n",
p.Owner, p.Repo, p.Number, err)
// empty-on-error: said above; a check handed no named build builds every module, as before issue 703
return nil
}
var named []link.CheckedBuild
for _, b := range read {
if b.Pull == pull && link.SameHead(b.Head, p.Commit) {
named = append(named, b)
}
}
return named
}
// checkBuildsOf are the modules a merge of the change would build, in the walk's order, as the check builds them
// for the walk to take (novox/hq ADR 0350 §1): a module built from the repository checked, at the branch it merges
// into, from the merge the check makes; any other — a module standing on one — from its own repository at the
// branch it follows, as the walk builds it. A module whose build reads the repository checked through a context,
// or that the merge deletes, cannot be built as the walk would: said, and nothing after it is built. A change that
// moves no module builds nothing.
func checkBuildsOf(p link.PullUpdated, scope checkScope, entries []inventory.Entry,
clone func(inventory.Source) (string, error)) []link.CheckBuild {
if len(scope.Reach.Plan.Tiers) == 0 {
return nil
}
m := link.SourceMoved{Owner: p.Owner, Repo: p.Repo, Base: p.Base, CloneURL: p.CloneURL}
byName := map[string]inventory.Entry{}
for _, e := range entries {
byName[e.Manifest.Module] = e
}
skip := map[string]string{}
for _, e := range scope.Reach.Packaging {
skip[e.Manifest.Module] = "its build reads " + p.Owner + "/" + p.Repo + " through a context, which a check " +
"cannot hand the merge"
}
for _, e := range scope.Reach.Deleted {
skip[e.Manifest.Module] = "the merge deletes it"
}
var out []link.CheckBuild
for tier, names := range scope.Reach.Plan.Tiers {
for _, name := range names {
e, known := byName[name]
b := link.CheckBuild{Module: name, Tier: tier, Skip: skip[name]}
switch {
case !known:
b.Skip = "the catalogue does not hold it"
case b.Skip != "":
case sourceIs(e.Source, m):
b.Merged, b.Path = true, e.Source.Path
default:
url, err := clone(e.Source)
if err != nil {
b.Skip = "its repository cannot be cloned: " + err.Error()
break
}
b.Repository, b.Path, b.Ref = url, e.Source.Path, followedBranch(e.Source.Ref)
}
out = append(out, b)
}
}
return out
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
Ref: p.Commit,
Held: heldBy(ctx),
Seats: seatBases(ctx),
Source: sourceOnSeat(source),
Check: &link.CheckRequest{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents, New: scope.New,
Manifests: scope.Manifests, Judge: scope.Judge},
}, nil
}
// besideRefs is the ref each repository is cloned at beside a check, by the directory it is found under:
@@ -514,13 +365,6 @@ const maxCheckReport = 60 << 10
// checked says a merge check's verdict as the controller's `checked`. Nothing is recorded or
// registered: a check builds nothing (issue 240's rule for a dry run, kept for a check).
func checked(ctx context.Context, result link.BuildResult) {
// **A check stopped by its key says nothing** (novox/hq ADR 0406 §4, issue 974, review 2): it stopped because the head or
// the trunk commit moved on, and the newer request will judge. Said as an error it would name the live head — red
// statuses on the forge, a delivery refused — and written to the key it would stand in the newer request's way.
if result.Failed == link.SupersededByKey {
fmt.Printf("%s: stopped, its pull request's key moved on; the newer check will judge\n", result.ID)
return
}
sayChecked(ctx, checkedOf(result))
}
@@ -538,9 +382,6 @@ func checkedOf(result link.BuildResult) link.Checked {
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and the commits of the
// repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
c.Built = result.Check.Built
c.Validator = result.Check.Validator
c.Siblings, c.SiblingsUnknown, c.JudgeFrom = result.Check.Siblings, result.Check.SiblingsUnknown, result.Check.JudgeFrom
if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
@@ -558,12 +399,6 @@ func checkedOf(result link.BuildResult) link.Checked {
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
c.RepoCheck = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
}
// **A group member's own check has no gate of its own** (novox/hq ADR 0403 clause 1): what ran was the repository's
// check alone, and its gate layer is "composed", never a pass. A failure or an error stays what it was.
if result.Checked != nil && result.Checked.Composed && result.Check != nil && c.Gate != nil &&
(c.Gate.Verdict == "pass" || c.Gate.Verdict == "warning") {
composedGate(&c)
}
if result.Checked != nil && c.Gate != nil && len(c.Gate.Modules) == 0 {
c.Gate.Modules = append(append([]string{}, result.Checked.Modules...), prefixedAll("new:", result.Checked.New)...)
}
@@ -594,20 +429,6 @@ func checkedOf(result link.BuildResult) link.Checked {
l.Verdict = "error"
}
}
// **A check run again says so** (novox/hq issue 594). The summaries are the lines the pull request
// shows, clipped by the forge to a few words, so they get only a short marker after the verdict's own
// words; the report gets the whole sentence. Layers copied, so the build's own outcome is left as it came.
if mark := link.RunMarker(result.Delivery); mark != "" {
c.Summary += " " + mark
for _, l := range []**link.CheckLayer{&c.Gate, &c.RepoCheck} {
if *l != nil {
layer := **l
layer.Summary += " " + mark
*l = &layer
}
}
c.Report = link.Restarted(result.Delivery) + "\n\n" + c.Report
}
if len(c.Report) > maxCheckReport {
c.Report = "…" + c.Report[len(c.Report)-maxCheckReport:]
}
@@ -1,56 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
func TestACheckStoppedByItsKeyIsAnnouncedAsNoVerdict(t *testing.T) {
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
legacy, err := js.Conn().JetStream()
if err != nil {
t.Fatal(err)
}
was := checkEvents
checkEvents = link.OverNATS{Conn: js.Conn(), JS: legacy}
t.Cleanup(func() { checkEvents = was })
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeyChecked), heard)
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 3}
checked(t.Context(), link.BuildResult{ID: "stopped", Ref: "abc", Checked: asked, Failed: link.SupersededByKey})
checked(t.Context(), link.BuildResult{ID: "broken", Ref: "abc", Checked: asked, Failed: "the facts snapshot cannot be read"})
select {
case m := <-heard:
var c link.Checked
_ = json.Unmarshal(m.Data, &c)
if c.ID != "broken" || c.Verdict != "error" {
t.Fatalf("the first verdict said is %+v; want only the broken check's error", c)
}
case <-time.After(5 * time.Second):
t.Fatal("nothing was said")
}
select {
case m := <-heard:
t.Fatalf("a second verdict was said: %s", m.Data)
case <-time.After(300 * time.Millisecond):
}
}
-97
View File
@@ -1,97 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"testing"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Acceptance 5 of milestone 2 (novox/hq ADR 0406 §3, issue 974), live, with the store and a real bus: two merges into one
// repository within a minute, with two open pull requests, leave each pull request with exactly one queued check, by the
// stream's rule. Each merge moves the trunk, and mesh-delivery asks the controller to check both open pull requests again
// against it (`delivery check`, which is the controller's own path for a head announced); nothing moves a request, nothing
// asks the forge, and the queue is read back whole.
func TestTwoMergesWithinAMinuteLeaveEachOpenPullRequestOneQueuedCheck(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
holdTheDeliverySeat(t, open)
was := askDeliveryOwner
t.Cleanup(func() { askDeliveryOwner = was })
askDeliveryOwner = func(_ context.Context, _ *nats.Conn, verb string, _ map[string]any) (json.RawMessage, error) {
if verb != "groups" {
t.Fatalf("asked the owner %s", verb)
}
return json.RawMessage(`[]`), nil
}
// The repository the two pull requests are into, as a module the mesh builds from it.
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
heads := map[int]string{7: "7777777777777777777777777777777777777777", 8: "8888888888888888888888888888888888888888"}
check := func(number int, trunk string) {
t.Helper()
p := orderMember{ID: fmt.Sprintf("novox/mesh-controller@%.12s", heads[number]), Repository: "novox/mesh-controller",
Base: "main", Head: heads[number], Number: number, Paths: []string{"main.go"}, ModuleDirsSaid: true,
BaseSHA: trunk, Ready: number == 8}.pull()
p.MergeCheck, p.MergeCheckSaid = true, true
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
}
// Announced against the trunk as it stands, then asked again after each of two merges.
for _, trunk := range []string{"1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222",
"3333333333333333333333333333333333333333"} {
check(7, trunk)
check(8, trunk)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
seen := map[int]string{}
for _, a := range q.Asks {
c := a.Request.Check
if c == nil {
t.Fatalf("%s is no check", a.ID)
}
if _, twice := seen[c.Number]; twice {
t.Errorf("pull request %d has two queued checks", c.Number)
}
seen[c.Number] = c.Trunk
if a.State != link.AskWaiting {
t.Errorf("%s is %s", a.ID, a.State)
}
if want := "3333333333333333333333333333333333333333"; c.Trunk != want {
t.Errorf("pull request %d is queued against %.8s; want the newest trunk %.8s", c.Number, c.Trunk, want)
}
}
if len(seen) != 2 {
t.Fatalf("queued checks for %v; want one for each of the two open pull requests", seen)
}
// By the streams' own count: the ready one on its stream, the other on its.
for stream, want := range map[string]uint64{broker.ChecksReadyStream: 1, broker.ChecksOtherStream: 1} {
info, err := js.Context().StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != want {
t.Errorf("%s holds %d request(s); want %d", stream, info.State.Msgs, want)
}
}
}
-3
View File
@@ -288,6 +288,3 @@ func TestARecheckedHeadsDirectVerdictNamesItsTrunk(t *testing.T) {
t.Fatalf("a rechecked head's direct verdict names trunk %q", c.Trunk)
}
}
// A check stopped by its pull request's key moving on says no verdict (issue 974, review 2): an error said for it would name the
// live head, and the newer request is the one that judges. A check that failed for any other reason still says its error.
@@ -1,68 +0,0 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A member of a group whose gate is the composed check's is asked of the build seat with no module: its repository's
// check alone (novox/hq ADR 0403 clause 1). With no word from mesh-delivery, or one that does not say the gate is
// composed, it is checked alone with its gate (clause 6).
func askedForMember(t *testing.T, answer []groupsLine, answerErr error) link.CheckRequest {
t.Helper()
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
was := groupsAsked
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, answerErr }
t.Cleanup(func() { groupsAsked = was })
p := link.PullUpdated{Owner: "novox", Repo: "mesh-controller", Number: 346, Base: "main", Head: "feat/x", Commit: headController,
Paths: []string{"main.go"}, ModuleDirsSaid: true, MergeCheck: true, MergeCheckSaid: true}
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil || len(q.Asks) != 1 {
t.Fatalf("asks %d: %v", len(q.Asks), err)
}
return *q.Asks[0].Request.Check
}
func TestAComposedMembersCheckIsItsRepositoryChecksAlone(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
line.ComposedGate = true
c := askedForMember(t, []groupsLine{line}, nil)
if len(c.Modules) != 0 || len(c.New) != 0 || len(c.Dependents) != 0 || !c.Composed {
t.Fatalf("the member's check is asked as %+v; want no module and composed", c)
}
}
func TestAMemberOfAGroupOfAnOlderMeshDeliveryKeepsItsOwnGate(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
c := askedForMember(t, []groupsLine{line}, nil)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed {
t.Fatalf("an older mesh-delivery's member is asked as %+v; want its own gate", c)
}
}
func TestAMemberWhoseGroupCannotBeLearnedIsCheckedAloneWithItsGate(t *testing.T) {
c := askedForMember(t, nil, context.DeadlineExceeded)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed ||
!strings.Contains(c.SiblingsUnknown, "reads no siblings") {
t.Fatalf("the member is asked as %+v; want its gate and why its siblings are unknown", c)
}
}
@@ -1,136 +0,0 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A delivery group's gate is the composed check's, and its members' own checks run their repository's check alone
// (novox/hq ADR 0403 clauses 1 and 7). Pure: no store, no bus.
// A member of a group whose line says composed_gate is a composed member; one whose group says nothing of it (an older
// mesh-delivery), one of a group of a single pull request and a lone pull request are not.
func TestAMemberOfAComposedGroupIsKnownByTheGroupsLine(t *testing.T) {
two := func(composed bool) groupsLine {
l := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))
l.ComposedGate = composed
return l
}
p := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
recheck := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Commit: headHost}
for name, c := range map[string]struct {
p link.PullUpdated
lines []groupsLine
want bool
}{
"composed": {p, []groupsLine{two(true)}, true},
"a recheck, by its number": {recheck, []groupsLine{two(true)}, true},
"an older mesh-delivery": {p, []groupsLine{two(false)}, false},
"another group": {p, []groupsLine{group("feat/y", member("novox/mesh-host", 1, headHost, "main", true))}, false},
"no group": {p, nil, false},
} {
if got := composedMember(c.p, c.lines); got != c.want {
t.Errorf("%s: composed %v, want %v", name, got, c.want)
}
}
alone := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))
alone.ComposedGate = true
if composedMember(p, []groupsLine{alone}) {
t.Error("a group of a single pull request is composed")
}
}
// A composed member's verdict never says its gate passed: the gate layer, and the verdict that is the gate's, read
// "composed"; a failure or an error stays what it was.
func TestAComposedMembersVerdictHasNoPassingGate(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114, Composed: true}
pass := link.BuildResult{ID: "b", Ref: headHost, Checked: asked, Check: &link.CheckOutcome{Verdict: "pass", Summary: "no module touched",
Gate: &link.CheckLayer{Verdict: "pass", Summary: "no module touched"}, RepoCheck: &link.CheckLayer{Verdict: "pass", Summary: "ok"}}}
c := checkedOf(pass)
if c.Gate == nil || c.Gate.Verdict != "composed" || c.Verdict != "composed" || c.RepoCheck == nil || c.RepoCheck.Verdict != "pass" {
t.Fatalf("the member's verdict says %+v gate %+v", c, c.Gate)
}
asked.Composed = false
if c := checkedOf(pass); c.Gate.Verdict != "pass" || c.Verdict != "pass" {
t.Fatalf("a member of no composed group has its own gate: %+v", c.Gate)
}
asked.Composed = true
failing := pass
failing.Check = &link.CheckOutcome{Verdict: "fail", Summary: "conflict", Gate: &link.CheckLayer{Verdict: "fail", Summary: "conflict"},
RepoCheck: &link.CheckLayer{Verdict: "fail", Summary: "conflict"}}
if c := checkedOf(failing); c.Gate.Verdict != "fail" || c.Verdict != "fail" {
t.Fatalf("a failure was read as composed: %+v", c.Gate)
}
broken := link.BuildResult{ID: "b", Ref: headHost, Checked: asked, Failed: "the facts snapshot cannot be read"}
if c := checkedOf(broken); c.Verdict != "error" || c.Gate.Verdict != "error" {
t.Fatalf("an error was read as composed: %+v", c)
}
}
// The composed verdict on the bus carries every member, each with its trunk commit, for the forge's holder to set each
// member's mesh/merge-gate from it.
func TestTheComposedVerdictCarriesEveryMember(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114, Group: "feat/x",
Members: []link.GroupMember{{Owner: "novox", Repo: "mesh-controller", Number: 346, Ref: headController}}}
c := checkedOf(link.BuildResult{ID: "g", Ref: headHost, Checked: asked, Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok",
Trunk: "aaaaaaaaaaaa", Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-controller", Commit: headController[:12], Trunk: "bbbbbbbbbbbb"}}}})
body, _ := json.Marshal(c)
for _, want := range []string{`"group":"feat/x"`, `"members":[{"owner":"novox","repo":"mesh-host","number":114,"commit":"` + headHost + `","trunk":"aaaaaaaaaaaa"}`,
`{"owner":"novox","repo":"mesh-controller","number":346,"commit":"` + headController + `","trunk":"bbbbbbbbbbbb"}`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the composed verdict lacks %s:\n%s", want, body)
}
}
}
// Members whose manifests order their modules in a cycle fail the composed check, naming every module and edge, and no
// ask is made of the build seat.
func TestAGroupInACycleFailsItsComposedCheckWithoutABuild(t *testing.T) {
members := []orderMember{
{ID: "novox/x@" + headHost[:12], Repository: "novox/x", Base: "main", Head: headHost, Number: 7, Paths: []string{"module.json"}},
{ID: "novox/y@" + headController[:12], Repository: "novox/y", Base: "main", Head: headController, Number: 9, Paths: []string{"module.json"}},
}
reach := map[string]orderReach{members[0].ID: {Moved: []string{"mx"}}, members[1].ID: {Moved: []string{"my"}}}
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
manifestAt = func(_ context.Context, _, repo, _, ref string) (catalogue.Manifest, bool, error) {
if ref == "main" {
return catalogue.Manifest{}, false, nil // what the pull requests merge into holds no such manifest
}
if repo == "x" {
return catalogue.Manifest{Module: "mx", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "a"}}}},
Invokes: []string{"seat:s.b"}}, true, nil
}
return catalogue.Manifest{Module: "my", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "b"}}}},
Invokes: []string{"seat:s.a"}}, true, nil
}
c, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, []inventory.Edge(nil), nil)
if err != nil || !cyclic {
t.Fatalf("no cycle found: %v", err)
}
if c.Verdict != "fail" || c.Group != "feat/cycle" || len(c.Members) != 2 || c.Gate == nil || c.Gate.Verdict != "fail" {
t.Fatalf("the verdict is %+v", c)
}
for _, w := range []string{"mx", "my", "s.a", "s.b"} {
if !strings.Contains(c.Summary, w) {
t.Errorf("the cycle does not name %s: %s", w, c.Summary)
}
}
// Members with no edge between them are no cycle.
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
manifestAt = func(context.Context, string, string, string, string) (catalogue.Manifest, bool, error) {
return catalogue.Manifest{}, false, nil
}
if _, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, nil, nil); err != nil || cyclic {
t.Fatalf("a cycle where there is none: %v %v", cyclic, err)
}
}
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
@@ -1,43 +0,0 @@
//go:build integration
package main
import (
"fmt"
"strings"
"testing"
"time"
)
func TestAVerdictCarriesAProbesFindingsInBrief(t *testing.T) {
var found []string
for i := 0; i < 689; i++ {
found = append(found, fmt.Sprintf("delivery d-%012x has been checking for 13h, past its bound of 45m", i))
}
run := doctorRun{Run: "run-1", Probes: []probeVerdict{{ID: "D14", Verdict: "failed", Found: found},
{ID: "D1", Verdict: "passed"}}}
answer, err := verdictAnswer(run, time.Now(), "")
if err != nil {
t.Fatal(err)
}
brief := answer["run"].(doctorRun).Probes[0]
if len(brief.Found) != overviewFindings || brief.FoundCount != 689 || !strings.Contains(brief.More, "probe=D14") {
t.Fatalf("the verdict carries %d findings of D14, count %d, more %q", len(brief.Found), brief.FoundCount, brief.More)
}
if len(run.Probes[0].Found) != 689 {
t.Fatal("the overview cut the run it was made from")
}
whole, err := verdictAnswer(run, time.Now(), "d14")
if err != nil {
t.Fatal(err)
}
if p := whole["run"].(doctorRun).Probes; len(p) != 1 || len(p[0].Found) != 689 || p[0].More != "" {
t.Fatalf("doctor probe=D14 answered %+v", p)
}
if _, err := verdictAnswer(run, time.Now(), "D99"); err == nil {
t.Fatal("a probe the run does not have was answered")
}
if !strings.Contains(doctorText(answer), "probe=D14` gives them all") {
t.Fatalf("the text does not say how to read them all:\n%s", doctorText(answer))
}
}
@@ -74,3 +74,35 @@ func TestAListOfConditionsCarriesEachInBrief(t *testing.T) {
// The self-check's verdict carries at most overviewFindings of one probe's findings, with their count and
// how to read them all; `doctor probe=` gives that probe whole (novox/hq issue 314).
func TestAVerdictCarriesAProbesFindingsInBrief(t *testing.T) {
var found []string
for i := 0; i < 689; i++ {
found = append(found, fmt.Sprintf("delivery d-%012x has been checking for 13h, past its bound of 45m", i))
}
run := doctorRun{Run: "run-1", Probes: []probeVerdict{{ID: "D14", Verdict: "failed", Found: found},
{ID: "D1", Verdict: "passed"}}}
answer, err := verdictAnswer(run, time.Now(), "")
if err != nil {
t.Fatal(err)
}
brief := answer["run"].(doctorRun).Probes[0]
if len(brief.Found) != overviewFindings || brief.FoundCount != 689 || !strings.Contains(brief.More, "probe=D14") {
t.Fatalf("the verdict carries %d findings of D14, count %d, more %q", len(brief.Found), brief.FoundCount, brief.More)
}
if len(run.Probes[0].Found) != 689 {
t.Fatal("the overview cut the run it was made from")
}
whole, err := verdictAnswer(run, time.Now(), "d14")
if err != nil {
t.Fatal(err)
}
if p := whole["run"].(doctorRun).Probes; len(p) != 1 || len(p[0].Found) != 689 || p[0].More != "" {
t.Fatalf("doctor probe=D14 answered %+v", p)
}
if _, err := verdictAnswer(run, time.Now(), "D99"); err == nil {
t.Fatal("a probe the run does not have was answered")
}
if !strings.Contains(doctorText(answer), "probe=D14` gives them all") {
t.Fatalf("the text does not say how to read them all:\n%s", doctorText(answer))
}
}
@@ -1,32 +0,0 @@
//go:build integration
package main
import (
"errors"
"strings"
"testing"
)
func TestUnreadableConditionsAreNotAWellMesh(t *testing.T) {
open := aMesh(t)
_, store := withConditionsInMemory(t)
store.SetFail(errors.New("the bus is away"))
asked, err := theThreeQuestions(t.Context(), open)
if err != nil {
t.Fatal(err)
}
if asked.well() || asked.conditionsUnread == "" {
t.Fatalf("well with its conditions unread: %+v", asked.conditionsUnread)
}
said := printed(t, func() error { return printStatus(asked) })
if !strings.HasPrefix(said, "the open conditions could NOT be read") || strings.Contains(said, "no open conditions") {
t.Fatalf("%s", said)
}
store.SetFail(nil)
asked, _ = theThreeQuestions(t.Context(), open)
said = printed(t, func() error { return printStatus(asked) })
if asked.well() && !strings.Contains(said, "no open conditions;") {
t.Fatalf("the all-well sentence does not say no conditions are open:\n%s", said)
}
}
+23
View File
@@ -1,6 +1,7 @@
package main
import (
"errors"
"strings"
"testing"
@@ -82,3 +83,25 @@ func TestASilenceThroughTheVerbHoldsAndTheConditionStaysOpen(t *testing.T) {
}
// **Conditions that cannot be read are not none open**: status says so, and is not well.
func TestUnreadableConditionsAreNotAWellMesh(t *testing.T) {
open := aMesh(t)
_, store := withConditionsInMemory(t)
store.SetFail(errors.New("the bus is away"))
asked, err := theThreeQuestions(t.Context(), open)
if err != nil {
t.Fatal(err)
}
if asked.well() || asked.conditionsUnread == "" {
t.Fatalf("well with its conditions unread: %+v", asked.conditionsUnread)
}
said := printed(t, func() error { return printStatus(asked) })
if !strings.HasPrefix(said, "the open conditions could NOT be read") || strings.Contains(said, "no open conditions") {
t.Fatalf("%s", said)
}
store.SetFail(nil)
asked, _ = theThreeQuestions(t.Context(), open)
said = printed(t, func() error { return printStatus(asked) })
if asked.well() && !strings.Contains(said, "no open conditions;") {
t.Fatalf("the all-well sentence does not say no conditions are open:\n%s", said)
}
}
-616
View File
@@ -1,616 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"os"
"reflect"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **The move of config** (novox/hq ADR 0348 §9 and its consequences, design 57 K5, issue 664): every config value a
// settings layer holds is set in mesh-config, one of the moves of data a release runs once after its delivery, after
// the move of placement. A module's layer on a node becomes its own value on that node (`module.<node>.<module>.<key>`).
// A whole-mesh layer has no module value for the whole mesh (ADR 0348 §3): its value becomes mesh config for the
// predefined key it falls back to where that module alone falls back to the key (claude-code's managed settings
// become `agent-managed-settings`), and otherwise the module's own value on each node it is assigned to, laid under
// that node's layer as composition lays it. A key a layer still says by its file's name (nodered's `timeZone`) is
// set under its config key (`time-zone`, issue 751).
//
// **Every node composes byte for byte as before.** The move sets the values, reads every module's resolved values
// back through mesh-config and compares them with what the layers lay today; only when they agree does it empty the
// layers it moved (each kept whole in its history first) and turn on the switch (inventory.FinishConfigMove), after
// which the controller reads mesh-config where it read the layers. A module still in the settings form keeps its
// layers and is said so. Anything that cannot move as it composes refuses the whole move, by name, before anything
// is set. Asked again, it sets nothing that is set already.
// configLevel is where mesh-config keeps a value: mesh config, node config, or a module's own value on a node.
type configLevel struct {
Level string `json:"level"`
Node string `json:"node,omitempty"`
Module string `json:"module,omitempty"`
}
func (l configLevel) valueKey(key string) string {
switch l.Level {
case "node":
return "node." + l.Node + "." + key
case "module":
return "module." + l.Node + "." + l.Module + "." + key
}
return "mesh." + key
}
func (l configLevel) String() string {
switch l.Level {
case "node":
return "node config on " + l.Node
case "module":
return l.Module + "'s own value on " + l.Node
}
return "mesh config"
}
// configSet is one value the move sets in mesh-config, and the layers it comes from.
type configSet struct {
configLevel
Key string `json:"key"`
Value any `json:"value"`
Trusted bool `json:"trusted,omitempty"`
// From is the layers it comes from, as `<module> on <where>: <key as the layer says it>`.
From []string `json:"from"`
// Already is true when mesh-config holds this value at this level already: nothing is set.
Already bool `json:"already,omitempty"`
}
// configMoveAnswer is what the move of config says it did, or would do.
type configMoveAnswer struct {
// DryRun is true when nothing was set: asked without --confirm.
DryRun bool `json:"dry-run"`
Sets []configSet `json:"sets"`
// Moved is each layer whose values the move sets, emptied once they read back.
Moved []string `json:"moved"`
// Stays is each layer of a module still in the settings form: kept, and composed from as before.
Stays []string `json:"stays,omitempty"`
// Unread is each layer that composes nothing (its module is not assigned where it says): kept in the history,
// emptied, and set nowhere.
Unread []string `json:"unread,omitempty"`
// Problems is why the move refuses, each by name; nothing is set while there is one.
Problems []string `json:"problems,omitempty"`
// Switched is true once the controller reads mesh-config where it read the layers.
Switched bool `json:"switched"`
}
// configMovePlan is the move worked out from the layers, before anything is asked of mesh-config.
type configMovePlan struct {
answer configMoveAnswer
// moved is the layers the move empties once the values read back.
moved []inventory.LayerToMove
// expected is, for each module the move reaches, keyed `<node>.<module>`, every value composition lays over
// its files today: what mesh-config must resolve for it from mesh config, node config or its own value.
expected map[string]map[string]any
// modules is each module in the config form, by the nodes it is assigned to: whose resolved values are read back.
modules map[string][]string
}
func whereOf(node string) string {
if node == "" {
return "the whole mesh"
}
return node
}
// planConfigMove works out the move from every layer that holds a value, the catalogue, and where each module is
// assigned.
func planConfigMove(layers []inventory.LayerToMove, shelf map[string]catalogue.Manifest,
on map[string][]string) configMovePlan {
p := configMovePlan{answer: configMoveAnswer{Sets: []configSet{}, Moved: []string{}},
expected: map[string]map[string]any{}, modules: map[string][]string{}}
problem := func(format string, a ...any) {
p.answer.Problems = append(p.answer.Problems, fmt.Sprintf(format, a...))
}
for name, m := range shelf {
if m.DeclaresConfig() {
p.modules[name] = on[name]
}
}
byModule := map[string]*layersOfModule{}
var names []string
for _, l := range layers {
said := fmt.Sprintf("%s on %s", l.Module, whereOf(l.Node))
m, known := shelf[l.Module]
if !known || !m.DeclaresConfig() {
p.answer.Stays = append(p.answer.Stays, said+" (its module is in the settings form)")
continue
}
if _, placement := catalogue.SplitPlacement(l.Values); len(placement) > 0 {
problem("%s still holds placement (%s): run move-placement first", said, strings.Join(movedKeys(placement), ", "))
continue
}
assigned := on[l.Module]
if (l.Node == "" && len(assigned) == 0) || (l.Node != "" && !containsString(assigned, l.Node)) {
p.answer.Unread = append(p.answer.Unread, said+" (the module is not assigned there, so it composes nothing)")
p.moved = append(p.moved, l)
continue
}
// The layer's keys as config keys: a key the file names otherwise is set under its config key.
as := catalogue.TakenAs(m)
values := map[string]any{}
from := map[string]string{}
for _, k := range movedKeys(l.Values) {
key := k
if _, declared := m.Settings[k]; !declared {
if renamed, ok := as[k]; ok {
key = renamed
} else {
problem("%s says %q, which %s does not declare: declare it, or clear it from the layer", said, k, l.Module)
continue
}
}
if _, both := values[key]; both {
// The layer says the key and the file's name for it: the config key wins, as composition has it (issue 751).
if key != k {
continue
}
}
values[key] = l.Values[k]
from[key] = said + ": " + k
}
om := byModule[l.Module]
if om == nil {
om = &layersOfModule{nodes: map[string]map[string]any{}, saidAt: map[string]map[string]string{}}
byModule[l.Module] = om
names = append(names, l.Module)
}
om.saidAt[l.Node] = from
if l.Node == "" {
om.mesh = values
} else {
om.nodes[l.Node] = values
}
p.moved = append(p.moved, l)
p.answer.Moved = append(p.answer.Moved, said)
}
sort.Strings(names)
// Which predefined key each module falls back to, so a whole-mesh value goes to mesh config only where no other
// module's value moves with it.
fallingBack := map[string][]string{}
for name, m := range shelf {
for _, d := range m.Settings {
if d.Predefined != "" && !containsString(fallingBack[d.Predefined], name) {
fallingBack[d.Predefined] = append(fallingBack[d.Predefined], name)
}
}
}
for _, name := range names {
m, om := shelf[name], byModule[name]
asked := catalogue.AskedKeys(m)
keys := map[string]bool{}
for k := range om.mesh {
keys[k] = true
}
for _, values := range om.nodes {
for k := range values {
keys[k] = true
}
}
for _, key := range movedKeys(keys) {
d := m.Settings[key]
meshValue, meshSays := om.mesh[key]
alone := d.Predefined != "" && len(fallingBack[d.Predefined]) == 1
if meshSays && alone {
p.set(configLevel{Level: "mesh"}, d.Predefined, meshValue, d, om.said("", key))
}
for _, node := range on[name] {
nodeValue, nodeSays := om.nodes[node][key]
value := meshValue
switch {
case nodeSays && meshSays:
_, under := meshValue.(map[string]any)
_, over := nodeValue.(map[string]any)
if under && over && asked[key] {
problem("%s on %s lays an object for %q over the whole mesh's, and a placeholder takes it whole "+
"where its file merges the two: say the value on %s whole, then move", name, node, key, node)
continue
}
value = catalogue.LaidOver(meshValue, nodeValue)
case nodeSays:
value = nodeValue
case !meshSays:
continue
}
p.expect(node, name, key, value)
if nodeSays || !alone {
saidBy := om.said(node, key)
if meshSays {
saidBy = append(om.said("", key), saidBy...)
}
p.set(configLevel{Level: "module", Node: node, Module: name}, key, value, d, saidBy)
}
}
}
}
sort.Strings(p.answer.Problems)
return p
}
// layersOfModule is one module's layers as the move reads them: its whole-mesh layer's config values, each node's,
// and where each key was said, by node ("" for the whole mesh).
type layersOfModule struct {
mesh map[string]any
nodes map[string]map[string]any
saidAt map[string]map[string]string
}
func (om *layersOfModule) said(node, key string) []string {
if s, ok := om.saidAt[node][key]; ok {
return []string{s}
}
return nil
}
// set is one value the move sets, judged against its declaration as mesh-config will judge it.
func (p *configMovePlan) set(level configLevel, key string, value any, d catalogue.SettingDeclaration, from []string) {
p.answer.Sets = append(p.answer.Sets, configSet{configLevel: level, Key: key, Value: value, Trusted: d.Trusted,
From: from})
for _, wrong := range catalogue.ValueProblems(d, value) {
p.answer.Problems = append(p.answer.Problems, fmt.Sprintf("%s at %s: %s", key, level, wrong))
}
}
func (p *configMovePlan) expect(node, module, key string, value any) {
k := node + "." + module
if p.expected[k] == nil {
p.expected[k] = map[string]any{}
}
p.expected[k][key] = value
}
func movedKeys[V any](m map[string]V) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
// configService is mesh-config as the move asks it: what it holds at each level, what it resolved, and a value set
// as the controller carries it from its terminal.
type configService interface {
// Values is every value set, by its key in mesh-config_values (`mesh.<key>`, `node.<node>.<key>`,
// `module.<node>.<module>.<key>`).
Values(ctx context.Context) (map[string]json.RawMessage, error)
// Resolved is every resolved value.
Resolved(ctx context.Context) ([]resolvedEntry, error)
// Set sets one value at its level, by whom and why.
Set(ctx context.Context, level configLevel, key string, value any, by, why string) error
}
// runConfigMove is the move against a store and mesh-config: a dry run says what it would set.
func runConfigMove(ctx context.Context, inv *inventory.Inventory, config configService, apply bool, by,
why string) (configMoveAnswer, error) {
layers, holding, err := inv.LayersToMove(ctx)
if err != nil {
return configMoveAnswer{}, err
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return configMoveAnswer{}, err
}
shelf := map[string]catalogue.Manifest{}
on := map[string][]string{}
for _, e := range entries {
shelf[e.Manifest.Module] = e.Manifest
on[e.Manifest.Module] = e.On
}
p := planConfigMove(layers, shelf, on)
if len(holding) > 0 && len(p.answer.Problems) == 0 {
p.answer.Problems = append(p.answer.Problems, "a layer still holds placement ("+strings.Join(holding, "; ")+
"): run move-placement first")
}
moved, err := inv.ConfigMoved(ctx)
if err != nil {
return configMoveAnswer{}, err
}
p.answer.Switched = moved.Moved
if config == nil {
return p.answer, errors.New("mesh-config cannot be asked, so nothing was moved")
}
held, err := config.Values(ctx)
if err != nil {
return p.answer, fmt.Errorf("what mesh-config holds cannot be read, so nothing was moved: %w", err)
}
for i := range p.answer.Sets {
s := &p.answer.Sets[i]
if was, ok := held[s.valueKey(s.Key)]; ok && sameJSON(was, s.Value) {
s.Already = true
}
}
p.answer.DryRun = !apply
if !apply || len(p.answer.Problems) > 0 {
if apply {
return p.answer, fmt.Errorf("the move of config refuses, and nothing was set: %s",
strings.Join(p.answer.Problems, "; "))
}
return p.answer, nil
}
if len(p.moved) == 0 && moved.Moved {
return p.answer, nil
}
for _, s := range p.answer.Sets {
if s.Already {
continue
}
said := "moved from the settings layer of " + strings.Join(s.From, " and ") + " (issue 664): " + why
if err := config.Set(ctx, s.configLevel, s.Key, s.Value, by, said); err != nil {
return p.answer, fmt.Errorf("%s at %s was not set, and no layer was emptied; ask the move again: %w",
s.Key, s.configLevel, err)
}
}
// Read back through mesh-config before anything points at it (design 57): every module the move reaches, and
// every other in the config form, must resolve exactly what its layers lay today.
if differs, err := readsBack(ctx, config, p); err != nil {
return p.answer, err
} else if len(differs) > 0 {
return p.answer, fmt.Errorf("mesh-config does not resolve what the layers lay today, so no layer was emptied and "+
"the controller still composes from the layers: %s", strings.Join(differs, "; "))
}
if err := inv.FinishConfigMove(ctx, p.moved, by, why); err != nil {
return p.answer, err
}
p.answer.Switched = true
return p.answer, nil
}
// readsBack compares what mesh-config resolved with what the layers lay, for every module in the config form on
// every node it is assigned to, and says each difference. A value the layers do not lay and mesh-config resolves from
// a level would be laid once the switch is on: a difference too.
func readsBack(ctx context.Context, config configService, p configMovePlan) ([]string, error) {
var resolved []resolvedEntry
var err error
// mesh-config writes the resolved values with each set; a pass that missed one is put right by its next
// resolution, so a difference is read again for a little while before it is said.
for wait := 0; ; wait++ {
if resolved, err = config.Resolved(ctx); err != nil {
return nil, fmt.Errorf("mesh-config's resolved values cannot be read back, so no layer was emptied: %w", err)
}
differs, err := differences(resolved, p)
if err != nil || len(differs) == 0 || wait >= readBackTries {
return differs, err
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(readBackEvery):
}
}
}
// How often, and how many times, the move reads mesh-config's resolved values back before it says they differ.
var (
readBackEvery = 2 * time.Second
readBackTries = 15
)
func differences(resolved []resolvedEntry, p configMovePlan) ([]string, error) {
var differs []string
for _, module := range movedKeys(p.modules) {
for _, node := range p.modules[module] {
laid, err := configLaid(resolved, node, module)
if err != nil {
return nil, err
}
want := p.expected[node+"."+module]
if want == nil {
want = map[string]any{}
}
for _, k := range movedKeys(mergedKeys(laid, want)) {
got, has := laid[k]
w, wants := want[k]
switch {
case !wants:
differs = append(differs, fmt.Sprintf("%s on %s resolves %s from a level no layer says; clear it "+
"there, or set it in a layer first", module, node, k))
case !has:
differs = append(differs, fmt.Sprintf("%s on %s does not resolve %s from the value moved", module, node, k))
case !reflect.DeepEqual(normalJSON(got), normalJSON(w)):
differs = append(differs, fmt.Sprintf("%s on %s resolves %s as %s, and its layers lay %s", module,
node, k, compactJSON(got), compactJSON(w)))
}
}
}
}
return differs, nil
}
func mergedKeys(a, b map[string]any) map[string]bool {
out := map[string]bool{}
for k := range a {
out[k] = true
}
for k := range b {
out[k] = true
}
return out
}
func normalJSON(v any) any {
raw, _ := json.Marshal(v)
var out any
_ = json.Unmarshal(raw, &out)
return out
}
func compactJSON(v any) string {
raw, _ := json.Marshal(v)
return string(raw)
}
func sameJSON(raw json.RawMessage, v any) bool {
var held any
if err := json.Unmarshal(raw, &held); err != nil {
return false
}
return reflect.DeepEqual(held, normalJSON(v))
}
// busConfig is mesh-config on this controller's bus: its buckets read directly, a value set through its verb in
// the controller's own name, so mesh-config takes a trusted value as given at the controller's terminal (ADR 0348 §8).
type busConfig struct{}
func (busConfig) Values(ctx context.Context) (map[string]json.RawMessage, error) {
out := map[string]json.RawMessage{}
err := onTheBus(func(conn *nats.Conn) error {
js, err := jetstream.New(conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
kv, err := js.KeyValue(ctx, broker.BucketName("mesh-config", "values"))
if err != nil {
return err
}
w, err := kv.WatchAll(ctx, jetstream.IgnoreDeletes())
if err != nil {
return err
}
defer func() { _ = w.Stop() }()
for e := range w.Updates() {
if e == nil {
return nil
}
var v struct {
Value json.RawMessage `json:"value"`
}
if err := json.Unmarshal(e.Value(), &v); err != nil {
return fmt.Errorf("mesh-config holds %q, which is not a value: %w", e.Key(), err)
}
out[e.Key()] = v.Value
}
return errors.New("mesh-config's values stopped answering")
})
return out, err
}
func (busConfig) Resolved(ctx context.Context) ([]resolvedEntry, error) { return readResolved(ctx) }
func (busConfig) Set(ctx context.Context, level configLevel, key string, value any, by, why string) error {
args := map[string]any{"action": "set", "key": key, "value": value, "why": why, "by": by}
if level.Node != "" {
args["node"] = level.Node
}
if level.Module != "" {
args["module"] = level.Module
}
return onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskMeshSeatToolAs(ctx, conn, broker.ControllerName, "mesh-config", "config", args, 30*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return errors.New(answer.Error)
}
return nil
})
}
// moveConfigCommand is `move-config`: the move of config (issue 664).
func moveConfigCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("move-config", flag.ContinueOnError)
asJSON := set.Bool("json", false, "answer as JSON")
confirm := set.Bool("confirm", false, "set the values and switch, rather than only say what would move")
f := addHandActFlags(set)
positionals, err := parseAround(set, args)
if err != nil {
return err
}
if len(positionals) != 0 {
return errors.New("move-config [--json] [--confirm --why <text>]")
}
if !*confirm && f.given() {
return errors.New("move-config takes --why only with --confirm: without it nothing is moved. Nothing was done")
}
if *confirm {
if err := f.require("move-config"); err != nil {
return err
}
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
if *confirm {
f.record(ctx, "move-config", nil)
}
why := ""
if *confirm {
why = strings.TrimSpace(*f.why)
}
answer, err := runConfigMove(ctx, open.inventory, busConfig{}, *confirm, "the move of config, run by "+link.Caller(), why)
if *asJSON {
encoder := json.NewEncoder(os.Stdout)
encoder.SetIndent("", " ")
if encErr := encoder.Encode(answer); encErr != nil {
return encErr
}
return err
}
fmt.Print(describeConfigMove(answer))
return err
}
// describeConfigMove says the answer as a person reads it.
func describeConfigMove(a configMoveAnswer) string {
var b strings.Builder
verb := "set"
if a.DryRun {
verb = "would set"
}
for _, s := range a.Sets {
state := verb
if s.Already {
state = "holds already"
}
fmt.Fprintf(&b, "%s %s = %s at %s (from %s)\n", state, s.Key, compactJSON(s.Value), s.configLevel,
strings.Join(s.From, " and "))
}
for _, s := range a.Unread {
fmt.Fprintf(&b, "emptied, set nowhere: %s\n", s)
}
for _, s := range a.Stays {
fmt.Fprintf(&b, "stays: %s\n", s)
}
for _, s := range a.Problems {
fmt.Fprintf(&b, "refused: %s\n", s)
}
switch {
case len(a.Problems) > 0:
b.WriteString("nothing was set: the move refuses until each is put right\n")
case a.DryRun:
fmt.Fprintf(&b, "%d layer(s), %d value(s); nothing was set — with --confirm --why <text> they are\n",
len(a.Moved), len(a.Sets))
case a.Switched:
fmt.Fprintf(&b, "%d layer(s) moved into mesh-config, each kept whole in its history first; the controller "+
"reads mesh-config where it read them\n", len(a.Moved))
}
return b.String()
}
@@ -1,467 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"sort"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/overlay"
)
func aConfiguredMesh(t *testing.T) (*stores, configuredMesh, map[string]map[string]any) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for i, name := range []string{"workstation", "desktop"} {
record, err := inv.AddNode(ctx, name)
if err != nil {
t.Fatal(err)
}
if err := inv.SetPlace(ctx, name, name+".example:51820", "here", false, fmt.Sprintf("10.77.0.%d", i+3)); err != nil {
t.Fatal(err)
}
if err := inv.RecordProfile(ctx, record.ID, map[string]any{"capabilities": []any{
map[string]any{"name": "container-runtime", "present": true},
map[string]any{"name": "systemd", "present": true}}}); err != nil {
t.Fatal(err)
}
if err := inv.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := inv.RecordOverlayKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, name, overlay.Name); err != nil {
t.Fatal(err)
}
}
raw, err := os.ReadFile(filepath.Join("testdata", "config-moved", "mesh.json"))
if err != nil {
t.Fatal(err)
}
var w configuredMesh
if err := json.Unmarshal(raw, &w); err != nil {
t.Fatal(err)
}
list, err := catalogue.ParsePredefined(w.Predefined)
if err != nil {
t.Fatal(err)
}
catalogue.UsePredefined(list)
served := map[string]map[string]any{}
for _, text := range w.Modules {
var shape map[string]json.RawMessage
if err := json.Unmarshal(text, &shape); err != nil {
t.Fatal(err)
}
var name string
_ = json.Unmarshal(shape["module"], &name)
if s, ok := shape["serves-checked"]; ok {
var serves map[string]any
if err := json.Unmarshal(s, &serves); err != nil {
t.Fatal(err)
}
served[name] = serves
delete(shape, "serves-checked")
}
text, err := json.Marshal(shape)
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(text)
if err != nil {
t.Fatalf("%s: %v", name, err)
}
register(t, open, m)
}
for _, l := range w.Layers {
values, err := json.Marshal(l.Values)
if err != nil {
t.Fatal(err)
}
if l.Node == "" {
inv.ExecForTest(t, `insert into settings (node, module, values) values (null, $1, $2)`, l.Module, string(values))
continue
}
inv.ExecForTest(t, `insert into settings (node, module, values) select id, $2, $3 from node where name = $1`,
l.Node, l.Module, string(values))
}
for _, module := range movedKeys(w.On) {
for _, node := range w.On[module] {
if _, err := assign(ctx, open, node, module); err != nil {
t.Fatalf("assign %s %s: %v", node, module, err)
}
}
}
return open, w, served
}
func TestEveryMachineIsToldTheSameAfterTheMoveOfConfig(t *testing.T) {
open, w, served := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
before := toldEveryMachine(t, open, served, w.On)
listedBefore := effectiveEverywhere(t, open, w)
factsBefore, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
asked, err := runConfigMove(ctx, inv, config, false, "the move of config, run by a test", "")
if err != nil || !asked.DryRun || asked.Switched || len(asked.Problems) > 0 || len(config.sets) > 0 {
t.Fatalf("the move, not confirmed, says %+v (%v) and set %v", asked, err, config.sets)
}
// nodered's live layer says the file's name for its time zone: set under the config key (issue 751).
if s := setAt(asked, "module.anchor.nodered.time-zone"); len(s) != 1 || s[0].Value != "Europe/Brussels" ||
!strings.HasSuffix(s[0].From[0], ": timeZone") {
t.Errorf("nodered's time zone moves as %+v", s)
}
for _, s := range asked.Sets {
if s.Key == "timeZone" {
t.Errorf("a key is set under the file's name: %+v", s)
}
}
// claude-code's whole-mesh managed settings become mesh config for agent-managed-settings (ADR 0348's
// consequences), and its whole-mesh layer sets no module value.
if s := setAt(asked, "mesh.agent-managed-settings"); len(s) != 1 || s[0].From[0] != "claude-code on the whole mesh: managed_settings" {
t.Errorf("claude-code's managed settings move as %+v", s)
}
for _, node := range fourMachines {
if s := setAt(asked, "module."+node+".claude-code.managed_settings"); len(s) > 0 {
t.Errorf("claude-code's managed settings are also set on %s: %+v", node, s)
}
}
// A whole-mesh layer of a value no predefined key holds goes to each machine it is assigned to, under that
// machine's layer: power's on the workstation is the workstation's, its own elsewhere.
if s := setAt(asked, "module.workstation.power.handle-power-key"); len(s) != 1 || s[0].Value != "suspend" {
t.Errorf("power's key on the workstation moves as %+v", s)
}
if s := setAt(asked, "module.anchor.power.handle-power-key"); len(s) != 1 || s[0].Value != "poweroff" {
t.Errorf("power's key on the anchor moves as %+v", s)
}
for _, s := range asked.Sets {
if s.Level == "node" {
t.Errorf("a machine's layer became node config: %+v", s)
}
}
var stays []string
for _, s := range asked.Stays {
stays = append(stays, strings.Fields(s)[0])
}
sort.Strings(stays)
if want := []string{"bazarr", "bazarr", "bookshelf", "jackett", "kernel", "kometa", "lidarr", "nzbget", "ombi",
"plex", "qbittorrent", "radarr", "sonarr", "tautulli"}; len(stays) < 10 || !containsString(stays, "kernel") {
t.Errorf("the layers that stay are %v, want those of %v", stays, want)
}
if after := toldEveryMachine(t, open, served, w.On); !reflect.DeepEqual(after, before) {
t.Fatal("a dry run changed what a machine is told")
}
moved, err := runConfigMove(ctx, inv, config, true, "the move of config, run by a test", "release 57 is delivered")
if err != nil || moved.DryRun || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if len(config.sets) != len(asked.Sets) {
t.Fatalf("the move set %d values, and said it would set %d", len(config.sets), len(asked.Sets))
}
if held := inv.QueryForTest(t, `select module from settings s where values <> '{}'
and module in (select name from module where manifest ? 'config')`); len(held) != 0 {
t.Fatalf("a layer of a module in the config form still holds values: %v", held)
}
after := toldEveryMachine(t, open, served, w.On)
for _, node := range fourMachines {
if after[node] != before[node] {
t.Errorf("%s is told something else after the move:\n%s\nwas:\n%s", node, after[node], before[node])
}
}
if listedAfter := effectiveEverywhere(t, open, w); !reflect.DeepEqual(listedAfter, listedBefore) {
t.Errorf("settings lists other values after the move:\n%v\nwas:\n%v", listedAfter, listedBefore)
}
// And config lists what settings listed: each module's resolved value on each machine is the value settings
// said before the move.
resolved, err := config.Resolved(ctx)
if err != nil {
t.Fatal(err)
}
for _, r := range resolved {
said, listed := listedBefore[r.Node+"."+r.Module+"."+r.Key]
if r.Source == "unset" {
if listed && said != nil {
t.Errorf("%s of %s on %s is unset in config, and settings said %v", r.Key, r.Module, r.Node, said)
}
continue
}
var v any
_ = json.Unmarshal(r.Value, &v)
if !listed || !reflect.DeepEqual(v, said) {
t.Errorf("config resolves %s of %s on %s as %v, and settings said %v", r.Key, r.Module, r.Node, v, said)
}
}
// The facts snapshot carries mesh-config's values for the merge gate, as each machine's layer of each module, and
// each machine's declaration composes to the digest it did before the move.
factsAfter, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
// Machines are named in a snapshot by stand-ins of its own; each is found by what it composes.
digests := func(f snapshotFacts) ([]string, string) {
var out []string
zone := ""
for _, m := range f.Machines {
out = append(out, m.Declaration.Digest)
for _, l := range m.Settings {
if l.Module == "nodered" {
zone = compactJSON(l.Values["time-zone"])
}
}
}
sort.Strings(out)
return out, zone
}
was, _ := digests(factsBefore)
now, zone := digests(factsAfter)
if zone != `"Europe/Brussels"` {
t.Errorf("the facts snapshot does not carry nodered's time zone from mesh-config: %s", zone)
}
if len(now) != len(fourMachines) || now[0] == "" || !reflect.DeepEqual(now, was) {
t.Errorf("the declarations in the facts snapshot are %v after the move, %v before", now, was)
}
for _, s := range factsAfter.Settings {
if _, isConfig := map[string]bool{"betterbird": true, "claude-code": true, "power": true, "xdg": true}[s.Module]; isConfig &&
len(s.Values) > 0 {
t.Errorf("the facts snapshot still carries the whole mesh's layer of %s: %v", s.Module, s.Values)
}
}
config.sets = nil
again, err := runConfigMove(ctx, inv, config, true, "the move of config, run by a test", "again")
if err != nil || len(config.sets) != 0 || len(again.Moved) != 0 {
t.Fatalf("asked again, the move set %v and says %+v (%v)", config.sets, again, err)
}
if after := toldEveryMachine(t, open, served, w.On); !reflect.DeepEqual(after, before) {
t.Fatal("the move asked again changed what a machine is told")
}
}
func TestTheMoveOfConfigRefusesWhatWouldChangeAMachine(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
inv.ExecForTest(t, `update settings set values = values || '{"endpoints": {"web": {"reach": "public"}}}'
where module = 'nodered'`)
inv.ExecForTest(t, `update settings set values = values || '{"colour": "blue"}' where module = 'unifi'`)
inv.ExecForTest(t, `update settings set values = '{"font-size": 900}' where module = 'dunst'`)
answer, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered")
if err == nil || answer.Switched || len(config.sets) > 0 {
t.Fatalf("the move went ahead: %+v (%v), set %v", answer, err, config.sets)
}
for _, want := range []string{"nodered on anchor still holds placement (endpoints): run move-placement first",
`unifi on anchor says "colour", which unifi does not declare`, "font-size at dunst's own value on workstation"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
inv.ExecForTest(t, `update settings set values = values - 'endpoints' where module = 'nodered'`)
inv.ExecForTest(t, `update settings set values = values - 'colour' where module = 'unifi'`)
inv.ExecForTest(t, `update settings set values = '{"font-size": 13, "width": 500}' where module = 'dunst'`)
// Set in mesh-config before the move, on a module whose layers say nothing of it: the switch would lay it.
config.values["module.anchor.grafana."+movedKeys(func() map[string]catalogue.SettingDeclaration {
shelf, _ := inv.Catalogue(ctx)
return shelf["grafana"].Settings
}())[0]] = "set before the move"
answer, err = runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered")
if err == nil || !strings.Contains(err.Error(), "grafana on anchor resolves") || answer.Switched {
t.Fatalf("a value no layer says was laid by the switch: %+v (%v)", answer, err)
}
if moved, _ := inv.ConfigMoved(ctx); moved.Moved {
t.Fatal("the switch is on after a refused move")
}
if held := inv.QueryForTest(t, `select count(*) from settings where values <> '{}'`); held[0][0] == "0" {
t.Fatal("a refused move emptied the layers")
}
}
func TestTheSettingsVerbForwardsToConfigOnceTheValuesMoved(t *testing.T) {
open, w, served := aConfiguredMesh(t)
ctx := t.Context()
inv := open.inventory
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
savedRead, savedAsk := readConfigValues, configAsk
readConfigValues, configAsk = config.laid, config.ask
t.Cleanup(func() { readConfigValues, configAsk = savedRead, savedAsk })
if _, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil {
t.Fatal(err)
}
shown := stdoutOf(t, func() error { return settingsCommand(ctx, []string{"show", "nodered", "--node", "anchor"}) })
for _, want := range []string{`"time-zone": "Europe/Brussels"`, `time-zone = "Europe/Brussels" (module)`,
"mesh-config.config", `"mqtt": {`} {
if !strings.Contains(shown, want) {
t.Errorf("settings show does not say %q:\n%s", want, shown)
}
}
if strings.Contains(shown, "timeZone") {
t.Errorf("settings show still says the file's name:\n%s", shown)
}
config.asked = nil
said := stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12, "width": 500}`, "--node", "workstation"})
})
if config.values["module.workstation.dunst.font-size"] != float64(12) || !strings.Contains(said, "~ font-size") ||
containsString(config.asked, "set module.workstation.dunst.width") {
t.Fatalf("settings set reached mesh-config as %v (%v):\n%s", config.values, config.asked, said)
}
told := toldEveryMachine(t, open, served, w.On)["workstation"]
if !strings.Contains(told, "12") {
t.Error("the workstation is not told the font size set through settings")
}
if err := settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12}`, "--node", "workstation"}); err == nil ||
!strings.Contains(err.Error(), "would no longer set width") {
t.Fatalf("a set dropping a value without --replace: %v", err)
}
if err := settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12}`}); err == nil ||
!strings.Contains(err.Error(), "no own value for the whole mesh") {
t.Fatalf("a module value for the whole mesh: %v", err)
}
t.Setenv(verbVar, "settings")
for _, c := range []struct{ module, node, values, key, held, why string }{
// A file root reads: refused as before the move (novox/hq issue 339).
{"hostname", "anchor", `{"hostname": "elsewhere"}`, "module.anchor.hostname.hostname", "Ace", "issue 339"},
{"messenger", "laptop", `{"desktop-machines": ["g14", "shanks"], "time-zone": "UTC"}`,
"module.laptop.messenger.time-zone", "Europe/Brussels", "issue 339"},
} {
err := settingsCommand(ctx, []string{"set", c.module, c.values, "--node", c.node})
if err == nil || !strings.Contains(err.Error(), c.why) || config.values[c.key] != c.held {
t.Fatalf("a trusted value of %s through a verb: %v, holds %v", c.module, err, config.values[c.key])
}
}
// And a trusted value no file makes the terminal's is refused as trusted, since mesh-config takes the controller's
// word for it (ADR 0348 §8).
if err := refuseTrustedThroughAVerb(ctx, inv, "messenger", "laptop", []string{"time-zone"}); err == nil ||
!strings.Contains(err.Error(), "ADR 0348 §8") {
t.Fatalf("a trusted value through a verb: %v", err)
}
if err := refuseTrustedThroughAVerb(ctx, inv, "dunst", "workstation", []string{"width"}); err != nil {
t.Fatalf("a value nobody trusts through a verb: %v", err)
}
stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"set", "dunst", `{"font-size": 12, "width": 600}`, "--node", "workstation"})
})
if config.values["module.workstation.dunst.width"] != float64(600) {
t.Fatalf("a value nobody trusts was not set through a verb: %v", config.values["module.workstation.dunst.width"])
}
t.Setenv(verbVar, "")
stdoutOf(t, func() error { return settingsCommand(ctx, []string{"clear", "dunst", "--node", "workstation"}) })
for _, k := range []string{"font-size", "width"} {
if _, held := config.values["module.workstation.dunst."+k]; held {
t.Errorf("settings clear left dunst's %s on the workstation", k)
}
}
config.asked = nil
proposed := stdoutOf(t, func() error {
return settingsCommand(ctx, []string{"propose", "hostname", `{"hostname": "elsewhere"}`, "--node", "anchor"})
})
if !containsString(config.asked, "propose module.anchor.hostname.hostname") || !strings.Contains(proposed, "p-1") {
t.Fatalf("settings propose reached mesh-config as %v:\n%s", config.asked, proposed)
}
config.asked = nil
kept := stdoutOf(t, func() error { return settingsCommand(ctx, []string{"show", "kernel", "--node", "laptop"}) })
if !strings.Contains(kept, "transparent-huge-pages") || len(config.asked) > 0 {
t.Fatalf("a module in the settings form is shown as %q, asking mesh-config %v", kept, config.asked)
}
}
func TestAnApprovalRaisedBeforeTheMoveOfConfigIsRefusedAfterIt(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
key := movedKeys(func() map[string]catalogue.SettingDeclaration { return shelf["grafana"].Settings }())[0]
values := map[string]any{key: "raised before the move"}
p := settingsProposal{Module: "grafana", Node: "anchor", Values: values, From: "g14/claude-code", At: time.Now(),
Digest: layerDigest(values), BeforeDigest: layerDigest(nil)}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has {
t.Fatalf("grafana has a layer on anchor before the test: %v %v", has, err)
}
if moved, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Fatalf("grafana does not read config after the move: %v %v", reads, err)
}
_, err = setLayerIn(open)(ctx, p, "s30", "approved by the operator")
if err == nil || !strings.Contains(err.Error(), "moved to mesh-config since the operator was shown the change") ||
!strings.Contains(err.Error(), "propose it again") {
t.Fatalf("the stale approval: %v", err)
}
if layer, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has && len(layer) > 0 {
t.Errorf("the refused approval wrote the layer: %v %v %v", layer, has, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Errorf("the refused approval turned ReadsConfig off: %v %v", reads, err)
}
}
func TestAPlacementOnlyApprovalRaisedBeforeTheMoveIsSetAfterIt(t *testing.T) {
open, _, _ := aConfiguredMesh(t)
ctx := t.Context()
readBackTries = 0
t.Cleanup(func() { readBackTries = 15 })
inv := open.inventory
config := &standInMeshConfig{open: open, values: map[string]any{}}
inv.ReadsConfigFrom(config.laid)
values := map[string]any{catalogue.PlacesSetting: map[string]any{"data": "/srv/grafana-data"}}
p := settingsProposal{Module: "grafana", Node: "anchor", Values: values, From: "g14/claude-code", At: time.Now(),
Digest: layerDigest(values), BeforeDigest: layerDigest(nil)}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || has {
t.Fatalf("grafana has a layer on anchor before the test: %v %v", has, err)
}
if moved, err := runConfigMove(ctx, inv, config, true, "a test", "release 57 is delivered"); err != nil || !moved.Switched {
t.Fatalf("the move says %+v (%v)", moved, err)
}
if _, err := setLayerIn(open)(ctx, p, "s31", "approved by the operator"); err != nil {
t.Fatalf("the placement-only approval after the move: %v", err)
}
if _, has, err := inv.Layer(ctx, "anchor", "grafana"); err != nil || !has {
t.Errorf("the approval did not set the layer: %v %v", has, err)
}
if reads, err := inv.ReadsConfig(ctx, "grafana"); err != nil || !reads {
t.Errorf("the approval turned ReadsConfig off: %v %v", reads, err)
}
}
-336
View File
@@ -1,336 +0,0 @@
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/link"
)
// configuredMesh is every live settings layer (issue 664) — read with the settings verb on 2026-10-15, on the four
// machines as anchor, laptop, workstation and desktop, the whole mesh's with no node; personal values stood in for,
// and the placement words left out, which move-placement moves first — with each module in the config form shaped
// as its definition on the feature branch: its config, its files (templates inlined) and directories, placeholders
// of kinds this test's mesh does not hold stood in for. A module still in the settings form (the media catalogue's,
// and kernel) stands in as a mergeable file that takes every key. `on` is where each is assigned, as live.
type configuredMesh struct {
Modules []json.RawMessage `json:"modules"`
Layers []configuredLayer `json:"layers"`
On map[string][]string `json:"on"`
// Predefined is the catalogue's list of predefined keys, which a module falls back to.
Predefined json.RawMessage `json:"predefined"`
}
type configuredLayer struct {
Node string `json:"node"`
Module string `json:"module"`
Values map[string]any `json:"values"`
}
var fourMachines = []string{"anchor", "laptop", "workstation", "desktop"}
// aConfiguredMesh is aMesh with two machines more, every module of the configured mesh registered and assigned, and
// its layers held as the store holds them once move-placement has run.
// standInMeshConfig is mesh-config as this test holds it: each value at its level, and resolved from the catalogue
// the store holds as mesh-config resolves (novox/hq ADR 0348 §2): the module's own value on the node, else node
// config then mesh config for the predefined key it falls back to, else its default; a required value nothing sets is
// unset.
type standInMeshConfig struct {
open *stores
values map[string]any
sets []string
// asked is each call of its verb: the action and the level's key.
asked []string
}
func (c *standInMeshConfig) Values(context.Context) (map[string]json.RawMessage, error) {
out := map[string]json.RawMessage{}
for k, v := range c.values {
raw, err := json.Marshal(v)
if err != nil {
return nil, err
}
out[k] = raw
}
return out, nil
}
func (c *standInMeshConfig) Set(_ context.Context, level configLevel, key string, value any, by, why string) error {
if by == "" || why == "" {
return fmt.Errorf("%s at %s is set without who or why", key, level)
}
c.values[level.valueKey(key)] = normalJSON(value)
c.sets = append(c.sets, level.valueKey(key))
return nil
}
func (c *standInMeshConfig) Resolved(ctx context.Context) ([]resolvedEntry, error) {
entries, err := c.open.inventory.Catalogued(ctx)
if err != nil {
return nil, err
}
var out []resolvedEntry
for _, e := range entries {
m := e.Manifest
if !m.DeclaresConfig() {
continue
}
for _, node := range e.On {
for _, key := range movedKeys(m.Settings) {
d := m.Settings[key]
r := resolvedEntry{Node: node, Module: m.Module, Key: key, Source: "default"}
value, has := c.values["module."+node+"."+m.Module+"."+key]
switch {
case has:
r.Source = "module"
case d.Predefined != "" && c.values["node."+node+"."+d.Predefined] != nil:
value, r.Source = c.values["node."+node+"."+d.Predefined], "node"
case d.Predefined != "" && c.values["mesh."+d.Predefined] != nil:
value, r.Source = c.values["mesh."+d.Predefined], "mesh"
case d.Required:
r.Source = "unset"
default:
value = d.Default
}
if r.Source != "unset" {
if r.Value, err = json.Marshal(value); err != nil {
return nil, err
}
}
out = append(out, r)
}
}
}
return out, nil
}
// laid is the stand-in as composition reads it once the switch is on (inventory.ConfigValues).
func (c *standInMeshConfig) laid(ctx context.Context, node, module string) (map[string]any, error) {
resolved, err := c.Resolved(ctx)
if err != nil {
return nil, err
}
return configLaid(resolved, node, module)
}
// snapshotFacts is the facts snapshot as gatherFacts gives it.
type snapshotFacts = snapshot.Facts
// toldEveryMachine is what each of the four machines is told — every resource and the filter's rules — and what
// each module that serves something serves, settled from its layers, as JSON a person can diff.
func toldEveryMachine(t *testing.T, open *stores, served map[string]map[string]any, on map[string][]string) map[string]string {
t.Helper()
ctx := t.Context()
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
told := map[string]string{}
for _, node := range fourMachines {
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Allocating)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
composed, err := plan.Compose(with)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
rules, err := plan.Rules(with)
if err != nil {
t.Fatalf("%s: %v", node, err)
}
settles := map[string]any{}
for module, serves := range served {
if !containsString(on[module], node) {
continue
}
layers, err := open.inventory.SettingsFor(ctx, node, module)
if err != nil {
t.Fatal(err)
}
for name, s := range serves {
fact, _ := s.(map[string]any)
settled, err := catalogue.Settle(fact, catalogue.WithDefaults(shelf[module], layers))
if err != nil {
settles[module+" serves "+name] = "error: " + err.Error()
continue
}
settles[module+" serves "+name] = settled
}
}
got, err := json.MarshalIndent(map[string]any{"resources": composed.Resources, "left-out": composed.LeftOut,
"rules": rules, "served": settles}, "", " ")
if err != nil {
t.Fatal(err)
}
told[node] = string(got)
}
return told
}
// effectiveEverywhere is each module's every value on each machine it is assigned to, as `settings show` and
// `settings preferences` say them (catalogue.Effective): the value, never where it came from, which the move
// changes by design; a key a layer says by its file's name is read as its config key.
func effectiveEverywhere(t *testing.T, open *stores, w configuredMesh) map[string]any {
t.Helper()
ctx := t.Context()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
out := map[string]any{}
for _, module := range movedKeys(w.On) {
if !shelf[module].DeclaresConfig() {
continue
}
for _, node := range w.On[module] {
layers, err := open.inventory.SettingsFor(ctx, node, module)
if err != nil {
t.Fatal(err)
}
// A layer's old spelling of a key (nodered's timeZone) is listed beside the key it is the file's name
// for, and reaches the file as that key does (issue 751): read as that key, which the move sets.
as := catalogue.TakenAs(shelf[module])
for _, v := range catalogue.Effective(shelf[module], layers) {
if key, old := as[v.Key]; old {
out[node+"."+module+"."+key] = normalJSON(v.Value)
continue
}
if _, said := out[node+"."+module+"."+v.Key]; said && v.FromDefault {
continue
}
out[node+"."+module+"."+v.Key] = normalJSON(v.Value)
}
}
}
return out
}
func setAt(a configMoveAnswer, key string) []configSet {
var out []configSet
for _, s := range a.Sets {
if s.valueKey(s.Key) == key {
out = append(out, s)
}
}
return out
}
// **The settings layers move into mesh-config and no machine is told anything new** (novox/hq ADR 0348 §9, design
// 57 K5, issue 664). Every live layer is seeded as the store holds it once move-placement has run, on modules shaped
// as their definitions, and every machine composed. The move, asked without confirming, says each value it would set
// and where from, and sets nothing; confirmed, it sets them in mesh-config, reads them back, empties the layers it
// moved and switches. Each machine is then told byte for byte what it was told before, every module serves the same,
// and `settings` lists the same values `config` resolves. Asked again, it sets nothing.
// **The move refuses what it cannot move as it composes, and sets nothing** (issue 664): a layer that still holds
// placement (move-placement has not run), a key its module does not declare, and a value its declaration refuses —
// each said by name; and a value mesh-config resolves from a level no layer says (it would be laid once the switch
// is on) leaves every layer and the switch as they were.
// The verb's command line: a dry run unless confirmed, and a confirmed move says why, recorded as a person's step of
// the release, never a repair (issue 664).
func TestTheMoveOfConfigIsADryRunUnlessConfirmed(t *testing.T) {
for _, c := range []struct {
args map[string]any
want string
}{
{map[string]any{}, "move-config --json"},
{map[string]any{"confirm": "true", "why": "release 57 is delivered"}, "move-config --json --confirm --why release 57 is delivered"},
} {
argv, err := argvFor("move-config", c.args)
if err != nil || strings.Join(argv, " ") != c.want {
t.Errorf("%v: %q %v, want %q", c.args, argv, err, c.want)
}
}
for _, args := range []map[string]any{{"confirm": "true"}, {"why": "tidy"}, {"node": "anchor"}} {
if argv, err := argvFor("move-config", args); err == nil {
t.Errorf("%v was composed as %q", args, argv)
}
}
if repairingCommand([]string{"move-config", "--json", "--confirm", "--why", "x"}) != "move-config" {
t.Error("a confirmed move through the generic verb would go unrecorded")
}
if !personsDecision(link.HandAct{Verb: "move-config"}) {
t.Error("a move of the release would count toward a healer the mesh lacks")
}
}
// ask is mesh-config's verb as the stand-in answers it: list, set, clear and propose.
func (c *standInMeshConfig) ask(ctx context.Context, args map[string]any) (json.RawMessage, error) {
str := func(k string) string { s, _ := args[k].(string); return s }
level := configLevel{Level: "mesh"}
switch {
case str("module") != "" && str("node") != "":
level = configLevel{Level: "module", Node: str("node"), Module: str("module")}
case str("node") != "":
level = configLevel{Level: "node", Node: str("node")}
}
c.asked = append(c.asked, str("action")+" "+level.valueKey(str("key")))
switch str("action") {
case "list":
resolved, err := c.Resolved(ctx)
if err != nil {
return nil, err
}
out := map[string]any{"resolved": []resolvedEntry{}, "set": []any{}}
var listed []resolvedEntry
for _, r := range resolved {
if r.Module == str("module") && (str("node") == "" || r.Node == str("node")) {
listed = append(listed, r)
}
}
out["resolved"] = listed
var set []any
for _, k := range movedKeys(c.values) {
parts := strings.SplitN(k, ".", 4)
if len(parts) == 4 && parts[0] == "module" && parts[2] == str("module") {
set = append(set, map[string]any{"level": configLevel{Level: "module", Node: parts[1], Module: parts[2]},
"key": parts[3], "value": c.values[k], "by": "a test"})
}
}
out["set"] = set
return json.Marshal(out)
case "set":
if str("why") == "" || str("by") == "" {
return nil, fmt.Errorf("a set without why or by: %v", args)
}
return nil, c.Set(ctx, level, str("key"), args["value"], str("by"), str("why"))
case "clear":
if _, had := c.values[level.valueKey(str("key"))]; !had {
return nil, fmt.Errorf("nothing is set for %s", str("key"))
}
delete(c.values, level.valueKey(str("key")))
return json.RawMessage(`{}`), nil
case "propose":
return json.RawMessage(`{"proposal":"p-1"}`), nil
}
return nil, fmt.Errorf("the stand-in does not %s", str("action"))
}
// **`settings` answers by forwarding to `config`** (novox/hq ADR 0348 §1, design 57 K5, issue 664): once the move
// has run, a module in the config form is shown from mesh-config, its values set and cleared there as its own on the
// machine (placement still beside the assignment), and proposed through it; a trusted value is refused through a
// verb, a module value for the whole mesh is refused, and a module still in the settings form keeps today's verb.
// **An approval for a module whose config has moved is refused, and writes nothing** (issue 830, novox/hq ADR 0348):
// a proposal raised while grafana read its layer (it had none, so the layer is as the operator was shown it) and
// approved after the move would put its values back into the layer, ReadsConfig would turn false, and the module
// would compose from emptied layers everywhere.
// **A proposal of placement only is set after the move, as before** (issue 830): forwardPropose sends it down the
// old path on purpose, so refusing it would loop the operator (refused, proposed again, refused).
-128
View File
@@ -1,128 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// **Where the controller reads config values once the move has run** (novox/hq ADR 0348 §9, design 57 K5, issue 664):
// mesh-config_resolved, the bucket mesh-config alone writes, keyed `<node>.<module>.<key>`, each value with the
// source it came from. Read only for a module the switch reaches (inventory.ReadsConfig), so a controller before the
// move never asks the bus for config. The exception to ADR 0348 §9 ("the controller holds no config value") that
// ADR 0349 clause 17's old way needs, until design 57's R1 hands every module to the provisioner.
// resolvedEntry is one key of mesh-config_resolved as the controller reads it: the value and where it came from.
type resolvedEntry struct {
Node string `json:"node"`
Module string `json:"module"`
Key string `json:"key"`
Value json.RawMessage `json:"value,omitempty"`
Source string `json:"source"`
}
// The sources a resolved value is laid over a module's files from: mesh config, node config and the module's own
// value. Its manifest's default is composition's already, and a taken key's default is never laid over its file;
// a required value nothing sets (`unset`) is refused by composition by name, as it is today.
var laidSources = map[string]bool{"mesh": true, "node": true, "module": true}
// configLaid is the values of one module on one node that are laid over its files, from the resolved entries.
func configLaid(entries []resolvedEntry, node, module string) (map[string]any, error) {
out := map[string]any{}
for _, e := range entries {
if e.Node != node || e.Module != module || !laidSources[e.Source] {
continue
}
var v any
if err := json.Unmarshal(e.Value, &v); err != nil {
return nil, fmt.Errorf("mesh-config's value %s of %s on %s cannot be read: %w", e.Key, module, node, err)
}
out[e.Key] = v
}
return out, nil
}
// resolvedRead is what one read of the whole bucket held, and when it was read.
type resolvedRead struct {
mu sync.Mutex
at time.Time
entries []resolvedEntry
}
// resolvedFresh is how long one read of the bucket answers: a plan reads every module of a machine, and one read
// serves them all; a value set since is read by the next composition.
const resolvedFresh = 5 * time.Second
var theResolved resolvedRead
// readConfigValues is where an opened inventory reads config values once the switch is on; a test replaces it.
var readConfigValues inventory.ConfigValues = busConfigValues
// busConfigValues reads one module's config values on one node from mesh-config_resolved (inventory.ConfigValues).
func busConfigValues(ctx context.Context, node, module string) (map[string]any, error) {
theResolved.mu.Lock()
defer theResolved.mu.Unlock()
if theResolved.entries == nil || time.Since(theResolved.at) > resolvedFresh {
entries, err := readResolved(ctx)
if err != nil {
return nil, err
}
theResolved.entries, theResolved.at = entries, time.Now()
}
return configLaid(theResolved.entries, node, module)
}
// readResolved is every entry of mesh-config_resolved, read on the bus.
func readResolved(ctx context.Context) ([]resolvedEntry, error) {
entries := []resolvedEntry{}
err := onTheBus(func(conn *nats.Conn) error {
js, err := jetstream.New(conn)
if err != nil {
return err
}
ctx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
kv, err := js.KeyValue(ctx, broker.ConfigResolvedBucket)
if err != nil {
return fmt.Errorf("%s cannot be opened: %w", broker.ConfigResolvedBucket, err)
}
w, err := kv.WatchAll(ctx, jetstream.IgnoreDeletes())
if err != nil {
return err
}
defer func() { _ = w.Stop() }()
for {
select {
case <-ctx.Done():
return fmt.Errorf("%s was not read whole: %w", broker.ConfigResolvedBucket, ctx.Err())
case e, open := <-w.Updates():
if !open {
return errors.New(broker.ConfigResolvedBucket + " stopped answering")
}
if e == nil {
return nil
}
var r resolvedEntry
if err := json.Unmarshal(e.Value(), &r); err != nil {
return fmt.Errorf("%s holds %q, which is not a resolved value: %w", broker.ConfigResolvedBucket,
e.Key(), err)
}
if r.Node == "" || r.Module == "" || r.Key == "" || !strings.HasPrefix(e.Key(), r.Node+"."+r.Module+".") {
return fmt.Errorf("%s holds %q, whose value names another key", broker.ConfigResolvedBucket, e.Key())
}
entries = append(entries, r)
}
}
})
return entries, err
}
-305
View File
@@ -1,305 +0,0 @@
package main
import (
"context"
"crypto/ecdh"
"crypto/x509"
"encoding/base64"
"encoding/json"
"encoding/pem"
"errors"
"fmt"
"slices"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The core's own credentials, sealed by the vault (novox/hq ADR 0349 §16, issue 755; the lead's decision on issue 755,
// notes 4 and 5).
//
// For a machine whose engine opens the vault's box (link.VaultBoxContract), each own secret of a module on the core's
// list (catalogue.CoreCredentialModules) travels as the vault sealed it to that machine's key, which the controller
// asks for on the send path (`get`, answered to the controller alone and sealed only to the key that machine's engine
// stated: mesh-vault's core.go) and carries without being able to open it. A machine below that contract is sent the
// controller's own sealed value, exactly as before.
//
// **The vault seals every answer afresh**, so the box last got is kept (inventory.CoreBox) and carried again while the
// vault's generation and the machine's key are the same: a declaration that changed with every answer would never be
// current. Every question but a send reads the kept box and asks nothing.
//
// **Until the vault holds a machine's core credentials, the controller seals them itself** (novox/hq ADR 0396, issue
// 908; it replaces the ADR 0349 §16 rule that the controller never does). When the vault does not answer, a credential
// with a kept box keeps it — the machine keeps the file it has — and one with none has its module composed as for an
// engine below the contract: with the controller's own sealed value, never left out (a core module left out took the
// bus's port from the packet filter and cut the remote machines off). Either way the send raises a condition naming
// the vault and the machine, cleared by the first send the vault answers whole. Only a machine with no sealing key, or
// one whose key cannot be read, leaves its module out.
// engineOpensVaultBox says whether a machine's node-engine opens the vault's box, by its own newest statement.
func engineOpensVaultBox(ctx context.Context, inv *inventory.Inventory, node string) (bool, error) {
stated, had, err := inv.HealthOf(ctx, node)
if err != nil {
return false, err
}
return had && stated.Contract >= link.VaultBoxContract, nil
}
// coreAnswer is the vault's `get` answer as the controller reads it: the name, the generation and the box, never
// opened.
type coreAnswer struct {
Name string `json:"name"`
Generation int `json:"generation"`
Sealed json.RawMessage `json:"sealed"`
}
// coreAsk asks the vault for one secret sealed to the given key.
type coreAsk func(name, publicKey string) (coreAnswer, error)
// coreBoxes is what the vault's boxes make of one machine's composition.
type coreBoxes struct {
// sealed is each core credential's box, by module and name (catalogue.Rendering.VaultSealed).
sealed map[string]map[string]json.RawMessage
// notPlaced is each core module left out, with why (catalogue.Rendering.NotPlaced).
notPlaced map[string]string
// keep is each box newly answered, to keep.
keep []inventory.CoreBox
// byController is each core module the vault did not answer for and no box was kept for, which is composed with
// the controller's own seal until the vault holds it (ADR 0396).
byController []string
// held is each credential the vault did not answer for whose kept box is carried again; silent is why the vault
// did not answer, per credential.
held, silent []string
}
// coreModulesOf is every module of the plan on the core's list whose own secrets this machine's declaration carries,
// with their plain own secret names, sorted: one handed to the provisioner has its secrets from the vault through the
// client, and one issued outside the mesh is never the vault's to seal here.
func coreModulesOf(modules []catalogue.Manifest, handed func(catalogue.Manifest) bool) map[string][]string {
out := map[string][]string{}
for _, m := range modules {
if !catalogue.CarriesVaultSealed(m.Module) || (handed != nil && handed(m)) {
continue
}
var names []string
for name, own := range m.OwnSecrets.Plain() {
if own.IssuedBy != catalogue.IssuedOutside {
names = append(names, name)
}
}
if len(names) > 0 {
sort.Strings(names)
out[m.Module] = names
}
}
return out
}
// sealingKeyPEM is a machine's sealing key, recorded raw in base64, as the SPKI PEM the vault seals to.
func sealingKeyPEM(raw string) (string, error) {
b, err := base64.StdEncoding.DecodeString(strings.TrimSpace(raw))
if err != nil || len(b) != 32 {
return "", errors.New("the machine's sealing key is not a raw X25519 key")
}
pub, err := ecdh.X25519().NewPublicKey(b)
if err != nil {
return "", err
}
der, err := x509.MarshalPKIXPublicKey(pub)
if err != nil {
return "", err
}
return string(pem.EncodeToMemory(&pem.Block{Type: "PUBLIC KEY", Bytes: der})), nil
}
// coreBoxesOf decides each core credential of a machine: on a send, the vault's answer (the kept box while its
// generation and key are the same), the kept box where the vault did not answer, or the module left to the controller's
// own seal where nothing was kept; on any other question, the kept box, or a stand-in the next send asks for. Pure: what to keep and what to
// say is answered, not done.
func coreBoxesOf(node, sealingKey string, core map[string][]string, kept map[string]map[string]inventory.CoreBox,
choosing Choosing, ask coreAsk) coreBoxes {
out := coreBoxes{sealed: map[string]map[string]json.RawMessage{}, notPlaced: map[string]string{}}
modules := make([]string, 0, len(core))
for m := range core {
modules = append(modules, m)
}
sort.Strings(modules)
for _, module := range modules {
if sealingKey == "" {
out.notPlaced[module] = fmt.Sprintf("%s has no sealing key, so the vault can seal %s's credentials to "+
"nothing there (novox/hq ADR 0349 §16)", node, module)
continue
}
key, err := sealingKeyPEM(sealingKey)
if err != nil {
out.notPlaced[module] = fmt.Sprintf("%s's credentials cannot be asked of the vault: %v", module, err)
continue
}
boxes := map[string]json.RawMessage{}
for _, name := range core[module] {
k, has := kept[module][name]
valid := has && k.SealedTo == sealingKey && len(k.Box) > 0
if choosing != Allocating {
if valid {
boxes[name] = k.Box
} else {
boxes[name] = catalogue.ForeseenVaultBox
}
continue
}
secret := "own/" + node + "/" + module + "/" + name
got, err := ask(secret, key)
if err == nil && (got.Generation <= 0 || canonicalBox(got.Sealed) == nil) {
err = errors.New("it answered no box")
}
if err != nil {
out.silent = append(out.silent, fmt.Sprintf("%s: %v", secret, err))
if valid {
boxes[name] = k.Box
out.held = append(out.held, module+"/"+name)
continue
}
out.byController = append(out.byController, module)
break
}
if valid && k.Generation == got.Generation {
boxes[name] = k.Box
continue
}
box := canonicalBox(got.Sealed)
boxes[name] = box
out.keep = append(out.keep, inventory.CoreBox{Module: module, Name: name, Generation: got.Generation,
SealedTo: sealingKey, Box: box})
}
if _, left := out.notPlaced[module]; !left && !slices.Contains(out.byController, module) {
out.sealed[module] = boxes
}
}
return out
}
// canonicalBox is the vault's box in the one form the declaration carries — the sealed box's fields, in their order,
// and nothing else — or nil when it is not one.
func canonicalBox(raw json.RawMessage) json.RawMessage {
var b struct {
V int `json:"v"`
Eph string `json:"eph"`
IV string `json:"iv"`
Tag string `json:"tag"`
Ct string `json:"ct"`
}
if err := json.Unmarshal(raw, &b); err != nil || b.V <= 0 || b.Eph == "" || b.Ct == "" {
return nil
}
out, _ := json.Marshal(b)
return out
}
// coreVault is how a send asks the vault for a core credential: its `get`, in the controller's name. A test stands
// in for it.
var coreVault = func(ctx context.Context) coreAsk {
vault := theVault(ctx)
return func(name, publicKey string) (coreAnswer, error) {
raw, err := vault("get", map[string]any{"name": name, "public_key": publicKey})
if err != nil {
return coreAnswer{}, err
}
var a coreAnswer
if err := json.Unmarshal(raw, &a); err != nil {
return coreAnswer{}, fmt.Errorf("the vault's answer is not one: %w", err)
}
return a, nil
}
}
// coreBoxesFor is coreBoxesOf for one machine, read from and kept in the store, and said on a send.
func coreBoxesFor(ctx context.Context, inv *inventory.Inventory, node string, modules []catalogue.Manifest,
handed func(catalogue.Manifest) bool, choosing Choosing) (coreBoxes, error) {
core := coreModulesOf(modules, handed)
if len(core) == 0 {
if choosing == Allocating {
settleVaultSilent(ctx, node)
}
return coreBoxes{}, nil
}
key, err := inv.SealingKeyOf(ctx, node)
if err != nil {
return coreBoxes{}, err
}
kept, err := inv.KeptCoreBoxes(ctx, node)
if err != nil {
return coreBoxes{}, err
}
var ask coreAsk
if choosing == Allocating {
ask = coreVault(ctx)
}
out := coreBoxesOf(node, key, core, kept, choosing, ask)
for _, b := range out.keep {
if err := inv.KeepCoreBox(ctx, node, b); err != nil {
return coreBoxes{}, err
}
}
if choosing == Allocating {
sayVaultSilent(ctx, node, out)
}
return out, nil
}
// kindVaultSilent is a send that did not have the vault's box for a core credential of a machine.
const kindVaultSilent = "vault-silent"
// vaultSilentObservation is that, as a condition: `machine.<machine>.vault-silent`.
func vaultSilentObservation(node string, b coreBoxes) conditions.Observation {
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: node, Kind: kindVaultSilent, Token: kindVaultSilent,
Machine: node, Severity: conditions.Warning, Source: coreDowngradeSource}
var said []string
if len(b.notPlaced) > 0 {
o.Severity = conditions.Urgent
var left []string
for m := range b.notPlaced {
left = append(left, m)
}
sort.Strings(left)
said = append(said, "not placed: "+strings.Join(left, ", "))
}
if len(b.byController) > 0 {
said = append(said, "sealed by the controller until the vault holds it: "+strings.Join(b.byController, ", "))
}
if len(b.held) > 0 {
said = append(said, "kept as last sent: "+strings.Join(b.held, ", "))
}
o.Summary = fmt.Sprintf("the vault did not answer for the core's credentials on %s; %s", node, strings.Join(said, "; "))
o.Said = o.Summary + " — " + strings.Join(b.silent, "; ")
return o
}
// sayVaultSilent raises the condition for a send the vault did not answer whole, and clears it for one it did.
func sayVaultSilent(ctx context.Context, node string, b coreBoxes) {
if len(b.silent) == 0 && len(b.notPlaced) == 0 {
settleVaultSilent(ctx, node)
return
}
fmt.Printf("HELD: %s\n", vaultSilentObservation(node, b).Said)
if conditionsFrom == nil {
return
}
if _, err := conditionsFrom.Observe(ctx, vaultSilentObservation(node, b)); err != nil {
fmt.Printf("the condition for the vault's silence on %s could not be raised: %v\n", node, err)
}
}
// settleVaultSilent clears the condition once a send had every core credential of the machine from the vault.
func settleVaultSilent(ctx context.Context, node string) {
if conditionsFrom == nil {
return
}
key := vaultSilentObservation(node, coreBoxes{}).Key()
if _, err := conditionsFrom.Clear(ctx, key, "the vault answered for every core credential of "+node); err != nil {
fmt.Printf("the condition %s could not be cleared: %v\n", key, err)
}
}
@@ -1,129 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func aCoreMesh(t *testing.T, contract int) (*stores, *coreStandIn) {
t.Helper()
open := aMesh(t)
ctx := t.Context()
client := aClientModule()
register(t, open, client)
if _, err := assign(ctx, open, "laptop", client.Module); err != nil {
t.Fatal(err)
}
if err := issueProvisionerClient(ctx, open.inventory, client, "laptop", broker.Broker{Fingerprint: "ab"},
"10.0.0.1:4222"); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.RecordHealth(ctx, inventory.NodeHealth{Node: "laptop", Contract: contract,
SaidAt: time.Now()}); err != nil {
t.Fatal(err)
}
v := &coreStandIn{generation: 1}
withVault(t, v)
return open, v
}
func TestTheSendCarriesTheVaultsBoxForACoreCredentialAndKeepsIt(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
key, err := open.inventory.SealingKeyOf(t.Context(), "laptop")
if err != nil {
t.Fatal(err)
}
f := clientsBroker(sentTo(t, open, "laptop"))
if f == nil || f["sealed"] != nil || f[catalogue.VaultSealedField] == nil {
t.Fatalf("the core credential is sent as %v: the vault's box, never the controller's seal", f)
}
if want := []string{"own/laptop/mesh-provisioner-client/" + broker.ProvisionerSecret}; !reflect.DeepEqual(vault.asked, want) {
t.Fatalf("the vault was asked %v", vault.asked)
}
if want, _ := sealingKeyPEM(key); vault.keys[0] != want {
t.Fatalf("the vault was asked to seal to %q, not the machine's key", vault.keys[0])
}
first, _ := json.Marshal(f[catalogue.VaultSealedField])
// The vault answers afresh with the same generation: the kept box is carried again, so the declaration is stable.
again, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
if string(again) != string(first) {
t.Fatalf("a fresh box of the same generation changed the declaration: %s, then %s", first, again)
}
// A rotation in the vault (a newer generation) is carried.
vault.generation = 2
rotated, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
if string(rotated) == string(first) {
t.Fatal("a newer generation was not carried")
}
if _, found := vaultSilence(t, "laptop"); found {
t.Fatal("a send the vault answered raised its silence")
}
}
func TestASilentVaultHoldsTheKeptBoxAndSaysSo(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
first, _ := json.Marshal(clientsBroker(sentTo(t, open, "laptop"))[catalogue.VaultSealedField])
vault.silent = errors.New("no responders")
held := clientsBroker(sentTo(t, open, "laptop"))
got, _ := json.Marshal(held[catalogue.VaultSealedField])
if held == nil || held["sealed"] != nil || string(got) != string(first) {
t.Fatalf("a silent vault's credential is sent as %v: the box kept, never the controller's seal", held)
}
c, found := vaultSilence(t, "laptop")
if !found || !strings.Contains(c.Summary, "vault") || !strings.Contains(c.Summary, "laptop") {
t.Fatalf("no condition names the vault and the machine: %+v", c)
}
// The vault answers again: the condition clears.
vault.silent = nil
sentTo(t, open, "laptop")
if _, found := vaultSilence(t, "laptop"); found {
t.Fatal("the condition stayed after the vault answered")
}
}
func TestACoreModuleFirstPlacedWhileTheVaultIsSilentIsSealedByTheController(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
vault.silent = errors.New("no responders")
plan, settings, err := planFor(t.Context(), open, "laptop")
if err != nil {
t.Fatal(err)
}
declared, err := declarationFor(t.Context(), open, "laptop", plan, settings)
if err != nil {
t.Fatal(err)
}
f := clientsBroker(declared.Resources)
if f == nil || f["sealed"] == nil || f[catalogue.VaultSealedField] != nil {
t.Fatalf("the core credential is composed as %v: the controller's own seal, until the vault holds it", f)
}
if why, left := declared.leftOutWhy["mesh-provisioner-client"]; left {
t.Fatalf("the core module was left out: %v", why)
}
c, found := vaultSilence(t, "laptop")
if !found || !strings.Contains(c.Summary, "sealed by the controller until the vault holds it") {
t.Fatalf("the condition does not say why: %+v", c)
}
}
func TestAnEngineBelowTheContractIsSentTheControllersSealAsBefore(t *testing.T) {
open, vault := aCoreMesh(t, link.HandedContract)
f := clientsBroker(sentTo(t, open, "laptop"))
if f == nil || f["sealed"] == nil || f[catalogue.VaultSealedField] != nil {
t.Fatalf("an older engine is sent %v", f)
}
if len(vault.asked) != 0 {
t.Fatalf("the vault was asked %v for an older engine", vault.asked)
}
}
-131
View File
@@ -1,131 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// The core's own credentials travel sealed by the vault to their machine (novox/hq ADR 0349 §16, issue 755; the lead's
// decision, issue 755 notes 4 and 5): to an engine that opens the vault's box, the send carries the box the vault
// answered the controller and never the controller's own seal; when the vault does not answer, a kept box is carried
// again and a module with none is sealed by the controller (ADR 0396, issue 908), with a condition naming the vault and the machine. An engine below the
// contract composes exactly as before, and the vault is not asked.
// coreStandIn answers `get` with a box per name and generation, or its silence.
type coreStandIn struct {
asked []string
keys []string
generation int
silent error
}
func (v *coreStandIn) ask(name, key string) (coreAnswer, error) {
v.asked, v.keys = append(v.asked, name), append(v.keys, key)
if v.silent != nil {
return coreAnswer{}, v.silent
}
box := `{"v":1,"eph":"ZQ==","iv":"aQ==","tag":"dA==","ct":"` + strings.Repeat("c", len(v.asked)) + `"}`
return coreAnswer{Name: name, Generation: v.generation, Sealed: json.RawMessage(box)}, nil
}
func withVault(t *testing.T, v *coreStandIn) {
t.Helper()
before := coreVault
coreVault = func(context.Context) coreAsk { return v.ask }
t.Cleanup(func() { coreVault = before })
}
func clientsBroker(resources []map[string]any) map[string]any {
for _, r := range resources {
if r["id"] == "mesh-provisioner-client.needs-"+broker.ProvisionerSecret {
return r
}
}
return nil
}
func vaultSilence(t *testing.T, node string) (conditions.Condition, bool) {
t.Helper()
c, found, err := conditionsFrom.Get(t.Context(), vaultSilentObservation(node, coreBoxes{}).Key())
if err != nil {
t.Fatal(err)
}
return c, found
}
// Issue 908: a machine at the vault-box contract, the vault silent, no kept box: nats is composed with the
// controller's own seal, never left out, and the vault-silent condition names it.
func TestReplay908(t *testing.T) {
key := "AAECAwQFBgcICQoLDA0ODxAREhMUFRYXGBkaGxwdHh8="
core := map[string][]string{"nats": {"password"}}
ask := func(name, publicKey string) (coreAnswer, error) { return coreAnswer{}, errors.New("no responders") }
got := coreBoxesOf("novox", key, core, nil, Allocating, ask)
if _, sealed := got.sealed["nats"]; sealed || len(got.notPlaced) != 0 {
t.Fatalf("nats is vault-sealed %v or not placed %v", got.sealed, got.notPlaced)
}
if !reflect.DeepEqual(got.byController, []string{"nats"}) {
t.Fatalf("nats is not left to the controller's seal: %+v", got)
}
o := vaultSilentObservation("novox", got)
if o.Severity != conditions.Warning || !strings.Contains(o.Summary, "sealed by the controller until the vault holds it: nats") {
t.Fatalf("the condition does not name nats: %+v", o)
}
// A kept box is still carried while the vault is silent.
kept := map[string]map[string]inventory.CoreBox{"nats": {"password": {SealedTo: key, Box: json.RawMessage(`{"v":1}`)}}}
got = coreBoxesOf("novox", key, core, kept, Allocating, ask)
if string(got.sealed["nats"]["password"]) != `{"v":1}` || len(got.byController) != 0 {
t.Fatalf("the kept box was not held: %+v", got)
}
}
// An engine below VaultBoxContract is sent the controller's own seal, as before, and the vault is not asked.
// Every question but a send reads the kept box and asks nothing; with none kept, a stand-in the next send asks for.
func TestOnlyASendAsksTheVault(t *testing.T) {
core := map[string][]string{"nats": {"broker"}}
key := aPublicKey(t)
kept := map[string]map[string]inventory.CoreBox{"nats": {"broker": {Module: "nats", Name: "broker",
Generation: 1, SealedTo: key, Box: json.RawMessage(`{"v":1}`)}}}
never := func(string, string) (coreAnswer, error) {
t.Fatal("the vault was asked by a question that is not a send")
return coreAnswer{}, nil
}
for _, choosing := range []Choosing{Reading, Foreseeing} {
if got := coreBoxesOf("anchor", key, core, kept, choosing, never); string(got.sealed["nats"]["broker"]) != `{"v":1}` {
t.Errorf("read %s", got.sealed["nats"]["broker"])
}
if got := coreBoxesOf("anchor", key, core, nil, choosing, never); string(got.sealed["nats"]["broker"]) !=
string(catalogue.ForeseenVaultBox) {
t.Errorf("with none kept, read %s", got.sealed["nats"]["broker"])
}
}
// A box kept for another key of the machine is never carried.
other := coreBoxesOf("anchor", aPublicKey(t), core, kept, Allocating,
func(string, string) (coreAnswer, error) { return coreAnswer{}, errors.New("silent") })
if _, vaultSealed := other.sealed["nats"]; vaultSealed || !reflect.DeepEqual(other.byController, []string{"nats"}) {
t.Fatalf("a box sealed to the machine's old key was carried: %+v", other)
}
}
// The core's modules on a machine: on the list, not handed, with their own secrets but one issued outside.
func TestTheCoresModulesAreTheListsWithTheirOwnSecrets(t *testing.T) {
modules := []catalogue.Manifest{
{Module: "nats", OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/a"}, "given": {Path: "/b",
IssuedBy: catalogue.IssuedOutside}}},
{Module: "postgres", OwnSecrets: catalogue.OwnSecrets{"superuser": {Path: "/c"}}},
{Module: "mesh-provisioner", OwnSecrets: catalogue.OwnSecrets{"broker": {Path: "/d"}}},
}
got := coreModulesOf(modules, func(m catalogue.Manifest) bool { return m.Module == "mesh-provisioner" })
if want := map[string][]string{"nats": {"broker"}}; !reflect.DeepEqual(got, want) {
t.Fatalf("got %v, want %v", got, want)
}
}
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
@@ -1,148 +0,0 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/testbus"
)
func TestAMovedPathIsNoShrinkAndAShrinkThereStillIs(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
shelf := shelfFor(t, houseManifest)
declared := []inventory.DeclaredData{{Module: "house", Item: "config", Class: "irreplaceable", Owned: true}}
start := time.Now().Add(-6 * time.Hour)
measure := func(at time.Time, path string, size int64) []conditions.Observation {
t.Helper()
if _, err := inv.RecordData(ctx, "home", declared, map[string]map[string]inventory.Measurement{"house": {
"config": {Path: path, Size: bytesOf(size), MeasuredAt: when(at), LastWrite: when(at),
LastBackup: when(at)}}}, "", at); err != nil {
t.Fatal(err)
}
records, err := inv.Data(ctx)
if err != nil {
t.Fatal(err)
}
peaks, err := inv.DataPeaks(ctx, at.Add(-shrinkWindow))
if err != nil {
t.Fatal(err)
}
return dataFindings(records, peaks, shelf, nil, nil, at)
}
measure(start, "/home/operator/.claude", 94_700_000)
if got := findingsByKind(measure(start.Add(10*time.Minute), "/home/agent/.claude", 490)); got[kindDataShrank].Kind != "" {
t.Fatalf("a moved path raised a shrink: %+v", got[kindDataShrank])
}
measure(start.Add(2*time.Hour), "/home/agent/.claude", 300<<20)
got := findingsByKind(measure(start.Add(4*time.Hour), "/home/agent/.claude", 1<<20))[kindDataShrank]
if got.Severity != conditions.Urgent || !strings.Contains(got.Summary, "shrank") {
t.Fatalf("a genuine shrink at the new path was not raised: %+v", got)
}
}
func TestNatsUnassigningIrreplaceableDataRetiresItAndAnEmptyReplacementIsUrgent(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "keeper", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BackupSeat, Scope: catalogue.ScopeNode, Serves: []string{"backed-up", "now", "restore"}}}})
house, err := catalogue.ParseManifest([]byte(houseManifest))
if err != nil {
t.Fatal(err)
}
register(t, open, house)
if _, err := assign(ctx, open, "laptop", "house"); err == nil {
t.Fatal("irreplaceable data was assigned to a machine with nothing to back it up")
}
for _, node := range []string{"laptop", "anchor"} {
if _, err := assign(ctx, open, node, "keeper"); err != nil {
t.Fatal(err)
}
}
if _, err := assign(ctx, open, "laptop", "house"); err != nil {
t.Fatal(err)
}
// Sent, and applied: a holder is asked only once it has been (novox/hq issue 275).
for _, node := range []string{"laptop", "anchor"} {
pushedAndApplied(t, open, node)
}
conn := onATestBus(t)
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
laptop, anchor := &fakeHolder{}, &fakeHolder{}
laptop.serve(t, conn, "laptop")
anchor.serve(t, conn, "anchor")
now := time.Now()
heard := &watchdogs{last: &signalFacts{now: now, machines: []machineFacts{
{name: "laptop", lastHeard: now}, {name: "anchor", lastHeard: now}}}}
d := &doctor{open: open, js: js, watchdogs: heard}
var d13 probe
for _, p := range probeRegistry {
if p.ID == probeDataID {
d13 = p
}
}
run := func() []conditions.Observation {
t.Helper()
probing := context.WithValue(ctx, probeAsksKey{}, d13)
obs, err := probeData(probing, d)
if err != nil {
t.Fatal(err)
}
return obs
}
laptop.set(measuredHouse("/var/lib/house/config", 2<<30, now))
if obs := run(); len(obs) != 0 {
t.Fatalf("a measured, backed-up item raised %+v", obs)
}
r, err := inv.DataOf(ctx, "laptop", "house", "config")
if err != nil || r.Path != "/var/lib/house/config" || r.Size == nil || *r.Size != 2<<30 || r.LastBackup == nil {
t.Fatalf("what the holder measured was not kept: %+v, %v", r, err)
}
said, err := unassign(ctx, open, "laptop", "house")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(said, "house's config (/var/lib/house/config, 2.0 GB) stays where it is") {
t.Fatalf("the unassignment does not say the data stays:\n%s", said)
}
laptop.set()
run()
r, err = inv.DataOf(ctx, "laptop", "house", "config")
if err != nil || !r.Retired() || r.Path != "/var/lib/house/config" {
t.Fatalf("unassigned, the irreplaceable item is not kept retired: %+v, %v", r, err)
}
records, _ := inv.Data(ctx)
if rows := retiredData(records, time.Now()); len(rows) != 1 || rows[0].Path != "/var/lib/house/config" {
t.Fatalf("cleanup list: %+v", rows)
}
// Assigned on the anchor, onto an empty directory.
if _, err := assign(ctx, open, "anchor", "house"); err != nil {
t.Fatal(err)
}
anchor.set(measuredHouse("/var/lib/house/config", 300<<10, time.Now()))
obs := findingsByKind(run())
o, ok := obs[kindEmptyReplacement]
if !ok || o.Severity != conditions.Urgent || o.Machine != "anchor" || o.Also[0] != "laptop" {
t.Fatalf("an empty replacement of a module's data was not urgent: %+v", obs)
}
}
+134
View File
@@ -1,6 +1,7 @@
package main
import (
"context"
"encoding/json"
"strings"
"sync"
@@ -9,11 +10,14 @@ import (
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/testbus"
)
func bytesOf(n int64) *int64 { return &n }
@@ -169,6 +173,39 @@ func TestAShrinkOfMoreThanHalfIsUrgent(t *testing.T) {
// operator's own home (94.7 MB) to the agent account's fresh one (490 B), and `data-shrank` was raised
// for data that was never lost. A moved item is read against its new path only, so nothing is raised —
// and a genuine shrink at the new path, a week of history later, still is.
func TestAMovedPathIsNoShrinkAndAShrinkThereStillIs(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
shelf := shelfFor(t, houseManifest)
declared := []inventory.DeclaredData{{Module: "house", Item: "config", Class: "irreplaceable", Owned: true}}
start := time.Now().Add(-6 * time.Hour)
measure := func(at time.Time, path string, size int64) []conditions.Observation {
t.Helper()
if _, err := inv.RecordData(ctx, "home", declared, map[string]map[string]inventory.Measurement{"house": {
"config": {Path: path, Size: bytesOf(size), MeasuredAt: when(at), LastWrite: when(at),
LastBackup: when(at)}}}, "", at); err != nil {
t.Fatal(err)
}
records, err := inv.Data(ctx)
if err != nil {
t.Fatal(err)
}
peaks, err := inv.DataPeaks(ctx, at.Add(-shrinkWindow))
if err != nil {
t.Fatal(err)
}
return dataFindings(records, peaks, shelf, nil, nil, at)
}
measure(start, "/home/operator/.claude", 94_700_000)
if got := findingsByKind(measure(start.Add(10*time.Minute), "/home/agent/.claude", 490)); got[kindDataShrank].Kind != "" {
t.Fatalf("a moved path raised a shrink: %+v", got[kindDataShrank])
}
measure(start.Add(2*time.Hour), "/home/agent/.claude", 300<<20)
got := findingsByKind(measure(start.Add(4*time.Hour), "/home/agent/.claude", 1<<20))[kindDataShrank]
if got.Severity != conditions.Urgent || !strings.Contains(got.Summary, "shrank") {
t.Fatalf("a genuine shrink at the new path was not raised: %+v", got)
}
}
// Data said to be written all the time and not written; data with no backup or an old one — urgent when
// irreplaceable, a warning when valuable; and a new item given its bound before it is said.
@@ -276,6 +313,103 @@ func measuredHouse(path string, size int64, at time.Time) map[string]any {
// directory is an empty replacement — through the real stores and a real bus. The unassignment says the
// data stays; the self-check's next run retires it (kept, listed by cleanup), and when the module comes
// up on another machine with an empty directory while the full one waits retired, that is urgent.
func TestNatsUnassigningIrreplaceableDataRetiresItAndAnEmptyReplacementIsUrgent(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "keeper", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BackupSeat, Scope: catalogue.ScopeNode, Serves: []string{"backed-up", "now", "restore"}}}})
house, err := catalogue.ParseManifest([]byte(houseManifest))
if err != nil {
t.Fatal(err)
}
register(t, open, house)
if _, err := assign(ctx, open, "laptop", "house"); err == nil {
t.Fatal("irreplaceable data was assigned to a machine with nothing to back it up")
}
for _, node := range []string{"laptop", "anchor"} {
if _, err := assign(ctx, open, node, "keeper"); err != nil {
t.Fatal(err)
}
}
if _, err := assign(ctx, open, "laptop", "house"); err != nil {
t.Fatal(err)
}
// Sent, and applied: a holder is asked only once it has been (novox/hq issue 275).
for _, node := range []string{"laptop", "anchor"} {
pushedAndApplied(t, open, node)
}
conn := onATestBus(t)
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
laptop, anchor := &fakeHolder{}, &fakeHolder{}
laptop.serve(t, conn, "laptop")
anchor.serve(t, conn, "anchor")
now := time.Now()
heard := &watchdogs{last: &signalFacts{now: now, machines: []machineFacts{
{name: "laptop", lastHeard: now}, {name: "anchor", lastHeard: now}}}}
d := &doctor{open: open, js: js, watchdogs: heard}
var d13 probe
for _, p := range probeRegistry {
if p.ID == probeDataID {
d13 = p
}
}
run := func() []conditions.Observation {
t.Helper()
probing := context.WithValue(ctx, probeAsksKey{}, d13)
obs, err := probeData(probing, d)
if err != nil {
t.Fatal(err)
}
return obs
}
laptop.set(measuredHouse("/var/lib/house/config", 2<<30, now))
if obs := run(); len(obs) != 0 {
t.Fatalf("a measured, backed-up item raised %+v", obs)
}
r, err := inv.DataOf(ctx, "laptop", "house", "config")
if err != nil || r.Path != "/var/lib/house/config" || r.Size == nil || *r.Size != 2<<30 || r.LastBackup == nil {
t.Fatalf("what the holder measured was not kept: %+v, %v", r, err)
}
said, err := unassign(ctx, open, "laptop", "house")
if err != nil {
t.Fatal(err)
}
if !strings.Contains(said, "house's config (/var/lib/house/config, 2.0 GB) stays where it is") {
t.Fatalf("the unassignment does not say the data stays:\n%s", said)
}
laptop.set()
run()
r, err = inv.DataOf(ctx, "laptop", "house", "config")
if err != nil || !r.Retired() || r.Path != "/var/lib/house/config" {
t.Fatalf("unassigned, the irreplaceable item is not kept retired: %+v, %v", r, err)
}
records, _ := inv.Data(ctx)
if rows := retiredData(records, time.Now()); len(rows) != 1 || rows[0].Path != "/var/lib/house/config" {
t.Fatalf("cleanup list: %+v", rows)
}
// Assigned on the anchor, onto an empty directory.
if _, err := assign(ctx, open, "anchor", "house"); err != nil {
t.Fatal(err)
}
anchor.set(measuredHouse("/var/lib/house/config", 300<<10, time.Now()))
obs := findingsByKind(run())
o, ok := obs[kindEmptyReplacement]
if !ok || o.Severity != conditions.Urgent || o.Machine != "anchor" || o.Also[0] != "laptop" {
t.Fatalf("an empty replacement of a module's data was not urgent: %+v", obs)
}
}
// Data on redundant storage: the array it is on is watched, one condition per array as loud as the most
// precious item on it; an item said to be on redundancy and found on plain storage is said; an item
-117
View File
@@ -1,117 +0,0 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// A dead letter from a restricted stream ages out with its stream (novox/hq ADR 0376 §3, §6).
//
// DEAD_LETTERS lets such a message go when its stream's maximum age runs out, so a consumer's `max-deliveries`
// condition can clear with nobody having delivered it again or dropped it. That is said: the watchdogs remember,
// from one look to the next, when each restricted dead letter ages out, count those gone once their time passed,
// say each in the controller's log as it goes, and clear the condition with words saying how many aged out
// undelivered and from which stream.
// agingTally is what the watchdogs remember of restricted dead letters between looks.
type agingTally struct {
// seen is each consumer's restricted dead letters at the last look, by id, with when each ages out.
seen map[string]map[uint64]time.Time
// agedOut is how many of each consumer's aged out undelivered while its condition was open.
agedOut map[string]int
}
// look compares this look's restricted dead letters with the last: newly is how many of each consumer's aged out
// since, and cleared how many aged out in all for each consumer DEAD_LETTERS no longer holds anything of, when any
// did. held is every consumer's count this look; aging its restricted dead letters.
func (t *agingTally) look(now time.Time, held map[string]int, aging map[string][]link.Aging) (newly, cleared map[string]int) {
if t.seen == nil {
t.seen, t.agedOut = map[string]map[uint64]time.Time{}, map[string]int{}
}
newly, cleared = map[string]int{}, map[string]int{}
for key, before := range t.seen {
still := map[uint64]bool{}
for _, a := range aging[key] {
still[a.ID] = true
}
for id, agesOut := range before {
if !still[id] && !agesOut.After(now) {
newly[key]++
}
}
if newly[key] == 0 {
delete(newly, key)
}
t.agedOut[key] += newly[key]
}
for key, n := range t.agedOut {
if held[key] > 0 {
continue
}
if n > 0 {
cleared[key] = n
}
delete(t.agedOut, key)
}
t.seen = map[string]map[uint64]time.Time{}
for key, letters := range aging {
t.seen[key] = map[uint64]time.Time{}
for _, a := range letters {
t.seen[key][a.ID] = a.AgesOut
}
}
return newly, cleared
}
// agedOutResolved is the line a consumer's condition clears with when messages it gave up on aged out undelivered.
func agedOutResolved(key string, n int) string {
stream, _, _ := strings.Cut(key, ".")
messages := "a message"
if n > 1 {
messages = fmt.Sprintf("%d messages", n)
}
return fmt.Sprintf("Resolved: %s of the stream %s aged out undelivered", messages, stream)
}
// sayAgedOut says, in the log, what aged out since the last look, and clears the condition of every consumer whose
// dead letters are all gone with the words saying how many aged out. Before the rows are reconciled, so the
// condition is cleared with these words and not its own.
func (w *watchdogs) sayAgedOut(ctx context.Context, f *signalFacts) []string {
if f.advisoriesErr != nil || f.deadLettersAging == nil {
return nil
}
newly, cleared := w.aging.look(f.now, f.deadLetters, f.deadLettersAging)
var problems []string
for _, key := range countedKeys(newly) {
stream, consumer, _ := strings.Cut(key, ".")
fmt.Printf("%d message(s) %s gave up on aged out of %s undelivered: %s keeps them only for its maximum age "+
"(novox/hq ADR 0376)\n", newly[key], link.ConsumerInWords(stream, consumer), "DEAD_LETTERS", stream)
}
for _, key := range countedKeys(cleared) {
stream, consumer, _ := strings.Cut(key, ".")
fmt.Printf("%s no longer has a message kept: %d aged out of %s undelivered, so its condition clears "+
"(novox/hq ADR 0376)\n", link.ConsumerInWords(stream, consumer), cleared[key], stream)
if _, err := w.keeper.ClearSaying(ctx, conditions.Key(conditions.ScopeBus, key, link.AdvisoryMaxDeliveries),
fmt.Sprintf("its %d dead letter(s) of %s aged out undelivered", cleared[key], stream),
agedOutResolved(key, cleared[key])); err != nil {
problems = append(problems, err.Error())
}
}
return problems
}
// countedKeys is a count's keys in order.
func countedKeys(m map[string]int) []string {
out := make([]string, 0, len(m))
for k := range m {
out = append(out, k)
}
sort.Strings(out)
return out
}
@@ -1,69 +0,0 @@
package main
import (
"context"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **A dead letter that ages out clears its condition and says so** (novox/hq ADR 0376 §6), on a fake clock: while
// DEAD_LETTERS holds a restricted stream's messages for a consumer its max-deliveries condition is open; once they
// have aged out it clears, with words saying how many aged out undelivered and from which stream. One dropped before
// its age is not counted as aged out, and a consumer of EVENTS clears as before.
func TestADeadLetterThatAgesOutClearsItsConditionAndSaysSo(t *testing.T) {
now := time.Date(2026, 10, 15, 12, 0, 0, 0, time.UTC)
store := conditions.NewInMemory()
told := &conditions.Told{}
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store, Teller: told,
Now: func() time.Time { return now }})
w := &watchdogs{keeper: k, started: now.Add(-time.Hour)}
look := func(at time.Time, held map[string]int, aging map[string][]link.Aging) {
f := calm(at)
f.deadLetters, f.deadLettersAging = held, aging
w.see(t.Context(), f)
}
mail := "MAIL.ace_mailer"
look(now, map[string]int{mail: 3, "EVENTS.media_sonarr": 1}, map[string][]link.Aging{mail: {
{ID: 1, AgesOut: now.Add(time.Hour)}, {ID: 2, AgesOut: now.Add(2 * time.Hour)}, {ID: 3, AgesOut: now.Add(3 * time.Hour)}}})
if open, _ := k.Open(t.Context()); len(open) != 2 {
t.Fatalf("open: %+v", open)
}
// One dropped by a person before its age: not aged out.
look(now.Add(30*time.Minute), map[string]int{mail: 2, "EVENTS.media_sonarr": 1}, map[string][]link.Aging{mail: {
{ID: 2, AgesOut: now.Add(2 * time.Hour)}, {ID: 3, AgesOut: now.Add(3 * time.Hour)}}})
look(now.Add(150*time.Minute), map[string]int{mail: 1}, map[string][]link.Aging{mail: {{ID: 3, AgesOut: now.Add(3 * time.Hour)}}})
open, _ := k.Open(t.Context())
if len(open) != 1 || open[0].Key != conditions.Key(conditions.ScopeBus, mail, link.AdvisoryMaxDeliveries) {
t.Fatalf("after one aged out: %+v", open)
}
look(now.Add(4*time.Hour), map[string]int{}, map[string][]link.Aging{})
if open, _ := k.Open(t.Context()); len(open) != 0 {
t.Fatalf("after all aged out: %+v", open)
}
k.Close(context.Background()) // everything said is said once it closes
var mailCleared, sonarrCleared string
for _, e := range told.Said() {
if e.Change != conditions.ChangeCleared {
continue
}
switch e.Condition.Key {
case conditions.Key(conditions.ScopeBus, mail, link.AdvisoryMaxDeliveries):
mailCleared = e.Condition.Resolved
case conditions.Key(conditions.ScopeBus, "EVENTS.media_sonarr", link.AdvisoryMaxDeliveries):
sonarrCleared = e.Condition.Resolved
}
}
if mailCleared != "Resolved: 2 messages of the stream MAIL aged out undelivered" {
t.Fatalf("MAIL's condition cleared with %q", mailCleared)
}
if sonarrCleared != "Resolved: the messages it gave up on were delivered again or dropped" {
t.Fatalf("an EVENTS consumer's condition cleared with %q", sonarrCleared)
}
if why, ok := conditions.Plain(mailCleared); !ok {
t.Fatalf("not plain: %s", why)
}
}
@@ -1,88 +0,0 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
func servingDeadLetters(t *testing.T) *broker.JetStream {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
before := servingBus.Load()
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(before) })
return js
}
func TestDeadLettersListsDropsAndRecordsWhy(t *testing.T) {
js := servingDeadLetters(t)
if _, err := js.Context().Publish("mesh.mod.gitea.event.pull.merged", []byte(`{"n":1}`)); err != nil {
t.Fatal(err)
}
d, err := link.KeepDeadLetter(js.Context(), []byte(`{"stream":"EVENTS","consumer":"media_sonarr","stream_seq":1,"deliveries":5}`))
if err != nil {
t.Fatal(err)
}
answer, err := askDeadLetters(t, map[string]any{})
if err != nil {
t.Fatal(err)
}
listed := answer.(map[string]any)
if listed["held"] != 1 || len(listed["dead_letters"].([]link.DeadLetter)) != 1 {
t.Fatalf("listed %v", listed)
}
// Refused before anything is done: an act without why, an argument the shape passes over, both acts.
for _, args := range []map[string]any{
{"drop": "1"},
{"drop": "1", "why": "x", "limit": "3"},
{"drop": "1", "deliver": "1", "why": "x"},
{"why": "x"},
{"id": "nought"},
} {
if _, err := askDeadLetters(t, args); err == nil {
t.Errorf("%v was done", args)
}
}
if _, total, _ := link.DeadLetters(js.Context(), "", 0); total != 1 {
t.Fatalf("a refused call changed what is kept: %d left", total)
}
done, err := askDeadLetters(t, map[string]any{"drop": "1", "why": "the media server took the download in by hand"})
if err != nil {
t.Fatal(err)
}
if said := done.(map[string]any); said["recorded"] == nil || !strings.Contains(said["done"].(string), "dropped") {
t.Fatalf("answered %v", said)
}
acts, err := link.HandActs(context.Background(), js.Conn(), time.Now().Add(-time.Minute))
if err != nil || len(acts) != 1 || acts[0].Verb != "dead-letters drop" || acts[0].Cause != causeDeadLetter ||
!strings.Contains(acts[0].Condition, "media_sonarr") {
t.Fatalf("recorded %+v (%v)", acts, err)
}
if !personsDecision(acts[0]) {
t.Error("dropping a dead letter is counted as a repair, so S15 would want a healer for it")
}
if _, err := askDeadLetters(t, map[string]any{"id": "1"}); err == nil {
t.Errorf("dead letter %d is still answered after it was dropped", d.ID)
}
}
+76
View File
@@ -6,10 +6,31 @@ import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// The serving controller's bus, with the mesh's streams, for the verb to read and act on.
func servingDeadLetters(t *testing.T) *broker.JetStream {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
before := servingBus.Load()
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(before) })
return js
}
func askDeadLetters(t *testing.T, args map[string]any) (any, error) {
t.Helper()
@@ -20,6 +41,61 @@ func askDeadLetters(t *testing.T, args map[string]any) (any, error) {
return deadLettersAnswer(context.Background(), a)
}
func TestDeadLettersListsDropsAndRecordsWhy(t *testing.T) {
js := servingDeadLetters(t)
if _, err := js.Context().Publish("mesh.mod.gitea.event.pull.merged", []byte(`{"n":1}`)); err != nil {
t.Fatal(err)
}
d, err := link.KeepDeadLetter(js.Context(), []byte(`{"stream":"EVENTS","consumer":"media_sonarr","stream_seq":1,"deliveries":5}`))
if err != nil {
t.Fatal(err)
}
answer, err := askDeadLetters(t, map[string]any{})
if err != nil {
t.Fatal(err)
}
listed := answer.(map[string]any)
if listed["held"] != 1 || len(listed["dead_letters"].([]link.DeadLetter)) != 1 {
t.Fatalf("listed %v", listed)
}
// Refused before anything is done: an act without why, an argument the shape passes over, both acts.
for _, args := range []map[string]any{
{"drop": "1"},
{"drop": "1", "why": "x", "limit": "3"},
{"drop": "1", "deliver": "1", "why": "x"},
{"why": "x"},
{"id": "nought"},
} {
if _, err := askDeadLetters(t, args); err == nil {
t.Errorf("%v was done", args)
}
}
if _, total, _ := link.DeadLetters(js.Context(), "", 0); total != 1 {
t.Fatalf("a refused call changed what is kept: %d left", total)
}
done, err := askDeadLetters(t, map[string]any{"drop": "1", "why": "the media server took the download in by hand"})
if err != nil {
t.Fatal(err)
}
if said := done.(map[string]any); said["recorded"] == nil || !strings.Contains(said["done"].(string), "dropped") {
t.Fatalf("answered %v", said)
}
acts, err := link.HandActs(context.Background(), js.Conn(), time.Now().Add(-time.Minute))
if err != nil || len(acts) != 1 || acts[0].Verb != "dead-letters drop" || acts[0].Cause != causeDeadLetter ||
!strings.Contains(acts[0].Condition, "media_sonarr") {
t.Fatalf("recorded %+v (%v)", acts, err)
}
if !personsDecision(acts[0]) {
t.Error("dropping a dead letter is counted as a repair, so S15 would want a healer for it")
}
if _, err := askDeadLetters(t, map[string]any{"id": "1"}); err == nil {
t.Errorf("dead letter %d is still answered after it was dropped", d.ID)
}
}
// Open while DEAD_LETTERS holds a message for the consumer, in words the operator reads in one pass:
// what is held, and where to act.
func TestAConsumersDeadLettersAreSaidUntilActedOn(t *testing.T) {
+192 -142
View File
@@ -136,9 +136,8 @@ func letGo(ctx context.Context, inv *inventory.Inventory, id, by, why string) (i
return p, fmt.Errorf("%s was let go by %s at %s already", p.ID, p.Delivery.By,
p.Delivery.Go.Local().Format("15:04:05"))
}
// **One walk at a time beside a related one** (novox/hq ADR 0276, ADR 0407 decision 4, issue 973): the word is
// taken, and the walk starts once no started walk it is related to is open (advanceOnce), so the delivery's
// owner says it once and is not refused.
// **One walk at a time** (novox/hq ADR 0276): the word is taken, and the walk starts once no other walk is
// started (advanceOnce), so the delivery's owner says it once and is not refused.
now := time.Now().UTC()
p.Delivery.Go, p.Delivery.By, p.Delivery.Why = &now, by, why
p.Note = "let go by " + by + "; its first tier is asked next"
@@ -269,7 +268,7 @@ func stopWalkWithout(ctx context.Context, inv *inventory.Inventory, id, by, why
append(moves, foldMoves...)); err != nil {
return p, said, fmt.Errorf("%s is not stopped, and none of its merges moved: %w", p.ID, err)
}
// What the batch holds and names now: written again by the next look should this fail.
// What the batch holds and names now: written again by the cutter's next look should this fail.
if err := keepBatch(ctx, inv, &target, now, ""); err != nil {
fmt.Printf(" %s holds its merges; its record is written again at the next look: %v\n", target.ID, err)
}
@@ -579,12 +578,11 @@ type orderMember struct {
ModuleDirs []string `json:"module_dirs,omitempty"`
ModuleDirsSaid bool `json:"module_dirs_said,omitempty"`
CloneURL string `json:"clone_url,omitempty"`
// After are the repositories its pull request says it goes after (`after: <repository>`).
After []string `json:"after,omitempty"`
// BaseSHA is the base branch's head the forge last announced the head with (novox/hq ADR 0323): the trunk commit
// a verdict no build judges names when the head is checked again through mesh-delivery.
BaseSHA string `json:"base_sha,omitempty"`
// Ready says mesh-delivery wants the member's check queued before every other check (novox/hq ADR 0406 §3, issue
// 974). There is no ready mark any more (issue 1022): mesh-delivery says it when it asks for the Approve.
Ready bool `json:"ready,omitempty"`
}
// pull is the member as the forge announced it.
@@ -596,7 +594,7 @@ func (m orderMember) pull() link.PullUpdated {
}
return link.PullUpdated{Owner: owner, Repo: repo, Number: m.Number, Base: base, Commit: m.Head, BaseSHA: m.BaseSHA,
CloneURL: m.CloneURL, Paths: m.Paths, PathsTruncated: m.PathsTruncated, Removed: m.Removed,
ModuleDirs: m.ModuleDirs, ModuleDirsSaid: m.ModuleDirsSaid, Ready: m.Ready}
ModuleDirs: m.ModuleDirs, ModuleDirsSaid: m.ModuleDirsSaid}
}
// orderPair is one "before" among a group's members, why, and the rules it won over.
@@ -609,6 +607,14 @@ type orderPair struct {
Over []string `json:"over,omitempty"`
}
// orderContradiction is two members that rules of one rank order both ways: no precedence resolves it, so
// the group is refused, the rules named, and Said says how a person declares the order.
type orderContradiction struct {
Members []string `json:"members"`
Rules []orderPair `json:"rules"`
Said string `json:"said"`
}
// orderReach is what a member moves, as the planner says.
type orderReach struct {
Moved []string `json:"moved,omitempty"`
@@ -617,76 +623,114 @@ type orderReach struct {
Manifests []string `json:"manifests,omitempty"`
}
// groupOrder is a group's order: the members in it, every pair and why, and the cycle when there is one. The
// order is read from the module graph and the manifests' change alone (novox/hq ADR 0419): a cycle is the only
// refusal, and CycleSaid names every module and edge of it. Edges are the module edges the pairs come from.
// groupOrder is a group's order: the members in it, every pair and why, and the cycle when there is one —
// with each contradiction no precedence resolves.
type groupOrder struct {
Order []string `json:"order"`
Pairs []orderPair `json:"pairs,omitempty"`
Cycle []string `json:"cycle,omitempty"`
CycleSaid string `json:"cycle_said,omitempty"`
Edges []inventory.Edge `json:"edges,omitempty"`
Reach map[string]orderReach `json:"reach,omitempty"`
Order []string `json:"order"`
Pairs []orderPair `json:"pairs,omitempty"`
Cycle []string `json:"cycle,omitempty"`
Contradictions []orderContradiction `json:"contradictions,omitempty"`
Reach map[string]orderReach `json:"reach,omitempty"`
}
// orderBuiltBy is the reason a member goes before another that moves a module built by, standing on, packaging or
// declaring one of its modules: the module graph's own edge.
const orderBuiltBy = "built by"
// The reasons a pair is ordered, in the words the delivery plan shows.
const (
orderDeclared = "declared"
orderBuiltBy = "built by"
orderVersionSkew = "version skew"
orderEngineFirst = "engine before controller"
)
// orderOf is a group's order (novox/hq ADR 0419), pure: a member goes before another when a module it moves goes
// before a module the other moves by an edge of the module graph (built by), or by a rule of delivery_edges.go
// (verb before caller, version skew, engine before controller), which changes names the manifests' added verbs.
// No `after:` line is read. Ties break by repository then id, so every reading gives one order. A cycle through
// two members or more is named by its members, and none of them is ordered.
func orderOf(members []orderMember, reach map[string]orderReach, edges []inventory.Edge, changes map[string]moduleChange) groupOrder {
out := groupOrder{Reach: reach}
movedBy := map[string][]string{}
var all []string
for _, m := range members {
for _, n := range reach[m.ID].Moved {
movedBy[n] = append(movedBy[n], m.ID)
all = appendNew(all, n)
}
// orderRank is a rule's precedence when two rules order one pair both ways (novox/hq ADR 0249), the lower
// winning: what a person declared; then what the graph and the change say (built by, version skew); then
// the rollout default, which infers a direction from the modules' names alone.
func orderRank(why string) int {
switch why {
case orderDeclared:
return 0
case orderBuiltBy, orderVersionSkew:
return 1
case orderEngineFirst:
return 2
}
return 3
}
// orderOf is a group's order (novox/hq ADR 0239 decision 4, ADR 0249), pure. A member goes before another
// when:
//
// - its pull request is named in the other's `after:` lines (declared);
// - it moves a module the other's moved modules are built by, stand on, package or declare (built by);
// - it moves the controller and the other changes any module's manifest (version skew: the newer
// controller parses what the newer manifest says) — the node-engine's excepted, which goes first;
// - it moves the node-engine and the other moves the controller (the witness reads nothing the controller
// does not yet grant, and a controller sends nothing an older engine would refuse — ADR 0236's rollout
// order).
//
// When rules order one pair both ways, the one of higher precedence wins (orderRank) and the pair says
// what it won over. Rules of one rank both ways are a contradiction: both are kept as pairs, the members
// are the cycle, and the contradiction says how to declare the order. Otherwise, by repository then id, so
// every reading gives one order. A cycle through three members or more is named by its members, and none
// of them is ordered.
func orderOf(members []orderMember, reach map[string]orderReach, edges []inventory.Edge) groupOrder {
out := groupOrder{Reach: reach}
byID := map[string]orderMember{}
for _, m := range members {
byID[m.ID] = m
}
var claims []orderPair
add := func(before, after, why string) {
if before == after {
return
}
for _, p := range out.Pairs {
for _, p := range claims {
if p.Before == before && p.After == after && p.Why == why {
return
}
}
out.Pairs = append(out.Pairs, orderPair{Before: before, After: after, Why: why})
claims = append(claims, orderPair{Before: before, After: after, Why: why})
}
for _, e := range edges {
if !orders(e) {
continue
named := func(said, repository string) bool {
said = strings.TrimSuffix(strings.TrimSpace(said), ".git")
if strings.EqualFold(said, repository) {
return true
}
for _, before := range movedBy[e.To] {
for _, after := range movedBy[e.From] {
add(before, after, orderBuiltBy)
}
}
_, repo, _ := strings.Cut(repository, "/")
return strings.EqualFold(said, repo)
}
order := orderEdges(all, changes, edges)
out.Edges = order.Edges
out.CycleSaid = order.CycleSaid()
for _, e := range order.Edges {
for _, before := range movedBy[e.To] {
for _, after := range movedBy[e.From] {
add(before, after, e.Why)
for _, a := range members {
for _, b := range members {
if a.ID == b.ID {
continue
}
ra, rb := reach[a.ID], reach[b.ID]
for _, said := range b.After {
if named(said, a.Repository) {
add(a.ID, b.ID, orderDeclared)
}
}
for _, e := range edges {
if slices.Contains(ra.Moved, e.To) && slices.Contains(rb.Moved, e.From) && e.From != e.To {
add(a.ID, b.ID, orderBuiltBy)
break
}
}
if slices.Contains(ra.Moved, "mesh-controller") && len(rb.Manifests) > 0 &&
!slices.Contains(rb.Moved, "mesh-controller") && !slices.Contains(rb.Moved, "mesh-host") {
add(a.ID, b.ID, orderVersionSkew)
}
if slices.Contains(ra.Moved, "mesh-host") && slices.Contains(rb.Moved, "mesh-controller") &&
!slices.Contains(ra.Moved, "mesh-controller") {
add(a.ID, b.ID, orderEngineFirst)
}
}
}
out.Pairs, out.Contradictions = resolveClaims(claims, byID)
sort.Slice(out.Pairs, func(i, j int) bool {
if out.Pairs[i].Before != out.Pairs[j].Before {
return out.Pairs[i].Before < out.Pairs[j].Before
}
if out.Pairs[i].After != out.Pairs[j].After {
return out.Pairs[i].After < out.Pairs[j].After
}
return out.Pairs[i].Why < out.Pairs[j].Why
return out.Pairs[i].After < out.Pairs[j].After
})
// Kahn's walk, the next always the first by repository and id among those with nothing before them.
waiting := map[string]int{}
@@ -731,6 +775,89 @@ func orderOf(members []orderMember, reach map[string]orderReach, edges []invento
return out
}
// orderRules is the rules in the order a pair names them when more than one says the same.
var orderRules = []string{orderDeclared, orderBuiltBy, orderVersionSkew, orderEngineFirst}
// resolveClaims is the pairs the rules' claims come to (novox/hq ADR 0249): for each two members, the claims
// of the highest precedence decide, and the claims they overrule are said on the pair. Claims of one rank
// both ways are a contradiction: both pairs are kept, so the walk finds the cycle, and the contradiction
// says how a person declares the order.
func resolveClaims(claims []orderPair, byID map[string]orderMember) ([]orderPair, []orderContradiction) {
byTwo := map[[2]string][]orderPair{}
var keys [][2]string
for _, c := range claims {
k := [2]string{min(c.Before, c.After), max(c.Before, c.After)}
if _, ok := byTwo[k]; !ok {
keys = append(keys, k)
}
byTwo[k] = append(byTwo[k], c)
}
slices.SortFunc(keys, func(a, b [2]string) int {
if c := strings.Compare(a[0], b[0]); c != 0 {
return c
}
return strings.Compare(a[1], b[1])
})
ruleIndex := func(why string) int { return slices.Index(orderRules, why) }
var pairs []orderPair
var contradictions []orderContradiction
for _, k := range keys {
cs := byTwo[k]
slices.SortStableFunc(cs, func(a, b orderPair) int { return ruleIndex(a.Why) - ruleIndex(b.Why) })
top := orderRank(cs[0].Why)
var won []orderPair
ways := map[string]bool{}
for _, c := range cs {
if orderRank(c.Why) == top {
won = append(won, c)
ways[c.Before] = true
}
}
if len(ways) == 1 {
p := orderPair{Before: won[0].Before, After: won[0].After, Why: won[0].Why}
for _, c := range cs {
if c.Before != p.Before && !slices.Contains(p.Over, c.Why) {
p.Over = append(p.Over, c.Why)
}
}
pairs = append(pairs, p)
continue
}
// One rank both ways: no precedence resolves it.
var first, second *orderPair
var said []string
for i := range won {
c := won[i]
if c.Before == k[0] && first == nil {
first = &won[i]
}
if c.Before == k[1] && second == nil {
second = &won[i]
}
said = append(said, fmt.Sprintf("%s puts %s first", c.Why, c.Before))
}
pairs = append(pairs, orderPair{Before: first.Before, After: first.After, Why: first.Why},
orderPair{Before: second.Before, After: second.After, Why: second.Why})
a, b := byID[k[0]], byID[k[1]]
how := fmt.Sprintf("neither outranks the other: declare the order with a line `after: %s` in %s's "+
"description, or `after: %s` in %s's", a.Repository, pullName(b), b.Repository, pullName(a))
if top == orderRank(orderDeclared) {
how = "each pull request declares it goes after the other: remove one of the `after:` lines"
}
contradictions = append(contradictions, orderContradiction{Members: []string{k[0], k[1]}, Rules: won,
Said: fmt.Sprintf("%s and %s: %s; %s", k[0], k[1], strings.Join(said, ", "), how)})
}
return pairs, contradictions
}
// pullName is a member as a person finds it: its pull request, or its id when no number was said.
func pullName(m orderMember) string {
if m.Number > 0 {
return fmt.Sprintf("%s#%d", m.Repository, m.Number)
}
return m.ID
}
// reachOfMembers is each member's reach, as the planner says it.
func reachOfMembers(members []orderMember, entries []inventory.Entry, read map[string][]inventory.ReadRepository,
edges []inventory.Edge) map[string]orderReach {
@@ -764,8 +891,6 @@ func deliveryCommand(ctx context.Context, args []string) error {
removed := set.String("removed", "", "the files it deletes, comma-separated")
membersJSON := set.String("members", "", "a group's members, as JSON")
group := set.String("group", "", "a delivery group's id")
ready := set.Bool("ready", false, "the head's pull request is marked ready: its check is queued before every other check "+
"(novox/hq ADR 0406 §3)")
by := set.String("by", catalogue.DeliverySeat, "who says it")
why := set.String("why", "", "why")
without := set.String("without", "", "merges a stop leaves out, <repository>@<commit>, comma-separated: the walk's "+
@@ -781,12 +906,6 @@ func deliveryCommand(ctx context.Context, args []string) error {
if err := json.Unmarshal([]byte(*membersJSON), &members); err != nil {
return fmt.Errorf("the members are not readable: %w", err)
}
// Marked ready by the verb's word for every member it names, besides any member that says so itself.
if *ready {
for i := range members {
members[i].Ready = true
}
}
}
open, err := openStores(ctx)
if err != nil {
@@ -828,16 +947,12 @@ func deliveryCommand(ctx context.Context, args []string) error {
if err != nil {
return err
}
changes, err := changesOf(ctx, changeSourcesOf(members, entries, reachOfMembers(members, entries, read, edges)))
if err != nil {
return fmt.Errorf("what the members say of their manifests cannot be read: %w", err)
}
return answer(orderOf(members, reachOfMembers(members, entries, read, edges), edges, changes))
return answer(orderOf(members, reachOfMembers(members, entries, read, edges), edges))
case "check":
if *group == "" && len(members) == 1 {
// One head, checked again as the forge's announcement would have it (a recheck): the controller's
// own path for a pull request, run because the delivery's owner asked.
if err := sayingOnTheBus(ctx, inv, func() error { return (following{open: open, oneShot: true}).PullUpdated(ctx, members[0].pull()) }); err != nil {
if err := sayingOnTheBus(ctx, inv, func() error { return (following{open}).PullUpdated(ctx, members[0].pull()) }); err != nil {
return err
}
return answer(map[string]any{"rechecked": members[0].ID})
@@ -861,7 +976,7 @@ func deliveryCommand(ctx context.Context, args []string) error {
}); err != nil {
return err
}
fmt.Printf("%s (%s at %s) is let go by %s; its first tier is asked when the controller next looks at the plans, which it does as this ends\n", p.ID, p.Repository,
fmt.Printf("%s (%s at %s) is let go by %s; its first tier is asked at the next pass\n", p.ID, p.Repository,
short(p.Commit), *by)
return nil
case "stop":
@@ -884,19 +999,6 @@ func deliveryCommand(ctx context.Context, args []string) error {
fmt.Printf("%s stopped by %s at tier %d of %d; what was asked still builds and registers, nothing further "+
"is asked or sent\n", p.ID, *by, p.Tier, len(p.Tiers))
return nil
case "backups-again":
if len(positionals) != 1 || strings.TrimSpace(*why) == "" {
return errors.New("delivery backups-again <plan> --why <text> [--by <who>]")
}
var p inventory.Plan
var again []inventory.PlanBackup
if err := sayingOnTheBus(ctx, inv, func() (err error) {
p, again, err = backupsAgain(ctx, inv, positionals[0], *by, strings.TrimSpace(*why))
return err
}); err != nil {
return err
}
return answer(map[string]any{"plan": p.ID, "asked_again": again})
case "walks":
var walks []inventory.Plan
if *planID != "" {
@@ -937,21 +1039,19 @@ func deliveryCommand(ctx context.Context, args []string) error {
return answer(map[string]any{"held": deliverySeatHeld(entries), "walks": walks,
"own-path": sortedKeysOf(ownPathWords())})
}
return fmt.Errorf("delivery %s: plan, order, check, go, stop, backups-again or walks", sub)
return fmt.Errorf("delivery %s: plan, order, check, go, stop or walks", sub)
}
// appliedFrom answers a machine's own report of the send a walk made to it from the controller's `apply` durations
// (novox/hq issue 582); a lookup that fails is a report not read, which leaves the walk's end unknown rather than wrong.
// appliedFrom answers a machine's first report after a send from the controller's `apply` durations; a lookup
// that fails is a report not read, which leaves the walk's end unknown rather than wrong.
func appliedFrom(ctx context.Context, inv *inventory.Inventory) inventory.AppliedLookup {
return func(node string, send inventory.SentDeclaration, walkSent time.Time) (inventory.AppliedReport, bool) {
r, ok, err := inv.FirstAppliedAfter(ctx, node, send, walkSent)
return func(node string, sent time.Time) (inventory.AppliedReport, bool) {
r, ok, err := inv.FirstAppliedAfter(ctx, node, sent)
if err != nil {
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, walkSent.Format(time.RFC3339), err)
fmt.Fprintf(os.Stderr, "the report of %s after %s could not be read: %v\n", node, sent.Format(time.RFC3339), err)
return inventory.AppliedReport{}, false
}
// And, for each module the send handed to the provisioner, its client's report on that build (novox/hq ADR
// 0362): applied only once both said it, at the later; failed if the client failed.
return inventory.ProvisionedSend(node, send, r, ok, provLookup(ctx))
return r, ok
}
}
@@ -1027,46 +1127,6 @@ func theGraph(ctx context.Context, inv *inventory.Inventory) ([]inventory.Entry,
return entries, read, edges, nil
}
// cycleVerdict is the failing verdict of a group whose modules are ordered in a cycle, and whether there is one: the
// group's composed check, said without a build. The manifests' change is read from the forge; an answer it cannot give
// is an error, never read as no cycle.
func cycleVerdict(ctx context.Context, group string, members []orderMember, reach map[string]orderReach,
edges []inventory.Edge, entries []inventory.Entry) (link.Checked, bool, error) {
changes, err := changesOf(ctx, changeSourcesOf(members, entries, reach))
if err != nil {
return link.Checked{}, false, fmt.Errorf("what the group's members say of their manifests cannot be read: %w", err)
}
var moved []string
for _, m := range members {
moved = appendNew(moved, reach[m.ID].Moved...)
}
said := orderEdges(moved, changes, edges).CycleSaid()
if said == "" {
return link.Checked{}, false, nil
}
first := members[0].pull()
c := link.Checked{ID: link.NewBuildID(time.Now()), Owner: first.Owner, Repo: first.Repo, Number: first.Number,
Commit: first.Commit, Verdict: "fail", Summary: said, Report: said, Group: group,
Gate: &link.CheckLayer{Verdict: "fail", Summary: said}}
for _, m := range members {
if group == "" {
break // a lone pull request's verdict names no group and no member
}
p := m.pull()
c.Members = append(c.Members, link.CheckedMember{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Commit: p.Commit,
Trunk: twelve(p.BaseSHA)})
}
return c, true, nil
}
// twelve is a commit as twelve characters.
func twelve(c string) string {
if len(c) > 12 {
return c[:12]
}
return c
}
// groupPrimary is the head a group's composed check is run from: the one moving the controller, which then
// judges the group by itself; else the one moving the node-engine, whose validator judges; else the first.
func groupPrimary(members []orderMember, reach map[string]orderReach) int {
@@ -1089,14 +1149,6 @@ func askGroupCheck(ctx context.Context, open *stores, group string, members []or
return "", err
}
reach := reachOfMembers(members, entries, read, edges)
// **A cycle of edges fails the group's composed check, naming every module and edge** (novox/hq ADR 0419 clause 6),
// before the build seat is asked: no composition of the heads has an order to be sent in.
if failing, cyclic, err := cycleVerdict(ctx, group, members, reach, edges, entries); err != nil {
return "", err
} else if cyclic {
sayChecked(ctx, failing)
return failing.ID, nil
}
primary := groupPrimary(members, reach)
p := members[primary].pull()
scope := pullScope(p, entries, read, edges)
@@ -1111,8 +1163,6 @@ func askGroupCheck(ctx context.Context, open *stores, group string, members []or
return "", err
}
request.Check.Group = group
// A group's check is queued as ready when any member's pull request is marked ready.
request.Check.Ready = slices.ContainsFunc(members, func(m orderMember) bool { return m.Ready })
world, err := theRestOfTheMesh(ctx, open.inventory, shelfOf(entries), "")
if err != nil {
return "", err
-271
View File
@@ -1,271 +0,0 @@
package main
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"net/url"
"path"
"slices"
"strings"
"sync"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// What a change says of the manifests it touches, read from the forge (novox/hq ADR 0419 clause 3, the lead's settlement
// of issue 977 question 1): the controller asks the forge for each changed module.json at the commit and at what it
// replaced, once per commit. The composed check's result is not used: the plan is cut after the merge.
// changeSource is one change to a repository whose manifests are read: the commit after it and, when known, what it
// replaced (a pull request's base commit or branch). An empty Old is the first parent of New: the merge's base.
type changeSource struct {
Owner, Repo string
New, Old string
Paths []string
Truncated bool
Removed []string
// Since is the first commit of the repository in the change, when the change is several merges: what it replaced
// is the first parent of that one, so a verb an earlier merge added is added by the change too.
Since string
}
// manifestAt reads a repository's module.json at a ref from the forge; the bool says it exists there. A variable so a
// test needs no forge.
var manifestAt = func(ctx context.Context, owner, repo, file, ref string) (catalogue.Manifest, bool, error) {
var found struct {
Result struct {
Content string `json:"content"`
Encoding string `json:"encoding"`
} `json:"result"`
}
missing, err := askForgeApi(ctx, "/repos/"+owner+"/"+repo+"/contents/"+escapedPath(file)+"?ref="+url.QueryEscape(ref), &found)
if err != nil {
return catalogue.Manifest{}, false, err
}
if missing {
return catalogue.Manifest{}, false, nil
}
body := []byte(found.Result.Content)
if found.Result.Encoding == "base64" {
if body, err = base64.StdEncoding.DecodeString(strings.ReplaceAll(found.Result.Content, "\n", "")); err != nil {
return catalogue.Manifest{}, false, fmt.Errorf("%s/%s %s at %s cannot be read: %w", owner, repo, file, ref, err)
}
}
var m catalogue.Manifest
if err := json.Unmarshal(body, &m); err != nil {
return catalogue.Manifest{}, false, fmt.Errorf("%s/%s %s at %s is not a manifest: %w", owner, repo, file, ref, err)
}
return m, true, nil
}
// parentOf is the first parent of a commit, from the forge. A variable so a test needs no forge.
var parentOf = func(ctx context.Context, owner, repo, commit string) (string, error) {
var found struct {
Result struct {
Parents []struct {
SHA string `json:"sha"`
} `json:"parents"`
} `json:"result"`
}
if _, err := askForgeApi(ctx, "/repos/"+owner+"/"+repo+"/git/commits/"+url.PathEscape(commit), &found); err != nil {
return "", err
}
if len(found.Result.Parents) == 0 {
return "", nil
}
return found.Result.Parents[0].SHA, nil
}
func escapedPath(p string) string {
parts := strings.Split(p, "/")
for i := range parts {
parts[i] = url.PathEscape(parts[i])
}
return strings.Join(parts, "/")
}
// askForgeApi makes one read of the forge's API through its module's tool on the bus. It answers true when the forge
// says there is no such thing (404), and an error for anything else it cannot answer.
func askForgeApi(ctx context.Context, apiPath string, into any) (bool, error) {
js, err := aBus()
if err != nil {
return false, err
}
defer js.Close()
raw, _ := json.Marshal(map[string]any{"path": apiPath})
answer, err := link.Ask(ctx, link.OverNATS{Conn: js.Conn()}, "gitea", "gitea_api", raw, 30*time.Second)
if err != nil {
return false, err
}
if answer.Error != "" {
if strings.Contains(answer.Error, " 404 ") {
return true, nil
}
return false, fmt.Errorf("the forge did not answer %s: %s", apiPath, answer.Error)
}
if err := json.Unmarshal(answer.Result, into); err != nil {
return false, fmt.Errorf("the forge's answer to %s cannot be read: %w", apiPath, err)
}
return false, nil
}
// manifestsRead keeps what the forge said of a manifest at a commit, which never changes: one ask per commit.
var (
manifestsRead = map[string]manifestRead{}
manifestsReadMu sync.Mutex
)
type manifestRead struct {
manifest catalogue.Manifest
exists bool
}
func manifestCached(ctx context.Context, owner, repo, file, ref string) (catalogue.Manifest, bool, error) {
key := strings.Join([]string{owner, repo, file, ref}, "\x00")
if len(ref) == 40 {
manifestsReadMu.Lock()
got, ok := manifestsRead[key]
manifestsReadMu.Unlock()
if ok {
return got.manifest, got.exists, nil
}
}
m, exists, err := manifestAt(ctx, owner, repo, file, ref)
if err == nil && len(ref) == 40 {
manifestsReadMu.Lock()
if len(manifestsRead) > 512 {
clear(manifestsRead)
}
manifestsRead[key] = manifestRead{m, exists}
manifestsReadMu.Unlock()
}
return m, exists, err
}
// verbsServed are the seat verbs a manifest serves, as "<seat>.<verb>": those of the seats it defines, those of the
// seats it claims, and its own tools as "<module>.<tool>".
func verbsServed(m catalogue.Manifest) []string {
var out []string
for _, s := range m.DefinesSeats {
for _, v := range s.Serves {
out = appendNew(out, s.Name+"."+v.Name)
}
}
for _, c := range m.Claims {
for _, v := range c.ServesFor(m) {
out = appendNew(out, c.Name+"."+v)
}
}
for _, t := range m.Tools {
out = appendNew(out, m.Module+"."+t)
}
return out
}
// verbsInvoked are the verbs a manifest calls, as "<seat>.<verb>" or "<module>.<tool>".
func verbsInvoked(m catalogue.Manifest) []string {
var out []string
for _, i := range m.Invokes {
if i = strings.TrimPrefix(i, "seat:"); i != "*" && i != "" {
out = appendNew(out, i)
}
}
return out
}
func added(now, was []string) []string {
var out []string
for _, v := range now {
if !slices.Contains(was, v) {
out = append(out, v)
}
}
return out
}
// changesOf reads, for each manifest the changes touch, what they add to the verbs its module serves and invokes, by
// module. A change whose paths are unsaid or truncated names no manifest: nothing is read for it. An error is the
// forge's, never read as "no change": the plan is not cut on a guess.
func changesOf(ctx context.Context, sources []changeSource) (map[string]moduleChange, error) {
out := map[string]moduleChange{}
for _, s := range sources {
if s.Truncated {
continue
}
for _, p := range s.Paths {
if path.Base(p) != "module.json" || slices.Contains(s.Removed, p) {
continue
}
now, exists, err := manifestCached(ctx, s.Owner, s.Repo, p, s.New)
if err != nil {
return nil, err
}
if !exists || now.Module == "" {
continue
}
old := s.Old
if old == "" {
first := s.Since
if first == "" {
first = s.New
}
if old, err = parentOf(ctx, s.Owner, s.Repo, first); err != nil {
return nil, err
}
}
var was catalogue.Manifest
if old != "" {
if was, _, err = manifestCached(ctx, s.Owner, s.Repo, p, old); err != nil {
return nil, err
}
}
c := out[now.Module]
c.Manifest = true
for _, v := range added(verbsServed(now), verbsServed(was)) {
c.Serves = appendNew(c.Serves, v)
}
for _, v := range added(verbsInvoked(now), verbsInvoked(was)) {
c.Invokes = appendNew(c.Invokes, v)
}
out[now.Module] = c
}
}
return out, nil
}
// truncatedPaths are the manifests a change that names only a beginning of its files is read as changing: those of the
// modules it moves that are built from its repository. Said, never skipped: no edge may be lost to a long list.
func truncatedPaths(entries []inventory.Entry, owner, repo string, moved []string) []string {
var out []string
for _, e := range entries {
if slices.Contains(moved, e.Manifest.Module) && sameRepository(e.Source.Repository, link.SourceMoved{Owner: owner, Repo: repo}) {
out = append(out, path.Join(strings.Trim(e.Source.Path, "/"), moduleManifestFile))
}
}
fmt.Printf(" %s/%s changed more files than were listed: the manifests of its modules (%s) are read as changed\n",
owner, repo, strings.Join(out, ", "))
return out
}
// changeSourcesOf are a group's members as changes to read: each head over what its pull request merges into.
func changeSourcesOf(members []orderMember, entries []inventory.Entry, reach map[string]orderReach) []changeSource {
var out []changeSource
for _, m := range members {
p := m.pull()
if p.PathsTruncated {
p.Paths, p.PathsTruncated = truncatedPaths(entries, p.Owner, p.Repo, reach[m.ID].Moved), false
}
old := p.BaseSHA
if old == "" {
old = p.Base
}
out = append(out, changeSource{Owner: p.Owner, Repo: p.Repo, New: p.Commit, Old: old, Paths: p.Paths,
Truncated: p.PathsTruncated, Removed: p.Removed})
}
return out
}
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
-170
View File
@@ -1,170 +0,0 @@
package main
import (
"fmt"
"slices"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
)
// The order of a plan's modules is read from the module graph and from what the change says of the manifests
// (novox/hq ADR 0419), per module and never per repository: no `after:` line is read, and no member of a group
// is ordered against another. Pure, so a test needs no store.
// The rules that draw an edge of a plan's order, in the words the plan shows.
const (
ruleVerbBeforeCaller = "verb before caller"
ruleVersionSkew = "version skew"
ruleEngineFirst = "engine before controller"
)
// The two modules the rollout rules name.
const (
theController = "mesh-controller"
theEngine = "mesh-host"
)
// moduleChange is what a change says of one module's manifest: the seat verbs ("<seat>.<verb>") it adds to what the
// module serves, the seat verbs it adds to what the module invokes, and whether the manifest changes at all.
type moduleChange struct {
Serves []string
Invokes []string
Manifest bool
}
// planOrder is the edges the rules draw among a plan's modules, and the cycle they make when they make one.
type planOrder struct {
Edges []inventory.Edge
// Cycle is every edge of a cycle among the order's edges and the graph's, the module graph's included.
Cycle []inventory.Edge
}
// CycleSaid names every module and edge of the cycle, for a check to fail with (ADR 0419 clause 6).
func (o planOrder) CycleSaid() string {
if len(o.Cycle) == 0 {
return ""
}
var parts []string
for _, e := range o.Cycle {
parts = append(parts, edgeSaid(e))
}
return "the modules of the plan are ordered in a cycle: " + strings.Join(parts, "; ")
}
// edgeSaid is an edge as a person reads it: the module after, the module before, and why.
func edgeSaid(e inventory.Edge) string {
why := e.Why
if why == "" {
why = e.Kind
}
return fmt.Sprintf("%s after %s (%s)", e.From, e.To, why)
}
// orders is whether an edge of the module graph orders a plan's tiers (tiersOf).
func orders(e inventory.Edge) bool {
return e.Kind != inventory.EdgePackages && e.Kind != inventory.EdgeBuiltBy && e.From != e.To
}
// orderEdges draws the edges of a plan's order among its moved modules (ADR 0419 clause 3, 4): a module whose change
// adds a verb to a seat goes before one whose change adds a call of that verb; the controller, when the plan moves
// it, goes before every module whose manifest the plan changes, the node-engine excepted; and the node-engine goes
// before the controller by default, when the plan moves both and no edge orders them either way, transitively. An
// edge is "From after To". graph is the module graph, whose edges order too and are never drawn again.
func orderEdges(moved []string, changes map[string]moduleChange, graph []inventory.Edge) planOrder {
var out planOrder
sorted := append([]string(nil), moved...)
sort.Strings(sorted)
for _, server := range sorted {
for _, caller := range sorted {
if server == caller {
continue
}
for _, verb := range changes[server].Serves {
if slices.Contains(changes[caller].Invokes, verb) {
out.Edges = append(out.Edges, inventory.Edge{From: caller, To: server, Kind: edgeChangeOrder,
Why: ruleVerbBeforeCaller + ": " + verb})
}
}
}
}
if slices.Contains(sorted, theController) {
for _, m := range sorted {
if m != theController && m != theEngine && changes[m].Manifest {
out.Edges = append(out.Edges, inventory.Edge{From: m, To: theController, Kind: edgeChangeOrder,
Why: ruleVersionSkew})
}
}
}
if slices.Contains(sorted, theController) && slices.Contains(sorted, theEngine) {
all := orderingEdges(graph, out.Edges)
switch {
case pathBetween(all, theController, theEngine) != nil:
// An edge already puts the controller after the engine: the default says the same.
case pathBetween(all, theEngine, theController) != nil:
// An edge puts the engine after the controller: it overrules the default, and says so.
path := pathBetween(all, theEngine, theController)
for i := range out.Edges {
if out.Edges[i] == path[0] {
out.Edges[i].Why += " (overrules " + ruleEngineFirst + ")"
break
}
}
default:
out.Edges = append(out.Edges, inventory.Edge{From: theController, To: theEngine, Kind: edgeChangeOrder,
Why: ruleEngineFirst})
}
}
all := orderingEdges(graph, out.Edges)
for _, e := range out.Edges {
if back := pathBetween(all, e.To, e.From); back != nil {
out.Cycle = append(out.Cycle, e)
out.Cycle = append(out.Cycle, back...)
break
}
}
return out
}
// orderingEdges is the graph's edges that order and the edges drawn, together.
func orderingEdges(graph, drawn []inventory.Edge) []inventory.Edge {
var out []inventory.Edge
for _, e := range graph {
if orders(e) {
out = append(out, e)
}
}
return append(out, drawn...)
}
// pathBetween is the edges that lead from a module to another along "depends on" (From to To), the shortest, or nil
// when the first does not depend on the second.
func pathBetween(edges []inventory.Edge, from, to string) []inventory.Edge {
if from == to {
return nil
}
via := map[string]inventory.Edge{}
seen := map[string]bool{from: true}
queue := []string{from}
for len(queue) > 0 {
at := queue[0]
queue = queue[1:]
for _, e := range edges {
if e.From != at || seen[e.To] {
continue
}
seen[e.To] = true
via[e.To] = e
if e.To == to {
var path []inventory.Edge
for m := to; m != from; m = via[m].From {
path = append([]inventory.Edge{via[m]}, path...)
}
return path
}
queue = append(queue, e.To)
}
}
return nil
}
-212
View File
@@ -1,212 +0,0 @@
package main
import (
"context"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// The order of a plan's modules is read from the module graph and the manifests' change alone (novox/hq ADR 0419):
// no `after:` line, no member of a group. Pure, like orderOf was.
func hasEdge(edges []inventory.Edge, from, to string) (inventory.Edge, bool) {
for _, e := range edges {
if e.From == from && e.To == to {
return e, true
}
}
return inventory.Edge{}, false
}
// A module whose change adds a verb to a seat goes before the module whose change adds a call of it: the caller
// depends on the server, the plan puts them in two tiers, and the edge says why.
func TestAVerbIsServedBeforeItsCallerIsSent(t *testing.T) {
changes := map[string]moduleChange{
"server": {Serves: []string{"mesh-delivery.groups"}, Manifest: true},
"caller": {Invokes: []string{"mesh-delivery.groups"}, Manifest: true},
}
got := orderEdges([]string{"caller", "server"}, changes, nil)
e, ok := hasEdge(got.Edges, "caller", "server")
if !ok || !strings.Contains(e.Why, "verb before caller") || !strings.Contains(e.Why, "mesh-delivery.groups") {
t.Fatalf("the caller does not follow the server by the verb: %+v", got.Edges)
}
if len(got.Cycle) != 0 {
t.Fatalf("a cycle: %+v", got.Cycle)
}
if tiers := tiersOf([]string{"caller", "server"}, got.Edges); len(tiers) != 2 || tiers[0][0] != "server" {
t.Fatalf("tiers %v, want the server first", tiers)
}
}
// Edges of one rule both ways are a cycle: every module and edge is named.
func TestEdgesBothWaysAreACycleNamingEveryModuleAndEdge(t *testing.T) {
changes := map[string]moduleChange{
"x": {Serves: []string{"s.a"}, Invokes: []string{"s.b"}},
"y": {Serves: []string{"s.b"}, Invokes: []string{"s.a"}},
}
got := orderEdges([]string{"x", "y"}, changes, nil)
if len(got.Cycle) < 2 {
t.Fatalf("no cycle found: %+v", got)
}
said := got.CycleSaid()
for _, want := range []string{"x", "y", "s.a", "s.b"} {
if !strings.Contains(said, want) {
t.Fatalf("the cycle is said as %q, without %s", said, want)
}
}
}
// The node-engine goes before the controller by default, and says so; an edge of the manifests the other way
// overrules it, and the edge that did is said.
func TestTheEngineGoesBeforeTheControllerUnlessAnEdgeSaysOtherwise(t *testing.T) {
got := orderEdges([]string{"mesh-controller", "mesh-host"}, nil, nil)
e, ok := hasEdge(got.Edges, "mesh-controller", "mesh-host")
if !ok || e.Why != "engine before controller" {
t.Fatalf("the engine does not go first: %+v", got.Edges)
}
changes := map[string]moduleChange{
"mesh-host": {Serves: []string{"s.v"}},
"mesh-controller": {Invokes: []string{"s.v"}},
}
got = orderEdges([]string{"mesh-controller", "mesh-host"}, changes, nil)
if _, ok := hasEdge(got.Edges, "mesh-controller", "mesh-host"); !ok || len(got.Edges) != 1 ||
!strings.Contains(got.Edges[0].Why, "verb before caller") {
t.Fatalf("the verb edge does not place the controller after the engine: %+v", got.Edges)
}
// The other way: the engine calls what the controller brings.
changes = map[string]moduleChange{
"mesh-controller": {Serves: []string{"s.v"}},
"mesh-host": {Invokes: []string{"s.v"}},
}
got = orderEdges([]string{"mesh-controller", "mesh-host"}, changes, nil)
e, ok = hasEdge(got.Edges, "mesh-host", "mesh-controller")
if !ok || len(got.Edges) != 1 || !strings.Contains(e.Why, "overrules engine before controller") {
t.Fatalf("the verb edge does not say it overruled the default: %+v", got.Edges)
}
}
// The controller goes before every module whose manifest the plan changes (version skew), the node-engine excepted.
func TestTheControllerGoesBeforeEveryChangedManifest(t *testing.T) {
changes := map[string]moduleChange{"app": {Manifest: true}, "other": {}}
got := orderEdges([]string{"mesh-controller", "app", "other"}, changes, nil)
if e, ok := hasEdge(got.Edges, "app", "mesh-controller"); !ok || e.Why != "version skew" {
t.Fatalf("the changed manifest does not follow the controller: %+v", got.Edges)
}
if _, ok := hasEdge(got.Edges, "other", "mesh-controller"); ok {
t.Fatalf("a module with an unchanged manifest was ordered: %+v", got.Edges)
}
if got := orderEdges([]string{"app", "other"}, changes, nil); len(got.Edges) != 0 {
t.Fatalf("version skew ordered a plan that does not move the controller: %+v", got.Edges)
}
}
// The order is per module, not per repository: of two modules of one repository only the one with the edge moves.
func TestTheOrderIsPerModuleNotPerRepository(t *testing.T) {
changes := map[string]moduleChange{
"a": {Serves: []string{"s.v"}},
"c": {Invokes: []string{"s.v"}},
}
got := orderEdges([]string{"a", "b", "c"}, changes, nil)
tiers := tiersOf([]string{"a", "b", "c"}, got.Edges)
if len(tiers) != 2 || strings.Join(tiers[0], ",") != "a,b" || strings.Join(tiers[1], ",") != "c" {
t.Fatalf("tiers %v, want [a b] then [c]", tiers)
}
}
// The edges a plan's order was drawn with are kept on the plan, and its gates read them: a later tier waits for the
// earlier to be applied although the catalogue's graph holds no such edge (ADR 0419, the lead's settlement Q4).
func TestAPlansOwnOrderGatesItsTiersWhateverTheStoreGraphHolds(t *testing.T) {
p := inventory.Plan{Tiers: [][]string{{"server"}, {"caller"}}, Tier: 0,
Delivery: &inventory.PlanDelivery{Order: []inventory.Edge{{From: "caller", To: "server", Kind: edgeChangeOrder,
Why: ruleVerbBeforeCaller + ": s.v"}}}}
if got := gates(p, nil, func(string) bool { return true }); len(got) != 1 || got[0] != "server" {
t.Fatalf("the caller's tier waits for %v; want the server", got)
}
p.Delivery.Order = nil
if got := gates(p, nil, func(string) bool { return true }); len(got) != 0 {
t.Fatalf("a plan with no order waits for %v", got)
}
}
// A verb an earlier merge of the same repository in the batch added is added by the batch: the change is compared with
// what the repository held before the batch, not with the first parent of its latest merge.
func TestAVerbAnEarlierMergeOfTheBatchAddedDrawsItsEdge(t *testing.T) {
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
const first, second, before = "1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222", "0000000000000000000000000000000000000000"
parentOf = func(_ context.Context, _, _, commit string) (string, error) {
if commit == first {
return before, nil
}
return first, nil // the latest merge's parent holds the verb already
}
manifestAt = func(_ context.Context, _, _, _, ref string) (catalogue.Manifest, bool, error) {
m := catalogue.Manifest{Module: "server"}
if ref != before {
m.DefinesSeats = []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "v"}}}}
}
return m, true, nil
}
src := changeSource{Owner: "novox", Repo: "r", New: second, Paths: []string{"module.json"}}
if got, err := changesOf(context.Background(), []changeSource{src}); err != nil || len(got["server"].Serves) != 0 {
t.Fatalf("against the first parent of the latest merge: %+v %v", got, err)
}
src.Since = first
got, err := changesOf(context.Background(), []changeSource{src})
if err != nil || len(got["server"].Serves) != 1 || got["server"].Serves[0] != "s.v" {
t.Fatalf("against what the batch found: %+v %v", got, err)
}
}
// A change that names only a beginning of its files is read as changing the manifests of the modules it moves.
func TestATruncatedChangeIsReadAsChangingTheManifestsOfWhatItMoves(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "a"}, Source: inventory.Source{Repository: "novox/mesh-catalog", Path: "modules/a"}},
{Manifest: catalogue.Manifest{Module: "b"}, Source: inventory.Source{Repository: "novox/mesh-catalog", Path: "modules/b"}},
{Manifest: catalogue.Manifest{Module: "c"}, Source: inventory.Source{Repository: "novox/other", Path: "modules/c"}},
}
got := truncatedPaths(entries, "novox", "mesh-catalog", []string{"a", "c"})
if len(got) != 1 || got[0] != "modules/a/module.json" {
t.Fatalf("read as changed: %v", got)
}
members := []orderMember{{ID: "m", Repository: "novox/mesh-catalog", Head: "h", PathsTruncated: true, Paths: []string{"x"}}}
src := changeSourcesOf(members, entries, map[string]orderReach{"m": {Moved: []string{"a"}}})
if len(src) != 1 || src[0].Truncated || len(src[0].Paths) != 1 || src[0].Paths[0] != "modules/a/module.json" {
t.Fatalf("the member's change is read as %+v", src)
}
}
// A lone pull request's cycle fails its check, naming every module and edge, with no group and no members.
func TestALonePullRequestInACycleFailsWithNoGroup(t *testing.T) {
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
const head = "3333333333333333333333333333333333333333"
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
manifestAt = func(_ context.Context, _, _, file, ref string) (catalogue.Manifest, bool, error) {
if ref == "main" {
return catalogue.Manifest{}, false, nil
}
if file == "modules/x/module.json" {
return catalogue.Manifest{Module: "mx", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "a"}}}},
Invokes: []string{"seat:s.b"}}, true, nil
}
return catalogue.Manifest{Module: "my", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "b"}}}},
Invokes: []string{"seat:s.a"}}, true, nil
}
m := orderMember{ID: "novox/r@" + head[:12], Repository: "novox/r", Base: "main", Head: head, Number: 5,
Paths: []string{"modules/x/module.json", "modules/y/module.json"}}
c, cyclic, err := cycleVerdict(context.Background(), "", []orderMember{m}, map[string]orderReach{m.ID: {Moved: []string{"mx", "my"}}}, nil, nil)
if err != nil || !cyclic || c.Verdict != "fail" || c.Group != "" || len(c.Members) != 0 || !strings.Contains(c.Summary, "s.a") ||
!strings.Contains(c.Summary, "my") {
t.Fatalf("the verdict is %+v (cyclic %v, %v)", c, cyclic, err)
}
}
@@ -1,196 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := asksRecorded(t)
withConditionsInMemory(t)
for _, name := range []string{"app", "mesh-delivery"} {
m := catalogue.Manifest{Module: name, Version: "1"}
if name == "mesh-delivery" {
m.Claims = []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}
}
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/" + name, Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
merge := func(commit string, paths ...string) inventory.Plan {
t.Helper()
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit, Paths: paths,
ModuleDirs: []string{"modules/app", "modules/mesh-delivery"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
recent, err := inv.RecentPlans(ctx, 1)
if err != nil || len(recent) != 1 || recent[0].Commit != commit {
t.Fatalf("no plan for %s: %v %v", commit, recent, err)
}
return recent[0]
}
// finish ends a walk as done, so the next merge has nothing of it to take over.
finish := func(id string) {
t.Helper()
w, err := inv.PlanByID(ctx, id)
if err != nil {
t.Fatal(err)
}
w.State = inventory.PlanDone
if err := inv.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
}
// No holder on record: the merge starts its walk, as before.
p := merge("c1aaaaaaaa", "modules/app/index.ts")
if p.Waiting() || len(*asked) != 1 {
t.Fatalf("with no holder on record the walk waited (%v) or asked %v", p.Waiting(), *asked)
}
// The holder on record: the next merge waits, asking nothing, and an advance asks nothing either. One walk
// is open at a time (novox/hq ADR 0276): the first ends before the next merge's batch is cut.
finish(p.ID)
if _, err := inv.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
p = merge("c2bbbbbbbb", "modules/app/index.ts")
if !p.Waiting() || len(*asked) != 1 || !strings.Contains(p.Note, "plans go "+p.ID) {
t.Fatalf("the walk did not wait for its word: waiting %v, asked %v, note %q", p.Waiting(), *asked, p.Note)
}
advanceHeld(ctx, open)
if len(*asked) != 1 {
t.Fatalf("a waiting walk was advanced into asking: %v", *asked)
}
if line := planLine(p, p.Created); strings.Contains(line, "LATE") || !strings.Contains(line, "waits for") {
t.Errorf("a waiting walk reads %q", line)
}
// The delivery's word starts it.
if err := deliveryCommand(ctx, []string{"go", p.ID, "--by", catalogue.DeliverySeat, "--why", "its turn"}); err != nil {
t.Fatal(err)
}
if err := deliveryCommand(ctx, []string{"go", p.ID}); err == nil {
t.Fatal("a walk was let go twice")
}
advanceHeld(ctx, open)
if len(*asked) != 2 {
t.Fatalf("the word did not start the walk: %v", *asked)
}
got, err := inv.PlanByID(ctx, p.ID)
if err != nil || got.Delivery == nil || got.Delivery.By != catalogue.DeliverySeat || got.Delivery.Why != "its turn" {
t.Fatalf("the word was not kept: %+v %v", got.Delivery, err)
}
// The delivery's owner's own merge never waits for it. A walk that started is never taken over (novox/hq
// ADR 0276): the merge's batch is cut once it ended.
finish(p.ID)
p = merge("c3cccccccc", "modules/mesh-delivery/main.go")
if p.Waiting() || len(*asked) != 3 {
t.Fatalf("mesh-delivery's own walk waited for mesh-delivery: %v %v", p.Waiting(), *asked)
}
// The owner down: a person starts a waiting walk, with why.
finish(p.ID)
p = merge("c4dddddddd", "modules/app/index.ts")
if !p.Waiting() {
t.Fatal("the walk did not wait")
}
if err := plansCommand(ctx, []string{"go", p.ID}); err == nil || !strings.Contains(err.Error(), "--why") {
t.Fatalf("a walk was started by hand without why: %v", err)
}
if err := plansCommand(ctx, []string{"go", p.ID, "--why", "mesh-delivery is down"}); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
got, _ = inv.PlanByID(ctx, p.ID)
if got.Waiting() || !strings.HasPrefix(got.Delivery.By, "a person") || len(*asked) < 4 {
t.Fatalf("a person's word did not start the walk: %+v, asked %v", got.Delivery, *asked)
}
// The owner's stop: failed, said as stopped by it.
finish(p.ID)
p = merge("c5eeeeeeee", "modules/app/index.ts")
if err := deliveryCommand(ctx, []string{"stop", p.ID, "--why", "the operator stopped it", "--by",
"mesh-delivery for jochen"}); err != nil {
t.Fatal(err)
}
got, _ = inv.PlanByID(ctx, p.ID)
if got.State != inventory.PlanFailed || got.Delivery.Stopped != "mesh-delivery for jochen" ||
!strings.Contains(got.Note, "the operator stopped it") {
t.Fatalf("the stop was kept as %s %+v %q", got.State, got.Delivery, got.Note)
}
// Unassigned: the mesh is back on the controller's own path.
if err := inv.Unassign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
asks := len(*asked)
if p = merge("c6ffffffff", "modules/app/index.ts"); p.Waiting() || len(*asked) != asks+1 {
t.Fatalf("with the holder gone the walk waited: %v", p.Waiting())
}
}
func TestAWalkLetGoByAVerbIsSaidAsPlanMoved(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
url := testbus.URL(t)
js, err := broker.Dial(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeyPlanMoved), heard)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
waiting := inventory.Plan{ID: "plan-waits", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c7c7c7c7",
Created: time.Now().UTC(), State: inventory.PlanBuilding, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if err := deliveryCommand(ctx, []string{"go", waiting.ID, "--why", "its turn"}); err != nil {
t.Fatal(err)
}
select {
case m := <-heard:
var said inventory.Plan
if err := json.Unmarshal(m.Data, &said); err != nil || said.ID != waiting.ID || said.Delivery == nil ||
said.Delivery.Go == nil {
t.Fatalf("plan-moved said %s (%v)", m.Data, err)
}
case <-time.After(5 * time.Second):
t.Fatal("a walk let go by a verb was not said")
}
if checkEvents != nil || inventory.PlanSaved != nil {
t.Fatal("the command's own bus was left in place")
}
}
+274 -40
View File
@@ -1,44 +1,50 @@
package main
import (
"encoding/json"
"reflect"
"slices"
"strings"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// novox/hq ADR 0419: a group's order, from the module graph and the manifests' change alone, the same on every
// reading — what a module stands on before it, the controller before a manifest that needs it, the node-engine
// before the controller, a verb's server before its caller — and no `after:` line read.
func TestAGroupIsOrderedByTheGraphAndTheManifestsChange(t *testing.T) {
// novox/hq ADR 0239 decision 4: a group's order, from the graph and the pull requests, the same on every
// reading — the build agent before what it builds, the controller before a manifest that needs it, the
// node-engine before the controller, a declared `after:` — and a contradiction named, never ordered.
func TestAGroupIsOrderedByTheGraphAndWhatItsPullRequestsSay(t *testing.T) {
members := []orderMember{
{ID: "cat", Repository: "novox/mesh-catalog"},
{ID: "ctl", Repository: "novox/mesh-controller"},
{ID: "host", Repository: "novox/mesh-host"},
{ID: "base", Repository: "novox/base"},
{ID: "app", Repository: "novox/app"},
{ID: "agent", Repository: "novox/build-agent"},
{ID: "app", Repository: "novox/app", After: []string{"lab"}},
{ID: "lab", Repository: "novox/lab"},
}
reach := map[string]orderReach{
"cat": {Moved: []string{"gitea"}, Manifests: []string{"modules/gitea/module.json"}},
"ctl": {Moved: []string{"mesh-controller"}, Manifests: []string{"module.json"}},
"host": {Moved: []string{"mesh-host"}, Manifests: []string{"module.json"}},
"base": {Moved: []string{"base"}},
"app": {Moved: []string{"app"}},
"lab": {Moved: []string{"lab"}},
"cat": {Moved: []string{"gitea"}, Manifests: []string{"modules/gitea/module.json"}},
"ctl": {Moved: []string{"mesh-controller"}, Manifests: []string{"module.json"}},
"host": {Moved: []string{"mesh-host"}, Manifests: []string{"module.json"}},
"agent": {Moved: []string{"build-agent"}},
"app": {Moved: []string{"app"}},
"lab": {Moved: []string{"lab"}},
}
edges := []inventory.Edge{{From: "app", To: "base", Kind: inventory.EdgeStandsOn},
edges := []inventory.Edge{{From: "app", To: "build-agent", Kind: inventory.EdgeBuiltBy},
{From: "gitea", To: "postgres", Kind: inventory.EdgeDeclared}}
changes := map[string]moduleChange{"gitea": {Manifest: true}, "mesh-controller": {Manifest: true},
"mesh-host": {Manifest: true}, "lab": {Serves: []string{"seat.v"}}, "app": {Invokes: []string{"seat.v"}}}
got := orderOf(members, reach, edges, changes)
got := orderOf(members, reach, edges)
if len(got.Cycle) > 0 {
t.Fatalf("a cycle where there is none: %v", got.Cycle)
}
// By repository among those with nothing before them: app waits for base and lab; gitea's catalogue for the controller.
want := []string{"base", "lab", "app", "host", "ctl", "cat"}
// By repository among those with nothing before them: the build agent, the lab, then what waited on both.
want := []string{"agent", "lab", "app", "host", "ctl", "cat"}
if !reflect.DeepEqual(got.Order, want) {
t.Fatalf("ordered %v, wanted %v; pairs %+v", got.Order, want, got.Pairs)
}
@@ -46,44 +52,95 @@ func TestAGroupIsOrderedByTheGraphAndTheManifestsChange(t *testing.T) {
for _, p := range got.Pairs {
why[p.Before+">"+p.After] = p.Why
}
for pair, reason := range map[string]string{"host>ctl": ruleEngineFirst, "ctl>cat": ruleVersionSkew,
"ctl>host": "", "base>app": orderBuiltBy, "lab>app": ruleVerbBeforeCaller + ": seat.v"} {
for pair, reason := range map[string]string{"host>ctl": orderEngineFirst, "ctl>cat": orderVersionSkew,
"ctl>host": "", "agent>app": orderBuiltBy, "lab>app": orderDeclared} {
if why[pair] != reason {
t.Errorf("%s is ordered %q, wanted %q", pair, why[pair], reason)
}
}
// The same members in another order read the same.
shuffled := []orderMember{members[5], members[3], members[0], members[4], members[2], members[1]}
if again := orderOf(shuffled, reach, edges, changes); !reflect.DeepEqual(again.Order, want) {
if again := orderOf(shuffled, reach, edges); !reflect.DeepEqual(again.Order, want) {
t.Fatalf("another reading ordered %v", again.Order)
}
// A declared order against an inferred one wins (novox/hq ADR 0249), and the pair says what it won over.
members[1].After = []string{"mesh-catalog"}
got = orderOf(members, reach, edges)
if len(got.Cycle) > 0 || len(got.Contradictions) > 0 {
t.Fatalf("a declared order did not win: cycle %v, %+v", got.Cycle, got.Contradictions)
}
if want := []string{"agent", "lab", "app", "cat", "host", "ctl"}; !reflect.DeepEqual(got.Order, want) {
t.Fatalf("ordered %v, wanted %v", got.Order, want)
}
if !slices.ContainsFunc(got.Pairs, func(p orderPair) bool {
return p.Before == "cat" && p.After == "ctl" && p.Why == orderDeclared && reflect.DeepEqual(p.Over, []string{orderVersionSkew})
}) || slices.ContainsFunc(got.Pairs, func(p orderPair) bool { return p.Before == "ctl" && p.After == "cat" }) {
t.Fatalf("the declared pair does not say it won over version skew: %+v", got.Pairs)
}
// Declared both ways is a contradiction no precedence resolves: a cycle, and how to end it.
members[0].After = []string{"novox/mesh-controller"}
got = orderOf(members, reach, edges)
if !reflect.DeepEqual(got.Cycle, []string{"cat", "ctl"}) || len(got.Contradictions) != 1 {
t.Fatalf("the cycle is %v, ordered %v, contradictions %+v", got.Cycle, got.Order, got.Contradictions)
}
if c := got.Contradictions[0]; !reflect.DeepEqual(c.Members, []string{"cat", "ctl"}) ||
!strings.Contains(c.Said, "remove one of the `after:` lines") {
t.Fatalf("the contradiction is said %+v", c)
}
}
// A cycle is the one refusal, and it names every module and edge. A description's `after:` line changes nothing.
func TestAGroupInACycleIsNamedAndNothingElseIsRefused(t *testing.T) {
// novox/hq ADR 0249: when two rules order one pair both ways, the rule of higher precedence wins and the
// pair says what it overruled; rules of one rank both ways are refused, naming both and how to declare.
func TestAnOrderRuleYieldsToAStrongerOne(t *testing.T) {
members := []orderMember{
{ID: "x", Repository: "novox/x", Number: 7},
{ID: "y", Repository: "novox/y", Number: 9},
{ID: "ctl", Repository: "novox/mesh-controller", Number: 7},
{ID: "host", Repository: "novox/mesh-host", Number: 9},
}
reach := map[string]orderReach{"x": {Moved: []string{"mx"}}, "y": {Moved: []string{"my"}}}
changes := map[string]moduleChange{
"mx": {Serves: []string{"s.a"}, Invokes: []string{"s.b"}},
"my": {Serves: []string{"s.b"}, Invokes: []string{"s.a"}},
reach := map[string]orderReach{
"ctl": {Moved: []string{"mesh-controller", "build-agent"}},
"host": {Moved: []string{"mesh-host"}},
}
got := orderOf(members, reach, nil, changes)
if !reflect.DeepEqual(got.Cycle, []string{"x", "y"}) || len(got.Order) > 0 {
t.Fatalf("a cycle was ordered: %v, cycle %v", got.Order, got.Cycle)
builtBy := []inventory.Edge{{From: "mesh-host", To: "build-agent", Kind: inventory.EdgeBuiltBy}}
// Built by outranks engine before controller.
got := orderOf(members, reach, builtBy)
if !reflect.DeepEqual(got.Order, []string{"ctl", "host"}) || len(got.Cycle) > 0 {
t.Fatalf("ordered %v, cycle %v", got.Order, got.Cycle)
}
for _, w := range []string{"mx", "my", "s.a", "s.b"} {
if !strings.Contains(got.CycleSaid, w) {
t.Errorf("the cycle does not say %q: %s", w, got.CycleSaid)
if want := []orderPair{{Before: "ctl", After: "host", Why: orderBuiltBy, Over: []string{orderEngineFirst}}}; !reflect.DeepEqual(got.Pairs, want) {
t.Fatalf("pairs %+v, wanted %+v", got.Pairs, want)
}
// A declared line outranks both.
members[0].After = []string{"mesh-host"}
got = orderOf(members, reach, builtBy)
if !reflect.DeepEqual(got.Order, []string{"host", "ctl"}) || len(got.Cycle) > 0 ||
!reflect.DeepEqual(got.Pairs[0].Over, []string{orderBuiltBy}) || got.Pairs[0].Why != orderDeclared {
t.Fatalf("a declared order did not win: %v %+v", got.Order, got.Pairs)
}
// Without the build agent, engine before controller stands alone.
members[0].After = nil
got = orderOf(members, map[string]orderReach{"ctl": {Moved: []string{"mesh-controller"}}, "host": reach["host"]}, builtBy)
if !reflect.DeepEqual(got.Order, []string{"host", "ctl"}) || got.Pairs[0].Why != orderEngineFirst || len(got.Pairs[0].Over) > 0 {
t.Fatalf("engine before controller alone: %v %+v", got.Order, got.Pairs)
}
// Built by both ways is one rank against itself: refused, both rules named, and how to declare.
both := append(builtBy, inventory.Edge{From: "build-agent", To: "mesh-host", Kind: inventory.EdgeStandsOn})
got = orderOf(members, reach, both)
if !reflect.DeepEqual(got.Cycle, []string{"ctl", "host"}) || len(got.Order) > 0 || len(got.Contradictions) != 1 {
t.Fatalf("a contradiction was ordered: %v, cycle %v, %+v", got.Order, got.Cycle, got.Contradictions)
}
said := got.Contradictions[0].Said
for _, w := range []string{"built by puts ctl first", "built by puts host first",
"`after: novox/mesh-controller` in novox/mesh-host#9's description", "`after: novox/mesh-host` in novox/mesh-controller#7's"} {
if !strings.Contains(said, w) {
t.Errorf("the contradiction does not say %q: %s", w, said)
}
}
// Without the edges the same members are ordered by repository.
got = orderOf(members, reach, nil, nil)
if !reflect.DeepEqual(got.Order, []string{"x", "y"}) || len(got.Cycle) > 0 || len(got.Pairs) > 0 {
t.Fatalf("members with no edge: %v %v %+v", got.Order, got.Cycle, got.Pairs)
}
}
// novox/hq ADR 0239 decision 8: a walk waits for the delivery's owner only while the seat has a holder on
@@ -121,6 +178,137 @@ func TestAWalkWaitsOnlyWhileTheDeliverySeatIsHeldAndNeverForTheCore(t *testing.T
// novox/hq ADR 0239: with mesh-delivery on record, a merge opens its walk and asks nothing until the
// delivery's word; the word starts it; the core's own merge never waits; a person starts a waiting walk by
// hand when the owner is down, and the owner's stop ends one as stopped.
func TestAMergeWaitsForItsDeliverysWordAndThePersonsWordWorksWithoutIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := asksRecorded(t)
withConditionsInMemory(t)
for _, name := range []string{"app", "mesh-delivery"} {
m := catalogue.Manifest{Module: name, Version: "1"}
if name == "mesh-delivery" {
m.Claims = []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}
}
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/" + name, Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
merge := func(commit string, paths ...string) inventory.Plan {
t.Helper()
m := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: commit, Paths: paths,
ModuleDirs: []string{"modules/app", "modules/mesh-delivery"}, ModuleDirsSaid: true}
if err := (following{open: open}).SourceMoved(ctx, m); err != nil {
t.Fatal(err)
}
recent, err := inv.RecentPlans(ctx, 1)
if err != nil || len(recent) != 1 || recent[0].Commit != commit {
t.Fatalf("no plan for %s: %v %v", commit, recent, err)
}
return recent[0]
}
// finish ends a walk as done, so the next merge has nothing of it to take over.
finish := func(id string) {
t.Helper()
w, err := inv.PlanByID(ctx, id)
if err != nil {
t.Fatal(err)
}
w.State = inventory.PlanDone
if err := inv.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
}
// No holder on record: the merge starts its walk, as before.
p := merge("c1aaaaaaaa", "modules/app/index.ts")
if p.Waiting() || len(*asked) != 1 {
t.Fatalf("with no holder on record the walk waited (%v) or asked %v", p.Waiting(), *asked)
}
// The holder on record: the next merge waits, asking nothing, and an advance asks nothing either. One walk
// is open at a time (novox/hq ADR 0276): the first ends before the next merge's batch is cut.
finish(p.ID)
if _, err := inv.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
p = merge("c2bbbbbbbb", "modules/app/index.ts")
if !p.Waiting() || len(*asked) != 1 || !strings.Contains(p.Note, "plans go "+p.ID) {
t.Fatalf("the walk did not wait for its word: waiting %v, asked %v, note %q", p.Waiting(), *asked, p.Note)
}
advanceHeld(ctx, open)
if len(*asked) != 1 {
t.Fatalf("a waiting walk was advanced into asking: %v", *asked)
}
if line := planLine(p, p.Created); strings.Contains(line, "LATE") || !strings.Contains(line, "waits for") {
t.Errorf("a waiting walk reads %q", line)
}
// The delivery's word starts it.
if err := deliveryCommand(ctx, []string{"go", p.ID, "--by", catalogue.DeliverySeat, "--why", "its turn"}); err != nil {
t.Fatal(err)
}
if err := deliveryCommand(ctx, []string{"go", p.ID}); err == nil {
t.Fatal("a walk was let go twice")
}
advanceHeld(ctx, open)
if len(*asked) != 2 {
t.Fatalf("the word did not start the walk: %v", *asked)
}
got, err := inv.PlanByID(ctx, p.ID)
if err != nil || got.Delivery == nil || got.Delivery.By != catalogue.DeliverySeat || got.Delivery.Why != "its turn" {
t.Fatalf("the word was not kept: %+v %v", got.Delivery, err)
}
// The delivery's owner's own merge never waits for it. A walk that started is never taken over (novox/hq
// ADR 0276): the merge's batch is cut once it ended.
finish(p.ID)
p = merge("c3cccccccc", "modules/mesh-delivery/main.go")
if p.Waiting() || len(*asked) != 3 {
t.Fatalf("mesh-delivery's own walk waited for mesh-delivery: %v %v", p.Waiting(), *asked)
}
// The owner down: a person starts a waiting walk, with why.
finish(p.ID)
p = merge("c4dddddddd", "modules/app/index.ts")
if !p.Waiting() {
t.Fatal("the walk did not wait")
}
if err := plansCommand(ctx, []string{"go", p.ID}); err == nil || !strings.Contains(err.Error(), "--why") {
t.Fatalf("a walk was started by hand without why: %v", err)
}
if err := plansCommand(ctx, []string{"go", p.ID, "--why", "mesh-delivery is down"}); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
got, _ = inv.PlanByID(ctx, p.ID)
if got.Waiting() || !strings.HasPrefix(got.Delivery.By, "a person") || len(*asked) < 4 {
t.Fatalf("a person's word did not start the walk: %+v, asked %v", got.Delivery, *asked)
}
// The owner's stop: failed, said as stopped by it.
finish(p.ID)
p = merge("c5eeeeeeee", "modules/app/index.ts")
if err := deliveryCommand(ctx, []string{"stop", p.ID, "--why", "the operator stopped it", "--by",
"mesh-delivery for jochen"}); err != nil {
t.Fatal(err)
}
got, _ = inv.PlanByID(ctx, p.ID)
if got.State != inventory.PlanFailed || got.Delivery.Stopped != "mesh-delivery for jochen" ||
!strings.Contains(got.Note, "the operator stopped it") {
t.Fatalf("the stop was kept as %s %+v %q", got.State, got.Delivery, got.Note)
}
// Unassigned: the mesh is back on the controller's own path.
if err := inv.Unassign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
asks := len(*asked)
if p = merge("c6ffffffff", "modules/app/index.ts"); p.Waiting() || len(*asked) != asks+1 {
t.Fatalf("with the holder gone the walk waited: %v", p.Waiting())
}
}
// The verbs mesh-delivery asks with become the commands they name, and nothing a caller sends is passed over.
func TestTheDeliveryVerbsComposeTheirCommands(t *testing.T) {
@@ -157,3 +345,49 @@ func TestTheDeliveryVerbsComposeTheirCommands(t *testing.T) {
// A verb runs as a command of its own: what it keeps of a walk is still said as plan-moved, on a bus of the
// command's own, so the delivery's owner hears it at once and not only when it reads the walks back.
func TestAWalkLetGoByAVerbIsSaidAsPlanMoved(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
url := testbus.URL(t)
js, err := broker.Dial(url)
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeyPlanMoved), heard)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
waiting := inventory.Plan{ID: "plan-waits", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c7c7c7c7",
Created: time.Now().UTC(), State: inventory.PlanBuilding, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if err := deliveryCommand(ctx, []string{"go", waiting.ID, "--why", "its turn"}); err != nil {
t.Fatal(err)
}
select {
case m := <-heard:
var said inventory.Plan
if err := json.Unmarshal(m.Data, &said); err != nil || said.ID != waiting.ID || said.Delivery == nil ||
said.Delivery.Go == nil {
t.Fatalf("plan-moved said %s (%v)", m.Data, err)
}
case <-time.After(5 * time.Second):
t.Fatal("a walk let go by a verb was not said")
}
if checkEvents != nil || inventory.PlanSaved != nil {
t.Fatal("the command's own bus was left in place")
}
}
+11 -123
View File
@@ -5,7 +5,6 @@ import (
"encoding/json"
"errors"
"fmt"
"log"
"strings"
"time"
@@ -55,10 +54,6 @@ type deskGive struct {
trusted func(module string) (bool, error)
// ask asks one machine's node-launcher.secret and answers its result, or the holder's refusal.
ask func(machine string, args map[string]any) (json.RawMessage, error)
// promptFor is the one line the secret's declaration asks for it as (novox/hq issue 789): read from the
// controller's own catalogue, never from the call or the module's runtime. Nil, or an empty answer, asks by
// name alone, as the desk does today.
promptFor func(module, name string) (string, error)
// accept seals the value as `secret accept` does, and says whether it lives until the module's start.
accept func(value string) (untilStart bool, err error)
// record writes the act in the hand-act log.
@@ -72,9 +67,6 @@ type deskGive struct {
// nil keeps no record (a test that does not look). end closes it with how it ended.
open func(node, module, name, desk string) (int64, error)
end func(id int64, outcome string) error
// vault, when set, is the path of a module whose secrets are filled the new way (vault_secrets.go): the vault
// records the ask, holds its bounds and opens what is typed; open, end and accept are not used.
vault vaultAsker
}
// askedByName is the caller as the prompt names it: the first clause of what the bus said, in the characters
@@ -141,9 +133,6 @@ func (d deskGive) give(node, module, name, desk string) (string, error) {
"bus, where an agent may answer first (novox/hq ADR 0259 §10)", module, name, node, module, name)
}
}
if d.vault != nil {
return d.giveToVault(node, module, name, desk)
}
// The bounds, read and kept before anybody is asked to type (novox/hq ADR 0277): one open prompt per secret,
// few an hour. How the ask ends is recorded whatever happens below.
outcome := "failed"
@@ -164,28 +153,15 @@ func (d deskGive) give(node, module, name, desk string) (string, error) {
}
// By name, never by words: the holder writes the prompt from these, and says the controller asks, which
// the bus alone makes true (broker.ControllerOnly). Who asked is the bus's word on the caller, cut to a
// name's characters — never an argument of the call. `what` is the catalogue's own words for the secret
// (novox/hq issue 789), never a word the caller chose; omitted when the module declares none, so an old
// desk and an old module behave as today.
args := map[string]any{
// name's characters — never an argument of the call.
raw, err := d.ask(desk, map[string]any{
"module": module,
"secret": name,
"node": node,
"asked_by": askedByName(d.askedBy),
"seal_to": public,
"timeout_seconds": deskPromptWithin,
}
if d.promptFor != nil {
what, err := d.promptFor(module, name)
if err != nil {
outcome = "refused"
return "", fmt.Errorf("nobody was asked to type anything: %w", err)
}
if what != "" {
args["what"] = what
}
}
raw, err := d.ask(desk, args)
})
if err != nil {
outcome = "refused"
return "", fmt.Errorf("the desk on %s could not be asked: %w", desk, err)
@@ -260,9 +236,8 @@ func askAtDesk(ctx context.Context, node, module, name, desk string) error {
return err
}
defer open.Close()
askedBy := link.Caller()
d := deskGive{
askedBy: askedBy,
askedBy: link.Caller(),
open: func(node, module, name, desk string) (int64, error) {
return open.inventory.OpenSecretAsk(ctx, node, module, name, askedByName(link.Caller()), desk)
},
@@ -272,8 +247,7 @@ func askAtDesk(ctx context.Context, node, module, name, desk string) error {
_, err := open.inventory.NodeByName(ctx, machine)
return err
},
trusted: func(module string) (bool, error) { return open.inventory.RunsAsItsOwnAccount(ctx, module) },
promptFor: func(module, name string) (string, error) { return open.inventory.OwnSecretPrompt(ctx, module, name) },
trusted: func(module string) (bool, error) { return open.inventory.RunsAsItsOwnAccount(ctx, module) },
ask: func(machine string, args map[string]any) (json.RawMessage, error) {
var result json.RawMessage
err := onTheBus(func(conn *nats.Conn) error {
@@ -299,18 +273,7 @@ func askAtDesk(ctx context.Context, node, module, name, desk string) error {
return err
})
},
announce: func(node, module, name, how string) error {
return announceSecretGiven(ctx, node, module, name, how, askedBy)
},
}
// A module whose secrets are filled the new way is asked for by the vault (ADR 0349 clause 9); every other
// keeps today's path (clause 17).
ours, err := filledTheNewWay(module)
if err != nil {
return fmt.Errorf("nothing was asked: %w", err)
}
if ours {
d.vault = theVault(ctx)
announce: func(node, module, name, how string) error { return announceSecretGiven(ctx, node, module, name, how) },
}
words, err := d.give(node, module, name, desk)
if err != nil {
@@ -361,85 +324,10 @@ func secretNameWords(name string) string {
return strings.Join(strings.FieldsFunc(name, func(r rune) bool { return r == '-' || r == '_' || r == '.' }), " ")
}
// tellWithin is how long the operator channel's `tell` is given to answer: well under a human's patience,
// and short because nothing in the call waits on the operator — it asks nothing (novox/hq ADR 0380).
const tellWithin = 10 * time.Second
// askOperatorChannel asks the operator channel's seat (the holder issue 797 gives it) its `tell` verb,
// through the controller's grant (novox/hq ADR 0380 §8), over this controller's own bus connection; a seam
// a test replaces whole. The holder's own refusal is an error naming it; an absent holder or a refused
// grant answer as link.ErrNothingServes or link.ErrGrantRefused (checked with errors.Is), so the caller can
// tell the two apart from any other failure.
var askOperatorChannel = func(ctx context.Context, args map[string]any) (json.RawMessage, error) {
var result json.RawMessage
err := onTheBus(func(conn *nats.Conn) error {
answer, err := link.AskMeshSeatTool(ctx, conn, "operator-channel", "tell", args, tellWithin)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("operator-channel.tell refused: %s", answer.Error)
}
result = answer.Result
return nil
// announceSecretGiven raises it on this controller's keeper.
func announceSecretGiven(ctx context.Context, node, module, name, how string) error {
return withKeeper(ctx, func(k *conditions.Keeper) error {
_, err := k.Observe(ctx, secretGivenObservation(node, module, name, how, time.Now()))
return err
})
return result, err
}
// announceSecretGiven tells the operator, once, that a module's own secret was given (novox/hq ADR 0380):
// one plain message through the operator channel's `tell`, naming the secret, the module, the machine,
// where and when it was given and who asked — asking nothing, and opening no condition. **The controller's
// change waits for that verb at run time, not at merge** (clause 9): while nothing on the bus yet serves it,
// or the controller's own grant is refused, the secret-given condition is raised exactly as before ADR 0380
// — any other failure (the holder itself refusing, or answering something unreadable) is the announcement's
// own failure, said to the caller as it always was.
func announceSecretGiven(ctx context.Context, node, module, name, how, askedBy string) error {
at := time.Now()
body := map[string]any{
"headline": fmt.Sprintf("A password was entered for %s's %s on %s", module, secretNameWords(name), node),
"subject": node,
"lines": []string{
fmt.Sprintf("%s at %s, asked by %s.", how, at.Local().Format("15:04"), askedByName(askedBy)),
"If that was not you, give it again yourself at the controller's terminal.",
},
}
_, told := askOperatorChannel(ctx, body)
switch {
case told == nil:
clearSecretGivenConditions(ctx)
return nil
case errors.Is(told, link.ErrNothingServes), errors.Is(told, link.ErrGrantRefused):
log.Printf("secret-given: %s's %s on %s could not be told through the operator channel (%v); the "+
"secret-given condition is raised as it was before novox/hq ADR 0380 (its clause 9)", module, name,
node, told)
return withKeeper(ctx, func(k *conditions.Keeper) error {
_, err := k.Observe(ctx, secretGivenObservation(node, module, name, how, at))
return err
})
default:
return told
}
}
// clearSecretGivenConditions retires every open secret-given condition once a plain message has replaced
// it (novox/hq ADR 0380 §10): called after the first time a secret-given message is told successfully. No
// resolved line is sent — the condition is simply gone — and why is kept in the keeper's own history.
func clearSecretGivenConditions(ctx context.Context) {
if err := withKeeper(ctx, func(k *conditions.Keeper) error {
open, err := k.Open(ctx)
if err != nil {
return err
}
for _, c := range open {
if c.Kind != kindSecretGiven {
continue
}
if _, err := k.ClearSaying(ctx, c.Key, "retired by novox/hq ADR 0380", ""); err != nil {
return err
}
}
return nil
}); err != nil {
log.Printf("secret-given: the open secret-given conditions, retired by ADR 0380, could not be cleared: %v", err)
}
}
@@ -1,96 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/link"
)
func TestASecretValueIsNeverAcceptedThroughAVerb(t *testing.T) {
t.Setenv(verbVar, "mesh-controller.command")
for _, args := range [][]string{
{"accept", "anchor", "telegram", "telegram-token", "--from", "/dev/null"},
{"accept", "anchor", "app", "db", "--from", "/dev/null", "--provider", "store"},
} {
err := secretCommand(context.Background(), args)
if err == nil || !strings.Contains(err.Error(), "never through a verb") {
t.Errorf("%v: %v", args, err)
}
}
}
func TestATrustedPartysSecretGivenAtTheTerminalIsAnnouncedBeforeItIsKept(t *testing.T) {
var order []string
announce := func(fail bool) func() error {
return func() error {
order = append(order, "announce")
if fail {
return errors.New("no channel")
}
return nil
}
}
keep := func() (bool, error) { order = append(order, "keep"); return false, nil }
order = nil
if _, unannounced, err := keepGiven(true, announce(false), keep); err != nil || unannounced != nil ||
strings.Join(order, ",") != "announce,keep" {
t.Errorf("trusted: %v %v, order %v; want announced, then kept", unannounced, err, order)
}
order = nil
if _, _, err := keepGiven(true, announce(true), keep); err == nil || strings.Join(order, ",") != "announce" {
t.Errorf("trusted, announcement failed: %v, order %v; want refused and nothing kept", err, order)
}
order = nil
if _, unannounced, err := keepGiven(false, announce(true), keep); err != nil || unannounced == nil ||
strings.Join(order, ",") != "keep,announce" {
t.Errorf("not trusted: %v %v, order %v; want kept, then the failed announcement said", unannounced, err, order)
}
order = nil
failing := func() (bool, error) { order = append(order, "keep"); return false, errors.New("store away") }
if _, _, err := keepGiven(false, announce(false), failing); err == nil || strings.Join(order, ",") != "keep" {
t.Errorf("not trusted, keep failed: %v, order %v; want refused and nothing announced", err, order)
}
}
func TestASecretAskIsNeverASecretRead(t *testing.T) {
t.Setenv(verbVar, "mesh-controller.secret-ask")
for _, args := range [][]string{
{"ask", "anchor", "telegram", "telegram-token", "the-value"},
{"ask", "anchor", "telegram"},
{"ask", "anchor", "telegram", "telegram-token", "--from", "/dev/null"},
} {
if err := secretCommand(context.Background(), args); err == nil {
t.Errorf("secret %v was taken", args)
}
}
for _, args := range [][]string{{"recover", "anchor", "telegram", "telegram-token"}, {"export"}} {
if err := terminalOnly(append([]string{"secret"}, args...)); err == nil {
t.Errorf("secret %v passed the terminal rule", args)
}
}
}
func TestATrustedPartysSecretAtTheTerminalIsNotKeptWhenTellAndTheConditionBothFail(t *testing.T) {
ctx := context.Background()
askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, link.ErrNothingServes })
// conditionsFrom is left nil and there is no bus to dial here, so the fallback condition cannot be
// raised either: both of ADR 0380's ways to tell the operator fail.
kept := false
if _, _, err := keepGiven(true,
func() error {
return announceSecretGiven(ctx, "anchor", "telegram", "telegram-token", "at the controller's terminal", "me")
},
func() (bool, error) { kept = true; return false, nil }); err == nil {
t.Error("a trusted party's secret was kept though telling the operator and raising the condition both failed")
}
if kept {
t.Error("the value was kept despite the failure")
}
}
+58 -194
View File
@@ -263,6 +263,18 @@ func TestOnlyTheControllerMayAskTheDesksPrompt(t *testing.T) {
// A value for a secret comes from the terminal or the desk, never through a verb (the review of 2026-10-09,
// M4): `secret accept` with a value, run for a verb, is refused before anything is read.
func TestASecretValueIsNeverAcceptedThroughAVerb(t *testing.T) {
t.Setenv(verbVar, "mesh-controller.command")
for _, args := range [][]string{
{"accept", "anchor", "telegram", "telegram-token", "--from", "/dev/null"},
{"accept", "anchor", "app", "db", "--from", "/dev/null", "--provider", "store"},
} {
err := secretCommand(context.Background(), args)
if err == nil || !strings.Contains(err.Error(), "never through a verb") {
t.Errorf("%v: %v", args, err)
}
}
}
// The `give` and `secret-ask` verbs' own line passes the terminal-only rule of ADR 0266, and no `secret accept`
// does: a value, a file, a provider or an extra word is still the terminal's alone (novox/hq ADR 0277).
@@ -343,6 +355,39 @@ func TestOnlyTheDeskMachinesLauncherMayAnswerItsPrompt(t *testing.T) {
// N1-give at the controller's terminal (the confirmation review of 2026-10-09): a trusted party's secret is
// announced before it is kept, and not kept when the announcement fails; another module's is kept first and
// a failed announcement is said, not undone.
func TestATrustedPartysSecretGivenAtTheTerminalIsAnnouncedBeforeItIsKept(t *testing.T) {
var order []string
announce := func(fail bool) func() error {
return func() error {
order = append(order, "announce")
if fail {
return errors.New("no channel")
}
return nil
}
}
keep := func() (bool, error) { order = append(order, "keep"); return false, nil }
order = nil
if _, unannounced, err := keepGiven(true, announce(false), keep); err != nil || unannounced != nil ||
strings.Join(order, ",") != "announce,keep" {
t.Errorf("trusted: %v %v, order %v; want announced, then kept", unannounced, err, order)
}
order = nil
if _, _, err := keepGiven(true, announce(true), keep); err == nil || strings.Join(order, ",") != "announce" {
t.Errorf("trusted, announcement failed: %v, order %v; want refused and nothing kept", err, order)
}
order = nil
if _, unannounced, err := keepGiven(false, announce(true), keep); err != nil || unannounced == nil ||
strings.Join(order, ",") != "keep,announce" {
t.Errorf("not trusted: %v %v, order %v; want kept, then the failed announcement said", unannounced, err, order)
}
order = nil
failing := func() (bool, error) { order = append(order, "keep"); return false, errors.New("store away") }
if _, _, err := keepGiven(false, announce(false), failing); err == nil || strings.Join(order, ",") != "keep" {
t.Errorf("not trusted, keep failed: %v, order %v; want refused and nothing announced", err, order)
}
}
// A machine the mesh does not know, as the secret's or as the desk, is refused before anybody is asked to type.
func TestAGiveNamingAMachineTheMeshDoesNotKnowAsksNobody(t *testing.T) {
@@ -411,52 +456,6 @@ func TestThePromptNamesWhoAskedFromTheBussWordAlone(t *testing.T) {
}
}
// The desk's prompt names which account of a family is asked for, from the catalogue's own words (novox/hq
// issue 789): a family declaring `prompt` with `{member}` sends `what` with the member's suffix filled in.
func TestThePromptNamesWhichAccountOfAFamilyIsAsked(t *testing.T) {
d, _, _, asked := aDesk(t, sealedTo(t, typed))
d.declares = func(module, name string) error {
if module != "mesh-mail" || name != "mail-credential-gmail" {
return errors.New(module + " does not declare " + name + " as an own secret")
}
return nil
}
d.promptFor = func(module, name string) (string, error) {
if module == "mesh-mail" && name == "mail-credential-gmail" {
return "the password of the mail account gmail", nil
}
return "", nil
}
if _, err := d.give("anchor", "mesh-mail", "mail-credential-gmail", "laptop"); err != nil {
t.Fatal(err)
}
if got := (*asked)[0]["what"]; got != "the password of the mail account gmail" {
t.Errorf(`"what" = %v`, got)
}
}
// Without a declared prompt, no `what` is sent: an old module, or one that declares none, is asked by name alone,
// as the desk does today (novox/hq issue 789).
func TestWithNoDeclaredPromptNoWhatIsSent(t *testing.T) {
d, _, _, asked := aDesk(t, sealedTo(t, typed))
d.promptFor = func(module, name string) (string, error) { return "", nil }
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if _, there := (*asked)[0]["what"]; there {
t.Errorf(`"what" was sent with no declared prompt: %v`, (*asked)[0])
}
// And with no promptFor wired at all (an older caller of deskGive), same thing: `give` still works, asking
// nothing more than it does today.
d2, _, _, asked2 := aDesk(t, sealedTo(t, typed))
if _, err := d2.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if _, there := (*asked2)[0]["what"]; there {
t.Errorf(`"what" was sent with promptFor nil: %v`, (*asked2)[0])
}
}
// An ask for a secret is bounded before anybody is asked to type (ADR 0277): the record refuses it, nothing is
// asked; and how every ask ends is recorded.
func TestASecretAskIsBoundedAndItsEndRecorded(t *testing.T) {
@@ -494,155 +493,20 @@ func TestASecretAskIsBoundedAndItsEndRecorded(t *testing.T) {
// A secret ask cannot be turned into a secret read: the line carries no value, the answer carries none, and every
// other `secret` line is the terminal's.
// askOperatorChannelFake replaces askOperatorChannel for the test and records every call it was given.
func askOperatorChannelFake(t *testing.T, answer func(args map[string]any) (json.RawMessage, error)) *[]map[string]any {
t.Helper()
var calls []map[string]any
was := askOperatorChannel
askOperatorChannel = func(_ context.Context, args map[string]any) (json.RawMessage, error) {
calls = append(calls, args)
return answer(args)
}
t.Cleanup(func() { askOperatorChannel = was })
return &calls
}
// announceWithRealTell wires a deskGive's announce to the real announceSecretGiven, as askAtDesk does,
// rather than the fakes the tests above use to look at what `give` does around it.
func announceWithRealTell(ctx context.Context, askedBy string) func(node, module, name, how string) error {
return func(node, module, name, how string) error {
return announceSecretGiven(ctx, node, module, name, how, askedBy)
}
}
// novox/hq ADR 0380 clauses 1-2: a secret given at the desk is told to the operator once, in a plain
// message through the operator channel's `tell`, naming the secret, the module, where and when and who
// asked, and opens no condition — the kind the review of 2026-10-09 (M4) raised is retired.
func TestASecretGivenAtTheDeskIsToldOnceAndOpensNoCondition(t *testing.T) {
ctx := context.Background()
k, _ := withConditionsInMemory(t)
calls := askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, nil })
d, _, _, _ := aDesk(t, sealedTo(t, typed))
d.announce = announceWithRealTell(ctx, "mail-seats-835@novox")
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
}
if len(*calls) != 1 {
t.Fatalf("told the operator %d times, want exactly 1: %v", len(*calls), *calls)
}
call := (*calls)[0]
if call["headline"] != "A password was entered for telegram's telegram token on anchor" {
t.Errorf("headline %v", call["headline"])
}
if call["subject"] != "anchor" {
t.Errorf("subject %v", call["subject"])
}
lines, _ := call["lines"].([]string)
if len(lines) != 2 || !strings.Contains(lines[0], "at the desk on laptop at") ||
!strings.Contains(lines[0], "asked by mail-seats-835@novox") ||
lines[1] != "If that was not you, give it again yourself at the controller's terminal." {
t.Errorf("lines %v", lines)
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
if len(open) != 0 {
t.Errorf("a condition was raised though the operator was told: %+v", open)
}
}
// novox/hq ADR 0380 §9: until the operator channel's `tell` is delivered — nothing yet serves it, or the
// controller's own grant is refused — the secret-given condition is raised exactly as before.
func TestASecretGivenAtTheDeskRaisesTheConditionWhileNothingServesTell(t *testing.T) {
ctx := context.Background()
for _, failure := range []error{link.ErrNothingServes, link.ErrGrantRefused} {
k, _ := withConditionsInMemory(t)
askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, failure })
d, _, _, _ := aDesk(t, sealedTo(t, typed))
d.announce = announceWithRealTell(ctx, "mesh-mail-835, novox")
if _, err := d.give("anchor", "telegram", "telegram-token", "laptop"); err != nil {
t.Fatal(err)
func TestASecretAskIsNeverASecretRead(t *testing.T) {
t.Setenv(verbVar, "mesh-controller.secret-ask")
for _, args := range [][]string{
{"ask", "anchor", "telegram", "telegram-token", "the-value"},
{"ask", "anchor", "telegram"},
{"ask", "anchor", "telegram", "telegram-token", "--from", "/dev/null"},
} {
if err := secretCommand(context.Background(), args); err == nil {
t.Errorf("secret %v was taken", args)
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
if len(open) != 1 || open[0].Kind != kindSecretGiven {
t.Fatalf("%v: the condition was not raised as before: %+v", failure, open)
}
for _, args := range [][]string{{"recover", "anchor", "telegram", "telegram-token"}, {"export"}} {
if err := terminalOnly(append([]string{"secret"}, args...)); err == nil {
t.Errorf("secret %v passed the terminal rule", args)
}
}
}
// novox/hq ADR 0380 §10: once the operator channel is told successfully, every secret-given condition
// still open from before this change is retired, without a resolved line of its own.
func TestASuccessfulTellRetiresEveryOpenSecretGivenCondition(t *testing.T) {
ctx := context.Background()
k, _ := withConditionsInMemory(t)
for _, module := range []string{"telegram", "mesh-mail"} {
if _, err := k.Observe(ctx, secretGivenObservation("anchor", module, "a-secret", "at the desk on laptop",
time.Now())); err != nil {
t.Fatal(err)
}
}
askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, nil })
if err := announceSecretGiven(ctx, "anchor", "telegram", "telegram-token", "at the desk on laptop", "me"); err != nil {
t.Fatal(err)
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
if len(open) != 0 {
t.Fatalf("the open secret-given conditions were not retired: %+v", open)
}
}
// novox/hq ADR 0380 clause 7: a trusted party's secret given at the controller's terminal is announced
// before it is kept, and not kept when the announcement fails. It still fails when telling the operator
// AND raising the fallback condition both fail — nothing silently keeps a trusted party's value unheard.
// Review round 2 of mesh-controller (feat/mail-seats, head 5f24a47d): ADR 0380 §9's fallback — raising the
// secret-given condition — answers only link.ErrNothingServes and link.ErrGrantRefused. Any other failure of
// the operator channel's `tell` (the holder itself refusing, or answering something unreadable) is the
// announcement's own failure, surfaced to the caller, and opens no condition.
func TestAnOperatorChannelFailureOtherThanNothingServesOrGrantRefusedSurfacesAndRaisesNoCondition(t *testing.T) {
ctx := context.Background()
k, _ := withConditionsInMemory(t)
refused := errors.New("operator-channel.tell refused: too many lines")
askOperatorChannelFake(t, func(map[string]any) (json.RawMessage, error) { return nil, refused })
if err := announceSecretGiven(ctx, "anchor", "telegram", "telegram-token", "at the desk on laptop", "me"); err != refused {
t.Fatalf("announceSecretGiven = %v, want the refusal itself surfaced", err)
}
open, err := k.Open(ctx)
if err != nil {
t.Fatal(err)
}
if len(open) != 0 {
t.Errorf("a condition was raised though the failure was neither ErrNothingServes nor ErrGrantRefused: %+v", open)
}
}
// Review round 2 of mesh-controller: a secret accepted at the controller's terminal writes a hand act naming
// who asked, where and when it was given — and never the value (novox/hq ADR 0380 §6).
func TestASecretAcceptedAtTheTerminalWritesAHandActNamingWhoWhereAndNeverTheValue(t *testing.T) {
act := terminalGiveHandAct("anchor", "telegram", "telegram-token", "mail-seats-835@novox")
if act.Verb != "secret accept" || act.Cause != "given-at-the-terminal" {
t.Fatalf("act %+v", act)
}
if !strings.Contains(act.Why, "at the controller's terminal") {
t.Errorf("the act does not name where: %q", act.Why)
}
if !strings.Contains(act.Why, "asked by mail-seats-835@novox") {
t.Errorf("the act does not name who: %q", act.Why)
}
if strings.Contains(act.Why, typed) {
t.Error("the act carries the value")
}
// When is RecordHandAct's own job (it stamps At with time.Now when the act leaves none, and By with the
// caller when the act leaves that empty too): this act leaves both for it to fill.
if act.By != "" || !act.At.IsZero() {
t.Errorf("the act sets its own who/when instead of leaving them to RecordHandAct: %+v", act)
}
}
-9
View File
@@ -146,11 +146,6 @@ var probeRegistry = []probe{
{ID: probeFillID, Asserts: "every module's bucket and log holds less than three quarters of its cap; one " +
"said filling is cleared once it holds less than seven tenths", From: "ADR 0297, issue 501",
Kind: kindBucketOrLogFilling, Raises: []string{kindBucketOrLogUnread}, Phase: 1, run: probeFill},
// A grant of a restricted stream that composes nothing (novox/hq ADR 0376 §2): the module is carried by the
// machine's tool runner, which never hears a restricted stream, so it hears nothing it was granted.
{ID: probeWithheldID, Asserts: "no grant of a restricted stream names a module that composes nothing for it: every " +
"module granted one runs as an account of its own", From: "ADR 0376 §2", Kind: kindGrantComposesNothing,
Phase: 1, run: probeWithheldHearing},
{ID: "DW", Asserts: "the watchdogs of the signals table ran within three of their intervals",
From: "ADR 0227 rule 6: the watchers are watched", Kind: "watchdogs-silent", Phase: 1, run: probeWatchdogs},
// The core's health definitions (novox/hq to-be 45 §8, ADR 0236): what a core component's new build is
@@ -174,10 +169,6 @@ var probeRegistry = []probe{
{ID: busStepProbe, Asserts: "a bus upgrade a person started is said while it runs, and is followed by the bus's " +
"health within its bound — or is said failed, with its snapshot as the way back", From: "ADR 0236, to-be 45 §8",
Kind: kindBusMaintenance, Raises: []string{kindBusUpgradeFailed}, Phase: 4, run: probeBusStep},
// A delivery plan held for a backup that failed or stayed silent (ADR 0404 decision 4): one condition per plan.
{ID: backupHeldProbe, Asserts: "no delivery plan is held for a backup before its send that failed or did not " +
"answer in ten minutes", From: "ADR 0404 decision 4, issue 1066", Kind: kindBackupHeld, Phase: 1,
run: probeBackupHeld},
}
// probeVerdict is one probe's outcome in a run.

Some files were not shown because too many files have changed in this diff Show More