novox/mesh-controller
13
0
Fork 0
Code Issues Pull Requests 3 Releases Wiki Activity

Compare commits

merge into: novox/mesh-controller:feat/r4b-sdk-users
Branches Tags
novox/mesh-controller:main novox/mesh-controller:feat/r4b-sdk-users novox/mesh-controller:feat/chain-on-the-bus novox/mesh-controller:fix/701-a-name-nothing-answers-is-not-used novox/mesh-controller:fix/270-a-dead-bus-watch-is-heard novox/mesh-controller:fix/630-config-values-from-four-sources novox/mesh-controller:fix/613-node-smb-server novox/mesh-controller:feat/328-libraries-and-module-users novox/mesh-controller:feat/325-identities-and-kept-folders novox/mesh-controller:feat/bus-view-principal novox/mesh-controller:test/plan-diff-left-out-line novox/mesh-controller:fix/the-registry-trust-test-reads-the-runtimes-module novox/mesh-controller:fix/the-runtime-file-is-the-runtimes novox/mesh-controller:feat/build-edges-are-recorded novox/mesh-controller:fix/several-contributions-collide-in-grants-v2
..
pull from: novox/mesh-controller:feat/328-libraries-and-module-users
Branches Tags
novox/mesh-controller:feat/r4b-sdk-users novox/mesh-controller:main novox/mesh-controller:feat/chain-on-the-bus novox/mesh-controller:fix/701-a-name-nothing-answers-is-not-used novox/mesh-controller:fix/270-a-dead-bus-watch-is-heard novox/mesh-controller:fix/630-config-values-from-four-sources novox/mesh-controller:fix/613-node-smb-server novox/mesh-controller:feat/328-libraries-and-module-users novox/mesh-controller:feat/325-identities-and-kept-folders novox/mesh-controller:feat/bus-view-principal novox/mesh-controller:test/plan-diff-left-out-line novox/mesh-controller:fix/the-registry-trust-test-reads-the-runtimes-module novox/mesh-controller:fix/the-runtime-file-is-the-runtimes novox/mesh-controller:feat/build-edges-are-recorded novox/mesh-controller:fix/several-contributions-collide-in-grants-v2
6 Commits
feat/r4b-sdk-users .. feat/328-libraries-and-module-users
Author SHA1 Message Date
jschoubben 5b9908bed2 Hold a joining build for a person without superseding its delivery, join only on an engine that keeps own groups, and keep a held module's settings (hq ADR 0328 §8, ADR 0334; re-review of 96e15f86) 
A delivery does not send a joining build held for its library or its engine: the module waits for a person,
the plan stays open saying so and judges its other modules, and sends it once nothing holds it. A held move is
no move for the gate. A module joins only on an engine at contract 7, and a held module is composed with the
settings it last ran with. The runner's verdicts are said for their modules and judged by whether a move
changes them. mesh-host pinned at #97 (7a41e120), its bundle now examples/foundation-first-node.lock.
 2026-10-13 00:55:54 +02:00
jschoubben 1de9bbc448 Own directories of a joining module carry setgid, to an engine that takes it (hq ADR 0334 v4) 2026-10-13 00:31:54 +02:00
jschoubben 96e15f8693 Hold a joining module at the build it ran, own its directories by its own group, and take a library only when its act is done (hq ADR 0328 review, ADR 0334) 
A held module is composed exactly as before it joined, at the build its machine runs, so its port, resources and
contributions stay. A joining module's directories belong to its user and its own group, and its tools' directories
are refused closed to that group; the runner joins it where the engine can say so. A take records the root and is
taken only when the engine states the act done. Libraries' paths and verdicts count modules pulled in by a
requirement, and a unit takes the identity.
 2026-10-13 00:03:18 +02:00
jschoubben 7ea6d36eab Pin the node-engine at its ADR 0328 merge (#95), and speak its contract: a library's verdict is said for the modules that use it 2026-10-12 23:02:31 +02:00
jschoubben f634e33792 Libraries and the modules that join them, on the controller's side (hq ADR 0328) 2026-10-12 22:29:35 +02:00
jschoubben fa0a5f1c88 Carry over the ADR 0325 controller work as the base for ADR 0328 2026-10-12 21:57:29 +02:00
1250 changed files with 46589 additions and 118280 deletions
Expand all files Collapse all files
Dockerfile
+1 -3
View File
@@ -27,11 +27,9 @@ WORKDIR /src
COPY go.mod go.sum ./
COPY vendor/ vendor/
# `COPY . .` brings `.git` along, so Go is told not to stamp the commit: the version is said by -X, and a
# rebuild of an unchanged program is the same bytes (novox/hq ADR 0267 rule 6, issue 363).
COPY . .
ARG VERSION=development
RUN CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPROXY=off go build -trimpath -buildvcs=false \
RUN CGO_ENABLED=0 GOFLAGS=-mod=vendor GOPROXY=off go build -trimpath \
-ldflags "-s -w -X main.version=${VERSION}" \
-o /mesh-controller ./cmd/mesh-controller
Makefile
+3 -7
View File
@@ -95,7 +95,7 @@ PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build --build-arg GO_BASE=$(GO_BASE) -f modules/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -110,18 +110,14 @@ proxy-image:
# after formatting never ran through it. The script is the one list of steps; this target only gives it
# a database (MESH_TEST_POSTGRES, exported above).
#
# **Two levels** (issue 895): the unit level, `go test ./...` untagged, timed and held to a bound; and the integration
# level, `-tags integration`, the tests that start a bus, a store or another process. Each is said with its time.
#
# **Packages in parallel, under the race detector, each test on a bus of its own** (internal/testbus),
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# The unit level alone (untagged: no bus, no store, no other process), which a developer runs continuously. The tests
# that need a bus or the store carry `//go:build integration` and run in `check`, not here; without a database the
# store's tests would skip rather than fail anyway.
# Without a database the store's tests skip rather than fail, so this is the honest subset and not
# the gate.
test:
go test -timeout 15m ./...
RELEASE-57.md
-6
View File
@@ -1,6 +0,0 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
check-nested-modules.sh
-29
View File
@@ -1,29 +0,0 @@
#!/bin/sh
# The unit tests of every Go module of its own below this repository's root, run from the root, for merge-check.sh
# and for the test that tries it on a tree (merge_check_nested_test.go). Its own file so that the test needs no
# particular merge-check.sh: the check that judges a change to merge-check.sh is main's.
set -eu
cgo="${cgo:-0}"
race="${race:-}"
# nested_modules vets and tests every Go module of its own below the root, `modules/<name>/go.mod` (novox/hq ADR 0420
# §5, issue 1025): `./...` stops at a nested go.mod, so a module that lives whole in this repository would have its
# unit tests run by nobody. None is a pass. Each runs in its own directory, outside the vendor directory this
# repository's own packages are built from; a nested module vendors what the forge holds (its own `vendor/`, so the
# check runs offline), as this repository's root module does (issue 1019). A failing module fails the check; the others are still run.
nested_modules() {
nested=0
for gomod in modules/*/go.mod; do
[ -e "$gomod" ] || continue
dir=$(dirname "$gomod")
echo "nested module $dir:"
if ! (cd "$dir" && GOFLAGS= CGO_ENABLED=0 go vet ./... &&
GOFLAGS= CGO_ENABLED="${cgo:-0}" go test ${race:-} -count=1 -timeout 30m ./...); then
echo "nested module $dir FAILED"
nested=1
fi
done
return $nested
}
nested_modules
cmd/mesh-builder/Dockerfile
+1 -1
View File
@@ -11,7 +11,7 @@ WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -buildvcs=false -ldflags '-s -w' -o /mesh-builder ./cmd/mesh-builder
RUN CGO_ENABLED=0 go build -trimpath -ldflags '-s -w' -o /mesh-builder ./cmd/mesh-builder
FROM ${ALPINE_BASE}
# git to clone what it is asked to build, and the docker client to build and push it. The daemon
cmd/mesh-builder/busy.go
-146
View File
@@ -1,146 +0,0 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// **The build agent says whether it is busy, and being asked drains it** (novox/hq ADR 0335 §1, §5). Its
// container declares `busy` as `mesh-builder busy`: the node-engine runs it inside the container before it
// replaces it. It exits 0 while a build is in hand and 1 when none is; and from the first ask the agent takes
// no new build request, so its answer ends once the build in hand ends — a replaced builder finishes the check
// it runs, instead of the build queue handing it out again from the start (issue 594).
//
// Two marks in the container's own temporary directory, as the ready mark (ready.go): a new container starts
// without them, and a restarted one removes them before it takes work again.
//
// - in hand: written by the agent when it claims a request, before it reads whether it is drained; removed
// when the build ends.
// - drained: written by the ask, before it reads whether a build is in hand.
//
// Each side writes its own mark before it reads the other's, so an ask racing a fetch sees the build in hand,
// or the agent sees the drain and hands the request back; never neither.
func inHandMark() string { return filepath.Join(os.TempDir(), "mesh-builder.in-hand") }
func drainMark() string { return filepath.Join(os.TempDir(), "mesh-builder.drained") }
// The drain lapses when no ask has come for drainLapse: the node-engine asks every 20 seconds for as long as it
// really waits, so an agent whose replacement was withdrawn — the declaration put back, the walk stopped —
// takes work again rather than staying drained for ever (review of ADR 0335). And an agent drained for longer
// than the node-engine's bound and the slack says it is not ready: the replacement it was drained for never
// came, which is a fault to see.
const (
drainLapse = 2 * time.Minute
drainTooLong = 15*time.Minute + 5*time.Minute
)
// drainedAt is when the drain began and when it was last asked, from its mark; false when there is no mark.
func drainedAt() (began, asked time.Time, ok bool) {
info, err := os.Stat(drainMark())
if err != nil {
return time.Time{}, time.Time{}, false
}
asked = info.ModTime()
began = asked
if raw, err := os.ReadFile(drainMark()); err == nil {
if t, err := time.Parse(time.RFC3339, strings.TrimSpace(string(raw))); err == nil && !t.After(asked) {
began = t
}
}
return began, asked, true
}
// drained says the agent was asked whether it is busy within drainLapse: it takes no new build request.
func drained() bool { return drainedSince(time.Now()) }
func drainedSince(now time.Time) bool {
_, asked, ok := drainedAt()
return ok && now.Sub(asked) < drainLapse
}
// drainedTooLong is the health check's second half: nil unless the agent has been drained, asked all along, for
// longer than its node-engine's bound and the slack.
func drainedTooLong(now time.Time) error {
began, asked, ok := drainedAt()
if !ok || now.Sub(asked) >= drainLapse || now.Sub(began) <= drainTooLong {
return nil
}
return fmt.Errorf("drained since %s and still asked: the replacement its node-engine waits for has not come "+
"within %s (novox/hq ADR 0335)", began.UTC().Format(time.RFC3339), drainTooLong)
}
// claim marks a request in hand, unless the agent is drained: then nothing is in hand, and the request goes
// back to the queue.
func claim() bool {
// Made, never written over: one build at a time, and a mark already there is a build in hand.
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
// Unmarked, an ask would say idle while this build runs: the build is not taken.
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
_, err = f.WriteString(time.Now().UTC().Format(time.RFC3339) + "\n")
if closeErr := f.Close(); err == nil {
err = closeErr
}
if err != nil {
_ = os.Remove(inHandMark())
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
if drained() {
_ = os.Remove(inHandMark())
fmt.Fprintln(os.Stderr, "asked whether busy, so this agent takes no new build: handed back")
return false
}
return true
}
// holding says what is in hand, for the ask to print: written once the request is read.
func holding(what string) {
_ = os.WriteFile(inHandMark(), []byte(time.Now().UTC().Format(time.RFC3339)+" "+what+"\n"), 0o644)
}
// released ends the build in hand.
func released() {
if err := os.Remove(inHandMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(os.Stderr, "cannot remove the mark of the build in hand (%s): %v\n", inHandMark(), err)
}
}
// undrained clears both marks at the start of a holder: a container started again is asked afresh.
func undrained() error {
for _, mark := range []string{drainMark(), inHandMark()} {
if err := os.Remove(mark); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove %s: %w", mark, err)
}
}
return nil
}
// busy is the ask: it drains the agent, then says whether a build is in hand. Nil — exit 0 — while one is.
func busy() (string, error) {
// The mark keeps when the drain began while it is asked within drainLapse, and is written again on every
// ask: its time is the last ask.
began := time.Now().UTC()
if was, _, ok := drainedAt(); ok && drained() {
began = was.UTC()
}
if err := os.WriteFile(drainMark(), []byte(began.Format(time.RFC3339)+"\n"), 0o644); err != nil {
// Not drained, it could take the next build: said, and answered busy only for what is in hand now.
fmt.Fprintf(os.Stderr, "cannot mark this agent drained (%s): %v\n", drainMark(), err)
}
held, err := os.ReadFile(inHandMark())
if err != nil {
return "", errors.New("idle: no build in hand, and this agent takes no new one")
}
since, what, _ := strings.Cut(strings.TrimSpace(string(held)), " ")
if what == "" {
what = "a build"
}
return fmt.Sprintf("busy: %s, in hand since %s; taking no new build", what, since), nil
}
cmd/mesh-builder/busy_test.go
-162
View File
@@ -1,162 +0,0 @@
package main
import (
"os"
"strings"
"sync"
"testing"
"time"
)
// `mesh-builder busy` answers 0 while a build is in hand and non-zero when idle, and after its first ask the
// agent takes no new build request (novox/hq ADR 0335 §5).
func TestBusyAnswersWhileABuildIsInHandAndDrainsTheAgent(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("an agent never asked is drained")
}
// A build is claimed and runs.
if !claim() {
t.Fatal("an agent never asked refused a build")
}
holding("novox/mesh-controller (check-280)")
said, err := busy()
if err != nil || !strings.Contains(said, "novox/mesh-controller (check-280)") {
t.Fatalf("an agent with a build in hand did not answer busy with what it holds: %q %v", said, err)
}
// From that ask on it takes nothing new: the fetch is not made, and one fetched is handed back.
if !drained() {
t.Fatal("the ask did not drain the agent")
}
if claim() {
t.Fatal("a drained agent took a new build")
}
// The build in hand is still in hand after the refused claim, and busy goes on saying so.
if _, err := busy(); err != nil {
t.Fatalf("a refused claim ended the build in hand: %v", err)
}
// It ends: idle, and still drained.
released()
if _, err := busy(); err == nil {
t.Fatal("an agent with nothing in hand answered busy")
}
if claim() {
t.Fatal("a drained agent took a build after its last one ended")
}
// Started again, it is asked afresh.
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() || !claim() {
t.Fatal("a holder started again is still drained")
}
released()
}
// An ask racing a fetch sees the build in hand, or the agent sees the drain and hands the request back: never
// neither, which would replace the container under a build just taken.
func TestAnAskRacingAFetchNeverMissesTheBuild(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
for i := 0; i < 200; i++ {
if err := undrained(); err != nil {
t.Fatal(err)
}
var claimed bool
var askErr error
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); claimed = claim() }()
go func() { defer wg.Done(); _, askErr = busy() }()
wg.Wait()
if claimed && askErr != nil {
t.Fatalf("round %d: the agent took a build and the ask said idle", i)
}
released()
}
}
// A drain no longer asked lapses: an agent whose replacement was withdrawn after the first ask (a put-back, a
// revert, a stopped walk) takes work again after drainLapse, rather than staying drained with its health green.
func TestADrainNoLongerAskedLapses(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if !drained() || claim() {
t.Fatal("an agent just asked takes work")
}
// The last ask was three minutes ago.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("a drain no longer asked for three minutes still holds")
}
if !claim() {
t.Fatal("an agent whose drain lapsed takes no work")
}
released()
}
// An agent drained, and asked all along, for longer than its node-engine's bound and the slack is not ready: the
// replacement it was drained for never came. One asked again after a lapse starts a new drain.
func TestAnAgentDrainedTooLongIsNotReady(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
taking("mesh-build-machine")
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
now := time.Now()
if err := drainedTooLong(now); err != nil {
t.Fatalf("an agent drained a moment ago is not ready: %v", err)
}
// The drain began 25 minutes ago, and the node-engine asked a moment ago.
began := now.Add(-25 * time.Minute).UTC().Format(time.RFC3339)
if err := os.WriteFile(drainMark(), []byte(began+"\n"), 0o644); err != nil {
t.Fatal(err)
}
err := drainedTooLong(time.Now())
if err == nil || !strings.Contains(err.Error(), "has not come") {
t.Fatalf("an agent drained for 25 minutes reads ready: %v", err)
}
// Each ask keeps when the drain began.
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if err := drainedTooLong(time.Now()); err == nil {
t.Fatal("an ask started the drain again")
}
// The asks stopped: the drain lapsed, and so did the fault.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if err := drainedTooLong(time.Now()); err != nil {
t.Fatalf("a lapsed drain still reads not ready: %v", err)
}
}
// An ask this build does not know exits non-zero: an older image asked `busy` reads idle, not busy for the
// node-engine's whole bound.
func TestAnUnknownCommandFails(t *testing.T) {
was := os.Args
t.Cleanup(func() { os.Args = was })
os.Args = []string{"mesh-builder", "some-later-word"}
if err := run(); err == nil {
t.Fatal("an unknown command exited 0")
}
os.Args = []string{"mesh-builder", "help"}
if err := run(); err != nil {
t.Fatalf("help failed: %v", err)
}
}
cmd/mesh-builder/check_spec_test.go
-38
View File
@@ -1,38 +0,0 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
)
// The build seat hands the check the judge cache the build agent declares (MESH_JUDGE_CACHE), never a path of its own.
func TestTheBuildSeatHandsTheCheckTheJudgeCacheItDeclares(t *testing.T) {
t.Setenv("MESH_JUDGE_CACHE", "/var/lib/mesh/build-agent/judge-cache")
if got := judgeCache(); got != "/var/lib/mesh/build-agent/judge-cache" {
t.Fatalf("the judge cache the manifest declares is read as %q", got)
}
t.Setenv("MESH_JUDGE_CACHE", "")
if got := judgeCache(); got != "" {
t.Fatalf("with no judge cache declared, %q is used", got)
}
}
// The siblings a check is asked to read reach the build seat's spec and, as read, the outcome (novox/hq ADR 0382).
func TestTheSiblingsAnAskNamesReachTheSpecAndTheOutcome(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main",
Siblings: []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", Dir: "mesh-controller"}},
SiblingsUnknown: "why"}})
if len(spec.Siblings) != 1 || spec.Siblings[0] != (builder.Sibling{Repository: "novox/mesh-controller", Number: 346,
Head: "a", Base: "main", Dir: "mesh-controller"}) || spec.SiblingsUnknown != "why" {
t.Fatalf("the spec reads %+v", spec)
}
out := siblingsOut([]builder.JudgedSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", BaseCommit: "b"}})
if len(out) != 1 || out[0].BaseCommit != "b" || out[0].Number != 346 {
t.Fatalf("the outcome reads %+v", out)
}
if plain := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "o", Repo: "r"}}); plain.Siblings != nil {
t.Fatalf("a request that names none has siblings %+v", plain.Siblings)
}
}
cmd/mesh-builder/holder.go
+1 -32
View File
@@ -14,8 +14,8 @@ import (
"github.com/nats-io/nats.go/micro"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
@@ -41,9 +41,6 @@ type holder struct {
say func(link.HolderState) error
// remove runs what a kill needs outside the build: the containers left behind.
remove builder.Runner
// scratch is where check builds are kept (novox/hq ADR 0350 §5): what the week's sweep lets go of. Nil where
// this machine publishes to no artifact store.
scratch builder.Scratch
mu sync.Mutex
paused bool
@@ -134,34 +131,6 @@ func (h *holder) stateWhilePaused(ctx context.Context, every time.Duration) {
}
}
// expireChecks lets go of every check build older than a week (novox/hq ADR 0350 §5), at start and every while
// after. Every holder sweeps the one store; what one let go of, another finds gone.
func (h *holder) expireChecks(ctx context.Context, every time.Duration, now func() time.Time) {
sweep := func() {
if h.scratch == nil {
return
}
n, err := h.scratch.Expire(ctx, builder.CheckBuildsKept, now())
switch {
case err != nil:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d let go of, and not all: %v\n", builder.CheckBuildsKept, n, err)
case n > 0:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d check(s)' builds let go of\n", builder.CheckBuildsKept, n)
}
}
sweep()
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
sweep()
}
}
}
// begin records the build this machine took, with the cancel that ends it.
func (h *holder) begin(request link.BuildRequest, cancel context.CancelFunc) *running {
r := &running{request: request, started: time.Now(), cancel: cancel, done: make(chan struct{})}
cmd/mesh-builder/main.go
+67 -373
View File
@@ -18,16 +18,12 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log"
"net/url"
"os"
"os/signal"
"sort"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
@@ -58,8 +54,6 @@ is dialled except the broker.
MESH_BINDING a file the mesh wrote saying where the artifact store is
MESH_PACKAGE_BINDING a file the mesh wrote saying where the package registry is
MESH_NPM_TOKEN_FILE a file the mesh sealed holding the token for it
MESH_GENESIS_PACKAGE_REGISTRY the document the installer sealed at genesis, read while no
binding is present (novox/hq ADR 0336)
MESH_NPM_REGISTRY a package registry URL, when the mesh has not said (a person, the bootstrap)
MESH_NPM_TOKEN the token for it, likewise
MESH_NPM_SCOPE the scope it answers for (default: @novox)
@@ -72,17 +66,6 @@ broker to take work from or a registry to publish into:
Without --registry the artifacts stay in this machine's container runtime, named by the
digest of their own configuration. The result is printed as JSON.
mesh-builder ready
exits 0 once the agent running in this container is bound to its build work and pulls it, and
1 before: its container's health check.
mesh-builder busy
exits 0 while the agent running in this container has a build in hand, and 1 when it has none;
from the first ask the agent takes no new build: its container's busy command, asked before the
container is replaced (novox/hq ADR 0335).
`
func run() error {
@@ -93,42 +76,12 @@ func run() error {
return nil
case "build":
return buildOnce(context.Background(), os.Args[2:])
case "ready":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work, and
// once it has been drained longer than its node-engine would wait to replace it (ADR 0335).
if err := ready(); err != nil {
return err
}
return drainedTooLong(time.Now())
case "busy":
// The container's busy command (novox/hq ADR 0335): drains the agent, and exits non-zero once
// nothing is in hand.
said, err := busy()
if err != nil {
return err
}
fmt.Println(said)
return nil
case "help", "-h", "--help":
default:
fmt.Print(usage)
return nil
default:
// **An ask this build does not know fails** (novox/hq ADR 0335): an image older than `busy`, asked
// it, must read as idle — exiting 0 would hold its container for the node-engine's whole bound.
fmt.Fprint(os.Stderr, usage)
return fmt.Errorf("%q is not a command this mesh-builder knows", os.Args[1])
}
}
if err := notReady(); err != nil {
return err
}
// **The builder's own version is part of every reuse key** (novox/hq ADR 0350 §1): a builder that builds
// differently never takes a build another one made.
builder.Version = builder.SelfVersion(version)
if err := undrained(); err != nil {
return err
}
credential, err := brokerFrom()
if err != nil {
return err
@@ -181,13 +134,7 @@ func run() error {
// **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder
// recreated mid-check left the check's store and bus running, and the ask went to another machine.
h.removeLeftBehind()
// Paused or drained, it takes nothing new (novox/hq ADR 0219, ADR 0335): a drained agent finishes the build
// in hand and then says idle, and its container is replaced between builds.
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{
Paused: func() bool { return h.Paused() || drained() },
Taking: func() { taking(seat) },
Claim: claim,
})
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused})
defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
@@ -211,13 +158,8 @@ func run() error {
fmt.Fprintf(os.Stderr, "building for the mesh, publishing to %s\n", registry)
publisher := builder.Registry{Address: registry, Run: builder.Command}
// Check builds are let go of after a week (novox/hq ADR 0350 §5).
h.scratch = publisher
go h.expireChecks(ctx, time.Hour, time.Now)
return machine.Take(ctx, func(ctx context.Context, work link.Build) {
defer released()
holding(work.Request().Repository + " (" + work.Request().ID + ")")
answer(ctx, publisher, on, workspace, work, h)
})
}
@@ -274,12 +216,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// role's own events under the build's id (novox/hq ADR 0157) — so whoever asked, and anybody
// watching, reads the same lines this container's log holds, live, and after the fact from the
// stream. Said first, before anything runs, so a build that hangs is one that visibly started.
// One line at a time: a check's builds for its merge say their lines while its tests say theirs (novox/hq
// issue 716).
var saying sync.Mutex
say := func(step, message string) {
saying.Lock()
defer saying.Unlock()
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
work.Say(step, message)
if mine != nil && step != "run" && step != "output" {
@@ -291,26 +228,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if err := work.Began(ctx); err != nil {
fmt.Fprintf(os.Stderr, "cannot say a build started: %v\n", err)
}
// **A running check watches its pull request's key** (novox/hq ADR 0406 §4, issue 974): when the head or the trunk
// commit in it moves, the check is for a state nobody wants any more, and ends itself — nothing is asked of the forge
// (issue 726). A group's check, a try and a build watch nothing.
var superseded atomic.Bool
if w, ok := work.(interface {
WatchPull(context.Context, func(link.PullKey)) (func(), error)
}); ok && request.Check != nil {
stopWatching, err := w.WatchPull(building, func(k link.PullKey) {
superseded.Store(true)
say("superseded", fmt.Sprintf("the pull request's key moved to %.12s on %.12s while this checked %.12s on %.12s: stopped",
k.Head, k.Trunk, request.Ref, request.Check.Trunk))
cancel()
})
if err != nil {
fmt.Fprintf(os.Stderr, "the key of %s/%s#%d could not be watched, so its check runs on if the key moves: %v\n",
request.Check.Owner, request.Check.Repo, request.Check.Number, err)
} else {
defer stopWatching()
}
}
result := link.BuildResult{
ID: request.ID, Repository: request.Repository, Path: request.Path,
@@ -325,11 +242,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
}
say("build", what)
npmrc, err := packagesFrom(on)
if err == nil {
say("packages", saysRegistry(npmrc))
}
registryHere, isRegistry := publisher.(builder.Registry)
npmrc, err := packagesFrom()
var built builder.Result
if request.Check != nil {
// **A pull request's merge check, not a build** (novox/hq to-be 45 §9): nothing is built,
@@ -339,37 +252,17 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if r, ok := publisher.(builder.Registry); ok {
registry = r.Address
}
spec := checkSpecOf(request)
// The judge cache the build agent declares as its own data, never in the workspace: only judge builds and
// mesh-lab's replays mount it, never a check's containers, so no pull request's script can write what a judge
// is built with or kept as, or what the replays run (novox/hq ADR 0358 §6, issue 709).
spec.JudgeCache = judgeCache()
// What the merge would move, built by the check for the walk to take (novox/hq ADR 0350 §1): into the
// scratch namespace. **Never with the package registry's credential**: this is code nobody has approved
// yet, and a recipe could read the credential it is handed (issue 462). A module that resolves packages
// at build time has no reuse key anyway, so nothing is lost by refusing it here.
if isRegistry && len(spec.Builds) > 0 {
spec.BuildWith = &builder.CheckBuilding{Publish: publisher, Scratch: registryHere, Held: request.Held,
Running: request.Check.Running, Seats: request.Seats, Named: namedOf(request.Check.Named)}
}
var v builder.CheckVerdict
v, err = builder.Check(building, builder.Command, spec, workspace, registry, forgeFrom(), say)
v, err = builder.Check(building, builder.Command, checkSpecOf(request), workspace, registry, forgeFrom(), say)
if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built),
Validator: v.Validator, Siblings: siblingsOut(v.Siblings), SiblingsUnknown: v.SiblingsUnknown, JudgeFrom: v.JudgeFrom}
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk})
}
// A branch tried before its pull request reads its outcome from this log, whole (novox/hq issue 686).
if request.Check.Try {
if line, err := json.Marshal(triedOf(v)); err == nil {
say(link.StepTried, string(line))
}
}
}
} else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a
@@ -377,22 +270,9 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// after a clone that then fails at npm ci.
// Every container it starts is labelled with its id, so a kill finds what outlived the
// docker client (ADR 0219).
if request.Reuse != nil && isRegistry && !request.DryRun {
// **A walk's build takes the check's build of the same key** (novox/hq ADR 0350 §2).
reuse := builder.Reuse{Scratch: registryHere}
for _, p := range request.Reuse.Pulls {
reuse.Pulls = append(reuse.Pulls, builder.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number})
}
// Only what a verdict names is taken (ADR 0350 §2).
reuse.Named = namedOf(request.Reuse.Builds)
built, err = builder.BuildReusing(building, builder.Labelled(builder.Command, request.ID), publisher, reuse,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
} else {
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
// What it copied into the store, whatever became of the build (novox/hq ADR 0257).
result.Mirrored = built.Mirrored
@@ -402,11 +282,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if mine != nil {
killed = h.returned(mine) && err != nil
}
if superseded.Load() && err != nil {
// Ended by its own key moving on (ADR 0406 §4): said as the check's failure, as a kill is.
result.Failed = link.SupersededByKey
say("failed", link.SupersededByKey)
} else if killed {
if killed {
// **Killed by hand is the outcome, whatever the build was doing** (novox/hq ADR 0219): the
// error it ended with is the kill's consequence, not a fault of the source.
result.Failed = link.KilledByHand
@@ -432,20 +308,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
}
result.Against = built.Against
result.SourceFingerprint = built.Source
result.SourceInputs = built.SourceInputs
// What the walk did instead of building, or why it built (novox/hq ADR 0350 §3, §4).
result.ReuseKey = built.ReuseKey
if built.Reused != nil {
f := built.Reused.From
result.Reused = built.Reused.Words()
result.ReusedFrom = &link.ReusedFrom{Pull: link.PullRef{Owner: f.Pull.Owner, Repo: f.Pull.Repo,
Number: f.Pull.Number}, Head: f.Head, Trunk: f.Trunk, Check: f.Check, Key: f.Key}
} else if request.Reuse != nil {
result.BuiltWhy = built.Why
if result.BuiltWhy == "" {
result.BuiltWhy = "it has no artifact to build"
}
}
result.Trunk, result.OnTrunk, result.Branches = built.Trunk, built.OnTrunk, built.Branches
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
@@ -453,11 +315,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
for _, s := range built.Sources {
result.Sources = append(result.Sources, link.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths})
}
// The list of predefined keys the manifest was judged against, for the controller to publish
// (novox/hq ADR 0348 §7).
if p := built.Predefined; p != nil {
result.Predefined = &link.PredefinedList{Keys: p.Keys, Generation: p.Generation}
}
say("built", built.Manifest.Module+" from "+short(built.Commit))
}
}
@@ -497,19 +354,10 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
spec := builder.CheckSpec{ID: request.ID, Repository: request.Repository, Ref: request.Ref,
Owner: c.Owner, Repo: c.Repo, Number: c.Number, Paths: c.Paths, Beside: map[string]builder.Beside{},
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, SiblingsUnknown: c.SiblingsUnknown}
for _, s := range c.Siblings {
spec.Siblings = append(spec.Siblings, builder.Sibling{Repository: s.Repository, Number: s.Number, Head: s.Head,
Base: s.Base, Dir: s.Dir})
}
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held)}
for dir, b := range c.Beside {
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref}
}
for _, b := range c.Builds {
spec.Builds = append(spec.Builds, builder.CheckBuildSpec{Module: b.Module, Repository: b.Repository, Path: b.Path,
Ref: b.Ref, Merged: b.Merged, Tier: b.Tier, Skip: b.Skip})
}
for _, m := range c.Members {
spec.Group = append(spec.Group, builder.GroupHead{Owner: m.Owner, Repo: m.Repo, Repository: m.Repository,
Ref: m.Ref, Paths: m.Paths, Base: m.Base})
@@ -517,68 +365,6 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
return spec
}
// siblingsOut is the siblings a check read as the outcome carries them (novox/hq ADR 0382 §4).
func siblingsOut(read []builder.JudgedSibling) []link.CheckSibling {
var out []link.CheckSibling
for _, s := range read {
out = append(out, link.CheckSibling{Repository: s.Repository, Number: s.Number, Head: s.Head, Base: s.Base,
BaseCommit: s.BaseCommit})
}
return out
}
// judgeCache is the judge cache the build agent's manifest declares (MESH_JUDGE_CACHE, novox/hq ADR 0358 §6): a
// directory mounted at the same path on the host, so a judge build's mounts name what the agent sees. Empty when the
// agent declares none, and every judge is then built cold under its check's root.
func judgeCache() string { return os.Getenv("MESH_JUDGE_CACHE") }
// triedOf is a try's outcome as its log line says it (novox/hq issue 686): the repository's own check's verdict,
// or the check's where that did not run — a head that does not merge — and the lines naming what failed.
func triedOf(v builder.CheckVerdict) link.Tried {
t := link.Tried{Verdict: v.Verdict, Summary: v.Summary, Took: v.Took.Round(time.Second).String(),
Head: v.Head, Trunk: v.Trunk}
if v.Repo != nil && (v.Gate == nil || v.Gate.Verdict == "pass") {
t.Verdict, t.Summary = v.Repo.Verdict, v.Repo.Summary
}
if v.Repo != nil && v.Repo.Failed != "" {
t.Failed = strings.Split(v.Repo.Failed, "\n")
}
if t.Verdict == "" {
t.Verdict = "error"
}
return t
}
// namedOf are the builds the seat's verdicts name, as the builder believes a scratch record by them (novox/hq ADR
// 0350 §2, issue 703): each module's key with every artifact's digest.
func namedOf(builds []link.CheckedBuild) []builder.NamedBuild {
var out []builder.NamedBuild
for _, b := range builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
out = append(out, named)
}
return out
}
// checkBuiltOf is a check's builds for its merge as the outcome carries them (novox/hq ADR 0350 §1).
func checkBuiltOf(built []builder.CheckBuilt) []link.CheckBuilt {
var out []link.CheckBuilt
for _, b := range built {
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String(),
Kept: b.Kept, Same: b.Same, SameAs: b.SameAs}
for _, m := range b.Made {
c.Made = append(c.Made, link.MadeArtifact{Name: m.Name, Kind: m.Kind, Reference: m.Reference})
}
out = append(out, c)
}
return out
}
// layerOf is one layer of a check as the outcome carries it.
func layerOf(l *builder.Layer) *link.CheckLayer {
if l == nil {
@@ -588,171 +374,79 @@ func layerOf(l *builder.Layer) *link.CheckLayer {
}
// packagesFrom is where a build resolves the mesh's own published packages — the SDK above all
// (novox/hq ADR 0076, issue 053) — on this machine.
// (novox/hq ADR 0076, issue 053).
//
// **The binding wins, then the genesis document** (novox/hq ADR 0336 §3, §4). The build agent's
// npm-package-registry requirement is optional: on genesis's node, before the gitea module is assigned,
// nothing provides it, and the installer sealed the forge it raised to this agent as one document
// instead. Once the requirement binds, the binding names the endpoint the way the artifact store's
// does and a sealed token file the credential, and the genesis document is left unread. A binding
// file named and absent is a requirement not bound here, never a fault: the environment names the
// path whether or not the mesh wrote it.
//
// The environment variables remain for a builder run by a person. With none of the three the result
// is disabled and says this machine, so a build that needs the registry is refused naming it (§6),
// and a build that needs none builds anyway.
func packagesFrom(machine string) (builder.Npmrc, error) {
// Preferably from the mesh: a package-registry binding names the endpoint the way the artifact
// store's binding does, and a sealed token file the credential the way the broker's does. The
// environment variables remain for a builder run by a person, and for the bootstrap, where there is
// no registry yet — there the result is disabled and a build that needs no mesh-published dependency
// builds anyway.
func packagesFrom() (builder.Npmrc, error) {
scope := strings.TrimSpace(os.Getenv("MESH_NPM_SCOPE"))
if scope == "" {
scope = "@novox"
}
none := builder.Npmrc{Machine: machine}
registry := strings.TrimSpace(os.Getenv("MESH_NPM_REGISTRY"))
var username string
if path := strings.TrimSpace(os.Getenv("MESH_PACKAGE_BINDING")); path != "" {
raw, err := os.ReadFile(path)
switch {
case errors.Is(err, os.ErrNotExist):
// Not bound on this machine: the genesis document, or nothing.
case err != nil:
return none, fmt.Errorf("cannot read what the mesh said about the package registry: %w", err)
default:
return fromBinding(path, raw, scope, machine)
}
}
if path := strings.TrimSpace(os.Getenv("MESH_GENESIS_PACKAGE_REGISTRY")); path != "" {
raw, err := os.ReadFile(path)
switch {
case errors.Is(err, os.ErrNotExist):
// Not this mesh's genesis node, or the installer has not sealed it yet.
case err != nil:
return none, fmt.Errorf("cannot read the genesis document for the package registry: %w", err)
default:
return fromGenesis(path, raw, scope, machine)
}
}
// A builder run by a person.
registry := strings.TrimSpace(os.Getenv("MESH_NPM_REGISTRY"))
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
username := strings.TrimSpace(os.Getenv("MESH_NPM_USER"))
if registry == "" && secret == "" {
return none, nil
}
n := builder.Npmrc{Scope: scope, Registry: registry, From: builder.FromEnvironment, Machine: machine}
if username != "" {
n.Username, n.Password = username, secret
} else {
n.Token = secret
}
return n, nil
}
// fromBinding composes the registry from the npm-package-registry binding the mesh wrote and the
// credential it sealed beside it. The mesh authenticates the ordinary way — a generated password the
// provider only applies (novox/hq ADR 0048) — so with a username the secret is a password (basic
// auth); without one it is a bearer token a provider minted.
func fromBinding(path string, raw []byte, scope, machine string) (builder.Npmrc, error) {
none := builder.Npmrc{Machine: machine}
var told struct {
From string `json:"from"`
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil {
return none, fmt.Errorf("%s is not a binding: %w", path, err)
}
if told.At == "" {
return none, fmt.Errorf(
"%s says the package registry is on %q and gives no address for it", path, told.From)
}
// Composed from what the provider serves, so nothing here knows gitea's URL shape from
// another registry's: it states its port, the path its registry answers on, and the scheme.
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
port, ok := told.Serves["port"]
if !ok {
return none, fmt.Errorf("%s says nothing about which port the package registry answers on", path)
}
npmPath, ok := told.Serves["npm-path"]
if !ok {
return none, fmt.Errorf("%s says nothing about the path the package registry answers on", path)
}
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if file := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); file != "" {
raw, err := os.ReadFile(file)
if err != nil {
return none, fmt.Errorf("cannot read this builder's package-registry credential: %w", err)
return builder.Npmrc{}, fmt.Errorf("cannot read what the mesh said about the package registry: %w", err)
}
var told struct {
From string `json:"from"`
At string `json:"at"`
As string `json:"as"`
Serves map[string]any `json:"serves"`
}
if err := json.Unmarshal(raw, &told); err != nil {
return builder.Npmrc{}, fmt.Errorf("%s is not a binding: %w", path, err)
}
if told.At == "" {
return builder.Npmrc{}, fmt.Errorf(
"%s says the package registry is on %q and gives no address for it", path, told.From)
}
// Composed from what the provider serves, so nothing here knows gitea's URL shape from
// another registry's: it states its port, the path its registry answers on, and the scheme.
scheme := "https"
if s, ok := told.Serves["scheme"]; ok {
scheme = fmt.Sprintf("%v", s)
}
port, ok := told.Serves["port"]
if !ok {
return builder.Npmrc{}, fmt.Errorf("%s says nothing about which port the package registry answers on", path)
}
npmPath, ok := told.Serves["npm-path"]
if !ok {
return builder.Npmrc{}, fmt.Errorf("%s says nothing about the path the package registry answers on", path)
}
registry = fmt.Sprintf("%s://%s:%v%v", scheme, told.At, port, npmPath)
username = told.As
}
// The credential the mesh sealed to this machine. The mesh authenticates the ordinary way — a
// generated password the provider only applies (novox/hq ADR 0048) — so with a username this is
// a password (basic auth); without one it is a bearer token a provider minted.
secret := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN"))
if path := strings.TrimSpace(os.Getenv("MESH_NPM_TOKEN_FILE")); path != "" {
raw, err := os.ReadFile(path)
if err != nil {
return builder.Npmrc{}, fmt.Errorf("cannot read this builder's package-registry credential: %w", err)
}
secret = strings.TrimSpace(string(raw))
}
n := builder.Npmrc{Scope: scope, Registry: fmt.Sprintf("%s://%s:%v%v", scheme, told.At, port, npmPath),
From: builder.FromBinding, Machine: machine}
if told.As != "" {
n.Username, n.Password = told.As, secret
} else {
n.Token = secret
if u := strings.TrimSpace(os.Getenv("MESH_NPM_USER")); u != "" {
username = u
}
return n, nil
}
// GenesisRegistry is the genesis document (novox/hq ADR 0336 §3): what the installer sealed to the
// build agent on its node at genesis's packages step, as the `genesis-package-registry` own-secret —
// the forge it raised, the port this node gave the package registry, its npm path, and the account it
// made with that account's password. mesh-host writes it; the field names are the two repositories'
// agreement.
type GenesisRegistry struct {
Scheme string `json:"scheme"`
At string `json:"at"`
Port int `json:"port"`
NpmPath string `json:"npm-path"`
As string `json:"as"`
Password string `json:"password"`
}
// fromGenesis composes the registry from the genesis document.
func fromGenesis(path string, raw []byte, scope, machine string) (builder.Npmrc, error) {
none := builder.Npmrc{Machine: machine}
var g GenesisRegistry
if err := json.Unmarshal(raw, &g); err != nil {
return none, fmt.Errorf("%s is not the genesis document for the package registry: %w", path, err)
if registry == "" && secret == "" {
return builder.Npmrc{}, nil
}
var missing []string
for name, value := range map[string]string{"at": g.At, "npm-path": g.NpmPath, "as": g.As, "password": g.Password} {
if strings.TrimSpace(value) == "" {
missing = append(missing, name)
}
if username != "" {
return builder.Npmrc{Scope: scope, Registry: registry, Username: username, Password: secret}, nil
}
if g.Port <= 0 {
missing = append(missing, "port")
}
if len(missing) > 0 {
sort.Strings(missing)
return none, fmt.Errorf("%s, the genesis document for the package registry, says nothing of %s",
path, strings.Join(missing, ", "))
}
scheme := g.Scheme
if scheme == "" {
scheme = "http"
}
return builder.Npmrc{Scope: scope, Registry: fmt.Sprintf("%s://%s:%d%s", scheme, g.At, g.Port, g.NpmPath),
Username: g.As, Password: g.Password, From: builder.FromGenesis, Machine: machine}, nil
}
// saysRegistry is the line each build's log carries about the package registry (novox/hq ADR 0336
// §4): which of the binding and the genesis document it resolved from, or that it has neither here.
func saysRegistry(n builder.Npmrc) string {
if n.Enabled() {
return "the package registry is resolved from " + n.From
}
machine := n.Machine
if machine == "" {
machine = "this machine"
}
return fmt.Sprintf("no package registry: %s is unbound on %s and no genesis document is here; "+
"a build needing it is refused", builder.RegistryRequirement, machine)
return builder.Npmrc{Scope: scope, Registry: registry, Token: secret}, nil
}
// forgeFrom is the git credential this builder may offer a clone, composed from the same binding
cmd/mesh-builder/once.go
+1 -6
View File
@@ -84,15 +84,10 @@ func buildOnce(ctx context.Context, args []string) error {
}
fmt.Fprintln(os.Stderr)
machine := os.Getenv("MESH_NODE")
if machine == "" {
machine, _ = os.Hostname()
}
npmrc, err := packagesFrom(machine)
npmrc, err := packagesFrom()
if err != nil {
return err
}
fmt.Fprintf(os.Stderr, " [packages] %s\n", saysRegistry(npmrc))
built, buildErr := builder.Build(ctx, builder.Command, publisher, repository, *path, *ref, where, bases, npmrc,
forgeFrom(),
func(step, message string) { fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message) })
cmd/mesh-builder/packages_test.go
-102
View File
@@ -1,102 +0,0 @@
package main
import (
"os"
"path/filepath"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/builder"
)
// The build agent's package registry (novox/hq ADR 0336, issue 121): its npm-package-registry
// requirement is optional, so the binding may be absent, and on genesis's node the installer seals
// the forge it raised as one genesis document instead.
// registryFiles names the three paths the build agent's environment always names, writing only those
// given.
func registryFiles(t *testing.T, binding, token, genesis string) {
t.Helper()
dir := t.TempDir()
for env, content := range map[string]string{
"MESH_PACKAGE_BINDING": binding,
"MESH_NPM_TOKEN_FILE": token,
"MESH_GENESIS_PACKAGE_REGISTRY": genesis,
} {
path := filepath.Join(dir, env)
t.Setenv(env, path)
if content != "" {
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
}
}
for _, env := range []string{"MESH_NPM_REGISTRY", "MESH_NPM_TOKEN", "MESH_NPM_USER", "MESH_NPM_SCOPE"} {
t.Setenv(env, "")
}
}
const (
aBinding = `{"from":"gitea","at":"anchor.mesh","as":"anchor-agent",
"serves":{"scheme":"http","port":3000,"npm-path":"/api/packages/novox/npm/"}}`
aGenesisDocument = `{"scheme":"http","at":"127.0.0.1","port":3100,"npm-path":"/api/packages/novox/npm/",
"as":"mesh-builder","password":"genesis-secret"}`
)
func TestUntilTheRequirementBindsTheGenesisDocumentIsRead(t *testing.T) {
registryFiles(t, "", "", aGenesisDocument)
n, err := packagesFrom("anchor")
if err != nil {
t.Fatalf("an absent binding beside a genesis document was refused: %v", err)
}
if n.From != builder.FromGenesis || n.Registry != "http://127.0.0.1:3100/api/packages/novox/npm/" ||
n.Username != "mesh-builder" || n.Password != "genesis-secret" || n.Scope != "@novox" {
t.Fatalf("the genesis document was not what the registry came from: %+v", n)
}
file, err := n.File()
if err != nil || !strings.Contains(file, "//127.0.0.1:3100/api/packages/novox/npm/:username=mesh-builder") {
t.Fatalf("the .npmrc is not written from the genesis document: %v\n%s", err, file)
}
if got := saysRegistry(n); !strings.Contains(got, "genesis document") {
t.Errorf("the log does not say the genesis document: %s", got)
}
}
func TestARealBindingWinsOverTheGenesisDocument(t *testing.T) {
registryFiles(t, aBinding, "bound-secret\n", aGenesisDocument)
n, err := packagesFrom("anchor")
if err != nil {
t.Fatal(err)
}
if n.From != builder.FromBinding || n.Registry != "http://anchor.mesh:3000/api/packages/novox/npm/" ||
n.Username != "anchor-agent" || n.Password != "bound-secret" {
t.Fatalf("the binding did not win over the genesis document: %+v", n)
}
if got := saysRegistry(n); !strings.Contains(got, "binding") {
t.Errorf("the log does not say the binding: %s", got)
}
}
func TestWithNeitherTheAgentHasNoRegistryAndSaysWhere(t *testing.T) {
registryFiles(t, "", "", "")
n, err := packagesFrom("workstation")
if err != nil {
t.Fatalf("an unbound optional requirement was a fault: %v", err)
}
if n.Enabled() || n.Machine != "workstation" {
t.Fatalf("a registry came from nowhere: %+v", n)
}
got := saysRegistry(n)
for _, want := range []string{"npm-package-registry", "unbound", "workstation"} {
if !strings.Contains(got, want) {
t.Errorf("the log does not say %q: %s", want, got)
}
}
}
func TestAGenesisDocumentMissingAFieldIsRefusedByName(t *testing.T) {
registryFiles(t, "", "", `{"at":"127.0.0.1","port":3000,"npm-path":"/api/packages/novox/npm/","as":"mesh-builder"}`)
if _, err := packagesFrom("anchor"); err == nil || !strings.Contains(err.Error(), "password") {
t.Fatalf("a genesis document with no password was taken: %v", err)
}
}
cmd/mesh-builder/ready.go
-44
View File
@@ -1,44 +0,0 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
)
// **The build agent says it is ready** (novox/hq issue 544). It listens on nothing, so no http or tcp
// check can see it work; it writes a mark once it is bound to its worker and pulls build work, and its
// container's health check runs `mesh-builder ready`, which passes only while the mark is there. Without
// it the node-engine could judge the agent only alive, and every walk that moved it waited out the whole
// grace on each machine.
//
// The mark lives in the container's own temporary directory, never in the workspace the machine keeps:
// a new container starts without it, and a restarted one removes it before it binds again.
func readyMark() string { return filepath.Join(os.TempDir(), "mesh-builder.ready") }
// notReady removes the mark: a holder starting is not taking work until it says so again.
func notReady() error {
if err := os.Remove(readyMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove the mark that this agent takes build work (%s): %w", readyMark(), err)
}
return nil
}
// taking writes the mark. A mark that cannot be written is said, and the agent goes on building: its
// check then says it is not ready, which is the truth the gate should hear.
func taking(seat string) {
if err := os.WriteFile(readyMark(), []byte(seat+"\n"), 0o644); err != nil {
fmt.Fprintf(os.Stderr, "cannot mark that this agent takes build work (%s): %v\n", readyMark(), err)
return
}
fmt.Fprintf(os.Stderr, "taking build work from %s\n", seat)
}
// ready is the health check: nil when this container's agent is bound and pulling build work.
func ready() error {
if _, err := os.Stat(readyMark()); err != nil {
return fmt.Errorf("not taking build work yet: %s is not there", readyMark())
}
return nil
}
cmd/mesh-builder/ready_test.go
-32
View File
@@ -1,32 +0,0 @@
package main
import (
"os"
"testing"
)
// The health check passes only between the agent saying it takes work and its next start (novox/hq
// issue 544).
func TestTheAgentIsReadyOnlyOnceItTakesWork(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := ready(); err == nil {
t.Fatal("an agent that never took work is ready")
}
taking("mesh-build-machine")
if err := ready(); err != nil {
t.Fatalf("an agent taking work is not ready: %v", err)
}
// Started again, it is not ready until it binds again.
if err := notReady(); err != nil {
t.Fatal(err)
}
if err := ready(); err == nil {
t.Fatal("an agent starting again is ready before it takes work")
}
if err := notReady(); err != nil {
t.Fatalf("removing a mark that is not there failed: %v", err)
}
if _, err := os.Stat(readyMark()); err == nil {
t.Fatal("the mark is still there")
}
}
cmd/mesh-builder/stdout_test.go
-1
View File
@@ -16,7 +16,6 @@ func TestBuilderDiagnosticsStayOffStdout(t *testing.T) {
"version)": true, // --version
`"stopping")`: true, // the loop.s shutdown line
"usage)": true, // --help text, for a human
"said)": true, // busy: the answer its node-engine reads (novox/hq ADR 0335)
}
for _, file := range []string{"once.go", "main.go"} {
src, err := os.ReadFile(file)
cmd/mesh-builder/tried_test.go
-31
View File
@@ -1,31 +0,0 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// A try's outcome, as the build seat says it in the build's log (novox/hq issue 686): the repository's own
// check's verdict, how long it took, and each line naming what failed.
func TestATrysOutcomeNamesWhatFailed(t *testing.T) {
tried := triedOf(builder.CheckVerdict{Verdict: "pass", Summary: "the change touches no module", Took: 303 * time.Second,
Head: "c0ffee", Trunk: "ca3f43c", Gate: &builder.Layer{Verdict: "pass"},
Repo: &builder.Layer{Verdict: "fail", Summary: "its merge-check.sh failed: TestX",
Failed: "--- FAIL: TestX (0.01s)\n x_test.go:12: got 1, want 2"}})
if tried.Verdict != "fail" || tried.Took != "5m3s" || len(tried.Failed) != 2 || tried.Trunk != "ca3f43c" ||
tried.Summary != "its merge-check.sh failed: TestX" {
t.Fatalf("a failed try: %+v", tried)
}
// A head that does not merge fails both layers: the check's own word.
conflict := triedOf(builder.CheckVerdict{Verdict: "fail", Summary: "does not merge", Gate: &builder.Layer{Verdict: "fail"},
Repo: &builder.Layer{Verdict: "fail", Summary: "does not merge"}})
if conflict.Verdict != "fail" || conflict.Summary != "does not merge" {
t.Fatalf("a try that does not merge: %+v", conflict)
}
// No verdict is an error, never a pass.
if none := triedOf(builder.CheckVerdict{}); none.Verdict != "error" {
t.Fatalf("a try with no verdict: %+v", none)
}
}
cmd/mesh-controller/accept_outside.go
-147
View File
@@ -1,147 +0,0 @@
package main
import (
"context"
"fmt"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A provision whose credential only a person outside the mesh can give (novox/hq ADR 0337, issue 164, issue 165):
// given once for its provider, at the controller's terminal, and sealed in one act to every consumer that binds the
// provision from that provider now. The value is never read back, printed or kept.
// outsideOffer is the provider module's own name when `secret accept <node> <module> <name>` names a provision that
// module offers with a credential issued outside the mesh, and "" when it names anything else (an own secret, or a
// pair with --provider).
func outsideOffer(ctx context.Context, module, name, provider string) (string, error) {
if provider != "" {
return "", nil
}
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
if m, known := shelf[module]; known && m.IssuedOutsideOf(name) {
return module, nil
}
return "", nil
}
// offeredOutsideBy is a module that offers the provision with a credential issued outside the mesh, or "".
func offeredOutsideBy(ctx context.Context, provision string) (string, error) {
open, err := openStores(ctx)
if err != nil {
return "", err
}
defer open.Close()
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
module, _ := catalogue.OffersIssuedOutside(shelf, provision)
return module, nil
}
// consumersFrom is every consumer that binds a provision from one provider now, found by resolving every machine as
// its next send would, and the machines that could not be resolved, with why.
func consumersFrom(ctx context.Context, open *stores, provider, providerModule, provision string) (
[]inventory.PairEnd, []string, error) {
// One reading of the mesh for every machine this looks at.
ctx = withInputCache(ctx)
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, nil, err
}
var out []inventory.PairEnd
var unresolved []string
for _, n := range nodes {
plan, _, err := resolveNode(ctx, open, n.Name)
if err != nil {
unresolved = append(unresolved, fmt.Sprintf("%s: %s", n.Name, oneLine(err.Error())))
continue
}
out = append(out, bindsFrom(plan, n.Name, provider, providerModule, provision)...)
}
return out, unresolved, nil
}
// bindsFrom is every consumer on one machine whose resolution answers the provision from that provider. Pure.
func bindsFrom(plan catalogue.Resolution, machine, provider, providerModule, provision string) []inventory.PairEnd {
var out []inventory.PairEnd
for _, need := range plan.Needs {
if need.ByRecord || need.Name != provision || need.From != provider {
continue
}
if need.Module != "" && need.Module != providerModule {
continue
}
out = append(out, inventory.PairEnd{Consumer: machine, ConsumerModule: need.For, Local: need.Local})
}
sort.Slice(out, func(i, j int) bool {
return out[i].ConsumerModule+"\x00"+out[i].Local < out[j].ConsumerModule+"\x00"+out[j].Local
})
return out
}
// acceptForProvision gives a provision's credential issued outside the mesh, once for its provider (ADR 0337 §3).
func acceptForProvision(ctx context.Context, provider, providerModule, provision, value string) error {
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
consumers, unresolved, err := consumersFrom(ctx, open, provider, providerModule, provision)
if err != nil {
return err
}
accepted, err := open.inventory.AcceptSecretForProvision(ctx, provision, provider, providerModule, value, consumers)
value = ""
if err != nil {
return err
}
fmt.Print(acceptedSaid(provider, providerModule, provision, accepted, unresolved))
if len(accepted.Sealed) == 0 {
return nil
}
if err := sendTo(ctx, open, accepted.Machines); err != nil {
return fmt.Errorf("%s is sealed and could not be sent: %w; `push %s` sends it", provision, err,
strings.Join(accepted.Machines, "`, `push "))
}
fmt.Printf(" sent %s; each consumer starts again on it\n", strings.Join(accepted.Machines, ", "))
return nil
}
// acceptedSaid is what one accept for a provider did, in words that never carry the value. Pure.
func acceptedSaid(provider, providerModule, provision string, a inventory.ProvisionAccepted, unresolved []string) string {
var b strings.Builder
if len(a.Sealed) == 0 {
fmt.Fprintf(&b, "%s from %s on %s: no consumer binds it now, so nothing was sealed; a consumer that binds "+
"later waits for it, and this line given again seals it to every consumer then\n", provision,
providerModule, provider)
} else {
var names []string
for _, c := range a.Sealed {
names = append(names, fmt.Sprintf("%s on %s%s", c.ConsumerModule, c.Consumer, asLocal(c.Local)))
}
fmt.Fprintf(&b, "%s from %s on %s is given, sealed to %s and to %s\n", provision, providerModule, provider,
strings.Join(names, ", "), provider)
fmt.Fprintf(&b, " the mesh cannot read it back, never makes one of its own for it, and will not rotate it\n")
}
for _, c := range a.Unsealable {
fmt.Fprintf(&b, " not sealed to %s on %s: no sealing key on that machine; give it again once it has one\n",
c.ConsumerModule, c.Consumer)
}
for _, u := range unresolved {
fmt.Fprintf(&b, " not known whether a consumer on %s binds it, which did not resolve; give it again once it does\n", u)
}
return b.String()
}
cmd/mesh-controller/acting.go
+3 -11
View File
@@ -26,11 +26,8 @@ import (
//
// - **The serving controller** takes it before it does anything else — before it asserts the bus's
// objects, which are the controller's to write — waiting while another holds it, and renews it.
// Another holder seen, or the key's age passed with no renewal made, is the lease lost: the gate
// closes at once and the process exits, so its service manager restarts it as a candidate (serve,
// in push.go). A renewal the bus does not answer in time is not a loss (novox/hq issue 477): the
// lease is kept, no new act starts once half the key's age passed unrenewed, and the next renewal
// made opens the gate again.
// A renewal refused or failed is the lease lost: the gate closes at once and the process exits, so
// its service manager restarts it as a candidate (serve, in push.go).
// - **A command run at a shell** — `push` in the installer, the lab, a person repairing a mesh whose
// controller is down (issue 201) — acts **under the holder's epoch** when a controller holds the
// lease: it is the same mesh's word, composed and sent under the store's hold of each machine like
@@ -67,11 +64,6 @@ type actor struct {
// the highest epoch issued, and what was said of it; zero when it was not (S12).
reset time.Time
resetSaid string
// renewEvery and poll, when set, are how often the serving controller renews the lease it holds and
// looks again at one another holds, in place of the lease's own: a test's, so two controllers hand the
// lease over in a second rather than in a renewal of five (novox/hq issue 591). A renewal is given
// the lease's own RenewEvery to answer however often it is made, so a slow bus does not lose it.
renewEvery, poll time.Duration
}
// theLease is this process's standing to the lease.
@@ -231,7 +223,7 @@ func (a *actor) serveUnderTheLease(ctx context.Context, inv *inventory.Inventory
}
say := func(format string, args ...any) { fmt.Printf(format+"\n", args...) }
l, err := lease.Open(ctx, api, broker.LeaseBucket, lease.Options{Holder: holderOf(instance),
RenewEvery: a.renewEvery, RenewWithin: lease.RenewEvery, Poll: a.poll, Floor: inv.HighestEpoch, Say: say, Health: controllerHealth, Moved: func(was, floor uint64) {
Floor: inv.HighestEpoch, Say: say, Health: controllerHealth, Moved: func(was, floor uint64) {
a.mu.Lock()
defer a.mu.Unlock()
a.reset = time.Now()
cmd/mesh-controller/acts.go
+9 -16
View File
@@ -10,7 +10,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
@@ -203,9 +203,8 @@ func seatDependenciesOnAssign(ctx context.Context, open *stores, node string, mo
if _, err := catalogue.AssignRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err
}
// Two modules declaring one path or unit, or one package present and absent, is refused before anything
// is recorded (novox/hq ADR 0210, ADR 0303, 04-ISSUES/235): kept, the node would not resolve until one came
// off again. Two modules declaring one package present is not a collision (ADR 0303, issue 628).
// Two modules declaring one package, path or unit is refused before anything is recorded
// (novox/hq ADR 0210, 04-ISSUES/235): kept, the node would not resolve until one came off again.
if err := catalogue.CollisionRefusal(shelf, node, assigned, adding); err != nil {
return nil, nil, err
}
@@ -314,7 +313,7 @@ type afterAct struct {
// after the write, which can only find more unmet than the closure would: a seat said held there is
// held in the closure too.
func resolvedAfter(ctx context.Context, open *stores, shelf map[string]catalogue.Manifest, node string) afterAct {
plan, _, err := resolveNode(ctx, open, node)
plan, _, err := planFor(ctx, open, node)
if err == nil {
out := afterAct{plan: plan, unheld: plan.Unheld}
for _, m := range plan.Modules {
@@ -355,8 +354,6 @@ func splitModules(field string) []string {
// Nothing here can fail the act it reports on. A mesh that cannot be read is worth saying and is
// not a reason to claim the assignment did not happen — it did.
func blockedElsewhere(ctx context.Context, open *stores, except string) string {
// One reading of the mesh for every machine this looks at.
ctx = withInputCache(ctx)
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return "\n\nThe rest of the mesh could not be checked: " + err.Error()
@@ -366,7 +363,7 @@ func blockedElsewhere(ctx context.Context, open *stores, except string) string {
if n.Name == except {
continue
}
if _, _, err := resolveNode(ctx, open, n.Name); err != nil {
if _, _, err := planFor(ctx, open, n.Name); err != nil {
blocked[n.Name] = err.Error()
}
}
@@ -393,10 +390,10 @@ func blockedElsewhere(ctx context.Context, open *stores, except string) string {
}
// issueOnAssign gives a newly assigned module its bus credential, the way `module issue` does, and
// says what it did in one line. Nothing for a module that declares no broker secret and runs no
// provisioner client; nothing for one whose user is already minted (a credential is rotated on
// purpose, never by re-assigning); and when the bus cannot be reached from here, the line names the
// verb and the push that would refuse the module until it is run — never a silent placeholder (novox/hq issue 203).
// says what it did in one line. Nothing for a module that declares no broker secret; nothing for one
// whose user is already minted (a credential is rotated on purpose, never by re-assigning); and when
// the bus cannot be reached from here, the line names the verb and the push that would refuse the
// module until it is run — never a silent placeholder (novox/hq issue 203).
func issueOnAssign(ctx context.Context, open *stores, node, module string) string {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
@@ -408,10 +405,6 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return ""
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, reads := m.OwnSecrets["broker"]; !reads {
// A module running the provisioner client alone is minted the client's account (issue 917).
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
}
if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted {
return ""
}
cmd/mesh-controller/acts_test.go
+1 -3
View File
@@ -1,12 +1,10 @@
//go:build integration
package main
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
)
// What an assignment says about the machines it was not about.
cmd/mesh-controller/addresses_integration_test.go
-228
View File
@@ -1,228 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
)
func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aStore())
if _, err := assign(ctx, open, "anchor", "distribution"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5101}}); err != nil {
t.Fatal(err)
}
// A module the mesh built, recorded by digest and path, running on the other machine.
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "app", Commit: "abc",
Made: []inventory.Artifact{{Name: "server", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "app/server@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "app", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-app",
"image": catalogue.ArtifactStoreScheme + "app/server@" + aDigest}}})
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
// The runtime's trust is the runtime's module's to write (novox/hq ADR 0222): a stand-in for it
// asks where this machine reaches the store, as the docker module does.
register(t, open, catalogue.Manifest{Module: "runtime", Version: "1",
Resources: []map[string]any{{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json",
"into": "json", "content": `{"insecure-registries": ["${seat:mesh-artifact-store:reach}"]}` + "\n"}}})
if _, err := assign(ctx, open, "laptop", "runtime"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
if err != nil || !found || node != "anchor" || port != "5101" {
t.Fatalf("the store is found on %q:%q (%v, %v); the node put it on 5101", node, port, found, err)
}
var trust string
for _, r := range composed(t, open, "laptop").Resources {
if r["path"] == "/etc/docker/daemon.json" {
trust, _ = r["content"].(string)
}
if r["id"] == "app.server" && r["image"] != "anchor.internal:5101/app/server@"+aDigest {
t.Errorf("the image the mesh built is fetched as %v", r["image"])
}
}
if !strings.Contains(trust, "anchor.internal:5101") || strings.Contains(trust, ":5000") {
t.Fatalf("the runtime is told to trust %q; the node put the store on 5101", trust)
}
// And a replay to the catalogue says where the store is now.
announced, err := following{open: open}.Announceable(ctx)
if err != nil {
t.Fatal(err)
}
if len(announced) != 1 || announced[0].Made[0].Reference != "anchor.internal:5101/app/server@"+aDigest {
t.Fatalf("the replay announces %+v", announced)
}
}
func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "controller", Kind: catalogue.ArtifactBundle,
Reference: "https://registry.example/mesh-controller/controller.tar.gz", Digest: aDigest}})
if err != nil {
t.Fatal(err)
}
register(t, open, control)
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}, "image": "pg@" + aDigest}}})
register(t, open, catalogue.Manifest{Module: "lavinmq", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5671, From: catalogue.FromMesh},
{Port: 5672, From: catalogue.FromMesh}},
Guards: []int{15672},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}, "image": "mq@" + aDigest}}})
// The control plane's own bus user is the installer's, seeded at genesis before the controller
// runs (SeedBusUser); without it a push now refuses the credential nobody issued (issue 203).
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.mesh-controller",
Kind: inventory.BusController, Node: "anchor", Module: "mesh-controller"}, "bootstrap"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The store's module is registered and given its port, and NOT assigned: the state genesis
// leaves a given-port node in before the foundation is adopted as modules (04-ISSUES/085).
if err := open.inventory.SetSettings(ctx, "anchor", "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 6852}}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "lavinmq"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "lavinmq",
map[string]any{catalogue.PortsSetting: map[string]any{"5672": 5679}}); err != nil {
t.Fatal(err)
}
var env map[string]any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "mesh-controller.controller" {
env, _ = r["env"].(map[string]any)
}
}
if env == nil {
t.Fatal("the control plane's process is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
// Neither given nor assigned by the mesh: the manifest's own number is NOT the answer,
// because the sealed value beside it carries the port genesis wrote (finding F3).
"MESH_BROKER_ADDRESS_PORT": "",
"MESH_BROKER_MANAGEMENT_PORT": "",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node says %q", key, env[key], want)
}
}
}
func aLoneNode(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t)
licences.ForTest(t)
open, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(open.Close)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
}
}
record, err := open.inventory.AddNode(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
reported, _ := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}}})
var profile map[string]any
_ = json.Unmarshal(reported, &profile)
if err := open.inventory.RecordProfile(t.Context(), record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
return open
}
func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
open := aLoneNode(t)
ctx := t.Context()
register(t, open, aStore())
register(t, open, catalogue.Manifest{Module: "builder", Version: "1",
Requires: []string{catalogue.ArtifactStoreProvision},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-builder",
"image": "registry.example/mesh-builder@" + aDigest}}})
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "postgres", Commit: "abc",
Made: []inventory.Artifact{{Name: "runtime", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-postgres",
"image": catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}}})
for _, module := range []string{"distribution", "builder", "postgres"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5100}}); err != nil {
t.Fatal(err)
}
var image any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "postgres.runtime" {
image = r["image"]
}
}
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
cmd/mesh-controller/addresses_test.go
+214 -1
View File
@@ -2,10 +2,13 @@ package main
import (
"encoding/json"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
"github.com/novox/mesh-controller/internal/link"
)
@@ -83,12 +86,150 @@ func aStore() catalogue.Manifest {
// **The trust a machine writes for the store, and the address every built image is fetched
// through, say the port the node gave the store** — not the catalogue's number.
func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aStore())
if _, err := assign(ctx, open, "anchor", "distribution"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5101}}); err != nil {
t.Fatal(err)
}
// A module the mesh built, recorded by digest and path, running on the other machine.
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "app", Commit: "abc",
Made: []inventory.Artifact{{Name: "server", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "app/server@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "app", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-app",
"image": catalogue.ArtifactStoreScheme + "app/server@" + aDigest}}})
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
// The runtime's trust is the runtime's module's to write (novox/hq ADR 0222): a stand-in for it
// asks where this machine reaches the store, as the docker module does.
register(t, open, catalogue.Manifest{Module: "runtime", Version: "1",
Resources: []map[string]any{{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json",
"into": "json", "content": `{"insecure-registries": ["${seat:mesh-artifact-store:reach}"]}` + "\n"}}})
if _, err := assign(ctx, open, "laptop", "runtime"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
if err != nil || !found || node != "anchor" || port != "5101" {
t.Fatalf("the store is found on %q:%q (%v, %v); the node put it on 5101", node, port, found, err)
}
var trust string
for _, r := range composed(t, open, "laptop").Resources {
if r["path"] == "/etc/docker/daemon.json" {
trust, _ = r["content"].(string)
}
if r["id"] == "app.server" && r["image"] != "anchor.internal:5101/app/server@"+aDigest {
t.Errorf("the image the mesh built is fetched as %v", r["image"])
}
}
if !strings.Contains(trust, "anchor.internal:5101") || strings.Contains(trust, ":5000") {
t.Fatalf("the runtime is told to trust %q; the node put the store on 5101", trust)
}
// And a replay to the catalogue says where the store is now.
announced, err := following{open}.Announceable(ctx)
if err != nil {
t.Fatal(err)
}
if len(announced) != 1 || announced[0].Made[0].Reference != "anchor.internal:5101/app/server@"+aDigest {
t.Fatalf("the replay announces %+v", announced)
}
}
// **The control plane's own connections say the port the node gave the store and the broker.**
//
// Composed from the control plane's own manifest against a real inventory: the store's module is
// given 6852 on this node the way genesis or an operator gives it, and the control plane's
// process is told so beside the sealed connection genesis wrote.
func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "controller", Kind: catalogue.ArtifactBundle,
Reference: "https://registry.example/mesh-controller/controller.tar.gz", Digest: aDigest}})
if err != nil {
t.Fatal(err)
}
register(t, open, control)
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}, "image": "pg@" + aDigest}}})
register(t, open, catalogue.Manifest{Module: "lavinmq", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5671, From: catalogue.FromMesh},
{Port: 5672, From: catalogue.FromMesh}},
Guards: []int{15672},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}, "image": "mq@" + aDigest}}})
// The control plane's own bus user is the installer's, seeded at genesis before the controller
// runs (SeedBusUser); without it a push now refuses the credential nobody issued (issue 203).
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.mesh-controller",
Kind: inventory.BusController, Node: "anchor", Module: "mesh-controller"}, "bootstrap"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The store's module is registered and given its port, and NOT assigned: the state genesis
// leaves a given-port node in before the foundation is adopted as modules (04-ISSUES/085).
if err := open.inventory.SetSettings(ctx, "anchor", "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 6852}}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "lavinmq"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "lavinmq",
map[string]any{catalogue.PortsSetting: map[string]any{"5672": 5679}}); err != nil {
t.Fatal(err)
}
var env map[string]any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "mesh-controller.controller" {
env, _ = r["env"].(map[string]any)
}
}
if env == nil {
t.Fatal("the control plane's process is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
// Neither given nor assigned by the mesh: the manifest's own number is NOT the answer,
// because the sealed value beside it carries the port genesis wrote (finding F3).
"MESH_BROKER_ADDRESS_PORT": "",
"MESH_BROKER_MANAGEMENT_PORT": "",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node says %q", key, env[key], want)
}
}
}
// withSeatPorts is the control plane's manifest with the seat placeholders in its environment —
// added here until module.json carries them (the manifest lands one commit after the code that
@@ -133,7 +274,79 @@ func withSeatPorts(m catalogue.Manifest) catalogue.Manifest {
// aLoneNode is one capable machine with nothing placed on any network — the control-node during
// genesis, before the "network" step, which is after the store, the broker, the vault and the
// catalogue have each been built and pushed (finding F2).
func aLoneNode(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t)
licences.ForTest(t)
open, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(open.Close)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
}
}
record, err := open.inventory.AddNode(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
reported, _ := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}}})
var profile map[string]any
_ = json.Unmarshal(reported, &profile)
if err := open.inventory.RecordProfile(t.Context(), record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
return open
}
// **Before the network exists, the store's own node reaches it by loopback** — never refused,
// never handed the scheme: a genesis pushes the store, the broker, the vault and the catalogue to
// a node on no network, and builds the catalogue on a base it must be able to pull.
func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
open := aLoneNode(t)
ctx := t.Context()
register(t, open, aStore())
register(t, open, catalogue.Manifest{Module: "builder", Version: "1",
Requires: []string{catalogue.ArtifactStoreProvision},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-builder",
"image": "registry.example/mesh-builder@" + aDigest}}})
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "postgres", Commit: "abc",
Made: []inventory.Artifact{{Name: "runtime", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-postgres",
"image": catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}}})
for _, module := range []string{"distribution", "builder", "postgres"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5100}}); err != nil {
t.Fatal(err)
}
var image any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "postgres.runtime" {
image = r["image"]
}
}
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
cmd/mesh-controller/adopting_test.go
+2 -4
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
@@ -12,10 +10,10 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
)
// novox/hq ADR 0100: taking a module is its cutover; converging a node is one act, previewed, and
cmd/mesh-controller/adoption.go
+10 -31
View File
@@ -12,10 +12,9 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// A node is adopted or converged (novox/hq ADR 0100), and it is said to be adopted wherever the
@@ -244,7 +243,7 @@ func take(ctx context.Context, open *stores, node, module string, opts takeOptio
return "", err
}
if !slices.Contains(assigned, module) {
if plan, _, err := resolveNode(ctx, open, node); err == nil {
if plan, _, err := planFor(ctx, open, node); err == nil {
if why, runs := plan.Because[module]; runs {
return "", fmt.Errorf("%w: %s runs on %s because %s — assign it to %s to take it",
inventory.ErrNotAssigned, module, node, why, node)
@@ -425,9 +424,8 @@ func comparisonOf(module string, c comparison, opts takeOptions) (preview string
}
// And its settings on this machine, composed against its definition (rule 1, rule 6).
for _, layer := range c.layers {
said := layer.Whole()
keys := make([]string, 0, len(said))
for k := range said {
keys := make([]string, 0, len(layer.Values))
for k := range layer.Values {
keys = append(keys, k)
}
sort.Strings(keys)
@@ -650,7 +648,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
if err != nil {
return "", err
}
plan, _, err := resolveNode(ctx, open, node)
plan, settings, err := planFor(ctx, open, node)
if err != nil {
return "", err
}
@@ -707,19 +705,14 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
filter)
}
// What the filter is derived from: the rendering of this machine as a question reads it, with the private network.
input, err := inputFor(ctx, open)
gens, err := generators(ctx, open)
if err != nil {
return "", err
}
rendered, err := compose.Render(input.Input, node, compose.Reading)
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Reading)
if err != nil {
return "", err
}
with := rendered.Rendering
if with.Generators, err = compose.Generators(input.Input); err != nil {
return "", err
}
rules, err := plan.Rules(with)
if err != nil {
return "", err
@@ -729,7 +722,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
return "", err
}
derived := derivedFilter{rules: rules, foundation: with.Foundation, mesh: with.Mesh,
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks, local: with.LocalNetworks}
outward: plan.PublicDomain != "", outwardLinks: with.OutwardLinks}
filtering, err := inv.FilteringOf(ctx, node)
if err != nil {
return "", err
@@ -782,7 +775,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
if _, err := inv.Assign(ctx, node, filter); err != nil {
return "", err
}
if _, _, err := resolveNode(ctx, open, node); err != nil {
if _, _, err := planFor(ctx, open, node); err != nil {
_ = inv.Unassign(ctx, node, filter)
return "", fmt.Errorf("%s cannot run %s, so it was not converged: %w", node, filter, err)
}
@@ -937,9 +930,6 @@ type derivedFilter struct {
// The filter constrains what arrives on them; everything arriving elsewhere is this machine's
// own guest and is not filtered.
outwardLinks []string
// local is the machine's local networks, which a port from `local-network` stays open to (novox/hq
// ADR 0340).
local []string
}
// closesOutside is what a narrowing from everywhere to the private network is called: it closes.
@@ -987,17 +977,6 @@ func (d derivedFilter) fate(r inventory.Reach) string {
if !onMesh {
return fmt.Sprintf("%s — declared by %s from the mesh only", closesOutside, by)
}
case catalogue.FromLocalNetwork:
// The private network and this machine's local networks (novox/hq ADR 0340): it closes to
// everything else outside, and says which local networks stay.
if !onMesh {
if len(d.local) == 0 {
return fmt.Sprintf("%s — declared by %s from the local network, and this machine "+
"reported none", closesOutside, by)
}
return fmt.Sprintf("WILL CLOSE to everything outside the private network and %s — declared "+
"by %s from the local network", strings.Join(d.local, ", "), by)
}
}
return fmt.Sprintf("declared by %s (from %s)", by, rule.From)
}
cmd/mesh-controller/agent_account_integration_test.go
-161
View File
@@ -1,161 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-host/rootsearch"
)
func TestTheSelfCheckSaysAnAgentAccountThatCanBecomeRoot(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.NodeByName(ctx, n); err != nil {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAccount(ctx, n, "ops", ""); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
found = onlyMachine(found, "anchor")
if len(found) != 1 || found[0].Machine != "anchor" || found[0].Severity != conditions.Urgent ||
!strings.Contains(found[0].Said, "not judged") {
t.Fatalf("a named agent account with no verdict: %+v", found)
}
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if w.Headline == "" || w.Needs == "" || w.Resolved == "" {
t.Errorf("the condition has no plain words: %+v", w)
}
healthy := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: link.StateHealthy, Root: link.RootNever, Account: "agent"}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{healthy}}); err != nil {
t.Fatal(err)
}
if found, err = probeAgentAccounts(ctx, d); err != nil || len(onlyMachine(found, "anchor")) != 0 {
t.Fatalf("a judged agent account still fails: %+v %v", found, err)
}
if named, confined, why, err := agentConfined(ctx, inv, "anchor", time.Now()); err != nil || !named || !confined {
t.Fatalf("agentConfined on anchor: %v %v %q %v", named, confined, why, err)
}
if named, _, why, err := agentConfined(ctx, inv, "laptop", time.Now()); err != nil || named ||
!strings.Contains(why, "operator account") {
t.Fatalf("agentConfined on a machine naming none: %v %q %v", named, why, err)
}
}
func TestTheFactsCarryTheAgentAccountAsAPseudonym(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
if err := open.inventory.SetAccount(ctx, "anchor", "keeper", ""); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetAgentAccount(ctx, "anchor", "warden", ""); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
body, _ := f.Encode()
if strings.Contains(string(body), "warden") {
t.Error("the agent account's name is in the snapshot")
}
m, ok := f.Machine(snapshot.Pseudonym("machine", "anchor"))
if !ok || m.AgentAccount != snapshot.Pseudonym("account", "warden") || m.Account == m.AgentAccount {
t.Fatalf("the anchor's agent account reads as %q (operator %q)", m.AgentAccount, m.Account)
}
}
func TestASearchStillRunningAfterARestartIsNotUrgent(t *testing.T) {
if searchQuietFor != rootsearch.Quiet {
t.Fatalf("the quiet is %s and the node-engine's %s: they are one value", searchQuietFor, rootsearch.Quiet)
}
// A daily search that finishes within the quiet never leaves the account unjudged between two of them.
if rootsearch.FreshFor < rootsearch.Every+rootsearch.Quiet {
t.Fatalf("a complete search judges for %s, less than a day's search (%s) and the quiet (%s)",
rootsearch.FreshFor, rootsearch.Every, rootsearch.Quiet)
}
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.NodeByName(ctx, "anchor"); err != nil {
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
say := func(state, reason string) []conditions.Observation {
t.Helper()
// The engine's since is always now: it was just restarted.
v := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: state, Reason: reason, Root: link.RootNever,
Account: "agent", Since: time.Now()}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{v}}); err != nil {
t.Fatal(err)
}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
return onlyMachine(found, "anchor")
}
running := link.ReasonRootPending + ": the search for setuid programs, started at 19:21, has not finished yet"
healthy := func() {
t.Helper()
if found := say(link.StateHealthy, ""); len(found) != 0 {
t.Fatalf("a healthy verdict raised: %+v", found)
}
}
if found := say(link.StateUnknown, running); len(found) != 0 {
t.Fatalf("a search first seen now was raised: %+v", found)
}
if _, confined, why, _ := agentConfined(ctx, inv, "anchor", time.Now()); confined || !strings.Contains(why, "not judged") {
t.Fatalf("an account whose search runs was read as confined: %q", why)
}
// The engine restarted again and again, each statement's own since fresh: the controller's clock runs on.
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err)
}
healthy() // a complete verdict forgets when the waiting began …
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err) // … and this restart loop began past the bound
}
if found := say(link.StateUnknown, running); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search pending past the bound, by the controller's clock, was not urgent: %+v", found)
}
healthy()
incomplete := rootsearch.ReasonIncomplete + ": the search for setuid programs did not finish (last tried at " +
"19:23: timeout); it is tried again later"
if found := say(link.StateUnknown, incomplete); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search that did not finish was not urgent: %+v", found)
}
if found := say(link.StateUnhealthy, link.ReasonRoot+": in the group docker; "+running); len(found) != 1 ||
found[0].Severity != conditions.Urgent {
t.Fatalf("a way to root found while the search runs was not urgent: %+v", found)
}
}
cmd/mesh-controller/agent_account_test.go
+146 -1
View File
@@ -1,12 +1,14 @@
package main
import (
"github.com/novox/mesh-host/rootsearch"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -63,6 +65,55 @@ func TestAnAgentAccountIsConfinedOnlyOnAHealthyVerdictJudgedForRoot(t *testing.T
// DA raises an urgent condition, with plain words, on a machine whose agent account is not judged unable to
// become root; a machine that names none is not its to judge.
func TestTheSelfCheckSaysAnAgentAccountThatCanBecomeRoot(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.NodeByName(ctx, n); err != nil {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAccount(ctx, n, "ops", ""); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
found = onlyMachine(found, "anchor")
if len(found) != 1 || found[0].Machine != "anchor" || found[0].Severity != conditions.Urgent ||
!strings.Contains(found[0].Said, "not judged") {
t.Fatalf("a named agent account with no verdict: %+v", found)
}
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if w.Headline == "" || w.Needs == "" || w.Resolved == "" {
t.Errorf("the condition has no plain words: %+v", w)
}
healthy := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: link.StateHealthy, Root: link.RootNever, Account: "agent"}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{healthy}}); err != nil {
t.Fatal(err)
}
if found, err = probeAgentAccounts(ctx, d); err != nil || len(onlyMachine(found, "anchor")) != 0 {
t.Fatalf("a judged agent account still fails: %+v %v", found, err)
}
if named, confined, why, err := agentConfined(ctx, inv, "anchor", time.Now()); err != nil || !named || !confined {
t.Fatalf("agentConfined on anchor: %v %v %q %v", named, confined, why, err)
}
if named, _, why, err := agentConfined(ctx, inv, "laptop", time.Now()); err != nil || named ||
!strings.Contains(why, "operator account") {
t.Fatalf("agentConfined on a machine naming none: %v %q %v", named, why, err)
}
}
func TestTheAgentRootWordsArePlain(t *testing.T) {
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
@@ -83,6 +134,28 @@ func onlyMachine(obs []conditions.Observation, machine string) []conditions.Obse
// The snapshot a merge check composes from carries the agent account as a pseudonym (novox/hq ADR 0266),
// so a change is judged against machines that name one, and the name never leaves.
func TestTheFactsCarryTheAgentAccountAsAPseudonym(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
if err := open.inventory.SetAccount(ctx, "anchor", "keeper", ""); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetAgentAccount(ctx, "anchor", "warden", ""); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
body, _ := f.Encode()
if strings.Contains(string(body), "warden") {
t.Error("the agent account's name is in the snapshot")
}
m, ok := f.Machine(snapshot.Pseudonym("machine", "anchor"))
if !ok || m.AgentAccount != snapshot.Pseudonym("account", "warden") || m.Account == m.AgentAccount {
t.Fatalf("the anchor's agent account reads as %q (operator %q)", m.AgentAccount, m.Account)
}
}
// No verb runs a `node` command that sets something: through the generic `command` verb, `node account`,
// `node agent-account` and every other `node` subcommand but list and show are refused, naming the terminal.
@@ -132,3 +205,75 @@ func TestTheNodeVerbOnlyShows(t *testing.T) {
// controller first saw it waiting — never from the engine's own "since", which a restart resets, so an engine
// restarted in a loop does not keep it quiet. The account is still not confined, and `node show` still says not
// judged; a search that failed, or a way to root found, is urgent at once.
func TestASearchStillRunningAfterARestartIsNotUrgent(t *testing.T) {
if searchQuietFor != rootsearch.Quiet {
t.Fatalf("the quiet is %s and the node-engine's %s: they are one value", searchQuietFor, rootsearch.Quiet)
}
// A daily search that finishes within the quiet never leaves the account unjudged between two of them.
if rootsearch.FreshFor < rootsearch.Every+rootsearch.Quiet {
t.Fatalf("a complete search judges for %s, less than a day's search (%s) and the quiet (%s)",
rootsearch.FreshFor, rootsearch.Every, rootsearch.Quiet)
}
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.NodeByName(ctx, "anchor"); err != nil {
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
say := func(state, reason string) []conditions.Observation {
t.Helper()
// The engine's since is always now: it was just restarted.
v := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: state, Reason: reason, Root: link.RootNever,
Account: "agent", Since: time.Now()}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{v}}); err != nil {
t.Fatal(err)
}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
return onlyMachine(found, "anchor")
}
running := link.ReasonRootPending + ": the search for setuid programs, started at 19:21, has not finished yet"
healthy := func() {
t.Helper()
if found := say(link.StateHealthy, ""); len(found) != 0 {
t.Fatalf("a healthy verdict raised: %+v", found)
}
}
if found := say(link.StateUnknown, running); len(found) != 0 {
t.Fatalf("a search first seen now was raised: %+v", found)
}
if _, confined, why, _ := agentConfined(ctx, inv, "anchor", time.Now()); confined || !strings.Contains(why, "not judged") {
t.Fatalf("an account whose search runs was read as confined: %q", why)
}
// The engine restarted again and again, each statement's own since fresh: the controller's clock runs on.
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err)
}
healthy() // a complete verdict forgets when the waiting began …
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err) // … and this restart loop began past the bound
}
if found := say(link.StateUnknown, running); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search pending past the bound, by the controller's clock, was not urgent: %+v", found)
}
healthy()
incomplete := rootsearch.ReasonIncomplete + ": the search for setuid programs did not finish (last tried at " +
"19:23: timeout); it is tried again later"
if found := say(link.StateUnknown, incomplete); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search that did not finish was not urgent: %+v", found)
}
if found := say(link.StateUnhealthy, link.ReasonRoot+": in the group docker; "+running); len(found) != 1 ||
found[0].Severity != conditions.Urgent {
t.Fatalf("a way to root found while the search runs was not urgent: %+v", found)
}
}
cmd/mesh-controller/api_test.go
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
cmd/mesh-controller/apply_failing.go
+1 -47
View File
@@ -33,66 +33,20 @@ const packageManagerKey = "the package manager ("
func watchApplyFailing(f *signalFacts) []conditions.Observation {
heard := map[string]bool{}
for _, m := range f.machines {
if !m.asleep() && !m.lastHeard.IsZero() && !m.silent(f.now) {
if !m.asleep() && !m.lastHeard.IsZero() && f.now.Sub(m.lastHeard) <= heartbeatBound(m.every) {
heard[m.name] = true
}
}
var out []conditions.Observation
said := map[string]bool{}
for _, d := range f.doing {
if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] {
continue
}
out = append(out, applyFailing(d, f.now))
said[d.Node] = true
}
// **A module's files the provisioner failed to place are a failed apply too** (novox/hq ADR 0362 §3): its client's
// newest word, judged by the same bound — the client says a failure once, and fills again on every change —
// on a machine whose node-engine applied what it was sent.
failed := map[string][]inventory.ProvReport{}
for _, r := range f.provFailed {
if r.Status != inventory.ProvFailed || f.now.Sub(r.At) <= applyEvery || !heard[r.Node] || said[r.Node] {
continue
}
failed[r.Node] = append(failed[r.Node], r)
}
for _, node := range sortedNames(failed) {
out = append(out, provisioningFailing(node, failed[node], f.now))
}
return out
}
// provisioningFailing is one machine's S21 condition for the modules whose files its provisioner client failed to
// place.
func provisioningFailing(m string, failed []inventory.ProvReport, now time.Time) conditions.Observation {
sort.Slice(failed, func(a, b int) bool { return failed[a].Module < failed[b].Module })
var names, words []string
since := failed[0].At
for _, r := range failed {
names = append(names, r.Module)
words = append(words, r.Module+": "+oneLine(orNoReason(r.Reason)))
if r.At.Before(since) {
since = r.At
}
}
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Kind: kindApplyFailing, Machine: m,
Severity: conditions.Warning, Said: strings.Join(words, "; "),
Resolved: m + " applies what it is sent again"}
o.Summary = fmt.Sprintf("%s's provisioner has failed to place the files of %s since %s (bound %s)", m,
strings.Join(names, ", "), since.UTC().Format("2006-01-02 15:04 MST"), ago(applyEvery))
o.Headline = m + " is not applying its update"
o.Explanation = fmt.Sprintf("The provisioner on %s has not been able to write the files of %s for %s. The "+
"details say why.", m, strings.Join(names, ", "), humanDuration(now.Sub(since)))
return o
}
func orNoReason(s string) string {
if s == "" {
return "no reason given"
}
return s
}
// applyFailing is one machine's S21 condition.
func applyFailing(d inventory.Doing, now time.Time) conditions.Observation {
m := d.Node
cmd/mesh-controller/asker.go
+27 -65
View File
@@ -117,10 +117,13 @@ type asked struct {
// Proposal is a settings layer proposed through a verb (proposals.go, novox/hq ADR 0277): about no
// condition, set on Approve by the serving controller itself, and left alone by the reconciling of conditions.
Proposal *settingsProposal `json:"proposal,omitempty"`
// Take is a library the operator is asked to let the mesh take (kept_libraries.go, novox/hq ADR 0328 §7): about no
// condition, taken on Approve by the serving controller itself, asked once per library.
Take *libraryTake `json:"take,omitempty"`
}
// ofACondition says an ask is one of a condition's: not a rehearsal, not a proposal.
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil }
func (r asked) ofACondition() bool { return !r.Rehearsal && r.Proposal == nil && r.Take == nil }
// partKey is an ask's place among what is asked: its condition and its part.
func partKey(condition, part string) string { return condition + "#" + part }
@@ -192,6 +195,11 @@ type asker struct {
call func(ctx context.Context, a conditions.Action, args map[string]string) error
// setLayer sets a proposed settings layer on the operator's warrant (novox/hq ADR 0277); nil cannot.
setLayer setOnWarrant
// folders is every library waiting to be taken (novox/hq ADR 0328 §7), and take what takes one on the operator's
// warrant; nil asks about none. machines are the mesh's machine names, allowed in an ask's words.
folders func(ctx context.Context) ([]libraryTake, error)
take takeOnWarrant
machines func(ctx context.Context) ([]string, error)
// record writes the hand-act log.
record func(ctx context.Context, act link.HandAct) error
// routerRecord reads the router's record of an ask for a warrant missed; nil reads nothing.
@@ -208,11 +216,8 @@ type asker struct {
// raise keeps the asker's own condition (sourceAsker): which conditions needing the operator could not be
// asked, and why. Nil raises nothing (a test that does not look).
raise func(ctx context.Context, obs []conditions.Observation) error
// tell tells the operator one plain notice that needs no answer (operator-channel.tell, novox/hq ADR 0380):
// the refusal of an answer to a plan's ask (ADR 0449 §3). Nil tells nothing (a test that does not look).
tell func(ctx context.Context, headline string, lines []string) error
now func() time.Time
logf func(string, ...any)
now func() time.Time
logf func(string, ...any)
saidNoRouter bool
saidNoGrant bool
@@ -409,9 +414,7 @@ func (a *asker) reconcile(ctx context.Context) error {
for _, r := range all {
k := partKey(r.Condition, r.Part)
if r.State == string(asks.OutcomeChosen) && now.Sub(r.Ended) < askAgainAfterAnswer {
if prior, has := answered[k]; !has || r.Ended.After(prior.Ended) {
answered[k] = r
}
answered[k] = r
}
if r.State == string(asks.OutcomeRefused) {
if prior, has := refused[k]; !has || r.Opened.After(prior.Opened) {
@@ -502,9 +505,7 @@ func (a *asker) reconcile(ctx context.Context) error {
continue // refused lately, and nothing it was refused for has changed: asked again after the wait
}
}
// **A condition raised after the answered ask ended is a new one** (novox/hq ADR 0449 §2): a retried
// plan that fails again is asked again, not held back for the hour its earlier answer holds.
if r, done := answered[key]; done && sameAsked(r.Actions, p.actions) && !c.Raised.After(r.Ended) {
if r, done := answered[key]; done && sameAsked(r.Actions, p.actions) {
if _, held := byCondition[key]; !held {
continue
}
@@ -561,6 +562,10 @@ func (a *asker) reconcile(ctx context.Context) error {
return err
}
}
// And every library waiting to be taken, once per library (novox/hq ADR 0328 §7).
if err := a.askFolders(ctx); err != nil {
a.logf("the operator could not be asked about the libraries waiting to be taken: %v", err)
}
why := "the router refused the ask"
if len(refusedWords) > 0 {
why += ": " + refusedWords[0]
@@ -643,8 +648,6 @@ func doesWords(act conditions.Action) string {
return "the delivery starts"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["stop"] != "":
return "the delivery is stopped"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["retry"] != "":
return "the failed builds are asked again"
case strings.HasSuffix(act.Verb, ".restart"):
return "its service is restarted on " + act.Machine
}
@@ -827,7 +830,7 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
if err != nil {
return err
}
stillOpen := r.Rehearsal || r.Proposal != nil // a rehearsal and a proposal are about no condition
stillOpen := r.Rehearsal || r.Proposal != nil || r.Take != nil // a rehearsal, a proposal and a take are about no condition
for _, c := range open {
stillOpen = stillOpen || c.Key == r.Condition
}
@@ -885,6 +888,15 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
if acted == nil && outcome != "" && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case r.Take != nil && act.Verb == takeVerb:
// A library taken on the operator's warrant (novox/hq ADR 0328 §7): the act is composed again from the
// record and held to what the option bound before the engine is asked.
outcome, acted = a.decideTake(ctx, *r, act, w)
if acted == nil && !strings.HasPrefix(outcome, "nothing") {
outcome = "done: " + outcome
}
case r.Take != nil:
acted = fmt.Errorf("the take %s offers no act %s", r.ID, act.Verb)
case act.Arguments["silence"] != "":
acted = a.silence(ctx, act.Arguments["silence"], conditions.MaxSilence, byWords(w), why)
default:
@@ -893,7 +905,6 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
ended := a.now()
if acted != nil {
outcome = "failed: " + acted.Error()
a.tellRefused(ctx, *r, act, acted)
}
r.Ended, r.Acted = ended, outcome
if _, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
@@ -928,55 +939,6 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
return nil
}
// tellRefused tells the operator what became of the answer they gave to a plan's ask, when it was not done (novox/hq
// ADR 0449 §3): the controller's `plans` verb refuses a Retry, Start or Stop when the plan moved on between the ask and
// the answer, and the refusal was only recorded, so the operator who pressed the answer heard nothing. **A refusal is
// told as one, and nothing else is**: a verb that said no changed nothing, but a call that got no answer (a timeout, a
// refused grant) may have run, and is told as not known. One plain notice with the ask's headline, beside the hand act
// the caller still records. An answer to any other ask is not told here. A notice that cannot be told is said in the
// log and changes nothing.
func (a *asker) tellRefused(ctx context.Context, r asked, act conditions.Action, failure error) {
if a.tell == nil || act.Verb != broker.ControllerSeat+".plans" {
return
}
var lines []string
var refused verbRefused
if errors.As(failure, &refused) {
lines = []string{
fmt.Sprintf("Your answer, %s, could not be done: %s.", act.Label, strings.TrimRight(refusalWords(refused.words, act), ".")),
"Nothing was changed.",
}
} else {
lines = []string{
fmt.Sprintf("Your answer, %s, was given, and it is not known whether it was done.", act.Label),
"Look at the delivery before you answer again.",
}
}
if err := a.tell(ctx, r.Ask.Headline, lines); err != nil {
a.logf("the operator could NOT be told that %s was refused (ask %s): %v", act.Label, r.ID, err)
}
}
// refusalWords is a verb's refusal as the operator reads it: the plans named as "the plan" (plainRefusal), and, where
// the words still carry an identifier the operator's channel withholds, one sentence of the answer's own.
func refusalWords(words string, act conditions.Action) string {
return plainRefusalOr(words, refusalFallback(act))
}
// refusalFallback is what an answer's refusal says when its own words cannot be said: one sentence for each answer
// to a plan's ask, never another answer's.
func refusalFallback(act conditions.Action) string {
switch {
case act.Arguments["retry"] != "":
return "a retry would not run now; the details say why"
case act.Arguments["go"] != "":
return "the delivery could not be started now; the details say why"
case act.Arguments["stop"] != "":
return "the delivery could not be stopped now; the details say why"
}
return "the controller refused it; the details say why"
}
// handActWarrant is the verb an act the operator chose on a warrant is recorded under: a person's decision,
// never a repair (handActVerbs).
const handActWarrant = "warrant"
cmd/mesh-controller/asker_bus_test.go
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
cmd/mesh-controller/asker_retry_integration_test.go
-141
View File
@@ -1,141 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// 1315/AC3: the operator's Retry on a failed plan, acted on through the real callAction and the controller's own
// `plans` verb, sets the plan building again at its tier and ends its plan-failed condition (novox/hq ADR 0449 §1).
func TestAWarrantForRetryRetriesAFailedPlanInTheStoreAndEndsItsCondition(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asked := asksRecorded(t)
planMovedHeard(t) // the bus the verb says its moves on, and the hand-act log
twoTiers(t, open)
before := time.Now().UTC().Add(-time.Hour)
ended := before.Add(time.Minute)
failed := inventory.Plan{ID: "plan-1315-retry", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: before, State: inventory.PlanFailed, Tier: 0, Tiers: [][]string{{"a"}, {"b"}},
Note: "a failed to build in tier 0",
Delivery: &inventory.PlanDelivery{},
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1",
Why: link.KilledByHand}},
Times: &inventory.PlanTimes{Cut: &before, Started: &before, Ended: &ended, Class: inventory.ClassLeaf}}
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer conn.Close()
// The controller's seat answers `plans` as it does: by running the verb.
sub, err := conn.Subscribe(link.SeatToolSubject("mesh-controller", "plans"), func(m *nats.Msg) {
var args map[string]string
_ = json.Unmarshal(m.Data, &args)
answer := link.Answer{Result: json.RawMessage(`"ok"`)}
if len(args) != 1 || args["retry"] == "" {
answer = link.Answer{Error: "the retry was asked with more than the plan"}
} else if err := plansCommand(context.Background(), []string{"retry", args["retry"]}); err != nil {
answer = link.Answer{Error: err.Error()}
}
body, _ := json.Marshal(answer)
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
// What S23 raises for it, read from the store as the watchdog reads it.
raised := func(context.Context) ([]conditions.Condition, error) {
facts, err := gatherFailedPlans(ctx, open.inventory, time.Now())
if err != nil {
return nil, err
}
var out []conditions.Condition
for _, o := range watchPlansFailed(&signalFacts{now: time.Now(), failed: facts}) {
out = append(out, conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions})
}
return out, nil
}
now := time.Now()
store := memAskedStore{}
var sent []published
var acts []link.HandAct
a := &asker{open: raised, store: store,
silence: func(context.Context, string, time.Duration, string, string) error { return nil },
publish: func(_ context.Context, subject string, body []byte, id string) error {
sent = append(sent, published{subject, id, body})
return nil
},
call: callAction(conn),
record: func(_ context.Context, act link.HandAct) error { acts = append(acts, act); return nil },
now: func() time.Time { return now },
logf: t.Logf}
cs, _ := raised(ctx)
if len(cs) != 1 || len(cs[0].Actions) != 1 {
t.Fatalf("the failed plan raised %+v", cs)
}
if err := a.reconcile(ctx); err != nil {
t.Fatal(err)
}
var w asks.Warrant
for _, r := range store {
for _, o := range r.Ask.Options {
if o.Label == "Retry" {
w = asks.Warrant{Ask: r.ID, Asker: "mesh-controller", About: r.Condition, Outcome: asks.OutcomeChosen,
Option: o.ID, Label: o.Label, Level: o.Level, Channel: "telegram", Proofs: []string{"P1"}, At: now,
AskDigest: r.Ask.Digest(),
By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
}
}
}
if w.Ask == "" {
t.Fatalf("no ask offers Retry: %+v", store)
}
body, _ := json.Marshal(w)
if err := a.Decided(ctx, body); err != nil {
t.Fatal(err)
}
if err := a.Decided(ctx, body); err != nil { // heard again: nothing more
t.Fatal(err)
}
p, err := open.inventory.PlanByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
m := p.Modules["a"]
if p.State != inventory.PlanBuilding || p.Tier != 0 || m.State != "asked" || m.Build == "" || m.Build == "build-1" {
t.Fatalf("after the warrant the plan is %s at tier %d and a is %+v", p.State, p.Tier, m)
}
if len(*asked) != 1 {
t.Fatalf("the build was asked %d times: %v", len(*asked), *asked)
}
if got := store[w.Ask]; got.Acted != "done" {
t.Fatalf("the ask ended %q", got.Acted)
}
if len(acts) != 1 || acts[0].Verb != handActWarrant || !strings.Contains(strings.Join(acts[0].Args, " "), "retry="+failed.ID) {
t.Fatalf("the warrant was recorded as %+v", acts)
}
if cs, _ = raised(ctx); len(cs) != 0 {
t.Fatalf("the plan-failed condition did not end: %+v", cs)
}
}
cmd/mesh-controller/asker_test.go
-225
View File
@@ -77,15 +77,6 @@ type askerRig struct {
silenced []string
acts []link.HandAct
now time.Time
// refuses, when set, is what every verb the asker calls answers; told is every notice the operator was told.
refuses error
told []told
}
// told is one plain notice the asker told the operator.
type told struct {
headline string
lines []string
}
func newAskerRig(t *testing.T) *askerRig {
@@ -111,10 +102,6 @@ func newAskerRig(t *testing.T) *askerRig {
call: func(_ context.Context, a conditions.Action, args map[string]string) error {
raw, _ := json.Marshal(args)
r.called = append(r.called, a.Verb+"@"+a.Machine+" "+string(raw))
return r.refuses
},
tell: func(_ context.Context, headline string, lines []string) error {
r.told = append(r.told, told{headline, lines})
return nil
},
record: func(_ context.Context, act link.HandAct) error { r.acts = append(r.acts, act); return nil },
@@ -1026,215 +1013,3 @@ func TestTheServingAskerJudgesTheGrantForReal(t *testing.T) {
t.Errorf("held %v (%q, %v) with nothing to read", held, why, err)
}
}
// failedPlanCondition is the condition S23 raises for a failed delivery plan, as the keeper holds it.
func failedPlanCondition(plan string) conditions.Condition {
p := aFailedPlan(plan, nil)
f := calm(time.Date(2026, 10, 19, 18, 30, 0, 0, time.UTC))
f.failed = []failedPlanFacts{failedFactsFor(p)}
o := watchPlansFailed(f)[0]
return conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
}
// 1315/AC1: a failed plan's Retry is asked at the approve level, in the words "the failed builds are asked again".
func TestAFailedPlansRetryIsAskedAtApproveInItsOwnWords(t *testing.T) {
r := newAskerRig(t)
c := failedPlanCondition("plan-1792433168676870614")
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 {
t.Fatalf("asked %+v", sent)
}
o := sent[0].Options[0]
if o.Label != "Retry" || o.Level != asks.Approve || o.Does != "the failed builds are asked again" {
t.Errorf("the option is %+v", o)
}
if err := sent[0].Check(r.now); err != nil {
t.Errorf("the ask is refused as made: %v", err)
}
if strings.Contains(sent[0].Explanation, "mesh MCP server") {
t.Errorf("the ask still says where else to answer: %q", sent[0].Explanation)
}
}
// 1315/AC2: the warrant calls plans with retry=<plan> and nothing else, once, and is recorded as the hand act;
// the same warrant heard again does nothing.
func TestAWarrantForRetryCallsPlansRetryOnceAndIsRecorded(t *testing.T) {
plan := "plan-1792433168676870614"
r := newAskerRig(t)
c := failedPlanCondition(plan)
r.open = []conditions.Condition{c}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, c.Key, "Retry")
answerWith(t, r, w)
answerWith(t, r, w)
if want := `mesh-controller.plans@ {"retry":"` + plan + `"}`; len(r.called) != 1 || r.called[0] != want {
t.Fatalf("called %v, want exactly %s", r.called, want)
}
if len(r.acts) != 1 {
t.Fatalf("hand-acts %+v", r.acts)
}
act := r.acts[0]
if act.Verb != handActWarrant || strings.Join(act.Args, " ") != "mesh-controller.plans retry="+plan ||
act.Ask != w.Ask || act.Condition != c.Key || act.Outcome != "done" || act.Cause != conditions.CauseOperatorAnswer {
t.Errorf("the hand-act is %+v", act)
}
if got := r.store[w.Ask]; got.Acted != "done" {
t.Errorf("kept %+v", got)
}
}
// 1316/AC1 and AC2: a Retry or a Stop on a plan that moved on is refused by `plans`; the operator who pressed it is
// told once, with the ask's headline and the refusal's words, and the hand act records the failure as before.
func TestARefusedAnswerToAPlansAskIsToldOnceAndStillRecorded(t *testing.T) {
plan := "plan-1792433168676870614"
waiting := conditions.Condition{Key: "plan." + plan + ".waiting", Severity: conditions.Urgent,
Headline: "openrazer delivery waiting to start", Needs: "start it, or stop it.",
Explanation: "Needs you: start it, or stop it.", Actions: waitingActions(plan, conditions.Urgent)}
for _, tc := range []struct {
name string
c conditions.Condition
label string
}{
{"retry", failedPlanCondition(plan), "Retry"},
{"stop", waiting, "Stop"},
} {
t.Run(tc.name, func(t *testing.T) {
r := newAskerRig(t)
r.refuses = verbRefused{verb: "mesh-controller.plans", words: plan + " is done; there is nothing to retry"}
r.open = []conditions.Condition{tc.c}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, tc.c.Key, tc.label)
answerWith(t, r, w)
answerWith(t, r, w) // heard again: told once
if len(r.told) != 1 {
t.Fatalf("told %+v", r.told)
}
got := r.told[0]
if got.headline != tc.c.Headline {
t.Errorf("the headline is %q, not the ask's %q", got.headline, tc.c.Headline)
}
said := strings.Join(got.lines, " ")
if !strings.Contains(said, "the plan is done; there is nothing to retry") || !strings.Contains(said, tc.label) {
t.Errorf("the lines do not carry the answer and the refusal's words: %q", said)
}
if why, ok := conditions.Plain(said); !ok {
t.Errorf("the notice is not plain, so the channel would withhold it: %s: %q", why, said)
}
if len(r.acts) != 1 || !strings.HasPrefix(r.acts[0].Outcome, "failed: ") {
t.Errorf("the failure is not recorded as before: %+v", r.acts)
}
if kept := r.store[w.Ask]; !strings.HasPrefix(kept.Acted, "failed: ") {
t.Errorf("kept %+v", kept)
}
})
}
}
// 1316/AC3: an answer that is done tells nothing; and a refusal of an answer that is no plan's is not told here.
func TestADoneAnswerTellsNothingAndOnlyAPlansRefusalIsTold(t *testing.T) {
r := newAskerRig(t)
c := failedPlanCondition("plan-1792433168676870614")
r.open = []conditions.Condition{c}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, c.Key, "Retry"))
if len(r.told) != 0 {
t.Fatalf("a retry that was done told %+v", r.told)
}
r = newAskerRig(t)
r.refuses = verbRefused{verb: "node-service-manager.restart", words: "unit not found"}
module := conditions.Condition{Key: "module.openrazer.g14.unhealthy", Severity: conditions.Warning,
Headline: "openrazer not working on g14", Needs: "restart its service openrazer-daemon on g14.",
Explanation: "Needs you: restart it.", Actions: []conditions.Action{{Label: "Restart",
Verb: "node-service-manager.restart", Machine: "g14", Level: conditions.LevelApprove,
Arguments: map[string]string{"unit": "openrazer-daemon.service", "scope": "user"}}}}
r.open = []conditions.Condition{module}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, module.Key, "Restart"))
if len(r.told) != 0 || len(r.acts) != 1 {
t.Fatalf("a refused restart told %+v, acts %+v", r.told, r.acts)
}
}
// 1316, review round 1: a call that got no answer may have run, so it is told as not known, never as "nothing was
// changed"; and the sentence a refusal falls back to fits the answer given.
func TestAnAnswerThatGotNoAnswerIsToldAsNotKnownNotAsRefused(t *testing.T) {
plan := "plan-1792433168676870614"
r := newAskerRig(t)
r.refuses = errors.New("mesh-controller did not answer plans within 30s")
c := failedPlanCondition(plan)
r.open = []conditions.Condition{c}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, c.Key, "Retry"))
if len(r.told) != 1 {
t.Fatalf("told %+v", r.told)
}
said := strings.Join(r.told[0].lines, " ")
if !strings.Contains(said, "not known whether") || strings.Contains(said, "Nothing was changed") {
t.Errorf("a call that may have run is told as %q", said)
}
if why, ok := conditions.Plain(said); !ok {
t.Errorf("not plain: %s: %q", why, said)
}
}
func TestEachAnswersRefusalFallsBackToItsOwnSentence(t *testing.T) {
plan := "plan-1792433168676870614"
for _, tc := range []struct {
act conditions.Action
want string
}{
{retryAction(plan), "a retry would not run now"},
{conditions.Action{Label: "Start", Verb: "mesh-controller.plans", Arguments: map[string]string{"go": plan}}, "could not be started now"},
{conditions.Action{Label: "Stop", Verb: "mesh-controller.plans", Arguments: map[string]string{"stop": plan}}, "could not be stopped now"},
} {
// Words the operator's channel withholds: a hash and a dotted name.
got := refusalWords("build 48bda475dunst at mesh.seat.x refused", tc.act)
if !strings.Contains(got, tc.want) {
t.Errorf("%s: %q, want %q", tc.act.Label, got, tc.want)
}
if _, ok := conditions.Plain(got); !ok {
t.Errorf("%s: the fallback is not plain: %q", tc.act.Label, got)
}
}
}
// 1314/AC6, ADR 0449 §2: a retried plan that fails again within the hour of the answer is asked again, because the
// condition that comes back is a new one, raised after the answered ask ended; the same condition still open is not.
func TestARetriedPlanThatFailsAgainWithinTheHourIsAskedAgain(t *testing.T) {
r := newAskerRig(t)
c := failedPlanCondition("plan-1792433168676870614")
c.Raised = r.now.Add(-time.Minute)
r.open = []conditions.Condition{c}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, c.Key, "Retry"))
// Still open right after the answer: not asked again at once.
r.now = r.now.Add(time.Minute)
_ = r.a.reconcile(context.Background())
if n := len(r.asksSent(t)); n != 1 {
t.Fatalf("asked again while the same condition stayed open: %d", n)
}
// The retry ran, the plan left failed and the condition ended.
r.now = r.now.Add(time.Minute)
r.open = nil
_ = r.a.reconcile(context.Background())
// It fails again, ten minutes after the answer: a condition raised after the answered ask ended.
r.now = r.now.Add(8 * time.Minute)
again := failedPlanCondition("plan-1792433168676870614")
again.Raised = r.now
r.open = []conditions.Condition{again}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
if n := len(r.asksSent(t)); n != 2 {
t.Fatalf("the second failure was asked %d times in all, want 2", n)
}
if len(r.called) != 1 {
t.Fatalf("something retried without an answer: %v", r.called)
}
}
cmd/mesh-controller/asker_wire.go
+24 -26
View File
@@ -19,8 +19,8 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -135,12 +135,6 @@ func (b busAsked) All(ctx context.Context) ([]asked, error) {
return out, nil
}
// verbRefused is a verb's own refusal of an act: the seat answered, and said no. Any other error of callAction (a bus
// that did not answer, a grant that is refused) is not one: the verb may have run.
type verbRefused struct{ verb, words string }
func (e verbRefused) Error() string { return e.verb + " refused: " + e.words }
// callAction performs an action's verb as the controller, through the grant that names it.
func callAction(conn *nats.Conn) func(ctx context.Context, a conditions.Action, args map[string]string) error {
return func(ctx context.Context, a conditions.Action, args map[string]string) error {
@@ -174,23 +168,7 @@ func callAction(conn *nats.Conn) func(ctx context.Context, a conditions.Action,
return err
}
if answer.Error != "" {
return verbRefused{verb: a.Verb, words: answer.Error}
}
return nil
}
}
// tellThrough tells the operator one plain notice through the operator channel's `tell`, over the controller's own
// connection and grant (novox/hq ADR 0380); a holder's refusal is an error naming it.
func tellThrough(conn *nats.Conn) func(ctx context.Context, headline string, lines []string) error {
return func(ctx context.Context, headline string, lines []string) error {
answer, err := link.AskMeshSeatTool(ctx, conn, "operator-channel", "tell",
map[string]any{"headline": headline, "lines": lines}, tellWithin)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("operator-channel.tell refused: %s", answer.Error)
return fmt.Errorf("%s refused: %s", a.Verb, answer.Error)
}
return nil
}
@@ -223,7 +201,7 @@ func routerRecordOf(conn *nats.Conn, inv *inventory.Inventory) func(ctx context.
}
// asksRecords is the bucket the asks seat's declarer keeps its record of asks in.
func asksRecords(ctx context.Context, inv catalogueReader) (string, error) {
func asksRecords(ctx context.Context, inv *inventory.Inventory) (string, error) {
declared, err := inv.Catalogue(ctx)
if err != nil {
return "", err
@@ -594,11 +572,31 @@ func servingAsker(open *stores, conn *nats.Conn, js jetstream.JetStream, keeper
call: callAction(conn),
// A proposed settings layer is set by this controller itself on the warrant (novox/hq ADR 0277).
setLayer: setLayerIn(open),
// A shared folder waiting to be kept is asked about once, and taken on the warrant (novox/hq ADR 0325 §8).
folders: foldersIn(open),
take: takeFolderIn(open, func(ctx context.Context, act link.AccessActAsk) (link.HandOverAnswer, error) {
ident, err := open.Identity(ctx)
if err != nil {
return link.HandOverAnswer{}, fmt.Errorf("the mesh's signing key cannot be read, so nothing was asked of "+
"%s: %w", act.Node, err)
}
return link.AskTake(ctx, conn, ident, act, link.TakeWithin)
}),
machines: func(ctx context.Context) ([]string, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, err
}
names := make([]string, 0, len(nodes))
for _, n := range nodes {
names = append(names, n.Name)
}
return names, nil
},
record: func(ctx context.Context, act link.HandAct) error {
_, err := link.RecordHandAct(ctx, conn, act)
return err
},
tell: tellThrough(conn),
routerRecord: routerRecordOf(conn, open.inventory),
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open, conn),
cmd/mesh-controller/awaiting_push_integration_test.go
-193
View File
@@ -1,193 +0,0 @@
//go:build integration
package main
import (
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
func TestAModuleAssignedAndNotPushedIsAwaitingAPushNotUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop") // the machine was pushed before; then the module is assigned
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
// The read the rest of the mesh asks still refuses it, with the composer's typed error: nothing
// can be compared about a secret that does not exist (status), and nothing here string-matches.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
_, err = declarationWith(ctx, open, "laptop", plan, settings, gens, Reading)
var notMade *catalogue.NotMadeError
if !errors.As(err, &notMade) || notMade.Module != "keeper" || notMade.Name != "repository" {
t.Fatalf("a read composition did not say which secret is not made, typed: %v", err)
}
// Foreseen, it composes, names what the push makes, and made nothing.
foreseen, err := declarationWith(ctx, open, "laptop", plan, settings, gens, Foreseeing)
if err != nil || len(foreseen.foreseen) != 1 || foreseen.foreseen[0] != "keeper/repository" {
t.Fatalf("foreseen: %v %v", foreseen.foreseen, err)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "keeper", "repository"); err != nil || held {
t.Fatalf("asking ahead of the push made the secret (held %v, %v)", held, err)
}
// Within the bound: nothing at all — no urgent, no warning, nobody notified.
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a machine waiting for a push raised %+v", got)
}
// Past the bound: a warning, not urgent, saying what the push will make.
before := awaitingPushBound
awaitingPushBound = -time.Minute
t.Cleanup(func() { awaitingPushBound = before })
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.awaiting-push" || got[0].Severity != conditions.Warning ||
!strings.Contains(got[0].Summary, "keeper/repository") || !strings.Contains(got[0].Summary, "push laptop") {
t.Fatalf("a machine left un-pushed past the bound: %+v", got)
}
// Pushed: the secret is made and D1 says nothing.
pushedBy(t, open, "laptop")
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a pushed machine still raised %+v", got)
}
}
func TestASecretThePushCannotMakeIsStillUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aTalker())
if _, err := assign(ctx, open, "laptop", "talker"); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "module issue talker --node laptop") {
t.Fatalf("a push that will be refused was not said urgently: %+v", got)
}
// And a machine whose composition fails for anything else, with a secret waiting beside it, is
// uncomposable for that — the stand-in hides nothing.
register(t, open, aKeeper())
if _, err := assign(ctx, open, "anchor", "keeper"); err != nil {
t.Fatal(err)
}
one, two := rivals()
register(t, open, one)
register(t, open, two)
_, _ = assign(ctx, open, "anchor", "rival-one")
_, _ = assign(ctx, open, "anchor", "rival-two")
keys := map[string]conditions.Severity{}
for _, o := range d1(t, open) {
keys[o.Key()] = o.Severity
}
if keys["machine.anchor.uncomposable"] != conditions.Urgent {
t.Fatalf("a real failure beside a waiting secret: %v", keys)
}
}
func TestAGivenSecretUnderAnOldKeyIsNotForeseen(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "keeper", "repository", "given"); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || !strings.Contains(got[0].Summary, "issue it again") {
t.Fatalf("a given secret under an old key: %+v", got)
}
}
func TestAMachineAwaitsAPushSinceItsOldestAssignmentSinceTheLastSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
sent := time.Now()
since, err := open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.After(sent) {
t.Fatalf("nothing assigned since the send: waiting since %v (%v), the send was before %v", since, err, sent)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
since, err = open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.Before(sent.Add(-time.Second)) {
t.Fatalf("assigned after the send: waiting since %v (%v), not from the assignment", since, err)
}
}
func TestALastSendIsReadFromTheSendAndTheReport(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
// What a machine was sent is read from the commits its assignments name (issue 1109): a module is carried by a send only
// when it was built from a commit, and a hand send on a machine with nothing assigned leaves no key to mark unknown.
if err := open.inventory.RegisterModule(ctx, aKeeper(), inventory.Source{BuiltFrom: "c-keeper"}); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSent(ctx, record.ID, "carried", map[string]string{overlay.Name: "c-overlay"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
sends, err := readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("a holder assigned and not pushed is expected to answer")
}
if !sends["laptop"].settled(overlay.Name, time.Now().Add(time.Hour)) {
t.Fatal("a module the machine was sent long ago is not expected to answer")
}
pushedBy(t, open, "laptop")
sends, err = readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("pushed a moment ago and not reported, a holder is already expected")
}
if !sends["laptop"].settled("keeper", time.Now().Add(reportGrace+time.Minute)) {
t.Fatal("pushed past the grace, a holder is not expected")
}
if _, ok := sends["laptop"].carried["keeper"]; !ok {
t.Fatalf("the send's builds do not carry keeper: %v", sends["laptop"].carried)
}
pushedAndApplied(t, open, "laptop")
if sends, err = readDeliveries(ctx, open.inventory); err != nil || !sends["laptop"].settled("keeper", time.Now()) {
t.Fatalf("pushed and reported applied, a holder is not expected to answer (%v)", err)
}
}
cmd/mesh-controller/awaiting_push_test.go
+168 -16
View File
@@ -1,12 +1,15 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// Between `assign` and `push` a module's own secrets are not made yet: the push makes them. D1 composed
@@ -46,21 +49,7 @@ func pushedBy(t *testing.T, open *stores, node string) string {
t.Fatal(err)
}
digest := digestOf(body)
// A push records the build of each module it carries (plan.go, Allocating); a module is carried by a send only when it was
// built from a commit, for what a machine was sent is read from the commits its assignments name (issue 1109).
builds := declared.Builds
if builds == nil {
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
t.Fatal(err)
}
names := make([]string, 0, len(plan.Modules))
for _, m := range plan.Modules {
names = append(names, m.Module)
}
builds = carriedBuilds(names, nil, current, nil)
}
if err := open.inventory.RecordSent(ctx, record.ID, digest, builds); err != nil {
if err := open.inventory.RecordSent(ctx, record.ID, digest, declared.Builds); err != nil {
t.Fatal(err)
}
return digest
@@ -92,14 +81,140 @@ func d1(t *testing.T, open *stores) []conditions.Observation {
// **THE WINDOW, REPRODUCED**: assigned and not pushed, a module whose own secret the push makes is
// waiting, not uncomposable; past the bound it is a warning naming what the push makes; pushed, nothing.
func TestAModuleAssignedAndNotPushedIsAwaitingAPushNotUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop") // the machine was pushed before; then the module is assigned
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
// The read the rest of the mesh asks still refuses it, with the composer's typed error: nothing
// can be compared about a secret that does not exist (status), and nothing here string-matches.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
_, err = declarationWith(ctx, open, "laptop", plan, settings, gens, Reading)
var notMade *catalogue.NotMadeError
if !errors.As(err, &notMade) || notMade.Module != "keeper" || notMade.Name != "repository" {
t.Fatalf("a read composition did not say which secret is not made, typed: %v", err)
}
// Foreseen, it composes, names what the push makes, and made nothing.
foreseen, err := declarationWith(ctx, open, "laptop", plan, settings, gens, Foreseeing)
if err != nil || len(foreseen.foreseen) != 1 || foreseen.foreseen[0] != "keeper/repository" {
t.Fatalf("foreseen: %v %v", foreseen.foreseen, err)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "keeper", "repository"); err != nil || held {
t.Fatalf("asking ahead of the push made the secret (held %v, %v)", held, err)
}
// Within the bound: nothing at all — no urgent, no warning, nobody notified.
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a machine waiting for a push raised %+v", got)
}
// Past the bound: a warning, not urgent, saying what the push will make.
before := awaitingPushBound
awaitingPushBound = -time.Minute
t.Cleanup(func() { awaitingPushBound = before })
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.awaiting-push" || got[0].Severity != conditions.Warning ||
!strings.Contains(got[0].Summary, "keeper/repository") || !strings.Contains(got[0].Summary, "push laptop") {
t.Fatalf("a machine left un-pushed past the bound: %+v", got)
}
// Pushed: the secret is made and D1 says nothing.
pushedBy(t, open, "laptop")
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a pushed machine still raised %+v", got)
}
}
// **A REAL FAILURE IS STILL URGENT**: a secret the push would be refused on is not one it will make.
// A bus credential nobody issued (issue 203) fails the push, so D1 says it — urgent, in the push's words.
func TestASecretThePushCannotMakeIsStillUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aTalker())
if _, err := assign(ctx, open, "laptop", "talker"); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "module issue talker --node laptop") {
t.Fatalf("a push that will be refused was not said urgently: %+v", got)
}
// And a machine whose composition fails for anything else, with a secret waiting beside it, is
// uncomposable for that — the stand-in hides nothing.
register(t, open, aKeeper())
if _, err := assign(ctx, open, "anchor", "keeper"); err != nil {
t.Fatal(err)
}
one, two := rivals()
register(t, open, one)
register(t, open, two)
_, _ = assign(ctx, open, "anchor", "rival-one")
_, _ = assign(ctx, open, "anchor", "rival-two")
keys := map[string]conditions.Severity{}
for _, o := range d1(t, open) {
keys[o.Key()] = o.Severity
}
if keys["machine.anchor.uncomposable"] != conditions.Urgent {
t.Fatalf("a real failure beside a waiting secret: %v", keys)
}
}
// A given secret sealed to a key the machine no longer has is not the mesh's to make again: the push is
// refused on it, so D1 is too.
func TestAGivenSecretUnderAnOldKeyIsNotForeseen(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "keeper", "repository", "given"); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || !strings.Contains(got[0].Summary, "issue it again") {
t.Fatalf("a given secret under an old key: %+v", got)
}
}
// The bound is read from when the machine began waiting: the oldest assignment since its last send.
func TestAMachineAwaitsAPushSinceItsOldestAssignmentSinceTheLastSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
sent := time.Now()
since, err := open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.After(sent) {
t.Fatalf("nothing assigned since the send: waiting since %v (%v), the send was before %v", since, err, sent)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
since, err = open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.Before(sent.Add(-time.Second)) {
t.Fatalf("assigned after the send: waiting since %v (%v), not from the assignment", since, err)
}
}
// **D3 AND D13 WAIT FOR THE SEND**: a holder is expected to answer on a machine once the machine was sent
// it and had time to report — never between assign and push.
@@ -127,3 +242,40 @@ func TestAHolderIsExpectedOnlyOnceSentAndReported(t *testing.T) {
}
// And through the stores: assigned, not in the last send; pushed and reported, carried and settled.
func TestALastSendIsReadFromTheSendAndTheReport(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
sends, err := readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("a holder assigned and not pushed is expected to answer")
}
if !sends["laptop"].settled(overlay.Name, time.Now().Add(time.Hour)) {
t.Fatal("a module the machine was sent long ago is not expected to answer")
}
pushedBy(t, open, "laptop")
sends, err = readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("pushed a moment ago and not reported, a holder is already expected")
}
if !sends["laptop"].settled("keeper", time.Now().Add(reportGrace+time.Minute)) {
t.Fatal("pushed past the grace, a holder is not expected")
}
if _, ok := sends["laptop"].carried["keeper"]; !ok {
t.Fatalf("the send's builds do not carry keeper: %v", sends["laptop"].carried)
}
pushedAndApplied(t, open, "laptop")
if sends, err = readDeliveries(ctx, open.inventory); err != nil || !sends["laptop"].settled("keeper", time.Now()) {
t.Fatalf("pushed and reported applied, a holder is not expected to answer (%v)", err)
}
}
cmd/mesh-controller/away.go
-100
View File
@@ -1,100 +0,0 @@
package main
import (
"context"
"fmt"
"os"
"slices"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// A machine out of touch when a tier is sent is not sent to and not waited for (novox/hq ADR 0435 §1, issue 1161): the
// delivery goes on with the machines heard, says the others were away, and the away ones get the builds by the send
// they are given when heard again (ADR 0353, ADR 0360 §5). Away is the silent check's one verdict, machineFacts.silent
// over the watchman's presence (ADR 0433 §3), never a second rule; a sleeping machine is away once presence says so.
// awayOf is the machines out of touch at a moment: those the silent check calls silent (a sleeping machine
// is away once presence says so). A machine
// never heard from has not joined and is not away.
func awayOf(machines []machineFacts, now time.Time) map[string]bool {
out := map[string]bool{}
for _, m := range machines {
if m.silent(now) {
out[m.name] = true
}
}
return out
}
// judgeAway is the walk's reading of which machines are away as a tier is sent: every node with its last-heard time and
// the watchman's verdict on its presence key. A bus that cannot be asked leaves none away, which is what a walk did
// before: a machine is not passed on a reading that was not made.
var judgeAway = func(ctx context.Context, open *stores, now time.Time) (map[string]bool, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, err
}
address, err := broker.BusAddress()
if err != nil {
return nil, err
}
js, err := broker.Dial(address)
if err != nil {
return nil, err
}
defer js.Close()
machines := make([]machineFacts, len(nodes))
for i, n := range nodes {
machines[i] = machineFacts{name: n.Name, lastHeard: n.LastSeen}
}
if err := withPresence(ctx, js.Conn(), machines, now); err != nil {
return nil, err
}
return awayOf(machines, now), nil
}
// rememberAway keeps on the plan the machines of running that are away: sorted, each once.
func rememberAway(p *inventory.Plan, running []string, away map[string]bool) {
for _, n := range running {
if away[n] && !slices.Contains(p.Away, n) {
p.Away = append(p.Away, n)
}
}
sort.Strings(p.Away)
}
// runningHeard is the machines of running that the walk waits on for a module: those a send of the walk reached, and
// those it did not pass as away.
func runningHeard(p inventory.Plan, running []string, s *inventory.PlanModule) []string {
return slices.DeleteFunc(slices.Clone(running), func(n string) bool {
if !slices.Contains(p.Away, n) {
return false
}
_, sent := s.Rest[n]
return !sent
})
}
// awaySaying is what a plan says of the machines it passed as away, empty when it passed none.
func awaySaying(p inventory.Plan) string {
var said []string
for _, n := range p.Away {
said = append(said, n+" was away: sent when it is heard again")
}
return strings.Join(said, "; ")
}
// judgedAway reads which machines are away for a tier about to be sent, and says when it cannot.
func judgedAway(ctx context.Context, open *stores, now time.Time) map[string]bool {
away, err := judgeAway(ctx, open, now)
if err != nil {
fmt.Fprintf(os.Stderr, "which machines are away cannot be read, so none is passed over: %v\n", err)
return nil // empty-on-error: none is passed over on a reading that was not made; the error is said just above
}
return away
}
cmd/mesh-controller/away_integration_test.go
-228
View File
@@ -1,228 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// 1161/AC5, with the store and a real bus and no ticker: a plan of two tiers whose first tier runs on a machine away when it
// is sent finishes that tier without waiting for it and without its bound, sends the second tier to the machine heard
// alone, and says the machine was away (issue 1161). The machine away keeps its want: it was sent nothing of the plan's
// builds, so the send it gets when it is heard again carries them (AC3).
func TestAPlanGoesOnWithoutAMachineAwayWhenItsTierIsSent(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
// laptop is away as every tier is sent; anchor is heard.
wasAway := judgeAway
judgeAway = func(context.Context, *stores, time.Time) (map[string]bool, error) {
return map[string]bool{"laptop": true}, nil
}
t.Cleanup(func() { judgeAway = wasAway })
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
legacy, err := js.Conn().JetStream()
if err != nil {
t.Fatal(err)
}
checkEvents = link.OverNATS{Conn: js.Conn(), JS: legacy}
t.Cleanup(func() { checkEvents = nil })
sentHeard := make(chan *nats.Msg, 32)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeySent), sentHeard)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
wasSaved := inventory.PlanSaved
inventory.PlanSaved = servingPlanSaved(ctx, checkEvents, open)
t.Cleanup(func() { inventory.PlanSaved = wasSaved })
// The build machine and the machines, as far as the plan meets them: an ask is counted, a send is recorded and kept.
var mu sync.Mutex
var asks []string
var sends [][]string
wasAsk := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
mu.Lock()
defer mu.Unlock()
id := link.NewBuildID(time.Now().Add(time.Duration(len(asks)) * time.Millisecond))
asks = append(asks, source.Repository+"#"+id)
return id, nil
}
t.Cleanup(func() { askABuild = wasAsk })
digests := 0
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
mu.Lock()
defer mu.Unlock()
sends = append(sends, append([]string(nil), names...))
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
builds := map[string]string{}
for _, m := range []string{"a", "b"} {
builds[m] = current[m].Commit
}
for _, node := range names {
digests++
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), fmt.Sprintf("d-%s-%d", node, digests), builds); err != nil {
return nil, err
}
}
return names, nil
}
t.Cleanup(func() { sendRollout = wasSend })
register := func(m, commit string, asked time.Time) {
t.Helper()
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: commit},
inventory.Source{Repository: "novox/" + m, Seat: "git", Path: "modules/" + m, BuiltFrom: commit, Head: commit,
Asked: asked}); err != nil {
t.Fatal(err)
}
}
record := func(id, m, commit string, at time.Time) {
t.Helper()
b := inventory.Build{ID: id, Module: m, Commit: commit, Repository: "novox/" + m, Path: "modules/" + m, Asked: at, At: at}
b.Manifest, _ = json.Marshal(catalogue.Manifest{Module: m, Version: commit})
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
old := time.Now().Add(-2 * time.Hour)
for _, m := range []string{"a", "b"} {
record("build-"+m+"-1", m, "c1", old)
register(m, "c1", old)
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.Assign(ctx, n, m); err != nil {
t.Fatal(err)
}
}
}
for _, n := range []string{"anchor", "laptop"} {
if err := inv.RecordSent(ctx, nodeID(t, open, n), "d-"+n+"-c1", map[string]string{"a": "c1", "b": "c1"}); err != nil {
t.Fatal(err)
}
}
record("build-a-2", "a", "c2", time.Now().Add(-time.Minute))
register("a", "c2", time.Now().Add(-time.Minute))
// A machine's report on what it was sent, heard as the controller hears it.
listener := nudgingListener{Enrolment: link.Enrolment{Inventory: inv}}
seq := int64(0)
reports := func(node string) {
t.Helper()
last, found, err := inv.SentTo(ctx, node)
if err != nil || !found {
t.Fatalf("%s was sent nothing: %v", node, err)
}
seq++
body, _ := json.Marshal(map[string]any{"node": node, "applied": []string{"x.server"}, "declared": last.Digest,
"report_sequence": seq})
var report link.Report
if err := json.Unmarshal(body, &report); err != nil {
t.Fatal(err)
}
if _, err := listener.Heard(ctx, report); err != nil {
t.Fatal(err)
}
}
plan := func() inventory.Plan {
t.Helper()
p, err := inv.PlanByID(ctx, "plan-1160")
if err != nil {
t.Fatal(err)
}
return p
}
built := time.Now().UTC()
p := inventory.Plan{ID: "plan-1160", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c2", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}, {"b"}},
Delivery: &inventory.PlanDelivery{Order: []inventory.Edge{{From: "b", To: "a", Kind: edgeChangeOrder}}},
Modules: map[string]*inventory.PlanModule{
"a": {State: "built", BuiltAt: &built, Commit: "c2", Build: "build-a-2"},
"b": {},
}}
first, stopFirst := context.WithCancel(ctx)
defer stopFirst()
waitFirst := startRolloutEvents(first, open, js)
// Ends with the test before the cleanups restore what the look reads, which run after it (issue 1338).
t.Cleanup(func() { stopFirst(); waitFirst() })
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
eventually(t, "tier 0 sent to anchor alone", func() bool {
mu.Lock()
defer mu.Unlock()
return len(sends) == 1 && strings.Join(sends[0], ",") == "anchor"
})
reports("anchor")
// The note of the move to tier 1 is read in the step that sees it: later steps of the tier say their own.
var moved string
eventually(t, "tier 1 reached on anchor's report alone", func() bool {
if q := plan(); q.Tier == 1 {
moved = q.Note
return true
}
return false
})
if !strings.Contains(moved, "laptop was away: sent when it is heard again") || strings.Contains(moved, "anchor") {
t.Errorf("1161/AC2: the plan says %q; want laptop named away and not anchor", moved)
}
// The ask for tier 1's build is the next step, and records its build id on the module.
eventually(t, "tier 1 asked", func() bool { m := plan().Modules["b"]; return m != nil && m.Build != "" })
b := plan().Modules["b"]
askedAt, _ := link.BuildAskedAt(b.Build)
record(b.Build, "b", "c2", askedAt)
register("b", "c2", askedAt)
planBuilt(ctx, open, "b", "c2", "", askedAt, b.Build)
eventually(t, "tier 1 sent to anchor alone", func() bool {
mu.Lock()
defer mu.Unlock()
return len(sends) == 2 && strings.Join(sends[1], ",") == "anchor"
})
// 1161/AC3: laptop was sent nothing of the plan's builds, so it still wants them for the send it gets when heard again.
was, known, err := inv.SentBuilds(ctx, "laptop")
if err != nil || !known {
t.Fatalf("1161/AC3: laptop's last send: %v, known %v", err, known)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
t.Fatal(err)
}
if was["a"] == current["a"].Commit || was["a"] != "c1" {
t.Errorf("1161/AC3: laptop was last sent a at %q and a is now %q; want it still at c1, owed c2", was["a"], current["a"].Commit)
}
if got := plan().Away; strings.Join(got, ",") != "laptop" {
t.Errorf("1161/AC2: the plan keeps %v away; want laptop", got)
}
}
cmd/mesh-controller/away_test.go
-67
View File
@@ -1,67 +0,0 @@
package main
import (
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A machine out of touch when a tier is sent is not sent to and not waited for (novox/hq ADR 0435 §1, issue 1161): the
// walk judges it with the silent check's one verdict, machineFacts.silent, and a sleeping machine is away once presence says so.
// 1161/AC1: away is the silent check's verdict; AC4: a machine heard is not away.
func TestAMachineIsAwayAsTheSilentCheckJudgesIt(t *testing.T) {
now := time.Now().UTC()
machines := []machineFacts{
{name: "heard", lastHeard: now.Add(-time.Minute)},
{name: "quiet", lastHeard: now.Add(-silentBound - time.Minute)},
{name: "stuck", lastHeard: now, presence: presenceSilent},
{name: "back", lastHeard: now.Add(-time.Hour), presence: presenceHeard},
{name: "new"},
}
got := awayOf(machines, now)
for node, want := range map[string]bool{"heard": false, "quiet": true, "stuck": true, "back": false,
"new": false} {
if got[node] != want {
t.Errorf("1161/AC1: %s away is %v; want %v (away: %v)", node, got[node], want, got)
}
}
}
// 1161/AC1, AC2: the plan keeps the machines of the tier it passed as away, sorted and once.
func TestAPlanKeepsTheMachinesItPassedAsAway(t *testing.T) {
p := &inventory.Plan{}
away := map[string]bool{"g14": true, "ace": true}
rememberAway(p, []string{"anchor", "g14"}, away)
rememberAway(p, []string{"g14", "ace"}, away)
if !slices.Equal(p.Away, []string{"ace", "g14"}) {
t.Fatalf("1161/AC2: the plan keeps %v away; want ace, g14", p.Away)
}
}
// 1161/AC2: the plan says it, where its other passed machines are said.
func TestAPlanSaysTheMachinesItPassedAsAway(t *testing.T) {
say := awaySaying(inventory.Plan{Away: []string{"g14"}})
if !strings.Contains(say, "g14 was away: sent when it is heard again") {
t.Fatalf("1161/AC2: the plan says %q", say)
}
if got := awaySaying(inventory.Plan{}); got != "" {
t.Fatalf("1161/AC2: a plan that passed no machine says %q", got)
}
}
// 1161/AC1: the gate does not wait for a machine passed as away, unless a send reached it; AC4: a machine heard is waited for.
func TestAGateDoesNotWaitForAMachinePassedAsAway(t *testing.T) {
p := inventory.Plan{Away: []string{"g14"}}
none := &inventory.PlanModule{}
if got := runningHeard(p, []string{"anchor", "g14"}, none); !slices.Equal(got, []string{"anchor"}) {
t.Fatalf("1161/AC1: the gate waits on %v; want anchor alone", got)
}
sent := &inventory.PlanModule{Rest: map[string]inventory.SentDeclaration{"g14": {Digest: "d"}}}
if got := runningHeard(p, []string{"anchor", "g14"}, sent); !slices.Equal(got, []string{"anchor", "g14"}) {
t.Fatalf("1161/AC4: a machine a send reached is waited for: %v", got)
}
}
cmd/mesh-controller/backups_before_send_integration_test.go
-308
View File
@@ -1,308 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"os"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// aBackupSeatOnTheBus is one machine's node-backup holder as a test makes it: it counts the `now` asks it hears and
// answers `backed-up` with the last night it is told (novox/hq ADR 0404).
type aBackupSeatOnTheBus struct {
mu sync.Mutex
nows []string
night map[string]any // module → lastNight
}
func (f *aBackupSeatOnTheBus) serve(t *testing.T, conn *nats.Conn, node string) {
t.Helper()
reply := func(m *nats.Msg, result any) {
body, _ := json.Marshal(map[string]any{"result": result, "error": "", "node": node})
_ = m.Respond(body)
}
now, err := conn.Subscribe(link.NodeSeatToolSubject(catalogue.BackupSeat, "now", node), func(m *nats.Msg) {
var a struct {
Module string `json:"module"`
}
_ = json.Unmarshal(m.Data, &a)
f.mu.Lock()
f.nows = append(f.nows, a.Module)
f.mu.Unlock()
reply(m, map[string]any{"started": a.Module})
})
if err != nil {
t.Fatal(err)
}
read, err := conn.Subscribe(link.NodeSeatToolSubject(catalogue.BackupSeat, "backed-up", node), func(m *nats.Msg) {
f.mu.Lock()
defer f.mu.Unlock()
var out []map[string]any
for module, n := range f.night {
out = append(out, map[string]any{"module": module, "lastNight": n})
}
reply(m, out)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = now.Unsubscribe(); _ = read.Unsubscribe() })
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
}
func (f *aBackupSeatOnTheBus) answers(module string, ok bool, words string) {
f.mu.Lock()
defer f.mu.Unlock()
f.night = map[string]any{module: map[string]any{"ok": ok, "at": time.Now().UTC().Add(time.Second), "error": words}}
}
func (f *aBackupSeatOnTheBus) asks() int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.nows)
}
// Acceptance test 3 of design 47, on the controller's side: a delivery plan that moves a module keeping data waits
// for its backups on both machines before it asks a build; a backup that fails holds it, with the machine, the
// module and the backup's words, and a condition says the same; the verb the delivery's release calls asks the
// backups again, and the plan goes on. One bus and one store for the scenarios.
func TestADeliveryThatMovesABackedUpModuleWaitsForItsBackupAndAFailedOneHoldsItAndIsSaid(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := asksRecorded(t)
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "keeper", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BackupSeat, Scope: catalogue.ScopeNode, Serves: []string{"backed-up", "now", "restore"}}}})
notes, err := catalogue.ParseManifest([]byte(`{"module":"notes","version":"1","data":{"own":[` +
`{"id":"files","path":"${dir:files}","class":"valuable","why":"people's notes"}]},` +
`"resources":[{"id":"files","type":"directory"}]}`))
if err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, notes, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/notes", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
if _, err := assign(ctx, open, node, "keeper"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, node, "notes"); err != nil {
t.Fatal(err)
}
}
seats := map[string]*aBackupSeatOnTheBus{"anchor": {}, "laptop": {}}
for node, s := range seats {
s.serve(t, js.Conn(), node)
}
// Short waits: the bound and the look are variables, so nothing here waits for the minutes of a real backup.
wasWithin, wasEvery := backupsWithin, backupsPollEvery
backupsWithin, backupsPollEvery = time.Minute, 10*time.Millisecond
t.Cleanup(func() { backupsWithin, backupsPollEvery = wasWithin, wasEvery })
newPlan := func(id string) {
t.Helper()
p := inventory.Plan{ID: id, Repository: "novox/mesh-catalog", Branch: "main", Commit: "c1", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"notes"}}, Modules: map[string]*inventory.PlanModule{"notes": {}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
}
read := func(id string) inventory.Plan {
t.Helper()
p, err := inv.PlanByID(ctx, id)
if err != nil {
t.Fatal(err)
}
return p
}
// 1. Both machines are asked; while one has not answered, no build is asked.
newPlan("plan-one")
advancePlans(ctx, open)
if seats["anchor"].asks() != 1 || seats["laptop"].asks() != 1 || len(*asked) != 0 {
t.Fatalf("asks anchor=%d laptop=%d builds=%v", seats["anchor"].asks(), seats["laptop"].asks(), *asked)
}
seats["anchor"].answers("notes", true, "")
advancePlans(ctx, open)
if len(*asked) != 0 || read("plan-one").Backups.State != inventory.BackupPending {
t.Fatalf("a tier was asked while a backup was pending: %v %+v", *asked, read("plan-one").Backups)
}
seats["laptop"].answers("notes", true, "")
advancePlans(ctx, open)
if len(*asked) != 1 || read("plan-one").Backups.State != inventory.BackupDone {
t.Fatalf("the tier was not asked once every backup was done: %v %+v", *asked, read("plan-one").Backups)
}
finish := read("plan-one")
finish.State = inventory.PlanDone
if err := inv.SavePlan(ctx, &finish); err != nil {
t.Fatal(err)
}
// 2. A backup that fails holds the plan: not failed, nothing asked, the words kept, a condition raised.
seats["anchor"].night, seats["laptop"].night = nil, nil
newPlan("plan-two")
advancePlans(ctx, open)
seats["anchor"].answers("notes", true, "")
seats["laptop"].answers("notes", false, "no space left on device")
advancePlans(ctx, open)
held := read("plan-two")
if held.State != inventory.PlanBuilding || held.Backups.State != inventory.BackupFailed || len(*asked) != 1 {
t.Fatalf("the plan was not held: %s %+v builds=%v", held.State, held.Backups, *asked)
}
var failed *inventory.PlanBackup
for i := range held.Backups.Asked {
if held.Backups.Asked[i].State == inventory.BackupFailed {
failed = &held.Backups.Asked[i]
}
}
if failed == nil || failed.Machine != "laptop" || failed.Module != "notes" || failed.Words != "no space left on device" ||
strings.Contains(held.Note, "tried again") {
t.Fatalf("the failed backup: %+v note=%q", failed, held.Note)
}
advancePlans(ctx, open) // not retried on the next look
if again := read("plan-two"); again.Backups.State != inventory.BackupFailed || len(*asked) != 1 ||
seats["laptop"].asks() != 2 {
t.Fatalf("a held plan was asked again by itself: %+v builds=%v asks=%d", again.Backups, *asked, seats["laptop"].asks())
}
said, err := probeBackupHeld(ctx, &doctor{open: open})
if err != nil || len(said) != 1 || said[0].Kind != kindBackupHeld || !strings.Contains(said[0].Said, "no space left on device") {
t.Fatalf("the condition: %+v %v", said, err)
}
// 3. The verb the release calls: the failed backup is asked again, and only it, and the plan goes on.
seats["laptop"].answers("notes", false, "no space left on device")
if err := deliveryCommand(ctx, []string{"backups-again", "plan-two", "--why", "a person released it", "--by",
"mesh-delivery for jochen"}); err != nil {
t.Fatal(err)
}
if err := deliveryCommand(ctx, []string{"backups-again", "plan-two", "--why", "again"}); err == nil {
t.Fatal("a plan with no failed backup was asked again")
}
seats["laptop"].night = nil
advancePlans(ctx, open)
if seats["laptop"].asks() != 3 || seats["anchor"].asks() != 2 {
t.Fatalf("asked again: laptop=%d anchor=%d", seats["laptop"].asks(), seats["anchor"].asks())
}
seats["laptop"].answers("notes", true, "")
advancePlans(ctx, open)
if got := read("plan-two"); got.Backups.State != inventory.BackupDone || len(*asked) != 2 {
t.Fatalf("the plan did not go on: %+v builds=%v", got.Backups, *asked)
}
if said, _ := probeBackupHeld(ctx, &doctor{open: open}); len(said) != 0 {
t.Fatalf("the condition stays: %+v", said)
}
}
// Issue 1086: a plan cut from a merge asked its first tier at the cut, so a module keeping data was sent without the
// backup the plan promised. The cut now takes the backups first (askFirstTierAtCut, batches.go), and the walk's own
// step asks the tier once they are done.
func TestAPlanCutFromAMergeOfABackedUpModuleAsksItsBackupAndNotItsTierUntilItIsDone(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
everyHeadWaits(t)
asked := asksRecorded(t)
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "keeper", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BackupSeat, Scope: catalogue.ScopeNode, Serves: []string{"backed-up", "now", "restore"}}}})
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
notes, err := catalogue.ParseManifest([]byte(`{"module":"notes","version":"1","data":{"own":[` +
`{"id":"files","path":"${dir:files}","class":"valuable","why":"people's notes"}]},` +
`"resources":[{"id":"files","type":"directory"}]}`))
if err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, notes, inventory.Source{Repository: "novox/notes", Seat: "git", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []string{"keeper", "notes"} {
if _, err := assign(ctx, open, "anchor", m); err != nil {
t.Fatal(err)
}
}
seat := &aBackupSeatOnTheBus{}
seat.serve(t, js.Conn(), "anchor")
wasWithin, wasEvery := backupsWithin, backupsPollEvery
backupsWithin, backupsPollEvery = time.Minute, 10*time.Millisecond
t.Cleanup(func() { backupsWithin, backupsPollEvery = wasWithin, wasEvery })
// 1. A merge of notes is heard and its window closes: the batch is cut through cutBatch.
hear(t, open, repoMerge("notes", "c-notes", t0), t0.Add(time.Second))
cutAt(t, open, t0.Add(time.Second+90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walk(s)", len(ws))
}
// 2. Right after the cut no tier is asked, the record has anchor/notes pending, the seat heard one `now`.
cut, err := inv.PlanByID(ctx, ws[0].ID)
if err != nil {
t.Fatal(err)
}
if len(*asked) != 0 || askedInTier(cut) {
t.Fatalf("the first tier was asked at the cut: %v %+v", *asked, cut.Modules)
}
if cut.Backups == nil || cut.Backups.State != inventory.BackupPending || len(cut.Backups.Asked) != 1 ||
cut.Backups.Asked[0].Machine != "anchor" || cut.Backups.Asked[0].Module != "notes" {
t.Fatalf("the backups of the cut: %+v", cut.Backups)
}
if seat.asks() != 1 {
t.Fatalf("the seat heard %d `now`", seat.asks())
}
// 3. The backup answers done: the walk's own step asks the first tier, once, and asks no second backup.
seat.answers("notes", true, "")
advancePlans(ctx, open)
advancePlans(ctx, open)
after, err := inv.PlanByID(ctx, ws[0].ID)
if err != nil {
t.Fatal(err)
}
if len(*asked) != 1 || after.Backups.State != inventory.BackupDone || seat.asks() != 1 {
t.Fatalf("builds=%v backups=%+v asks=%d", *asked, after.Backups, seat.asks())
}
}
cmd/mesh-controller/batches.go
+153 -399
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"os"
"regexp"
"slices"
"sort"
"strconv"
@@ -26,10 +27,9 @@ import (
// commit in it, with at most one commit per repository, the latest of its branch, which contains the earlier
// ones. The walk names every merge it answers.
//
// - One walk at a time beside a related one (ADR 0407 decision 4, issue 973). Merges that come while a walk
// runs join the next batch; a closed batch that shares nothing with an open walk is cut beside it. A walk
// that started is never superseded; one still waiting for its delivery's word is folded into the walk of a
// batch related to it, which answers every merge it answered — the only supersession left.
// - One walk at a time. Merges that come while a walk runs join the next batch. A walk that started is never
// superseded; one still waiting for its delivery's word is folded into the next batch's walk, which
// answers every merge it answered — the only supersession left.
// - A merge heard after a later merge of its branch was put in a walk is answered by that walk: a batch
// still assembling, an open walk that builds everything it moves, or a walk done that did, which then
// names it at once. Otherwise it joins the next batch.
@@ -39,9 +39,8 @@ import (
//
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, ADR 0306 decisions
// 2 and 3, issue 525): it joins a batch it is related to, and one that is related to none and whose delivery
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Such a batch is cut
// at once, beside any open walk it shares nothing with, and queued behind the first one it is related to
// (cutUnrelated).
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Walks stay one at
// a time: such a batch is cut when no walk is open, and queued behind the one that is otherwise.
//
// A merge on the controller's own path (a module whose walk waits for nobody's word) never shares a batch with
// one that waits for mesh-delivery's: each kind has a batch of its own, an own-path batch is cut first and folds
@@ -62,17 +61,16 @@ var (
// mergeWindowFileVar names the file the controller's window settings are written to.
const mergeWindowFileVar = "MESH_MERGE_WINDOW_FILE"
// batchCutEvery is how often the controller looks whether a batch's window closed.
const batchCutEvery = 5 * time.Second
// batchLateAfter is how long past its maximum a batch may still be assembling, with no walk open, before
// the controller is said to have failed to cut it (ADR 0276 decision 9).
const batchLateAfter = time.Minute
// mergeGrace is how long before a batch's first merge a merge may have been made and still be told apart from one
// made before the controller began keeping merges (owedLate).
const mergeGrace = 10 * time.Minute
// edgeChangeOrder is the order the rules of a plan draw among its modules (novox/hq ADR 0419 clauses 3 and 4, delivery_edges.go)
// made an edge of its tiers: the module after depends on the module before, and is gated on it.
const edgeChangeOrder = "change-order"
// edgeGroupOrder is a delivery group's order made an edge of a walk's tiers (ADR 0276 decision 5): the
// modules of the member after depend on the modules of the member before, and are gated on them.
const edgeGroupOrder = "group-order"
// mergeWindowOf is the window and its maximum, from the controller's settings; a value unreadable is its
// default, and said.
@@ -127,7 +125,8 @@ func durationSetting(v any) (time.Duration, bool) {
// hearMerge puts a merge the forge announced into the batch that answers it, at the moment it was heard,
// and cuts what is due.
func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.Time) error {
// One merge acted on at a time, whoever hands it over (novox/hq issue 266).
// One merge acted on at a time, whoever hands it over: the bus, or the catch-up that reads back
// what the bus did not hand over (novox/hq issue 266).
actingOnMerges.Lock()
defer actingOnMerges.Unlock()
@@ -176,17 +175,12 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
repository, m.Base, m.Commit, already)
return nil
}
if already == 0 {
if done, err := f.firstBuilds(ctx, m, entries, read, now); err != nil || done {
return err
}
}
fmt.Print(nothingReadsIt(repository, m))
fmt.Printf("%s merged into %s (%.8s); nothing the mesh holds reads it\n", repository, m.Base, m.Commit)
return nil
}
// **Its delivery group's word, before the plans are held** (ADR 0306 decision 2): a slow answer holds up no walk.
own := ownPath(touches)
waits, holding, err := groupWaitHolding(ctx, inv, m, deliverySeatHeld(entries))
waits, err := groupWait(ctx, inv, m, deliverySeatHeld(entries), own)
if err != nil {
return notNow(err)
}
@@ -224,10 +218,10 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
}
if p.Batch() {
if err := keepBatch(ctx, inv, &p, now, ""); err != nil {
return unsaved(err)
return notNow(err)
}
} else if err := nameMerges(ctx, inv, &p); err != nil {
return unsaved(err)
return notNow(err)
}
fmt.Printf("%s merged into %s (%.8s) after a later merge of its branch was put in %s (%s): that %s "+
"answers it\n", repository, m.Base, m.Commit, p.ID, p.State, walkOrBatch(p))
@@ -242,55 +236,17 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
if added, err := inv.AddMerge(ctx, heard); err != nil || !added {
return notNow(err)
}
holdFor(&batch, m.Head, waits, holding, own, now)
if err := keepBatch(ctx, inv, &batch, now, ""); err != nil {
return unsaved(err)
return notNow(err)
}
fmt.Printf("%s merged into %s (%.8s); %s, it joins %s — %s\n", repository, m.Base, m.Commit, why, batch.ID,
batchWords(batch, now))
return cutBatchesHeld(ctx, f.open, now)
}
// holdFor keeps in a batch that a merge of a group joined whether that group still has a member to merge (ADR 0418
// clause 3): held while it has, the window running again from the moment the last hold is cleared. A group with a
// merge of the controller's own path makes its batch an own-path one: such a plan waits for nobody's word, and is
// started by the controller when the group's batch closes (clause 6).
func holdFor(batch *inventory.Plan, head, why string, holding, own bool, now time.Time) {
if batch.Delivery == nil {
batch.Delivery = &inventory.PlanDelivery{}
}
if batch.Delivery.Batch == nil {
batch.Delivery.Batch = &inventory.PlanBatch{Own: own}
}
w := batch.Delivery.Batch
w.Own = w.Own || own
if head == "" {
return
}
if holding {
if w.Holds == nil {
w.Holds = map[string]string{}
}
w.Holds[head] = why
return
}
if _, held := w.Holds[head]; held {
holds := map[string]string{}
for h, y := range w.Holds {
if h != head {
holds[h] = y
}
}
w.Holds = holds
if len(holds) == 0 {
w.Holds, w.HeldAt = nil, now
}
}
}
// owedLate says a merge that reads as history is still the record's to answer (review of this change): a cut
// marks every module it moved seen at that moment, so a merge of that branch
// handed over late — made before the cut, heard after it — reads as history and was dropped. One that touches
// marks every module it moved seen at that moment, so a merge of that branch the bus lost and the catch-up
// hands over later — made before the cut, heard after it — reads as history and was dropped. One that touches
// a module the mesh holds, of a branch whose merges the controller keeps, made since it began keeping them,
// is kept; one made before it began is old news, as before.
func owedLate(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, merged time.Time,
@@ -367,7 +323,7 @@ func buildsEvery(p inventory.Plan, modules []inventory.Entry) bool {
// ownPath says a merge touches a module on the controller's own path, whose walk waits for nobody's word.
func ownPath(touches []inventory.Entry) bool {
for _, e := range touches {
if onTheOwnPath(e.Manifest.Module, e.Manifest) {
if _, own := onTheControllersPath[e.Manifest.Module]; own {
return true
}
}
@@ -384,23 +340,14 @@ func ownPath(touches []inventory.Entry) bool {
// names why): a new batch, which waits for the window, beside the ones it shares nothing with; its words say
// why.
// - none, and no member of its group is still to merge: a new batch whose window is closed at once. Cut at
// once means only that: with a walk open that it is related to the batch is queued behind it as any closed
// batch is (ADR 0276 decision 4, ADR 0407 decision 4); beside one it shares nothing with it is cut.
// once means only that: walks stay one at a time (ADR 0276 decision 4), so with a walk open the batch is
// queued behind it as any closed batch is.
func batchToJoin(ctx context.Context, inv *inventory.Inventory, heard inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, now time.Time, own bool, waits string) (inventory.Plan, string, error) {
edges, err := inv.Dependencies(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
// **A group is one batch, whatever kinds its members are** (novox/hq ADR 0418 clause 6): a merge joins the batch
// of the other kind that holds a merge from its head.
if group, ok, err := batchOfGroup(ctx, inv, announcedOf(heard).Head, heard.Repository); err != nil {
return inventory.Plan{}, "", err
} else if ok && group.OwnPath() != own && group.Delivery != nil && group.Delivery.Batch != nil {
group.Delivery.Batch.CutAtOnce = false
return group, "its group " + announcedOf(heard).Head + " has a merge in " + group.ID + ", a batch of the other kind: " +
"a group is one batch", nil
}
joins, whys, _, err := relatedBatches(ctx, inv, []inventory.BatchedMerge{heard}, own, "", entries, read, edges)
if err != nil {
return inventory.Plan{}, "", err
@@ -459,9 +406,7 @@ func relatedBatches(ctx context.Context, inv *inventory.Inventory, merges []inve
}
carry := newFootprint()
if !own {
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch folds those it is related to
// (cutUnrelated). Every one is counted here, which only makes a merge join a batch more readily: it waits
// for the window, never walks beside what it shares something with (issue 973).
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch may fold every one.
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, nil, nil, err
@@ -599,11 +544,6 @@ func newBatch(ctx context.Context, inv *inventory.Inventory, first inventory.Bat
// keepBatch writes a batch as its merges and the clock say it now: the commits it holds, the merges they
// answer, its window, and assembling or queued behind a walk. Saved only when that changed.
func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind string) error {
return keepBatchBecause(ctx, inv, b, now, behind, "")
}
// keepBatchBecause is keepBatch for a batch queued behind a walk it is related to, which its words say (issue 973).
func keepBatchBecause(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind, why string) error {
merges, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return err
@@ -624,42 +564,29 @@ func keepBatchBecause(ctx context.Context, inv *inventory.Inventory, b *inventor
if b.Delivery == nil {
b.Delivery = &inventory.PlanDelivery{}
}
anchor := b.Created
if was := b.Delivery.Batch; was != nil && was.HeldAt.After(anchor) {
anchor = was.HeldAt // the maximum counts from the last hold's end (ADR 0418 clause 3)
}
closes, latest := windowOf(merges, anchor, window, atMost)
closes, latest := windowOf(merges, b.Created, window, atMost)
b.Delivery.Merges = named
var closed, cutAtOnce bool
var withheld []string
var waits string
var holds map[string]string
var heldAt time.Time
if was := b.Delivery.Batch; was != nil {
holds, heldAt = was.Holds, was.HeldAt
closed, withheld = was.Closed, was.Withheld // a stop's, which no look takes back (novox/hq ADR 0299)
cutAtOnce = was.CutAtOnce // the merge's, until another joins it (ADR 0304 decision 1)
waits = was.Waits // its group's, as said when it was made (ADR 0306 decision 2)
}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, BehindWhy: why, Own: b.OwnPath(),
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits, Holds: holds, HeldAt: heldAt}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, Own: b.OwnPath(),
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits}
b.State = inventory.PlanAssembling
if behind != "" && windowClosed(*b, now) {
b.State = inventory.PlanQueued
} else {
b.Delivery.Batch.Behind, b.Delivery.Batch.BehindWhy = "", ""
b.Delivery.Batch.Behind = ""
}
b.Note = batchWords(*b, now)
if unNoted(*b) == before {
return nil
}
if err := inv.SavePlan(ctx, b); err != nil {
return err
}
// The window's clock follows the record, only when the record moved: a merge made the quiet key again, a
// batch that opened made its most (ADR 0407 decision 1).
armWindow(ctx, *b, now)
return nil
return inv.SavePlan(ctx, b)
}
// windowOf is when a batch's window closes, unless another merge comes, and when at the latest: the last
@@ -678,20 +605,13 @@ func windowOf(merges []inventory.BatchedMerge, created time.Time, window, atMost
}
// windowClosed says a batch's window is closed: closed by a stop, closed at once for a merge that shares
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached. **A batch holding a group
// with a member still to merge is closed by none of the last three** (novox/hq ADR 0418 clause 3).
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached.
func windowClosed(b inventory.Plan, now time.Time) bool {
if b.Delivery == nil || b.Delivery.Batch == nil {
return true
}
w := b.Delivery.Batch
if w.Closed {
return true
}
if len(w.Holds) > 0 {
return false // a group's last member is still to merge: neither the window nor its maximum closes the batch
}
return w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
return w.Closed || w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
}
// carriedOf is a set of merges as a walk carries them: the latest merge of each repository's branch, and
@@ -747,9 +667,6 @@ func carriedOf(merges []inventory.BatchedMerge) ([]inventory.PlanCommit, []inven
type keptMerge struct {
link.SourceMoved
Moves []string `json:"moves,omitempty"`
// FirstBuild says the merge is a first merge into a repository the mesh builds nothing from, kept to ask the first
// build of its modules once mesh-delivery's word on it is kept (novox/hq ADR 0432, issue 1146).
FirstBuild bool `json:"first_build,omitempty"`
}
// announcedOf is a kept merge as the forge announced it, and what it moved; empty where the record says none.
@@ -771,7 +688,7 @@ func spelledOf(m inventory.BatchedMerge) string {
func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge {
k := announcedOf(m)
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves,
Head: k.HeadSHA, Merged: m.Merged, Heard: m.Heard}
Merged: m.Merged, Heard: m.Heard}
}
// laterMerge says a was merged after b: by the forge's merge time, then by when each was heard.
@@ -822,18 +739,33 @@ func nameMerges(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan
return inv.SavePlan(ctx, p)
}
// batchCutter cuts what is due on a timer, until the context ends: a window closes with no merge to say so.
func batchCutter(ctx context.Context, open *stores) {
tick := time.NewTicker(batchCutEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
release, err := open.inventory.HoldPlans(ctx, false)
if err != nil {
continue // whoever holds the plans moves them, and cuts at the end
}
if err := cutBatchesHeld(ctx, open, time.Now().UTC()); err != nil {
fmt.Printf("batches: %v\n", err)
}
release()
}
}
// cutBatchesHeld takes the step that is due, for a caller holding the plans: after a failed walk, the search
// for the merge that brought the failure; with no walk open, the next merge of that search walked alone; else
// each closed batch cut into a walk with the waiting walks it is related to folded in, unless it is related to a
// walk open, behind which it is said queued (cutUnrelated). While the search runs everything waits.
// for the merge that brought the failure; with no walk open, the next merge of that search walked alone, or a
// closed batch cut into one walk with the waiting walks folded in; with a walk open, a closed batch said
// queued behind it.
func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
inv := open.inventory
// A merge found in two records is the half-done move of a crash: put right before anything is decided from it.
if repaired, err := inv.RepairMerges(ctx); err != nil {
return err
} else if repaired > 0 {
fmt.Printf("%d merge(s) found in two plans, the half of a move a restart interrupted, are kept by the plan they were moved to\n", repaired)
}
if err := searchAfterFailures(ctx, inv); err != nil {
return err
}
@@ -848,7 +780,8 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
if err != nil {
return err
}
var started, waiting []inventory.Plan
var started *inventory.Plan
var waiting []inventory.Plan
for i := range plans {
p := plans[i]
if p.Release != nil {
@@ -856,19 +789,14 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
}
if p.Waiting() {
waiting = append(waiting, p)
} else {
started = append(started, p)
} else if started == nil {
started = &plans[i]
}
}
batches, err := inv.Batches(ctx)
if err != nil {
return err
}
// **A hold is looked at again at every cut** (ADR 0418 clause 3): a group whose last member merged or closed
// since is released, and its window runs again from now.
if batches, err = releaseHolds(ctx, inv, batches, now); err != nil {
return err
}
keepAll := func(behind string) error {
for i := range batches {
if err := keepBatch(ctx, inv, &batches[i], now, behind); err != nil {
@@ -877,155 +805,46 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
}
return nil
}
if started != nil {
// **One walk at a time** (ADR 0276 decision 4): every batch waits behind it, whatever its window says.
return keepAll(started.ID)
}
if len(alone) > 0 {
// A failed walk's search walks its merges alone, one at a time: everything waits for it (kept as it was,
// issue 973).
if len(started) > 0 {
return keepAll(started[0].ID)
}
if len(waiting) > 0 {
// The waiting walk is the one open; the search walks after it, and the batches wait behind both.
return keepAll(waiting[0].ID)
}
return walkAlone(ctx, open, alone[0], now)
}
return cutUnrelated(ctx, open, batches, started, waiting, now)
}
// releaseHolds asks mesh-delivery again about each group a batch holds, and clears the holds of those with no member
// left to merge. A group mesh-delivery cannot be asked about stays held: the safe error is to wait. It answers the
// batches as kept.
func releaseHolds(ctx context.Context, inv *inventory.Inventory, batches []inventory.Plan, now time.Time) ([]inventory.Plan, error) {
var entries []inventory.Entry
for i := range batches {
b := &batches[i]
if b.Delivery == nil || b.Delivery.Batch == nil || len(b.Delivery.Batch.Holds) == 0 {
continue
}
if entries == nil {
var err error
if entries, err = inv.Catalogued(ctx); err != nil {
return nil, err
}
}
changed := false
for head := range b.Delivery.Batch.Holds {
open, isGroup, _, err := openMembersOf(ctx, inv, deliverySeatHeld(entries), head, "")
if err != nil || (isGroup && len(open) > 0) {
continue
}
holdFor(b, head, "", false, false, now)
changed = true
fmt.Printf("%s: group %s has no member left to merge; its window runs again from now\n", b.ID, head)
}
if changed {
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return nil, err
}
}
if err := keepAll(""); err != nil {
return err
}
return batches, nil
}
// cutUnrelated cuts the closed batches that share nothing with an open walk, each into a walk beside the open
// ones, and says the others queued behind the first walk they are related to (novox/hq ADR 0407 decision 4,
// issue 973). A batch cut in this look counts as open for the batches after it.
//
// - An own-path batch goes first and folds no waiting walk; a catalogue batch folds the waiting walks it is
// related to, and those it shares nothing with keep waiting.
// - **The controller's own path is related to everything** (decided for issue 973): a walk or batch that
// replaces the controller, the node-engine, the tool runner, the bus or mesh-delivery never runs beside
// another, and nothing runs beside it.
// - A batch whose window is still open is assembling; one related to an open walk is queued behind it, and
// ADR 0276 decision 4 stands for it: nothing supersedes a walk that started.
func cutUnrelated(ctx context.Context, open *stores, batches, started, waiting []inventory.Plan, now time.Time) error {
inv := open.inventory
// The oldest batch whose window closed is cut, an own-path one before a catalogue one (it goes first, and
// the catalogue batch queues behind it rather than being cut and overtaken). Several of a kind may be open
// since a merge that shares nothing is cut at once (ADR 0304 decision 1): walks stay one at a time, so one
// is cut per look, and once its walk started the others are queued behind it.
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
var entries []inventory.Entry
var read map[string][]inventory.ReadRepository
var edges []inventory.Edge
judged := false
onPath, err := ownPathNow(ctx, inv)
if err != nil {
return err
}
openWalks, err := openWalksOf(ctx, inv, started, onPath)
if err != nil {
return err
}
for i := range batches {
b := &batches[i]
if !windowClosed(*b, now) {
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return err
}
continue
}
if !judged {
var err error
if entries, err = inv.Catalogued(ctx); err != nil {
if b.OwnPath() {
// A walk on the controller's own path folds no walk waiting for its word (decided during the build,
// 2026-10-10): that walk keeps waiting beside it, and starts once this one ended (advanceOnce).
waiting = nil
}
if err := cutBatch(ctx, open, b, waiting, now); err != nil {
return err
}
// The batches left wait behind the walk just cut when it started; the next look does the same.
if b.Open() && !b.Waiting() {
if batches, err = inv.Batches(ctx); err != nil {
return err
}
if read, err = readForPlanning(ctx, inv); err != nil {
return err
}
if edges, err = inv.Dependencies(ctx); err != nil {
return err
}
judged = true
}
held, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return err
}
fp := footprintAsCut(held, entries, read, edges)
var fold, keep []inventory.Plan
if !b.OwnPath() {
for _, w := range waiting {
ww, err := openWalkOf(ctx, inv, w, onPath)
if err != nil {
return err
}
if ok, _ := related(fp, ww.fp, edges); ok {
fold = append(fold, w)
fp = unionFootprint(fp, ww.fp)
} else {
keep = append(keep, w)
}
}
}
if behind, why := relatedOpen(fp, b.OwnPath(), openWalks, edges); behind != "" {
if err := keepBatchBecause(ctx, inv, b, now, behind, why); err != nil {
return err
}
continue
}
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return err
}
if err := cutBatch(ctx, open, b, fold, now); err != nil {
return err
}
waiting = keep
if c := windowClockOf(); c != nil {
c.Forget(ctx, b.ID) // a batch cut has no window left to close
}
cut, err := inv.PlanByID(ctx, b.ID)
if err != nil {
return err
}
if !cut.Open() {
continue
}
w, err := openWalkOf(ctx, inv, cut, onPath)
if err != nil {
return err
}
if cut.Waiting() {
waiting = append(waiting, cut)
} else {
openWalks = append(openWalks, w)
return keepAll(b.ID)
}
return nil
}
return nil
}
@@ -1120,16 +939,11 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
commits, named := carriedOf(merges)
var moved []string
var members []batchMember
unchanged := map[string]string{}
for _, m := range combinedMerges(merges) {
// **News, whatever was seen since** (review of this change): each merge was judged when it was heard, and
// a cut, a fold or a failed walk since marks its modules seen — so a merge walked alone after a failed
// walk, or folded, would read as history and move nothing. When it was made is not asked again here.
m.MergedAt = ""
// **Unchanged, said** (novox/hq issue 611): read before the merge is acted on, from the same records.
for n, why := range unchangedOfMerge(m, entries, read) {
unchanged[n] = why
}
names, err := movesOfMerge(ctx, inv, m, entries, read)
if err != nil {
return err
@@ -1157,13 +971,9 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
}
}
sort.Strings(also)
ordered, err := changeOrderEdges(ctx, merges, members, entries, moved, edges)
if err != nil {
return err
}
ordered := groupOrderEdges(members, entries, read, edges)
var withheld []string
plan := planOfMoves(moved, append(append([]inventory.Edge{}, edges...), ordered...))
placedBy(&plan, ordered)
// **Withheld, a stopped merge's modules** (novox/hq ADR 0299): a module is built at its branch, which holds the
// merge a stop left out of the walk these merges were walked again from — built here only as a dependent, it
// would deliver that merge. One a merge of this batch moves is built: that merge carries the stopped change,
@@ -1185,22 +995,12 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
plan.Delivery.Merges = named
plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone
plan.Delivery.Withheld = withheld
plan.Delivery.Order = ordered
// **Left where it is, a module whose program did not change** (novox/hq issue 721): its check found it the same as
// the build the mesh holds.
for n, why := range leaveSameAsRunning(ctx, inv, &plan, moved, append(append([]inventory.Edge{}, edges...), ordered...)) {
unchanged[n] = why
}
plan.Delivery.Unchanged = unchangedLeft(unchanged, plan.Modules)
// **Its moments and its class** (novox/hq ADR 0282 decision 6): measured, never acted on.
plan.Times = walkTimesAtCut(*batch, plan, entries, now)
if len(moved) == 0 || len(plan.Modules) == 0 {
if len(moved) == 0 {
plan.State = inventory.PlanDone
plan.Tiers = [][]string{}
plan.Note = "moved nothing the mesh holds: nothing to walk"
if len(plan.Delivery.Unchanged) > 0 {
plan.Note = "moved nothing the mesh holds: nothing to walk; " + unchangedSaid(plan.Delivery.Unchanged)
}
if err := inv.SavePlan(ctx, &plan); err != nil {
return err
}
@@ -1217,14 +1017,7 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
}
var tiers []string
for i, t := range plan.Tiers {
var names []string
for _, n := range t {
if m := plan.Modules[n]; m != nil && m.Placed != "" {
n += " " + m.Placed
}
names = append(names, n)
}
tiers = append(tiers, fmt.Sprintf("%d: %s", i, strings.Join(names, ", ")))
tiers = append(tiers, fmt.Sprintf("%d: %s", i, strings.Join(t, ", ")))
}
var answers []string
for _, n := range named {
@@ -1242,9 +1035,7 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil
}
// Written and its first tier asked as one act on the plans (novox/hq issue 213), under the caller's hold.
if err := askFirstTierAtCut(&plan,
func(p *inventory.Plan) (bool, error) { return backupsFirst(ctx, inv, p) },
func() error { return askTier(ctx, inv, &plan, plan.Tiers[plan.Tier]) }); err != nil {
if err := askTier(ctx, inv, &plan); err != nil {
return err
}
if err := inv.SavePlan(ctx, &plan); err != nil {
@@ -1254,65 +1045,6 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil
}
// askFirstTierAtCut asks a freshly cut plan's first tier, after the backups of the data it moves (novox/hq ADR 0404,
// issue 1086): the cut asks the tier itself, so advanceOnce, which asks the backups before a first tier, finds it
// asked and never takes them. A plan whose backups wait, failed or could not be worked out asks nothing; it is saved
// unasked and the walk's own step goes on from its record.
func askFirstTierAtCut(p *inventory.Plan, backups func(*inventory.Plan) (bool, error), ask func() error) error {
proceed, err := backups(p)
if err != nil {
p.Note = backupsNote + "not asked, the backups of the plan could not be worked out: " + err.Error() +
"; nothing is asked or sent, and the plan tries again"
return nil
}
if !proceed {
return nil
}
return ask()
}
// placedBy writes on each module of a plan the edges of its order that put it after another module of the plan, with the
// rule that drew each (novox/hq ADR 0419 clause 5).
func placedBy(plan *inventory.Plan, ordered []inventory.Edge) {
for _, e := range ordered {
m, in := plan.Modules[e.From]
if _, before := plan.Modules[e.To]; !in || !before {
continue
}
said := "after " + e.To + " (" + e.Why + ")"
if m.Placed == "" {
m.Placed = said
} else if !strings.Contains(m.Placed, said) {
m.Placed += "; " + said
}
}
}
// unchangedSaid names the modules a walk says matched what runs, for its note.
func unchangedSaid(unchanged map[string]string) string {
names := make([]string, 0, len(unchanged))
for n := range unchanged {
names = append(names, n)
}
sort.Strings(names)
return "unchanged, the build the mesh holds is theirs: " + strings.Join(names, ", ")
}
// unchangedLeft is what a walk says matched what runs: every module a merge left unchanged that the walk does not
// build anyway, as another merge's module or a dependent (novox/hq issue 611). Nil when none is.
func unchangedLeft(unchanged map[string]string, builds map[string]*inventory.PlanModule) map[string]string {
out := map[string]string{}
for n, why := range unchanged {
if _, built := builds[n]; !built {
out[n] = why
}
}
if len(out) == 0 {
return nil
}
return out
}
// withhold takes from a plan every module named that its merges do not move, and the tiers left empty: the
// modules taken.
func withhold(plan *inventory.Plan, names, moved []string) []string {
@@ -1395,9 +1127,8 @@ func classOfModules(modules []string, entries []inventory.Entry) string {
}
}
}
onPath := ownPathOf(entries)
for _, m := range modules {
if _, own := onPath[m]; own || resolvers[m] {
if _, own := onTheControllersPath[m]; own || resolvers[m] {
return inventory.ClassCore
}
}
@@ -1477,46 +1208,73 @@ type batchMember struct {
moved []string
}
// changeOrderEdges is the order of a batch's moved modules as edges of its plan's tiers (novox/hq ADR 0419): drawn
// per module from the module graph and from what the merges say of the manifests they touch (delivery_edges.go),
// whatever repository the merges are in and whether or not they form a delivery group. No `after:` line is read.
// A cycle is said, never lost: the plan puts the modules left in its last tier, and the composed check fails the
// group before it merges.
func changeOrderEdges(ctx context.Context, merges []inventory.BatchedMerge, members []batchMember,
entries []inventory.Entry, moved []string, edges []inventory.Edge) ([]inventory.Edge, error) {
// What each repository's branch held before the batch: the first parent of its earliest merge in it.
earliest := map[string]inventory.BatchedMerge{}
for _, k := range merges {
key := strings.ToLower(k.Repository) + "\x00" + k.Branch
if e, ok := earliest[key]; !ok || laterMerge(e, k) {
earliest[key] = k
}
}
var sources []changeSource
// afterLines are the `after: <repository>` lines of a pull request's description, as mesh-delivery reads them.
var afterLines = regexp.MustCompile(`(?im)^\s*after:\s*([A-Za-z0-9_.\-/]+)\s*$`)
// groupOrderEdges is the order of every delivery group among a batch's merges as edges of its walk's tiers
// (ADR 0276 decision 5, ADR 0249 unchanged): merges of two repositories or more from one head branch name are
// a group; each pair its order rules give puts the modules of the member after in a later tier than the
// member before's. A pair in a contradiction orders nothing: its group was refused before it merged.
func groupOrderEdges(members []batchMember, entries []inventory.Entry, read map[string][]inventory.ReadRepository,
edges []inventory.Edge) []inventory.Edge {
byHead := map[string][]batchMember{}
for _, m := range members {
s := changeSource{Owner: m.merge.Owner, Repo: m.merge.Repo, New: m.merge.Commit,
Paths: m.merge.Paths, Truncated: m.merge.PathsTruncated, Removed: m.merge.Removed}
if e, ok := earliest[strings.ToLower(m.merge.Owner+"/"+m.merge.Repo)+"\x00"+m.merge.Base]; ok {
s.Since = e.Commit
if m.merge.Head != "" {
byHead[m.merge.Head] = append(byHead[m.merge.Head], m)
}
if s.Truncated {
s.Paths, s.Truncated = truncatedPaths(entries, s.Owner, s.Repo, moved), false
}
var out []inventory.Edge
heads := make([]string, 0, len(byHead))
for h := range byHead {
heads = append(heads, h)
}
sort.Strings(heads)
for _, h := range heads {
group := byHead[h]
repos := map[string]bool{}
for _, m := range group {
repos[strings.ToLower(m.merge.Owner+"/"+m.merge.Repo)] = true
}
if len(repos) < 2 {
continue
}
var om []orderMember
moved := map[string][]string{}
for _, m := range group {
repository := m.merge.Owner + "/" + m.merge.Repo
id := repository + "@" + m.merge.Commit
var after []string
for _, l := range afterLines.FindAllStringSubmatch(m.merge.Body, -1) {
after = append(after, strings.TrimSuffix(l[1], ".git"))
}
om = append(om, orderMember{ID: id, Repository: repository, Base: m.merge.Base, Head: m.merge.Commit,
Number: m.merge.Number, Paths: m.merge.Paths, PathsTruncated: m.merge.PathsTruncated,
Removed: m.merge.Removed, ModuleDirs: m.merge.ModuleDirs, ModuleDirsSaid: m.merge.ModuleDirsSaid,
CloneURL: m.merge.CloneURL, After: after})
moved[id] = m.moved
}
order := orderOf(om, reachOfMembers(om, entries, read, edges), edges)
contradicted := map[string]bool{}
for _, c := range order.Contradictions {
for _, id := range c.Members {
contradicted[id] = true
}
}
for _, p := range order.Pairs {
if contradicted[p.Before] && contradicted[p.After] {
continue
}
for _, before := range moved[p.Before] {
for _, after := range moved[p.After] {
if before != after {
out = append(out, inventory.Edge{From: after, To: before, Kind: edgeGroupOrder})
}
}
}
fmt.Printf(" group %s: %s before %s (%s) inside the walk\n", h, p.Before, p.After, p.Why)
}
sources = append(sources, s)
}
changes, err := changesOf(ctx, sources)
if err != nil {
return nil, fmt.Errorf("what the merges say of their manifests cannot be read, so the order of their modules "+
"cannot be drawn: %w", err)
}
order := orderEdges(moved, changes, edges)
for _, e := range order.Edges {
fmt.Printf(" order: %s\n", edgeSaid(e))
}
if said := order.CycleSaid(); said != "" {
fmt.Printf(" %s\n", said)
}
return order.Edges, nil
return out
}
// searchAfterFailures starts the search a failed walk leaves (ADR 0276 decision 3): every merge it answered
@@ -1599,11 +1357,7 @@ func batchWords(b inventory.Plan, now time.Time) string {
grouped += " (on the controller's own path)"
}
if b.State == inventory.PlanQueued {
because := ""
if w.BehindWhy != "" {
because = " (" + w.BehindWhy + ")"
}
return fmt.Sprintf("queued behind %s%s; grouped: %s; plan not yet calculated", w.Behind, because, grouped)
return fmt.Sprintf("queued behind %s; grouped: %s; plan not yet calculated", w.Behind, grouped)
}
if w.CutAtOnce {
return fmt.Sprintf("cut at once, sharing nothing with a batch assembling; grouped: %s; plan not yet calculated",
cmd/mesh-controller/batches_integration_test.go
-698
View File
@@ -1,698 +0,0 @@
//go:build integration
package main
import (
"bytes"
"io"
"os"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func windowed(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []struct{ module, repository, path string }{
{"app", "novox/mesh-catalog", "modules/app"},
{"notes", "novox/mesh-catalog", "modules/notes"},
{"one", "novox/one", ""},
{"two", "novox/two", ""},
{"three", "novox/three", ""},
} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1"},
inventory.Source{Repository: m.repository, Seat: "git", Path: m.path, Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
}
return open
}
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
if err := (following{open: open}).hearMerge(t.Context(), m, now); err != nil {
t.Fatal(err)
}
}
func cutAt(t *testing.T, open *stores, now time.Time) {
t.Helper()
if err := cutBatchesHeld(t.Context(), open, now); err != nil {
t.Fatal(err)
}
}
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, claude, t0.Add(time.Second))
hear(t, open, dunst, t0.Add(18*time.Second))
if ws, bs := walks(t, open); len(ws) != 0 || len(bs) != 1 {
t.Fatalf("within the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
cutAt(t, open, t0.Add(18*time.Second+89*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window closed: %+v", ws)
}
cutAt(t, open, t0.Add(18*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if w.Commit != dunst.Commit || len(w.Commits) != 1 || w.Commits[0].Commit != dunst.Commit {
t.Fatalf("the walk is at %s %+v, not the later commit", w.Commit, w.Commits)
}
for _, m := range []string{"app", "notes"} {
if _, in := w.Modules[m]; !in {
t.Fatalf("the walk does not build %s, which one of its merges moved: %v", m, w.Modules)
}
}
want := []inventory.PlanMerge{{Repository: "novox/mesh-catalog", Commit: claude.Commit, Carried: dunst.Commit},
{Repository: "novox/mesh-catalog", Commit: dunst.Commit}}
if w.Delivery == nil || !slices.EqualFunc(w.Delivery.Merges, want, sameMerge) {
t.Fatalf("the walk answers %+v, want %+v", w.Delivery, want)
}
if len(*asked) != 2 || (*asked)[0][2] != "main" || (*asked)[1][2] != "main" {
t.Fatalf("the walk did not ask its modules at the branch, which holds the commit it carries: %v", *asked)
}
// Heard again, as the bus may hand it over twice: never a second merge, nor a second walk.
hear(t, open, claude, t0.Add(5*time.Minute))
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 {
t.Fatalf("a merge heard twice made %d walk(s) and %d batch(es)", len(ws), len(bs))
}
}
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if len(w.Commits) != 3 || len(w.Delivery.Merges) != 3 {
t.Fatalf("the walk carries %+v and answers %+v", w.Commits, w.Delivery.Merges)
}
for _, r := range []string{"one", "two", "three"} {
if w.CommitOf("novox/"+r) != "c-"+r {
t.Fatalf("the walk carries %s at %q", r, w.CommitOf("novox/"+r))
}
if _, in := w.Modules[r]; !in {
t.Fatalf("the walk does not build %s: %v", r, w.Modules)
}
}
// Each module's ask is recorded at its own repository's commit.
for _, r := range []string{"one", "two", "three"} {
q, found, err := open.inventory.BuildRequestByID(t.Context(), w.Modules[r].Build)
if err != nil || !found || q.Commit != "c-"+r {
t.Fatalf("%s's build was not recorded at its own commit: %+v %v %v", r, q, found, err)
}
}
}
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
}
cutAt(t, open, t0.Add(170*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || len(ws[0].Commits) != 2 {
t.Fatalf("the window did not close 90 seconds after its last merge: %+v", ws)
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
}
break
}
}
ws, _ := walks(t, busy)
if len(ws) != 1 {
t.Fatalf("ten minutes of merges a minute apart were never cut: %d walk(s)", len(ws))
}
}
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
claude.Number, claude.Title = 174, "claude-code: an agent proposes a section"
dunst := catalogueMerge("48bda475dunst", "notes", now.Add(-2*time.Second))
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
for _, want := range []string{"assembling: ", " s left (at the latest ", "grouped: novox/mesh-catalog@48bda475 " +
"(answers 553b7191), novox/one@a6bc0931; plan not yet calculated"} {
if !strings.Contains(first, want) {
t.Fatalf("plans' first line %q does not say %q", first, want)
}
}
// Under it, one line per repository: the pull request, what it answers, what it moves.
if len(lines) < 3 || strings.TrimSpace(lines[1]) != "mesh-catalog #175 dunst: the font the operator chose (answers "+
"#174 claude-code: an agent proposes a section) · app, notes" ||
strings.TrimSpace(lines[2]) != "one a6bc0931 · one" {
t.Fatalf("the grouped list reads %q", lines[1:4])
}
if len(said) == 0 || said[len(said)-1].State != inventory.PlanAssembling || said[len(said)-1].Delivery.Batch == nil ||
len(said[len(said)-1].Delivery.Merges) != 3 {
t.Fatalf("the batch was not said as assembling with its merges: %+v", said)
}
}
func captured(t *testing.T, run func() error) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
runErr := run()
os.Stdout = was
_ = w.Close()
var b bytes.Buffer
_, _ = io.Copy(&b, r)
if runErr != nil {
t.Fatal(runErr)
}
return b.String()
}
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first batch was not cut: %+v", ws)
}
first := ws[0]
hear(t, open, repoMerge("one", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(5*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
t.Fatalf("the merge during the walk: %d walk(s), batches %+v", len(ws), bs)
}
if got, _ := open.inventory.PlanByID(t.Context(), first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the next merge", got.State)
}
if !strings.HasPrefix(batchWords(bs[0], t0.Add(5*time.Minute)), "queued behind "+first.ID) {
t.Fatalf("a queued batch reads %q", batchWords(bs[0], t0.Add(5*time.Minute)))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(6*time.Minute))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut when the walk ended: %+v %+v", ws, bs)
}
}
func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
waiting := ws[0]
hear(t, open, repoMerge("one", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
ws, bs := walks(t, open)
if len(bs) != 0 || len(ws) != 2 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
folded, err := open.inventory.PlanByID(ctx, waiting.ID)
if err != nil {
t.Fatal(err)
}
newer := ws[0]
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
}
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c2" {
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
}
if _, in := newer.Modules["one"]; !in {
t.Fatalf("what the folded walk was to build is not built: %v", newer.Modules)
}
}
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, later, t0.Add(18*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
// Moves only notes, which the done walk built from the branch after it: answered there, at once.
hear(t, open, catalogueMerge("553b7191early", "notes", t0), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if got.State != inventory.PlanDone || len(got.Delivery.Merges) != 2 ||
!sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: "553b7191early",
Carried: later.Commit}) {
t.Fatalf("the late merge is not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// Moves app, which that walk never built: the next batch walks it.
hear(t, open, catalogueMerge("1111aaaaapp", "app", t0.Add(time.Second)), t0.Add(16*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "1111aaaaapp" {
t.Fatalf("a late merge moving what the walk never built did not join the next batch: %+v", bs)
}
}
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
oldest := catalogueMerge("aaaa0001", "app", t0.Add(-3*time.Minute))
middle := catalogueMerge("bbbb0002", "app", t0.Add(-3*time.Minute+10*time.Second))
newest := catalogueMerge("cccc0003", "notes", t0.Add(-3*time.Minute+20*time.Second))
for i, m := range []link.SourceMoved{oldest, middle, newest} {
hear(t, open, m, t0.Add(time.Duration(i)*10*time.Second+time.Second))
}
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
failed := ws[0]
failed.State, failed.Note = inventory.PlanFailed, "notes failed to build in tier 0"
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, failed.ID)
if got.State != inventory.PlanFailed || len(got.Delivery.Merges) != 3 {
t.Fatalf("the failed walk's record changed: %s %+v", got.State, got.Delivery.Merges)
}
ws, _ = walks(t, open)
alone := ws[0]
if alone.ID == failed.ID || alone.Commit != middle.Commit || len(alone.Delivery.Merges) != 1 ||
!sameMerge(alone.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: middle.Commit}) {
t.Fatalf("the newest earlier merge was not walked alone on its own commit: %+v", alone)
}
if _, in := alone.Modules["app"]; !in || (*asked)[len(*asked)-1] != [3]string{"novox/mesh-catalog", "modules/app",
middle.Commit} {
t.Fatalf("the merge walked alone does not build app at its own commit: %v, asked %v", alone.Modules, *asked)
}
// Retried, the failed walk would name merges the search answers now.
if _, err := retryPlan(ctx, open, failed.ID); err == nil || !strings.Contains(err.Error(), "walked alone") {
t.Fatalf("a failed walk whose merges are searched was retried: %v", err)
}
alone.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &alone); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(4*time.Minute))
ws, bs := walks(t, open)
if ws[0].ID != alone.ID || len(bs) != 0 {
t.Fatalf("the search went on after a merge was delivered: %+v", ws[0])
}
done, _ := open.inventory.PlanByID(ctx, alone.ID)
if len(done.Delivery.Merges) != 2 || !sameMerge(done.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog",
Commit: oldest.Commit, Carried: middle.Commit}) {
t.Fatalf("the oldest merge is not answered by the walk that delivered the one after it: %+v", done.Delivery.Merges)
}
// A stopped walk starts no search: a person ended it.
hear(t, open, catalogueMerge("dddd0004", "app", t0.Add(5*time.Minute)), t0.Add(5*time.Minute))
hear(t, open, catalogueMerge("eeee0005", "notes", t0.Add(5*time.Minute+time.Second)), t0.Add(5*time.Minute+time.Second))
cutAt(t, open, t0.Add(8*time.Minute))
ws, _ = walks(t, open)
if _, err := stopWalk(ctx, open.inventory, ws[0].ID, "mesh-delivery for jochen", "not now"); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(9*time.Minute))
if again, _ := walks(t, open); again[0].ID != ws[0].ID {
t.Fatalf("a stopped walk's earlier merge was walked alone: %+v", again[0])
}
}
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1"},
inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
host := repoMerge("mesh-host", "h1", t0)
ctl := repoMerge("mesh-controller", "k1", t0.Add(time.Second))
host.Head, ctl.Head = "feat/together", "feat/together"
if !grouped {
ctl.Head = "feat/alone"
}
hear(t, open, ctl, t0.Add(2*time.Second))
hear(t, open, host, t0.Add(3*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walks", len(ws))
}
tiers := ws[0].Tiers
if grouped {
if len(tiers) != 2 || !slices.Equal(tiers[0], []string{"mesh-host"}) ||
!slices.Equal(tiers[1], []string{"mesh-controller"}) {
t.Fatalf("the group's order is not the walk's tiers: %v", tiers)
}
} else if len(tiers) != 1 {
t.Fatalf("two merges of no group were ordered: %v", tiers)
}
}
}
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
}
cutAt(t, again, t0.Add(120*time.Second))
ws, bs := walks(t, again)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller did not cut the batch at its window: %+v %+v", ws, bs)
}
}
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
hear(t, open, c2, t0)
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0.Add(-60*time.Second)), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the late merge was not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// A walk of two that built only two: a late merge of one is not its to name.
hear(t, open, repoMerge("two", "d2", t0.Add(16*time.Minute)), t0.Add(16*time.Minute))
cutAt(t, open, t0.Add(18*time.Minute))
ws, _ = walks(t, open)
w = ws[0]
delete(w.Modules, "two")
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("two", "d1", t0.Add(15*time.Minute)), t0.Add(20*time.Minute))
if got, _ := open.inventory.PlanByID(ctx, w.ID); len(got.Delivery.Merges) != 1 {
t.Fatalf("a walk that never built what the late merge moves named it: %+v", got.Delivery.Merges)
}
if _, bs := walks(t, open); len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "d1" {
t.Fatalf("the late merge did not join the next batch: %+v", bs)
}
}
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(2*time.Minute))
started, _ := walks(t, open)
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/one", Branch: "main", Commit: "d1", Created: t0,
State: inventory.PlanBuilding, Tiers: [][]string{{"one"}}, Modules: map[string]*inventory.PlanModule{"one": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if _, err := letGo(ctx, open.inventory, waiting.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatalf("the word was refused beside %s: %v", started[0].ID, err)
}
before := len(*asked)
advanceHeld(ctx, open)
got, _ := open.inventory.PlanByID(ctx, waiting.ID)
if len(*asked) != before || !strings.Contains(got.Note, "starts once "+started[0].ID) {
t.Fatalf("a walk let go beside a started one asked (%d → %d) or does not say it waits: %q", before, len(*asked), got.Note)
}
// Read as a wait, an hour on: its note on the line, never LATE, and no tier of it late for S3.
later := time.Now().Add(time.Hour)
if line := planLine(got, later); !strings.Contains(line, "starts once "+started[0].ID) || strings.Contains(line, "LATE") {
t.Fatalf("a deferred walk reads %q", line)
}
facts, _, err := gatherPlans(ctx, open.inventory, later, nil)
if err != nil {
t.Fatal(err)
}
for _, f := range facts {
if f.id == got.ID {
t.Fatalf("a deferred walk is watched as a tier running late: %+v", f)
}
}
done := started[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) != before+1 {
t.Fatalf("the walk did not start once the started one ended: asked %v", *asked)
}
}
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("aaaa0001", "app", t0), t0)
hear(t, open, repoMerge("mesh-controller", "k1", t0.Add(10*time.Second)), t0.Add(10*time.Second))
hear(t, open, catalogueMerge("bbbb0002", "notes", t0.Add(20*time.Second)), t0.Add(20*time.Second))
_, bs := walks(t, open)
var ownBatch, catalogueBatch inventory.Plan
for _, b := range bs {
if b.OwnPath() {
ownBatch = b
} else {
catalogueBatch = b
}
}
if len(bs) != 2 || ownBatch.ID == "" || catalogueBatch.ID == "" || len(catalogueBatch.Delivery.Merges) != 2 ||
len(ownBatch.Delivery.Merges) != 1 {
t.Fatalf("a controller merge and two catalogue merges in one window: %+v", bs)
}
if !strings.Contains(batchWords(ownBatch, t0.Add(30*time.Second)), "own path") {
t.Fatalf("the own-path batch does not say so: %q", batchWords(ownBatch, t0.Add(30*time.Second)))
}
// Both windows closed, the controller's batch is cut first and starts; the catalogue batch queues behind it,
// is cut when the controller's walk ended, and waits for its word with both merges.
cutAt(t, open, t0.Add(2*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the controller's batch was not cut first into a started walk: %+v", ws)
}
ownWalk := ws[0]
for _, m := range []string{"app", "notes"} {
if _, in := ownWalk.Modules[m]; in {
t.Fatalf("the controller's walk builds %s, a catalogue module: it skipped its turn", m)
}
}
if len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != ownWalk.ID || bs[0].OwnPath() {
t.Fatalf("the catalogue batch does not queue behind the controller's walk: %+v", bs)
}
ownWalk.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &ownWalk); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
ws, bs = walks(t, open)
if len(bs) != 0 || ws[0].ID != catalogueBatch.ID || !ws[0].Waiting() || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the catalogue batch was not cut into a waiting walk once the controller's ended: %+v %+v", ws, bs)
}
for _, m := range []string{"app", "notes"} {
if _, in := ws[0].Modules[m]; !in {
t.Fatalf("the catalogue walk does not build %s: %v", m, ws[0].Modules)
}
}
for _, a := range *asked {
if a[1] == "modules/app" || a[1] == "modules/notes" {
t.Fatalf("a catalogue module was asked without the word: %v", *asked)
}
}
// A catalogue walk waiting for its word when an own-path batch is cut keeps waiting beside the own-path
// walk, is not folded into it, and its word is taken meanwhile: it starts once the own-path walk ended.
catalogueWalk := ws[0]
hear(t, open, repoMerge("mesh-controller", "k2", t0.Add(4*time.Minute)), t0.Add(4*time.Minute))
cutAt(t, open, t0.Add(6*time.Minute))
ws, _ = walks(t, open)
kept, _ := open.inventory.PlanByID(ctx, catalogueWalk.ID)
if !kept.Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k2" || ws[0].Waiting() {
t.Fatalf("the waiting catalogue walk was not kept waiting beside the controller's walk: %s %+v", kept.State, ws)
}
if _, in := ws[0].Modules["app"]; in {
t.Fatalf("the controller's walk folded the waiting catalogue walk in: %v", ws[0].Modules)
}
if _, err := letGo(ctx, open.inventory, catalogueWalk.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatal(err)
}
before := len(*asked)
advanceHeld(ctx, open)
if len(*asked) != before {
t.Fatalf("the catalogue walk started beside the controller's: asked %v", (*asked)[before:])
}
own2 := ws[0]
own2.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &own2); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) < before+1 || (*asked)[before][1] != "modules/app" {
t.Fatalf("the catalogue walk did not start once the controller's ended: %v", (*asked)[before:])
}
}
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
module string
claims []catalogue.Claim
}{
{"nats", nil},
{"mesh-delivery", []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1", Claims: m.claims},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/" + m.module, Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
mixed := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "m1xed", MergedAt: t0.Format(time.RFC3339Nano),
Paths: []string{"modules/nats/module.json", "modules/app/module.json"}, ModuleDirs: []string{"modules/nats", "modules/app"},
ModuleDirsSaid: true}
hear(t, open, mixed, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() {
t.Fatalf("the mixed merge's walk waited, or was not cut: %+v", ws)
}
done := ws[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("ear1ier", "app", t0.Add(-30*time.Second)), t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, done.ID)
_, bs := walks(t, open)
if len(got.Delivery.Merges) != 1 || len(bs) != 1 || bs[0].OwnPath() || bs[0].Delivery.Merges[0].Commit != "ear1ier" {
t.Fatalf("the late catalogue merge was answered by the own-path walk (%+v) rather than the next catalogue batch (%+v)",
got.Delivery.Merges, bs)
}
}
cmd/mesh-controller/batches_test.go
+720 -47
View File
@@ -1,16 +1,17 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"os"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -18,19 +19,42 @@ import (
// The suite's tests of a merge, but for the merge window's own (this file's), plan the merge as it is heard: a
// window of nothing closes at once, so a merge with no walk open is cut into its walk at once, as every merge
// was planned before novox/hq ADR 0276. The window's tests set it through the controller's settings.
func init() {
mergeWindowDefault = 0
// No forge in a test: no manifest is found at any commit, so a merge draws no edge of the manifests' change
// (delivery_edges.go). A test that wants some replaces these (forgeHolding).
manifestAt = func(context.Context, string, string, string, string) (catalogue.Manifest, bool, error) {
return catalogue.Manifest{}, false, nil
}
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
}
func init() { mergeWindowDefault = 0 }
// windowed is a mesh whose controller's settings give a merge window of 90 seconds and at most 10 minutes, as its
// settings file says them, and
// modules built from the catalogue (app, notes) and from three repositories of their own.
func windowed(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []struct{ module, repository, path string }{
{"app", "novox/mesh-catalog", "modules/app"},
{"notes", "novox/mesh-catalog", "modules/notes"},
{"one", "novox/one", ""},
{"two", "novox/two", ""},
{"three", "novox/three", ""},
} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1"},
inventory.Source{Repository: m.repository, Seat: "git", Path: m.path, Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
}
return open
}
// t0 is the moment a test's clock starts: merges are dated by it, and heard after it.
var t0 = time.Now().UTC().Truncate(time.Second)
@@ -59,16 +83,15 @@ func noGroups(t *testing.T) {
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupsWaiting stubs mesh-delivery's word on groups for a test of the merge window's own timing: each head named
// is one whose group mesh-delivery cannot say, so a merge from it waits for the window (ADR 0306 decision 2) and
// nothing holds the batch open; every other head names no group. A group with a member still to merge, which holds
// its batch open however long it takes (ADR 0418 clause 3), is groupStates'.
// groupsWaiting stubs mesh-delivery's word on groups for a test: each head named is a group with a member of another
// repository whose pull request is open (novox/elsewhere, ready), so a merge from it waits for the window (ADR 0306
// decision 2); every other head names no group.
func groupsWaiting(t *testing.T, heads ...string) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, head string) ([]groupMember, bool, error) {
if slices.Contains(heads, head) {
return nil, false, errors.New("the test's mesh-delivery does not say")
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
return nil, false, nil
}
@@ -76,37 +99,12 @@ func groupsWaiting(t *testing.T, heads ...string) {
}
// everyHeadWaits stubs mesh-delivery's word on groups for a test whose merges each wait for the window as they did
// before novox/hq ADR 0304: mesh-delivery cannot say any head's group.
// before novox/hq ADR 0304: every head is a group with a member of another repository still to merge.
func everyHeadWaits(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return nil, false, errors.New("the test's mesh-delivery does not say")
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupStates stubs mesh-delivery's word on one group: its members and their states, which a test changes as the
// members merge.
func groupStates(t *testing.T, head string, states map[string]string) {
t.Helper()
groupStatesFrom(t, head, states, true)
}
// groupStatesFrom is groupStates for a mesh-delivery that does (composed) or does not yet say the group's gate is the
// composed check's: only the first holds the group's batch open for its last member.
func groupStatesFrom(t *testing.T, head string, states map[string]string, composed bool) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, asked string) ([]groupMember, bool, error) {
if asked != head {
return nil, false, nil
}
var members []groupMember
for repository, state := range states {
members = append(members, groupMember{Repository: repository, State: state, Composed: composed})
}
return members, true, nil
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
@@ -126,8 +124,20 @@ func lone(m link.SourceMoved) link.SourceMoved {
}
// hear is a merge heard at a moment.
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
if err := (following{open: open}).hearMerge(t.Context(), m, now); err != nil {
t.Fatal(err)
}
}
// cutAt is the cutter looking at a moment.
func cutAt(t *testing.T, open *stores, now time.Time) {
t.Helper()
if err := cutBatchesHeld(t.Context(), open, now); err != nil {
t.Fatal(err)
}
}
// walks is every plan record that is a walk, newest first, and the batches not yet cut.
func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
@@ -148,34 +158,426 @@ func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
// The replay of issue 362: two catalogue merges 18 seconds apart are one batch, cut once into one walk at the
// later commit, which names both merges — the earlier carried by the later.
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, claude, t0.Add(time.Second))
hear(t, open, dunst, t0.Add(18*time.Second))
if ws, bs := walks(t, open); len(ws) != 0 || len(bs) != 1 {
t.Fatalf("within the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
cutAt(t, open, t0.Add(18*time.Second+89*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window closed: %+v", ws)
}
cutAt(t, open, t0.Add(18*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if w.Commit != dunst.Commit || len(w.Commits) != 1 || w.Commits[0].Commit != dunst.Commit {
t.Fatalf("the walk is at %s %+v, not the later commit", w.Commit, w.Commits)
}
for _, m := range []string{"app", "notes"} {
if _, in := w.Modules[m]; !in {
t.Fatalf("the walk does not build %s, which one of its merges moved: %v", m, w.Modules)
}
}
want := []inventory.PlanMerge{{Repository: "novox/mesh-catalog", Commit: claude.Commit, Carried: dunst.Commit},
{Repository: "novox/mesh-catalog", Commit: dunst.Commit}}
if w.Delivery == nil || !slices.EqualFunc(w.Delivery.Merges, want, sameMerge) {
t.Fatalf("the walk answers %+v, want %+v", w.Delivery, want)
}
if len(*asked) != 2 || (*asked)[0][2] != "main" || (*asked)[1][2] != "main" {
t.Fatalf("the walk did not ask its modules at the branch, which holds the commit it carries: %v", *asked)
}
// Heard again, as the bus may hand it over twice: never a second merge, nor a second walk.
hear(t, open, claude, t0.Add(5*time.Minute))
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 {
t.Fatalf("a merge heard twice made %d walk(s) and %d batch(es)", len(ws), len(bs))
}
}
// Merges of three repositories in one window are one walk, one commit for each.
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if len(w.Commits) != 3 || len(w.Delivery.Merges) != 3 {
t.Fatalf("the walk carries %+v and answers %+v", w.Commits, w.Delivery.Merges)
}
for _, r := range []string{"one", "two", "three"} {
if w.CommitOf("novox/"+r) != "c-"+r {
t.Fatalf("the walk carries %s at %q", r, w.CommitOf("novox/"+r))
}
if _, in := w.Modules[r]; !in {
t.Fatalf("the walk does not build %s: %v", r, w.Modules)
}
}
// Each module's ask is recorded at its own repository's commit.
for _, r := range []string{"one", "two", "three"} {
q, found, err := open.inventory.BuildRequestByID(t.Context(), w.Modules[r].Build)
if err != nil || !found || q.Commit != "c-"+r {
t.Fatalf("%s's build was not recorded at its own commit: %+v %v %v", r, q, found, err)
}
}
}
// Each merge restarts the window: a merge at second 80 keeps the batch open until second 170; the window
// closes at most ten minutes after its first merge however busy.
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
}
cutAt(t, open, t0.Add(170*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || len(ws[0].Commits) != 2 {
t.Fatalf("the window did not close 90 seconds after its last merge: %+v", ws)
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
}
break
}
}
ws, _ := walks(t, busy)
if len(ws) != 1 {
t.Fatalf("ten minutes of merges a minute apart were never cut: %d walk(s)", len(ws))
}
}
// `plans` lists the batch being assembled first, in the operator's words: how long is left, when at the latest,
// what is grouped, and that the plan is not yet calculated. It says so on the bus as plan-moved, too.
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
claude.Number, claude.Title = 174, "claude-code: an agent proposes a section"
dunst := catalogueMerge("48bda475dunst", "notes", now.Add(-2*time.Second))
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
for _, want := range []string{"assembling: ", " s left (at the latest ", "grouped: novox/mesh-catalog@48bda475 " +
"(answers 553b7191), novox/one@a6bc0931; plan not yet calculated"} {
if !strings.Contains(first, want) {
t.Fatalf("plans' first line %q does not say %q", first, want)
}
}
// Under it, one line per repository: the pull request, what it answers, what it moves.
if len(lines) < 3 || strings.TrimSpace(lines[1]) != "mesh-catalog #175 dunst: the font the operator chose (answers "+
"#174 claude-code: an agent proposes a section) · app, notes" ||
strings.TrimSpace(lines[2]) != "one a6bc0931 · one" {
t.Fatalf("the grouped list reads %q", lines[1:4])
}
if len(said) == 0 || said[len(said)-1].State != inventory.PlanAssembling || said[len(said)-1].Delivery.Batch == nil ||
len(said[len(said)-1].Delivery.Merges) != 3 {
t.Fatalf("the batch was not said as assembling with its merges: %+v", said)
}
}
// captured is what a command printed.
func captured(t *testing.T, run func() error) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
runErr := run()
os.Stdout = was
_ = w.Close()
var b bytes.Buffer
_, _ = io.Copy(&b, r)
if runErr != nil {
t.Fatal(runErr)
}
return b.String()
}
// A merge heard during a walk it is related to (the same repository, issue 973) joins the next batch, which is
// cut when the walk ends; the walk that started is never superseded (ADR 0276 decision 4, ADR 0407 decision 4).
// One walk at a time: a merge heard while a walk runs joins the next batch, which is cut when the walk ends;
// the walk that started is never superseded.
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first batch was not cut: %+v", ws)
}
first := ws[0]
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(5*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
t.Fatalf("the merge during the walk: %d walk(s), batches %+v", len(ws), bs)
}
if got, _ := open.inventory.PlanByID(t.Context(), first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the next merge", got.State)
}
if !strings.HasPrefix(batchWords(bs[0], t0.Add(5*time.Minute)), "queued behind "+first.ID) {
t.Fatalf("a queued batch reads %q", batchWords(bs[0], t0.Add(5*time.Minute)))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(6*time.Minute))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut when the walk ended: %+v %+v", ws, bs)
}
}
// A walk waiting for its delivery's word is folded into the next batch's walk, which answers its merges; it
// names the walk that took it over.
func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
waiting := ws[0]
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
ws, bs := walks(t, open)
if len(bs) != 0 || len(ws) != 2 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
folded, err := open.inventory.PlanByID(ctx, waiting.ID)
if err != nil {
t.Fatal(err)
}
newer := ws[0]
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
}
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c1" || newer.CommitOf("novox/two") != "c2" {
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
}
if _, in := newer.Modules["one"]; !in {
t.Fatalf("what the folded walk was to build is not built: %v", newer.Modules)
}
}
// A merge heard after a later merge of its branch was walked is answered by that walk when it builds all it
// moves — named at once on a walk done — and joins the next batch when it does not.
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, later, t0.Add(18*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
// Moves only notes, which the done walk built from the branch after it: answered there, at once.
hear(t, open, catalogueMerge("553b7191early", "notes", t0), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if got.State != inventory.PlanDone || len(got.Delivery.Merges) != 2 ||
!sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: "553b7191early",
Carried: later.Commit}) {
t.Fatalf("the late merge is not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// Moves app, which that walk never built: the next batch walks it.
hear(t, open, catalogueMerge("1111aaaaapp", "app", t0.Add(time.Second)), t0.Add(16*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "1111aaaaapp" {
t.Fatalf("a late merge moving what the walk never built did not join the next batch: %+v", bs)
}
}
// A failed walk marks nothing delivered: its earlier merges are walked alone, on their own commit, the newest
// first; the first delivered answers the older ones, and the search stops.
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
oldest := catalogueMerge("aaaa0001", "app", t0.Add(-3*time.Minute))
middle := catalogueMerge("bbbb0002", "app", t0.Add(-3*time.Minute+10*time.Second))
newest := catalogueMerge("cccc0003", "notes", t0.Add(-3*time.Minute+20*time.Second))
for i, m := range []link.SourceMoved{oldest, middle, newest} {
hear(t, open, m, t0.Add(time.Duration(i)*10*time.Second+time.Second))
}
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
failed := ws[0]
failed.State, failed.Note = inventory.PlanFailed, "notes failed to build in tier 0"
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, failed.ID)
if got.State != inventory.PlanFailed || len(got.Delivery.Merges) != 3 {
t.Fatalf("the failed walk's record changed: %s %+v", got.State, got.Delivery.Merges)
}
ws, _ = walks(t, open)
alone := ws[0]
if alone.ID == failed.ID || alone.Commit != middle.Commit || len(alone.Delivery.Merges) != 1 ||
!sameMerge(alone.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: middle.Commit}) {
t.Fatalf("the newest earlier merge was not walked alone on its own commit: %+v", alone)
}
if _, in := alone.Modules["app"]; !in || (*asked)[len(*asked)-1] != [3]string{"novox/mesh-catalog", "modules/app",
middle.Commit} {
t.Fatalf("the merge walked alone does not build app at its own commit: %v, asked %v", alone.Modules, *asked)
}
// Retried, the failed walk would name merges the search answers now.
if _, err := retryPlan(ctx, open, failed.ID); err == nil || !strings.Contains(err.Error(), "walked alone") {
t.Fatalf("a failed walk whose merges are searched was retried: %v", err)
}
alone.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &alone); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(4*time.Minute))
ws, bs := walks(t, open)
if ws[0].ID != alone.ID || len(bs) != 0 {
t.Fatalf("the search went on after a merge was delivered: %+v", ws[0])
}
done, _ := open.inventory.PlanByID(ctx, alone.ID)
if len(done.Delivery.Merges) != 2 || !sameMerge(done.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog",
Commit: oldest.Commit, Carried: middle.Commit}) {
t.Fatalf("the oldest merge is not answered by the walk that delivered the one after it: %+v", done.Delivery.Merges)
}
// A stopped walk starts no search: a person ended it.
hear(t, open, catalogueMerge("dddd0004", "app", t0.Add(5*time.Minute)), t0.Add(5*time.Minute))
hear(t, open, catalogueMerge("eeee0005", "notes", t0.Add(5*time.Minute+time.Second)), t0.Add(5*time.Minute+time.Second))
cutAt(t, open, t0.Add(8*time.Minute))
ws, _ = walks(t, open)
if _, err := stopWalk(ctx, open.inventory, ws[0].ID, "mesh-delivery for jochen", "not now"); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(9*time.Minute))
if again, _ := walks(t, open); again[0].ID != ws[0].ID {
t.Fatalf("a stopped walk's earlier merge was walked alone: %+v", again[0])
}
}
// A delivery group's order holds inside the walk (ADR 0249 unchanged): merges of two repositories from one head
// branch name, the node-engine's before the controller's, are tiered so; without the group, one tier.
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1"},
inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
host := repoMerge("mesh-host", "h1", t0)
ctl := repoMerge("mesh-controller", "k1", t0.Add(time.Second))
host.Head, ctl.Head = "feat/together", "feat/together"
if !grouped {
ctl.Head = "feat/alone"
}
hear(t, open, ctl, t0.Add(2*time.Second))
hear(t, open, host, t0.Add(3*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walks", len(ws))
}
tiers := ws[0].Tiers
if grouped {
if len(tiers) != 2 || !slices.Equal(tiers[0], []string{"mesh-host"}) ||
!slices.Equal(tiers[1], []string{"mesh-controller"}) {
t.Fatalf("the group's order is not the walk's tiers: %v", tiers)
}
} else if len(tiers) != 1 {
t.Fatalf("two merges of no group were ordered: %v", tiers)
}
}
}
// A restarted controller resumes the window where it stood: the batch, its merges and their times are in the
// store, a merge handed over again is not doubled, and the batch is cut when its window closes.
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
}
cutAt(t, again, t0.Add(120*time.Second))
ws, bs := walks(t, again)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller did not cut the batch at its window: %+v %+v", ws, bs)
}
}
// The window's settings reach the controller in its settings file, read at every look; one that says no
// duration, or no file, is the default.
@@ -220,9 +622,94 @@ func TestAWaitingWalkNamesTheMergesItAnswers(t *testing.T) {
// A merge heard after a later merge of its branch was cut is named by that walk however its modules read since
// the cut, in a repository of one module (review of this change); a walk that never built what it moves names
// nothing, and the merge joins the next batch.
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
hear(t, open, c2, t0)
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0.Add(-60*time.Second)), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the late merge was not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// A related walk waits for the delivery's word too: a waiting walk let go beside a started one it is related to
// (the same repository, issue 973) takes the word and starts once that one ended, asking nothing before.
// A walk of two that built only two: a late merge of one is not its to name.
hear(t, open, repoMerge("two", "d2", t0.Add(16*time.Minute)), t0.Add(16*time.Minute))
cutAt(t, open, t0.Add(18*time.Minute))
ws, _ = walks(t, open)
w = ws[0]
delete(w.Modules, "two")
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("two", "d1", t0.Add(15*time.Minute)), t0.Add(20*time.Minute))
if got, _ := open.inventory.PlanByID(ctx, w.ID); len(got.Delivery.Merges) != 1 {
t.Fatalf("a walk that never built what the late merge moves named it: %+v", got.Delivery.Merges)
}
if _, bs := walks(t, open); len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "d1" {
t.Fatalf("the late merge did not join the next batch: %+v", bs)
}
}
// One walk at a time holds against the delivery's word too: a waiting walk let go beside a started one takes the
// word and starts once that one ended, asking nothing before.
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(2*time.Minute))
started, _ := walks(t, open)
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/two", Branch: "main", Commit: "d1", Created: t0,
State: inventory.PlanBuilding, Tiers: [][]string{{"two"}}, Modules: map[string]*inventory.PlanModule{"two": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if _, err := letGo(ctx, open.inventory, waiting.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatalf("the word was refused beside %s: %v", started[0].ID, err)
}
before := len(*asked)
advanceHeld(ctx, open)
got, _ := open.inventory.PlanByID(ctx, waiting.ID)
if len(*asked) != before || !strings.Contains(got.Note, "starts once "+started[0].ID) {
t.Fatalf("a walk let go beside a started one asked (%d → %d) or does not say it waits: %q", before, len(*asked), got.Note)
}
// Read as a wait, an hour on: its note on the line, never LATE, and no tier of it late for S3.
later := time.Now().Add(time.Hour)
if line := planLine(got, later); !strings.Contains(line, "starts once "+started[0].ID) || strings.Contains(line, "LATE") {
t.Fatalf("a deferred walk reads %q", line)
}
facts, _, err := gatherPlans(ctx, open.inventory, later, nil)
if err != nil {
t.Fatal(err)
}
for _, f := range facts {
if f.id == got.ID {
t.Fatalf("a deferred walk is watched as a tier running late: %+v", f)
}
}
done := started[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) != before+1 {
t.Fatalf("the walk did not start once the started one ended: asked %v", *asked)
}
}
// A file a merge of the batch removed and a later one brought back is not removed; one removed last is.
func TestARemovedFileIsTheLastMergesWord(t *testing.T) {
@@ -242,10 +729,153 @@ func TestARemovedFileIsTheLastMergesWord(t *testing.T) {
}
}
// The catch-up hands over what the bus lost after its branch's later merges were cut, and the record keeps it:
// an earlier merge is named by the walk that carried it; a merge made before a cut and heard after it, which
// no later merge carries, joins the next batch — neither reads as history and is dropped (review of this change).
func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
base := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
hear(t, open, repoMerge("one", "c2", base.Add(10*time.Second)), base.Add(11*time.Second))
cutAt(t, open, base.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
lost := []link.AnnouncedMerge{
{SourceMoved: repoMerge("one", "c1", base), At: base.Add(time.Second)},
{SourceMoved: repoMerge("one", "c3lost", base.Add(20*time.Second)), At: base.Add(21 * time.Second)},
}
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return nil, nil, err
}
read, err := readForPlanning(ctx, open.inventory)
return entries, read, err
}
if err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announcedList(lost), open.inventory}, catalogued,
(following{open}).SourceMoved, t.Logf); err != nil {
t.Fatal(err)
}
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the earlier merge the catch-up handed over is not named by the walk that carried it: %+v",
got.Delivery.Merges)
}
if _, kept, err := open.inventory.MergeOf(ctx, "novox/one", "c3lost"); err != nil || !kept {
t.Fatalf("the merge made before the cut and heard after it was dropped: %v %v", kept, err)
}
}
// A merge on the controller's own path never shares a batch with one that waits for mesh-delivery's word
// (decided during the build, 2026-10-10): the two are batches of their own, cut one after the other; a walk on
// the own path folds no waiting catalogue walk but batches its merges again behind it; the catalogue's walk
// still waits for the word, so no catalogue delivery skips its turn.
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("aaaa0001", "app", t0), t0)
hear(t, open, repoMerge("mesh-controller", "k1", t0.Add(10*time.Second)), t0.Add(10*time.Second))
hear(t, open, catalogueMerge("bbbb0002", "notes", t0.Add(20*time.Second)), t0.Add(20*time.Second))
_, bs := walks(t, open)
var ownBatch, catalogueBatch inventory.Plan
for _, b := range bs {
if b.OwnPath() {
ownBatch = b
} else {
catalogueBatch = b
}
}
if len(bs) != 2 || ownBatch.ID == "" || catalogueBatch.ID == "" || len(catalogueBatch.Delivery.Merges) != 2 ||
len(ownBatch.Delivery.Merges) != 1 {
t.Fatalf("a controller merge and two catalogue merges in one window: %+v", bs)
}
if !strings.Contains(batchWords(ownBatch, t0.Add(30*time.Second)), "own path") {
t.Fatalf("the own-path batch does not say so: %q", batchWords(ownBatch, t0.Add(30*time.Second)))
}
// Both windows closed, the controller's batch is cut first and starts; the catalogue batch queues behind it,
// is cut when the controller's walk ended, and waits for its word with both merges.
cutAt(t, open, t0.Add(2*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the controller's batch was not cut first into a started walk: %+v", ws)
}
ownWalk := ws[0]
for _, m := range []string{"app", "notes"} {
if _, in := ownWalk.Modules[m]; in {
t.Fatalf("the controller's walk builds %s, a catalogue module: it skipped its turn", m)
}
}
if len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != ownWalk.ID || bs[0].OwnPath() {
t.Fatalf("the catalogue batch does not queue behind the controller's walk: %+v", bs)
}
ownWalk.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &ownWalk); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
ws, bs = walks(t, open)
if len(bs) != 0 || ws[0].ID != catalogueBatch.ID || !ws[0].Waiting() || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the catalogue batch was not cut into a waiting walk once the controller's ended: %+v %+v", ws, bs)
}
for _, m := range []string{"app", "notes"} {
if _, in := ws[0].Modules[m]; !in {
t.Fatalf("the catalogue walk does not build %s: %v", m, ws[0].Modules)
}
}
for _, a := range *asked {
if a[1] == "modules/app" || a[1] == "modules/notes" {
t.Fatalf("a catalogue module was asked without the word: %v", *asked)
}
}
// A catalogue walk waiting for its word when an own-path batch is cut keeps waiting beside the own-path
// walk, is not folded into it, and its word is taken meanwhile: it starts once the own-path walk ended.
catalogueWalk := ws[0]
hear(t, open, repoMerge("mesh-controller", "k2", t0.Add(4*time.Minute)), t0.Add(4*time.Minute))
cutAt(t, open, t0.Add(6*time.Minute))
ws, _ = walks(t, open)
kept, _ := open.inventory.PlanByID(ctx, catalogueWalk.ID)
if !kept.Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k2" || ws[0].Waiting() {
t.Fatalf("the waiting catalogue walk was not kept waiting beside the controller's walk: %s %+v", kept.State, ws)
}
if _, in := ws[0].Modules["app"]; in {
t.Fatalf("the controller's walk folded the waiting catalogue walk in: %v", ws[0].Modules)
}
if _, err := letGo(ctx, open.inventory, catalogueWalk.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatal(err)
}
before := len(*asked)
advanceHeld(ctx, open)
if len(*asked) != before {
t.Fatalf("the catalogue walk started beside the controller's: asked %v", (*asked)[before:])
}
own2 := ws[0]
own2.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &own2); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) < before+1 || (*asked)[before][1] != "modules/app" {
t.Fatalf("the catalogue walk did not start once the controller's ended: %v", (*asked)[before:])
}
}
// `plans` names a walk by what it moves (asked by the operator, 2026-10-10): the repository's pull request and
// title, the modules it moves and the machines running them, then the state words; a record naming no pull
@@ -291,3 +921,46 @@ func sameMerge(a, b inventory.PlanMerge) bool {
// A catalogue merge heard after a later merge of its branch was walked without the word (a merge that touched
// the bus too, on the controller's own path) is not answered by that walk while the delivery seat has a holder:
// its delivery would skip its turn. It joins the next catalogue batch.
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
module string
claims []catalogue.Claim
}{
{"nats", nil},
{"mesh-delivery", []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1", Claims: m.claims},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/" + m.module, Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
mixed := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "m1xed", MergedAt: t0.Format(time.RFC3339Nano),
Paths: []string{"modules/nats/module.json", "modules/app/module.json"}, ModuleDirs: []string{"modules/nats", "modules/app"},
ModuleDirsSaid: true}
hear(t, open, mixed, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() {
t.Fatalf("the mixed merge's walk waited, or was not cut: %+v", ws)
}
done := ws[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("ear1ier", "app", t0.Add(-30*time.Second)), t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, done.ID)
_, bs := walks(t, open)
if len(got.Delivery.Merges) != 1 || len(bs) != 1 || bs[0].OwnPath() || bs[0].Delivery.Merges[0].Commit != "ear1ier" {
t.Fatalf("the late catalogue merge was answered by the own-path walk (%+v) rather than the next catalogue batch (%+v)",
got.Delivery.Merges, bs)
}
}
cmd/mesh-controller/batches_watch.go
+2 -48
View File
@@ -3,8 +3,6 @@ package main
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
@@ -19,9 +17,6 @@ import (
const (
kindBatchNotCut = "batch-not-cut"
kindBatchBehindWalk = "batch-behind-walk"
// kindGroupNotMerged is raised for a group whose last member has not merged by the merge window's maximum after
// its first (novox/hq ADR 0418 clause 3).
kindGroupNotMerged = "group-not-merged"
)
// batchFacts is one batch not yet cut, as the watchdogs read it.
@@ -33,11 +28,6 @@ type batchFacts struct {
// tiers.
behind string
walkBound time.Duration
// holds are the groups the batch is held for, by head, with why; created is when the batch was made, the first
// merge of the group heard; atMostWindow is the maximum a window may take (the controller's setting).
holds map[string]string
created time.Time
atMostWindow time.Duration
}
// gatherBatches is every batch not yet cut, with the bound of the walk it waits behind.
@@ -52,10 +42,8 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
}
var out []batchFacts
for _, b := range batches {
_, atMostWindow := mergeWindowOf()
f := batchFacts{id: b.ID, state: b.State, grouped: groupedWords(b), created: b.Created, atMostWindow: atMostWindow}
f := batchFacts{id: b.ID, state: b.State, grouped: groupedWords(b)}
if w := b.Delivery; w != nil && w.Batch != nil {
f.holds = w.Batch.Holds
f.atMost, f.closed, f.behind = w.Batch.AtMost, w.Batch.ClosesAt, w.Batch.Behind
if f.atMost.Before(f.closed) {
f.closed = f.atMost
@@ -79,8 +67,7 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
func watchBatchesNotCut(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, b := range f.batches {
if b.state != inventory.PlanAssembling || b.atMost.IsZero() || f.now.Sub(b.atMost) <= batchLateAfter ||
len(b.holds) > 0 { // a batch held for a group waits on a member, which has its own condition
if b.state != inventory.PlanAssembling || b.atMost.IsZero() || f.now.Sub(b.atMost) <= batchLateAfter {
continue
}
late := f.now.Sub(b.atMost)
@@ -118,36 +105,3 @@ func watchBatchesBehind(f *signalFacts) []conditions.Observation {
}
return out
}
// watchGroupsNotMerged is the condition of a batch held for a group whose last member has not merged by the merge
// window's maximum after the first member merged, naming the member and why (novox/hq ADR 0418 clause 3): nothing of
// the group is delivered while it waits.
func watchGroupsNotMerged(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, b := range f.batches {
if len(b.holds) == 0 || b.created.IsZero() || f.now.Sub(b.created) <= b.atMostWindow {
continue
}
heads := make([]string, 0, len(b.holds))
for h := range b.holds {
heads = append(heads, h)
}
sort.Strings(heads)
var why []string
for _, h := range heads {
why = append(why, b.holds[h])
}
waited := f.now.Sub(b.created)
out = append(out, conditions.Observation{Scope: conditions.ScopePlan, ID: b.id,
Kind: kindGroupNotMerged, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s; its first member merged %s ago, past the merge window's maximum (%s), and nothing "+
"of the group is delivered while it waits (batch %s)", strings.Join(why, "; "), ago(waited), ago(b.atMostWindow), b.id),
Said: "a group waits for a member that has not merged",
Headline: "A change waits for the rest of its group to merge",
Explanation: fmt.Sprintf("The changes of a group are delivered together, once the last one is merged. "+
"One has not merged in %s. Nothing is lost; the merged ones are kept and delivered with it.",
humanDuration(waited)),
Resolved: "The group's last change merged"})
}
return out
}
cmd/mesh-controller/beside_test.go
-108
View File
@@ -1,108 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// An unrelated closed batch is cut into a walk beside an open one, and that walk asks its first tier at once
// (issue 973, novox/hq ADR 0407 decision 4): a hotfix is never held by an unrelated delivery.
func TestAnUnrelatedBatchWalksBesideAnOpenWalk(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first walk is not open: %+v", ws)
}
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("two", "c2", heard)), heard)
advanceHeld(ctx, open)
ws, bs := walks(t, open)
if len(ws) != 2 || len(bs) != 0 {
t.Fatalf("%d walk(s) and %d batch(es); want the unrelated batch cut beside the open walk", len(ws), len(bs))
}
for _, w := range ws {
if !w.Open() || w.Waiting() || deferred(w) || !askedAny(w) {
t.Errorf("%s (%s) is not walking: open %v, deferred %v, asked %v, note %q", w.ID, w.Named(), w.Open(),
deferred(w), askedAny(w), w.Note)
}
}
}
// A closed batch related to an open walk is queued behind it with the reason, while one that shares nothing is
// cut beside it; the walk that started is never superseded (ADR 0276 decision 4, issue 973).
func TestARelatedBatchWaitsBehindTheOpenWalk(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
first := func() inventory.Plan { ws, _ := walks(t, open); return ws[len(ws)-1] }()
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("one", "c2", heard)), heard)
hear(t, open, lone(repoMerge("three", "c3", heard.Add(time.Second))), heard.Add(time.Second))
ws, bs := walks(t, open)
if len(ws) != 2 || len(bs) != 1 {
t.Fatalf("%d walk(s) and %d batch(es); want the unrelated merge walking and the related one queued", len(ws), len(bs))
}
b := bs[0]
words := batchWords(b, heard)
if b.State != inventory.PlanQueued || b.Delivery.Batch.Behind != first.ID || b.CommitOf("novox/one") != "c2" ||
!strings.Contains(words, "queued behind "+first.ID) || !strings.Contains(words, "both change novox/one") {
t.Fatalf("the related batch is %s, behind %q, reading %q", b.State, b.Delivery.Batch.Behind, words)
}
if got, _ := open.inventory.PlanByID(ctx, first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the related merge", got.State)
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(2*time.Second))
if _, bs := walks(t, open); len(bs) != 0 {
t.Fatalf("the queued batch was not cut once the walk it waited for ended: %+v", bs)
}
}
// A batch on the controller's own path never walks beside another walk, and nothing walks beside it, whatever
// the two share (issue 973): it replaces the controller, so it goes alone.
func TestAnOwnPathBatchNeverWalksBeside(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
first := ws[0]
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("mesh-controller", "k1", heard)), heard)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || !bs[0].OwnPath() || bs[0].State != inventory.PlanQueued ||
bs[0].Delivery.Batch.Behind != first.ID || !strings.Contains(batchWords(bs[0], heard), "own path") {
t.Fatalf("an own-path batch beside an open walk: %d walk(s), batches %+v", len(ws), bs)
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(time.Second))
ws, _ = walks(t, open)
if len(ws) != 2 || !ws[0].Open() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the own-path batch was not cut once the walk ended: %+v", ws)
}
own := ws[0]
// Unrelated to the own-path walk by every footprint, a merge still waits for it.
later := heard.Add(time.Minute)
hear(t, open, lone(repoMerge("two", "c2", later)), later)
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != own.ID {
t.Fatalf("a merge beside the own-path walk: %d walk(s), batches %+v", len(ws), bs)
}
}
cmd/mesh-controller/bindings.go
+2 -4
View File
@@ -4,7 +4,7 @@ import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
@@ -73,8 +73,6 @@ func reportKept(ctx context.Context, plan catalogue.Resolution) {
// sent, on every machine — the resolver kept it there (said, urgent, until a person pins), or a pin
// moves it (said, so the data goes first), and never anything else.
func probeBindings(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
// One reading of the mesh for every machine this looks at.
ctx = withInputCache(ctx)
inv := d.open.inventory
nodes, err := inv.Nodes(ctx)
if err != nil {
@@ -82,7 +80,7 @@ func probeBindings(ctx context.Context, d *doctor) ([]conditions.Observation, er
}
var out []conditions.Observation
for _, n := range nodes {
plan, _, err := resolveNode(ctx, d.open, n.Name)
plan, _, err := planFor(ctx, d.open, n.Name)
if err != nil {
if unresolvable(err) {
// D1 says it, with the binding that refused it when that is why.
cmd/mesh-controller/bindings_integration_test.go
-102
View File
@@ -1,102 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
func TestTheIncidentAConsumerStaysBesideItsDataUntilAPersonPinsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
for _, m := range storeManifests() {
register(t, open, m)
}
assignAll := func(pairs ...[2]string) {
for _, a := range pairs {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
}
// The anchor's store and resolver hold the mesh's seats, on record; the laptop runs its own of each.
assignAll([2]string{"anchor", "store"}, [2]string{"anchor", "resolver"})
for _, seat := range [][2]string{{"mesh-store", "store"}, {"mesh-dns-resolver", "resolver"}} {
if err := inv.HoldSeat(ctx, seat[0], catalogue.ScopeMesh, "anchor", seat[1]); err != nil {
t.Fatal(err)
}
}
assignAll([2]string{"laptop", "store"}, [2]string{"laptop", "resolver"}, [2]string{"laptop", "network"},
[2]string{"laptop", "board"})
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "laptop" || n.Module != "store" || !n.KeepsData {
t.Fatalf("the consumer was bound to %s/%s (keeps data: %v); its data is beside it", n.From, n.Module, n.KeepsData)
}
if n := need(t, plan, "network", "wildcard-resolution"); n.From != "anchor" || n.KeepsData {
t.Fatalf("the resolver was bound to %s (keeps data: %v); its seat is held on anchor (issue 258)", n.From, n.KeepsData)
}
// Sent, and recorded: only the binding to data.
bindings := boundToData(plan, nil)
if len(bindings) != 1 || bindings[0].Provider != (catalogue.Chosen{Node: "laptop", Module: "store"}) {
t.Fatalf("recorded %+v", bindings)
}
if err := inv.RecordBindings(ctx, "laptop", bindings); err != nil {
t.Fatal(err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a mesh bound where it was sent: %+v, %v", found, err)
}
// The laptop's store taken away: refused, not moved to the anchor's empty one.
if err := inv.Unassign(ctx, "laptop", "store"); err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) ||
!strings.Contains(err.Error(), "board on laptop is bound to laptop/store") ||
!strings.Contains(err.Error(), "pin laptop postgres-database anchor store") {
t.Fatalf("the consumer's store went and it was answered elsewhere: %v", err)
}
// A person pins the anchor's: it moves, said before it is sent, and the record keeps where it was.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "anchor", "store"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "anchor" {
t.Fatalf("pinned to the anchor and bound to %s", n.From)
}
found, err := probeBindings(ctx, &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || found[0].Kind != kindBindingMoving || found[0].Severity != conditions.Warning ||
!strings.Contains(found[0].Summary, "board's postgres-database moves from laptop/store to anchor/store") {
t.Fatalf("a pinned move is not said before it is sent: %+v", found)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
all, err := inv.Bindings(ctx)
if err != nil || len(all) != 1 || all[0].MovedFrom != "laptop/store" {
t.Fatalf("%+v, %v", all, err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a move sent is still said: %+v, %v", found, err)
}
}
cmd/mesh-controller/bindings_test.go
+91 -1
View File
@@ -4,7 +4,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
@@ -39,6 +39,96 @@ func need(t *testing.T, plan catalogue.Resolution, consumer, provision string) c
// The incident through the stores: the laptop runs its own store and a consumer of it, the anchor's
// store holds the mesh's seat. The consumer stays beside its data, the resolver follows its seat, the
// binding is recorded as sent, and a pin — only a pin — moves it, said before the push that carries it.
func TestTheIncidentAConsumerStaysBesideItsDataUntilAPersonPinsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
for _, m := range storeManifests() {
register(t, open, m)
}
assignAll := func(pairs ...[2]string) {
for _, a := range pairs {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
}
// The anchor's store and resolver hold the mesh's seats, on record; the laptop runs its own of each.
assignAll([2]string{"anchor", "store"}, [2]string{"anchor", "resolver"})
for _, seat := range [][2]string{{"mesh-store", "store"}, {"mesh-dns-resolver", "resolver"}} {
if err := inv.HoldSeat(ctx, seat[0], catalogue.ScopeMesh, "anchor", seat[1]); err != nil {
t.Fatal(err)
}
}
assignAll([2]string{"laptop", "store"}, [2]string{"laptop", "resolver"}, [2]string{"laptop", "network"},
[2]string{"laptop", "board"})
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "laptop" || n.Module != "store" || !n.KeepsData {
t.Fatalf("the consumer was bound to %s/%s (keeps data: %v); its data is beside it", n.From, n.Module, n.KeepsData)
}
if n := need(t, plan, "network", "wildcard-resolution"); n.From != "anchor" || n.KeepsData {
t.Fatalf("the resolver was bound to %s (keeps data: %v); its seat is held on anchor (issue 258)", n.From, n.KeepsData)
}
// Sent, and recorded: only the binding to data.
bindings := boundToData(plan, nil)
if len(bindings) != 1 || bindings[0].Provider != (catalogue.Chosen{Node: "laptop", Module: "store"}) {
t.Fatalf("recorded %+v", bindings)
}
if err := inv.RecordBindings(ctx, "laptop", bindings); err != nil {
t.Fatal(err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a mesh bound where it was sent: %+v, %v", found, err)
}
// The laptop's store taken away: refused, not moved to the anchor's empty one.
if err := inv.Unassign(ctx, "laptop", "store"); err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) ||
!strings.Contains(err.Error(), "board on laptop is bound to laptop/store") ||
!strings.Contains(err.Error(), "pin laptop postgres-database anchor store") {
t.Fatalf("the consumer's store went and it was answered elsewhere: %v", err)
}
// A person pins the anchor's: it moves, said before it is sent, and the record keeps where it was.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "anchor", "store"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "anchor" {
t.Fatalf("pinned to the anchor and bound to %s", n.From)
}
found, err := probeBindings(ctx, &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || found[0].Kind != kindBindingMoving || found[0].Severity != conditions.Warning ||
!strings.Contains(found[0].Summary, "board's postgres-database moves from laptop/store to anchor/store") {
t.Fatalf("a pinned move is not said before it is sent: %+v", found)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
all, err := inv.Bindings(ctx)
if err != nil || len(all) != 1 || all[0].MovedFrom != "laptop/store" {
t.Fatalf("%+v, %v", all, err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a move sent is still said: %+v, %v", found, err)
}
}
// What one machine's resolution says against its record.
func TestBindingFindingsSayAKeptMoveAndAMoveNothingAskedFor(t *testing.T) {
cmd/mesh-controller/build.go
+31 -137
View File
@@ -7,7 +7,6 @@ import (
"flag"
"fmt"
"github.com/novox/mesh-controller/internal/conditions"
"io"
"os"
"slices"
"strings"
@@ -15,9 +14,8 @@ import (
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -153,9 +151,6 @@ func buildFrom(result link.BuildResult) inventory.Build {
Path: result.Path,
// What it was made from (novox/hq issue 280): two builds with one are one build.
SourceFingerprint: result.SourceFingerprint,
SourceInputs: result.SourceInputs,
// And its reuse key (novox/hq ADR 0350 §1), which the chain names (ADR 0408 §1).
ReuseKey: result.ReuseKey,
}
// When it was asked, which is what orders it against another build of the same module
// (novox/hq 04-ISSUES/219) — not when it was heard.
@@ -187,10 +182,8 @@ func buildFrom(result link.BuildResult) inventory.Build {
kept.Manifest = recordedManifest(result.Manifest, announced)
// The module name comes from the manifest, which only exists when the build got that far.
if len(kept.Manifest) > 0 {
if list, err := judgedWith(result); err == nil {
if m, err := catalogue.ParseManifestWith(kept.Manifest, list); err == nil {
kept.Module = m.Module
}
if m, err := catalogue.ParseManifest(kept.Manifest); err == nil {
kept.Module = m.Module
}
}
return kept
@@ -238,56 +231,35 @@ func buildsCommand(ctx context.Context, args []string) error {
}
for _, b := range builds {
printBuild(os.Stdout, b)
what := b.Module
if what == "" {
// It failed before knowing what it was building, which is most of the interesting
// failures. The repository is what a person has to go and look at.
what = "?"
}
outcome := "built " + short(b.Commit)
if !b.Worked() {
outcome = "failed"
}
fmt.Printf("%-18s %-14s %-10s %s %s\n",
what, outcome, b.On, b.At.Local().Format("2006-01-02 15:04"), b.ID)
fmt.Printf(" %s", b.Repository)
if b.Ref != "" {
fmt.Printf(" at %s", b.Ref)
}
fmt.Println()
for _, made := range b.Made {
fmt.Printf(" %-10s %s\n", made.Kind, made.Reference)
}
if !b.Worked() {
// The builder's own first line. The whole failure is often a build log, and printing
// it here would bury every other row.
fmt.Printf(" %s\n", firstLine(b.Failed))
}
}
return nil
}
// printBuild says one build: what, its outcome, where and when, what it made and published, its reuse key, and its
// source fingerprint (novox/hq issue 1204: the one tool that reads what the builder recorded it was made from).
func printBuild(w io.Writer, b inventory.Build) {
what := b.Module
if what == "" {
// It failed before knowing what it was building, which is most of the interesting
// failures. The repository is what a person has to go and look at.
what = "?"
}
outcome := "built " + short(b.Commit)
if !b.Worked() {
outcome = "failed"
}
fmt.Fprintf(w, "%-18s %-14s %-10s %s %s\n",
what, outcome, b.On, b.At.Local().Format("2006-01-02 15:04"), b.ID)
fmt.Fprintf(w, " %s", b.Repository)
if b.Ref != "" {
fmt.Fprintf(w, " at %s", b.Ref)
}
fmt.Fprintln(w)
for _, made := range b.Made {
fmt.Fprintf(w, " %-10s %s\n", made.Kind, made.Reference)
}
// What it published and under what reuse key (novox/hq ADR 0408 §1).
if p := b.Published; p != nil {
if p.Artifact != "" {
fmt.Fprintf(w, " published %s\n", p.Artifact)
}
if p.Package != nil {
fmt.Fprintf(w, " published %s@%s\n", p.Package.Name, p.Package.Version)
}
}
if b.ReuseKey != "" {
fmt.Fprintf(w, " reuse key %s\n", b.ReuseKey)
}
if b.SourceFingerprint != "" {
fmt.Fprintf(w, " source %s\n", b.SourceFingerprint)
}
if !b.Worked() {
// The builder's own first line. The whole failure is often a build log, and printing
// it here would bury every other row.
fmt.Fprintf(w, " %s\n", firstLine(b.Failed))
}
}
// builderCommand issues a build machine its own broker credential.
//
// **A build machine is not a node**, and giving it a node's account would let it read another
@@ -484,7 +456,7 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
return "", err
}
// Through a verb, only a repository the catalogue builds from (novox/hq ADR 0266).
if err := verbMayAsk(ctx, source, repository, path); err != nil {
if err := verbMayAsk(ctx, source, repository); err != nil {
return "", err
}
@@ -511,7 +483,6 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
Seats: seatBases(ctx),
DryRun: dryRun,
}
request.Reuse = reuseAsked(ctx, dryRun)
fmt.Printf("asked for %s", source)
if source.Seat != "" {
fmt.Printf(" (%s)", repository)
@@ -630,11 +601,6 @@ func saysWhenThePolicyActs(ctx context.Context, inv *inventory.Inventory, module
// written with the same values.
func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResult) (
catalogue.Manifest, inventory.Build, error) {
// The list of predefined keys as the store keeps it now: another controller process may have kept a newer one.
if err := loadPredefined(ctx, inv); err != nil {
return catalogue.Manifest{}, inventory.Build{}, fmt.Errorf("the list of predefined keys kept cannot be "+
"read: %w", err)
}
kept := buildFrom(result)
if err := inv.RecordBuild(ctx, kept); err != nil {
return catalogue.Manifest{}, kept, err
@@ -645,15 +611,7 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return catalogue.Manifest{}, kept, fmt.Errorf("%s could not build %s:\n%s",
result.On, result.Repository, result.Failed)
}
// Judged first against the list the build was judged against: the catalogue's at the commit built (novox/hq
// ADR 0348 §7), so a module naming a key its own commit adds is read; and again below against the list the
// mesh publishes, before it is registered.
list, err := judgedWith(result)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
result.Repository, short(result.Commit), err)
}
manifest, err := catalogue.ParseManifestWith(kept.Manifest, list)
manifest, err := catalogue.ParseManifest(kept.Manifest)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s and what came back is not a manifest: %w",
result.On, result.Repository, err)
@@ -711,12 +669,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
manifest.Module, short(result.Commit), err)
}
// **Never a build in the scratch namespace** (novox/hq ADR 0239 §9, ADR 0350 §2): what a check builds is
// registered only as the walk's copy in the module's own namespace.
if scratch := inScratch(result.Made); scratch != "" {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %s is in the scratch "+
"namespace, which only a walk's copy leaves (ADR 0350)", result.On, manifest.Module, short(result.Commit), scratch)
}
// **A build that failed its gate is never registered again** (novox/hq ADR 0236): an outcome heard
// twice, or replayed, would otherwise make the build a rollback put back what the module is again,
// and the next push would send it.
@@ -727,23 +679,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
"back: it is recorded and not registered again — a newer build is", result.On, manifest.Module,
short(result.Commit))
}
// **The newest trunk build's list of predefined keys is published** (novox/hq ADR 0348 §7): a build of the
// catalogue that may be registered keeps its list when its commit is newer than the one kept. Then the module
// is judged against the list the mesh publishes, as every manifest read again is: one naming a key that list
// does not hold, or of another type, is refused here and never enters the catalogue.
if carried, err := carriedList(result); err != nil {
return manifest, kept, err
} else if carried != nil {
if err := publishPredefined(ctx, inv, result, carried); err != nil {
fmt.Printf("%s: the list of predefined keys it carried is not published: %v\n", result.ID, err)
}
}
if manifest.NamesPredefined() {
if _, err := catalogue.ParseManifest(kept.Manifest); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: judged against the "+
"list of predefined keys the mesh publishes, %w", result.On, manifest.Module, short(result.Commit), err)
}
}
// **A build whose source is unchanged is never a move** (novox/hq issue 280): a rebuild made from
// what the build the mesh stands on was made from registers that build's artifacts at the new
// commit, so no machine is sent a new digest for a source nobody changed — an image is not
@@ -779,16 +714,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, nil
}
// inScratch is the first artifact a build made into the scratch namespace; empty when none is.
func inScratch(made []link.MadeArtifact) string {
for _, m := range made {
if path, kept := catalogue.InArtifactStore(catalogue.Recorded(m.Reference)); kept && strings.HasPrefix(path, builder.ScratchPrefix) {
return m.Reference
}
}
return ""
}
// errOffTheTrunk is a build of a commit off its repository's trunk, which is never published.
var errOffTheTrunk = errors.New("the commit is not on its repository's trunk: a commit off the trunk is checked, " +
"never published (ADR 0238) — merge it, and the merge builds it")
@@ -823,7 +748,7 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
if err != nil {
return err
}
if err := verbMayAsk(ctx, source, repository, path); err != nil {
if err := verbMayAsk(ctx, source, repository); err != nil {
return err
}
ident, err := openIdentity(ctx)
@@ -877,9 +802,6 @@ type answers struct {
sources map[string]inventory.Source
// waiting is every machine not running what the mesh would send it.
waiting []inventory.Machine
// recordedWaits is, per machine, every move of a recorded module that waits there for a person (novox/hq ADR
// 0377 §6): the machine is behind for these, and `push <node>` sends them.
recordedWaits map[string][]string
// reported is every machine's last word beside when it was last sent a declaration — the
// pair that answers "has it caught up", which waiting alone cannot (the sent digest is
// recorded at send, not at apply).
@@ -959,34 +881,6 @@ type answers struct {
//
// Routed through the artifact store as the network reaches it now (novox/hq 04-ISSUES/102): a
// base is recorded by digest and path, and a build machine needs something it can pull.
// runningBy is, keyed "<module>/<artifact>", each artifact of the build the mesh has registered for its module, as a
// builder reaches the artifact store (novox/hq issue 721): what a check compares its builds with. Empty when it cannot
// be read, said, and then nothing is said unchanged.
func runningBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
defer open.Close()
running, err := open.inventory.RegisteredHeld(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
address, err := whereABuilderReachesTheStore(ctx, open.inventory)
if err != nil || address == "" {
return running
}
routed := make(map[string]string, len(running))
for key, reference := range running {
routed[key] = catalogue.Rerouted(reference, address)
}
return routed
}
func heldBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
cmd/mesh-controller/build_integration_test.go
-131
View File
@@ -1,131 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func TestABuildHeardIsRecordedAndRegistered(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "shop", "version": "3"})
m, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, link.BuildResult{
ID: "b-1", Repository: "http://forge.internal:20000/novox/shop.git", Path: "modules/shop",
Ref: "main", On: "anchor", Commit: "abcdef0123", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/shop"},
}))
if err != nil {
t.Fatal(err)
}
if m.Module != "shop" {
t.Fatalf("registered %q", m.Module)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if _, held := shelf["shop"]; !held {
t.Fatal("the module a heard build produced is not in the catalogue")
}
src, err := open.inventory.SourceOf(ctx, "shop")
if err != nil || src.Seat != "git" || src.Repository != "novox/shop" || src.BuiltFrom != "abcdef0123" {
t.Fatalf("the source is the seat and the path, never the URL: %+v %v", src, err)
}
builds, err := open.inventory.Builds(ctx, "shop", 5)
if err != nil || len(builds) != 1 || builds[0].ID != "b-1" {
t.Fatalf("the build is not recorded once: %v %v", builds, err)
}
named, _ := json.Marshal(map[string]any{"module": "idp", "version": "1", "resources": []any{
map[string]any{"id": "server", "type": "container", "image": "x@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}}}})
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{
ID: "b-2", Repository: "/r", On: "anchor", Commit: "0123456789", Manifest: named})
if err == nil || !strings.Contains(err.Error(), "does not register it") {
t.Fatalf("a definition naming an installation was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["idp"].Module != "" {
t.Fatal("the refused module was registered anyway")
}
if builds, _ := open.inventory.Builds(ctx, "idp", 5); len(builds) != 1 {
t.Fatalf("the refused build was not recorded: %v", builds)
}
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{ID: "b-3", Repository: "/r", On: "anchor", Failed: "no compiler"})
if err == nil || !strings.Contains(err.Error(), "no compiler") {
t.Fatalf("a failure is said in the builder's words: %v", err)
}
}
func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "unifi", "version": "1"})
result := func(id, ref, commit string) link.BuildResult {
return link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/unifi", Ref: ref, On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result("b-1", "main", "1111111aaaa"))); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, result("b-2", "9c97a8a", "9c97a8a1d2c3")); err != nil {
t.Fatal(err)
}
src, err := open.inventory.SourceOf(ctx, "unifi")
if err != nil {
t.Fatal(err)
}
if src.Ref != "main" || src.BuiltFrom != "9c97a8a1d2c3" {
t.Errorf("after a build at a commit the module follows %q, built from %q; want main, 9c97a8a1d2c3", src.Ref, src.BuiltFrom)
}
// One new to the catalogue, first built at a commit, follows the default branch.
other, _ := json.Marshal(map[string]any{"module": "letta", "version": "1"})
r := result("b-3", "deadbeef", "deadbeefcafe")
r.Manifest, r.Path = other, "modules/letta"
if _, _, err := takeIn(ctx, open.inventory, r); err != nil {
t.Fatal(err)
}
if src, _ := open.inventory.SourceOf(ctx, "letta"); src.Ref != "" {
t.Errorf("a module first built at a commit follows %q, want the default branch", src.Ref)
}
}
func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
result := func(asked time.Time, image string) link.BuildResult {
manifest, _ := json.Marshal(map[string]any{"module": "postgres", "version": image})
return link.BuildResult{ID: link.NewBuildID(asked), Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result(newer, "4bcd5f73"))); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
if !errors.Is(err, inventory.ErrSuperseded) {
t.Fatalf("the older request's outcome was taken in as current: %v", err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "4bcd5f73" {
t.Errorf("postgres is %q; want the newer request's 4bcd5f73", got)
}
if builds, _ := open.inventory.Builds(ctx, "postgres", 5); len(builds) != 2 {
t.Errorf("the late build was not recorded: %v", builds)
}
}
cmd/mesh-controller/build_seat_test.go
+1 -1
View File
@@ -3,7 +3,7 @@ package main
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
cmd/mesh-controller/build_source.go
+8 -91
View File
@@ -10,7 +10,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -88,7 +88,7 @@ func newSourceForms(ctx context.Context, inv *inventory.Inventory) *sourceForms
if world == nil {
w := catalogue.World{}
if shelf, err := inv.Catalogue(ctx); err == nil {
if read, err := restOfTheMeshOf(ctx, inv, shelf, ""); err == nil {
if read, err := theRestOfTheMesh(ctx, inv, shelf, ""); err == nil {
w = read
}
}
@@ -389,31 +389,10 @@ func (f *sourceForms) onTheForge(repository, seat string) (owner, name string, o
// least one required status, no administrator merging past one.
//
// Anything else only when the build request was kept as asked at the controller's terminal, for this very
// repository and path; or, for the build of a merge, when mesh-delivery announced it on the operator's Approve of
// that head (novox/hq ADR 0420 §7 for a moved module, ADR 0432 for a new one), the forge's checks still holding. path is the module's directory as built; branch the trunk it is registered from.
// repository and path. path is the module's directory as built; branch the trunk it is registered from.
func mayRegisterFrom(ctx context.Context, inv *inventory.Inventory, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
return mayRegisterIn(ctx, inv, newSourceForms(ctx, inv), module, built, buildID, path, branch)
}
// catalogueReader is the catalogue's manifests, as the router's records are found in them.
type catalogueReader interface {
Catalogue(ctx context.Context) (map[string]catalogue.Manifest, error)
}
// registrations is what a registration's decision reads of the mesh: *inventory.Inventory, or a test's stand-in.
type registrations interface {
catalogueReader
SourceOf(ctx context.Context, module string) (inventory.Source, error)
SourceIdentity(ctx context.Context, module string) (int64, error)
Catalogued(ctx context.Context) ([]inventory.Entry, error)
BuildRequestByID(ctx context.Context, id string) (inventory.BuildRequest, bool, error)
MergeOf(ctx context.Context, repository, commit string) (inventory.BatchedMerge, bool, error)
}
// mayRegisterIn is mayRegisterFrom on what it reads of the mesh handed in.
func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
forms := newSourceForms(ctx, inv)
was, err := inv.SourceOf(ctx, module)
isNew := errors.Is(err, inventory.ErrNoSuchModule)
if err != nil && !isNew {
@@ -425,8 +404,7 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
}
refuse := func(why string) (int64, error) {
return 0, fmt.Errorf("%w: %s. A module is registered from such a source only by a build asked at the "+
"controller's terminal, or by the build of a merge mesh-delivery announced on the operator's Approve of that "+
"head, never through a verb: a registered module is what the next push sends, and whoever "+
"controller's terminal, never through a verb: a registered module is what the next push sends, and whoever "+
"may call a verb includes agents (novox/hq ADR 0266)", errNotItsSource, why)
}
@@ -457,23 +435,6 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
module, sourceWords(built.Repository, built.Seat))
}
}
// **Or the move the operator approved** (novox/hq ADR 0420 §7, issue 1025): a known module built from another
// repository by a merge, whose pull request mesh-delivery merged on the operator's Approve of that head.
// A verb alone, or a merge with no such warrant, is refused as before.
// **Or a new module from a repository nothing registered is built from, on the same warrant** (novox/hq ADR 0432,
// issue 1146): the build of a merge mesh-delivery announced on the operator's Approve of that head registers it.
// The forge's protection and the repository's identity are checked below as for every registration.
movedWhy := ""
if why != "" && !terminal {
accepted, err := approvedMove(ctx, inv, forms, buildID, built)
switch {
case err == nil:
movedWhy, why = accepted, ""
case !errors.Is(err, errNotAMerge):
fmt.Printf("%s: %s is built from another repository, and no approved move covers this build: %v\n",
buildID, module, err)
}
}
if why != "" && !terminal {
return refuse(why)
}
@@ -510,8 +471,7 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
}
// The same repository by the forge's id, not only its name: one deleted and made again is another.
recorded := map[string]int64{}
// A moved module is registered from the repository the operator approved, not the one it was registered from.
if !isNew && movedWhy == "" {
if !isNew {
id, err := inv.SourceIdentity(ctx, module)
if err != nil {
return 0, err
@@ -532,19 +492,12 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
owner, name, m, facts.ID, m, id))
}
}
if movedWhy != "" && isNew {
fmt.Printf("%s: %s is new, from %s/%s — registered at the operator's Approve: %s\n", buildID, module, owner, name,
movedWhy)
} else if movedWhy != "" {
fmt.Printf("%s: %s moved from %s to %s/%s — registered at the operator's Approve: %s\n", buildID, module,
sourceWords(was.Repository, was.Seat), owner, name, movedWhy)
}
return facts.ID, nil
}
// askedHereFor says the build was asked at the controller's terminal, for this repository and this path: a kept
// request marked so, whose source is the outcome's (novox/hq ADR 0266).
func askedHereFor(ctx context.Context, inv registrations, forms *sourceForms, buildID string,
func askedHereFor(ctx context.Context, inv *inventory.Inventory, forms *sourceForms, buildID string,
built inventory.Source, path string) (bool, error) {
r, found, err := inv.BuildRequestByID(ctx, buildID)
if err != nil || !found || !r.AtTerminal {
@@ -567,49 +520,13 @@ func sourceWords(repository, seat string) string {
return buildSource{Repository: repository, Seat: seat}.String()
}
// firstBuildWarrant is what the controller checked before it asks the first build of the modules a merge adds to a
// repository nothing is built from: the merge was made on the operator's Approve of its head (approvedAnnounced). It
// travels in the context of that ask alone, in this process, so no verb carries one.
type firstBuildWarrant struct {
repository string // owner/name on the git seat
paths map[string]bool
why string
}
type firstBuildWarrantKey struct{}
// withFirstBuildWarrant is ctx carrying the warrant for asking the first builds of paths in repository.
func withFirstBuildWarrant(ctx context.Context, repository string, paths []string, why string) context.Context {
w := &firstBuildWarrant{repository: repository, paths: map[string]bool{}, why: why}
for _, p := range paths {
w.paths[strings.Trim(p, "/")] = true
}
return context.WithValue(ctx, firstBuildWarrantKey{}, w)
}
func firstBuildWarrantIn(ctx context.Context) *firstBuildWarrant {
w, _ := ctx.Value(firstBuildWarrantKey{}).(*firstBuildWarrant)
return w
}
// covers says the warrant is for exactly this ask: the repository on the git seat, and a module path the merge holds.
func (w *firstBuildWarrant) covers(source buildSource, path string) bool {
return w != nil && source.Seat == gitSeat && strings.EqualFold(strings.Trim(source.Repository, "/"), w.repository) &&
w.paths[strings.Trim(path, "/")]
}
// verbMayAsk refuses, for a call through a verb, a build of a repository the catalogue builds no module from
// (novox/hq ADR 0266): the build node would run what an agent wrote, and its outcome could never be
// registered anyway. url is the repository as it is cloned. At the terminal anything may be asked.
func verbMayAsk(ctx context.Context, source buildSource, url, path string) error {
func verbMayAsk(ctx context.Context, source buildSource, url string) error {
if startedAtTheTerminal() {
return nil
}
// **Or the first build of a module a merge adds on the operator's Approve** (novox/hq ADR 0432, issue 1146): the
// controller's own ask, carrying the warrant it checked, for that repository and the module paths the merge holds.
if firstBuildWarrantIn(ctx).covers(source, path) {
return nil
}
open, err := openStores(ctx)
if err != nil {
return err
cmd/mesh-controller/build_source_integration_test.go
-353
View File
@@ -1,353 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func theCatalogue(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
for _, b := range []link.BuildResult{
onTrunk("build-sudo", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "1"}),
onTrunk("build-host", "novox/mesh-host", "git", "", map[string]any{"module": "mesh-host", "version": "1"}),
} {
keptAsked(t, open.inventory, b.ID, b.Source.Repository, b.Path, true)
if _, _, err := takeIn(ctx, open.inventory, b); err != nil {
t.Fatal(err)
}
}
return open
}
func TestABuildFromAnAgentsRepositoryIsNotRegisteredAsAModuleTheMeshHolds(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
for _, c := range []struct {
name, repository, path, module string
}{
{"its own repository naming sudo", "agent/sudo", "modules/sudo", "sudo"},
{"a fork of the catalogue", "agent/mesh-catalog", "modules/sudo", "sudo"},
{"a fork of the node-engine", "agent/mesh-host", "", "mesh-host"},
} {
t.Run(c.name, func(t *testing.T) {
id := "build-" + strings.ReplaceAll(c.repository, "/", "-")
keptAsked(t, open.inventory, id, c.repository, c.path, false) // through the build verb
evil := onTrunk(id, c.repository, "git", c.path, map[string]any{"module": c.module, "version": "evil"})
_, _, err := takeIn(ctx, open.inventory, evil)
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a build of %s was taken in as %s: %v", c.repository, c.module, err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf[c.module].Version; got != "1" {
t.Fatalf("%s is now %q, from %s", c.module, got, c.repository)
}
if _, found, _ := open.inventory.BuildByID(ctx, id); !found {
t.Errorf("the refused build %s is not recorded", id)
}
})
}
}
func TestANewModuleFromARepositoryTheCatalogueDoesNotBuildFromIsNotRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-new", "agent/tools", "", false)
_, _, err := takeIn(ctx, open.inventory, onTrunk("build-new", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from an agent's repository was taken in: %v", err)
}
// And one asked of nobody here — an outcome on the bus no request was kept for — the same.
_, _, err = takeIn(ctx, open.inventory, onTrunk("build-unasked", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome nobody asked for was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["agent-tools"].Module != "" {
t.Fatal("the refused module is in the catalogue")
}
}
func TestAtTheTerminalAnotherRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-moved", "novox/sudo", "", true)
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-moved", "novox/sudo", "git", "",
map[string]any{"module": "sudo", "version": "2"})); err != nil {
t.Fatalf("a move the operator asked for at the terminal was refused: %v", err)
}
if src, _ := open.inventory.SourceOf(ctx, "sudo"); src.Repository != "novox/sudo" {
t.Fatalf("sudo is built from %q", src.Repository)
}
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-external",
Repository: "http://forge.internal:20000/someone/app.git", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-external", "someone/app", "", "",
map[string]any{"module": "app", "version": "1"})); err != nil {
t.Fatalf("a new module the operator asked for at the terminal was refused: %v", err)
}
}
func TestADeliveryFromTheRegisteredRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
rebuilt := onTrunk("build-plan", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: rebuilt.ID,
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/sudo", For: "plan"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, rebuilt); err != nil {
t.Fatalf("a plan's build of the module's own repository was refused: %v", err)
}
added := onTrunk("build-merge", "novox/mesh-catalog", "git", "modules/zram", map[string]any{"module": "zram", "version": "1"})
if _, _, err := takeIn(ctx, open.inventory, added); err != nil {
t.Fatalf("a module a merge added to the catalogue repository was refused: %v", err)
}
shelf, _ := open.inventory.Catalogue(ctx)
if shelf["sudo"].Version != "2" || shelf["zram"].Module == "" {
t.Fatalf("not registered: sudo %q, zram %q", shelf["sudo"].Version, shelf["zram"].Module)
}
}
func TestAVerbAsksNoBuildOfARepositoryTheCatalogueDoesNotBuildFrom(t *testing.T) {
theCatalogue(t)
ctx := t.Context()
t.Setenv(verbVar, "build")
t.Setenv(link.CallerVar, "node-tools.anchor, through the mesh-controller seat")
err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "http://forge.internal:20000/agent/sudo.git", "")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("a verb's build of an agent's repository was asked: %v", err)
}
if err := verbMayAsk(ctx, buildSource{Repository: "novox/mesh-catalog", Seat: "git"},
"http://forge.internal:20000/novox/mesh-catalog.git", ""); err != nil {
t.Fatalf("a verb's build of the catalogue repository was refused: %v", err)
}
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
if err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "", ""); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
func TestARollbackNeverPutsBackABuildFromAnotherRepository(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
fork := onTrunk("build-1791500000000000000", "agent/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "evil"})
fork.Commit = "c0ffee0123456789" // the commit sudo was registered at
keptAsked(t, inv, fork.ID, "agent/mesh-catalog", "modules/sudo", false)
if _, _, err := takeIn(ctx, inv, fork); !errors.Is(err, errNotItsSource) {
t.Fatalf("the fork's build was taken in: %v", err)
}
// Asked after the registered build, whenever the test runs: an id naming a fixed moment read as older
// than the registered build once the clock passed it (2026-10-10 02:40 UTC), and the test failed on main.
failed := onTrunk(fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano()), "novox/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "2"})
failed.Commit = "badbadbad0123456"
if _, _, err := takeIn(ctx, inv, failed); err != nil {
t.Fatal(err)
}
record, _, err := inv.BuildByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
previous, found, err := inv.PreviousBuild(ctx, "sudo", "c0ffee0123456789", record)
if err != nil {
t.Fatal(err)
}
if found && previous.ID == fork.ID {
t.Fatalf("a rollback would put back the fork's build %s", previous.ID)
}
if !found || previous.ID != "build-sudo" {
t.Fatalf("a rollback puts back %q (found %v), want the registered build-sudo", previous.ID, found)
}
}
func TestATrunkTheForgeDoesNotProtectRegistersNothing(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/unguarded", forgeFacts{ID: 7, Why: "the rule main lets a person push to main directly"})
t.Cleanup(func() { theForge.Delete("novox/unguarded") })
first := onTrunk("build-unguarded-1", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/unguarded", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatalf("at the terminal: %v", err)
}
again := onTrunk("build-unguarded-2", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "2"})
if _, _, err := takeIn(ctx, open.inventory, again); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "push to main directly") {
t.Fatalf("a rebuild from an unprotected trunk was taken in: %v", err)
}
theForge.Store("novox/unguarded", errors.New("nothing serves gitea.gitea_api"))
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-unguarded-3", "novox/unguarded", "git", "",
map[string]any{"module": "unguarded", "version": "3"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a forge that could not say was read as a protected trunk: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["unguarded"].Version != "1" {
t.Fatalf("unguarded is %q", shelf["unguarded"].Version)
}
}
func TestARepositoryMadeAgainUnderItsNameIsNotTheModulesSource(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/remade", forgeFacts{ID: 100, Guarded: true})
t.Cleanup(func() { theForge.Delete("novox/remade") })
first := onTrunk("build-remade-1", "novox/remade", "git", "", map[string]any{"module": "remade", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/remade", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "remade"); id != 100 {
t.Fatalf("the forge's id was not recorded: %d", id)
}
theForge.Store("novox/remade", forgeFacts{ID: 101, Guarded: true}) // deleted, and made again by an agent
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-2", "novox/remade", "git", "",
map[string]any{"module": "remade", "version": "evil"})); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "made again") {
t.Fatalf("a repository made again under the name was taken in: %v", err)
}
// And a new module from it, beside the one registered from the first, the same.
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-3", "novox/remade", "git", "modules/other",
map[string]any{"module": "other", "version": "1"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from a repository made again was taken in: %v", err)
}
}
func TestATerminalRequestCoversOnlyWhatItAsked(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-asked", Repository: "novox/app",
Seat: "git", Path: "modules/app", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
other := onTrunk("build-asked", "agent/sudo", "git", "modules/app", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, other); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another repository under a terminal request's id was taken in: %v", err)
}
elsewhere := onTrunk("build-asked", "novox/app", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, elsewhere); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another path under a terminal request's id was taken in: %v", err)
}
}
func TestTheServingControllerIsNeverTheTerminal(t *testing.T) {
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
t.Setenv(servedVar, "")
if !startedAtTheTerminal() {
t.Fatal("a process started by hand is not the terminal")
}
markServed()
if startedAtTheTerminal() {
t.Fatal("the serving controller reads as the terminal")
}
child := exec.Command(os.Args[0], "-test.run=^$")
child.Env = os.Environ()
if !slices.Contains(child.Env, servedVar+"=1") {
t.Fatal("what the serving controller starts does not carry its mark")
}
}
func TestAnUnknownIdentityKeepsTheRecordedOne(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 100); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 0); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "sudo"); id != 100 {
t.Fatalf("the recorded id became %d", id)
}
}
func TestAModuleMovesOnTheOperatorsApproveOfTheMergeThatMovedIt(t *testing.T) {
const commit = "c0ffee0123456789"
for _, c := range []struct {
what string
change func(*anApprovedMerge)
unkept bool
request string // who asked: "merge", or "build" through a verb
moved bool
}{
{what: "the operator's Approve of the merged head", request: "merge", moved: true},
{what: "no merge mesh-delivery announced", unkept: true, request: "merge"},
{what: "a warrant for other heads", request: "merge", change: func(a *anApprovedMerge) {
a.merge.Heads = map[string]string{"novox/mesh-host@aaaaaaaaaaaa": "e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5"}
}},
{what: "a Decline", request: "merge", change: func(a *anApprovedMerge) {
a.warrant.Option, a.warrant.Label, a.merge.Option = "decline", "Decline", "decline"
}},
{what: "a person who is not the operator", request: "merge", change: func(a *anApprovedMerge) { a.warrant.By.Who = "agent" }},
{what: "a build asked through a verb", request: "build"},
} {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
a := approvedMerge(t)
a.merge.Owner, a.merge.Repo, a.merge.Number, a.merge.HeadSHA = "novox", "mesh-host", 7, "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1"
if c.change != nil {
c.change(&a)
}
wasKept, wasAsk := keptDeliveryMerge, routerAskOf
keptDeliveryMerge = func(_ context.Context, owner, repo string, number int, head string) (deliveryMerge, bool, error) {
return a.merge, !c.unkept && owner == "novox" && repo == "mesh-host" && number == 7 && head == a.merge.HeadSHA, nil
}
routerAskOf = func(_ context.Context, _ catalogueReader, _, id string) (*asks.Ask, *asks.Warrant, error) {
if id != a.merge.AskID {
t.Errorf("%s: the router was asked for %q", c.what, id)
}
return &a.ask, &a.warrant, nil
}
t.Cleanup(func() { keptDeliveryMerge, routerAskOf = wasKept, wasAsk })
event, _ := json.Marshal(map[string]any{"owner": "novox", "repo": "mesh-host", "base": "main", "number": 7,
"head_sha": a.merge.HeadSHA, "merge_commit_sha": commit})
if _, err := inv.AddMerge(ctx, inventory.BatchedMerge{Repository: "novox/mesh-host", Branch: "main", Commit: commit,
Merged: time.Now(), Heard: time.Now(), Event: event}); err != nil {
t.Fatal(err)
}
id := fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano())
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: id, Repository: "novox/mesh-host", Seat: "git",
Path: "modules/sudo", Ref: "main", Commit: commit, For: c.request}); err != nil {
t.Fatal(err)
}
moved := onTrunk(id, "novox/mesh-host", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
moved.Commit = commit
_, _, err := takeIn(ctx, inv, moved)
src, _ := inv.SourceOf(ctx, "sudo")
switch {
case c.moved && err != nil:
t.Errorf("%s: the move was refused: %v", c.what, err)
case c.moved && src.Repository != "novox/mesh-host":
t.Errorf("%s: sudo is built from %q", c.what, src.Repository)
case !c.moved && !errors.Is(err, errNotItsSource):
t.Errorf("%s: the move was not refused as ADR 0266 has it: %v", c.what, err)
case !c.moved && src.Repository != "novox/mesh-catalog":
t.Errorf("%s: a refused move changed sudo's source to %q", c.what, src.Repository)
}
}
}
cmd/mesh-controller/build_source_test.go
+262 -6
View File
@@ -3,10 +3,16 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"hash/fnv"
"os"
"os/exec"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -39,14 +45,141 @@ func keptAsked(t *testing.T, inv *inventory.Inventory, id, repository, path stri
}
// theCatalogue is a mesh whose sudo is built from the catalogue repository and whose node-engine from its own.
func theCatalogue(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
for _, b := range []link.BuildResult{
onTrunk("build-sudo", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "1"}),
onTrunk("build-host", "novox/mesh-host", "git", "", map[string]any{"module": "mesh-host", "version": "1"}),
} {
keptAsked(t, open.inventory, b.ID, b.Source.Repository, b.Path, true)
if _, _, err := takeIn(ctx, open.inventory, b); err != nil {
t.Fatal(err)
}
}
return open
}
func TestABuildFromAnAgentsRepositoryIsNotRegisteredAsAModuleTheMeshHolds(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
for _, c := range []struct {
name, repository, path, module string
}{
{"its own repository naming sudo", "agent/sudo", "modules/sudo", "sudo"},
{"a fork of the catalogue", "agent/mesh-catalog", "modules/sudo", "sudo"},
{"a fork of the node-engine", "agent/mesh-host", "", "mesh-host"},
} {
t.Run(c.name, func(t *testing.T) {
id := "build-" + strings.ReplaceAll(c.repository, "/", "-")
keptAsked(t, open.inventory, id, c.repository, c.path, false) // through the build verb
evil := onTrunk(id, c.repository, "git", c.path, map[string]any{"module": c.module, "version": "evil"})
_, _, err := takeIn(ctx, open.inventory, evil)
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a build of %s was taken in as %s: %v", c.repository, c.module, err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf[c.module].Version; got != "1" {
t.Fatalf("%s is now %q, from %s", c.module, got, c.repository)
}
if _, found, _ := open.inventory.BuildByID(ctx, id); !found {
t.Errorf("the refused build %s is not recorded", id)
}
})
}
}
func TestANewModuleFromARepositoryTheCatalogueDoesNotBuildFromIsNotRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-new", "agent/tools", "", false)
_, _, err := takeIn(ctx, open.inventory, onTrunk("build-new", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from an agent's repository was taken in: %v", err)
}
// And one asked of nobody here — an outcome on the bus no request was kept for — the same.
_, _, err = takeIn(ctx, open.inventory, onTrunk("build-unasked", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome nobody asked for was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["agent-tools"].Module != "" {
t.Fatal("the refused module is in the catalogue")
}
}
// The operator at the terminal may still move a module, or add one from a new repository.
func TestAtTheTerminalAnotherRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-moved", "novox/sudo", "", true)
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-moved", "novox/sudo", "git", "",
map[string]any{"module": "sudo", "version": "2"})); err != nil {
t.Fatalf("a move the operator asked for at the terminal was refused: %v", err)
}
if src, _ := open.inventory.SourceOf(ctx, "sudo"); src.Repository != "novox/sudo" {
t.Fatalf("sudo is built from %q", src.Repository)
}
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-external",
Repository: "http://forge.internal:20000/someone/app.git", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-external", "someone/app", "", "",
map[string]any{"module": "app", "version": "1"})); err != nil {
t.Fatalf("a new module the operator asked for at the terminal was refused: %v", err)
}
}
// The delivery's flow is untouched: a merge's rebuild of a module from its own repository, and a merge adding
// a module to a repository the catalogue builds from (novox/hq issue 300), are registered with no terminal.
func TestADeliveryFromTheRegisteredRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
rebuilt := onTrunk("build-plan", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: rebuilt.ID,
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/sudo", For: "plan"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, rebuilt); err != nil {
t.Fatalf("a plan's build of the module's own repository was refused: %v", err)
}
added := onTrunk("build-merge", "novox/mesh-catalog", "git", "modules/zram", map[string]any{"module": "zram", "version": "1"})
if _, _, err := takeIn(ctx, open.inventory, added); err != nil {
t.Fatalf("a module a merge added to the catalogue repository was refused: %v", err)
}
shelf, _ := open.inventory.Catalogue(ctx)
if shelf["sudo"].Version != "2" || shelf["zram"].Module == "" {
t.Fatalf("not registered: sudo %q, zram %q", shelf["sudo"].Version, shelf["zram"].Module)
}
}
// Through a verb, a build of a repository the catalogue builds nothing from is not even asked: the build node
// would run what the agent wrote.
func TestAVerbAsksNoBuildOfARepositoryTheCatalogueDoesNotBuildFrom(t *testing.T) {
theCatalogue(t)
ctx := t.Context()
t.Setenv(verbVar, "build")
t.Setenv(link.CallerVar, "node-tools.anchor, through the mesh-controller seat")
err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "http://forge.internal:20000/agent/sudo.git")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("a verb's build of an agent's repository was asked: %v", err)
}
if err := verbMayAsk(ctx, buildSource{Repository: "novox/mesh-catalog", Seat: "git"},
"http://forge.internal:20000/novox/mesh-catalog.git"); err != nil {
t.Fatalf("a verb's build of the catalogue repository was refused: %v", err)
}
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
if err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, ""); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
// asTheOperator keeps a build as asked at the controller's terminal, as the operator's first build of a module
// from a repository the catalogue does not yet build from is (novox/hq ADR 0266), and hands it back.
@@ -66,6 +199,40 @@ func asTheOperator(t *testing.T, inv *inventory.Inventory, b link.BuildResult) l
// A rollback puts back only a build of the module's own repository: an agent's build of the module's name,
// recorded and refused, at the very commit the machine ran before (a fork carries it), is never registered by
// the back door of a failed gate.
func TestARollbackNeverPutsBackABuildFromAnotherRepository(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
fork := onTrunk("build-1791500000000000000", "agent/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "evil"})
fork.Commit = "c0ffee0123456789" // the commit sudo was registered at
keptAsked(t, inv, fork.ID, "agent/mesh-catalog", "modules/sudo", false)
if _, _, err := takeIn(ctx, inv, fork); !errors.Is(err, errNotItsSource) {
t.Fatalf("the fork's build was taken in: %v", err)
}
// Asked after the registered build, whenever the test runs: an id naming a fixed moment read as older
// than the registered build once the clock passed it (2026-10-10 02:40 UTC), and the test failed on main.
failed := onTrunk(fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano()), "novox/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "2"})
failed.Commit = "badbadbad0123456"
if _, _, err := takeIn(ctx, inv, failed); err != nil {
t.Fatal(err)
}
record, _, err := inv.BuildByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
previous, found, err := inv.PreviousBuild(ctx, "sudo", "c0ffee0123456789", record)
if err != nil {
t.Fatal(err)
}
if found && previous.ID == fork.ID {
t.Fatalf("a rollback would put back the fork's build %s", previous.ID)
}
if !found || previous.ID != "build-sudo" {
t.Fatalf("a rollback puts back %q (found %v), want the registered build-sudo", previous.ID, found)
}
}
// theForge is what the forge says in a test, by owner/name: a repository not named here is protected as a
// trunk must be, with an id of its own.
@@ -89,15 +256,97 @@ func init() {
// A module's trunk the forge does not protect — direct pushes, no required status — or a forge that cannot say,
// registers nothing through a verb: the trunk rule means nothing on a branch anyone pushes to.
func TestATrunkTheForgeDoesNotProtectRegistersNothing(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/unguarded", forgeFacts{ID: 7, Why: "the rule main lets a person push to main directly"})
t.Cleanup(func() { theForge.Delete("novox/unguarded") })
first := onTrunk("build-unguarded-1", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/unguarded", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatalf("at the terminal: %v", err)
}
again := onTrunk("build-unguarded-2", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "2"})
if _, _, err := takeIn(ctx, open.inventory, again); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "push to main directly") {
t.Fatalf("a rebuild from an unprotected trunk was taken in: %v", err)
}
theForge.Store("novox/unguarded", errors.New("nothing serves gitea.gitea_api"))
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-unguarded-3", "novox/unguarded", "git", "",
map[string]any{"module": "unguarded", "version": "3"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a forge that could not say was read as a protected trunk: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["unguarded"].Version != "1" {
t.Fatalf("unguarded is %q", shelf["unguarded"].Version)
}
}
// A repository deleted and made again under the module's repository's name is another repository: the forge's
// id, recorded at registration, tells them apart.
func TestARepositoryMadeAgainUnderItsNameIsNotTheModulesSource(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/remade", forgeFacts{ID: 100, Guarded: true})
t.Cleanup(func() { theForge.Delete("novox/remade") })
first := onTrunk("build-remade-1", "novox/remade", "git", "", map[string]any{"module": "remade", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/remade", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "remade"); id != 100 {
t.Fatalf("the forge's id was not recorded: %d", id)
}
theForge.Store("novox/remade", forgeFacts{ID: 101, Guarded: true}) // deleted, and made again by an agent
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-2", "novox/remade", "git", "",
map[string]any{"module": "remade", "version": "evil"})); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "made again") {
t.Fatalf("a repository made again under the name was taken in: %v", err)
}
// And a new module from it, beside the one registered from the first, the same.
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-3", "novox/remade", "git", "modules/other",
map[string]any{"module": "other", "version": "1"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from a repository made again was taken in: %v", err)
}
}
// The terminal's mark is the operator's for the repository and path they asked: an outcome of another, under that
// build's id, is not theirs.
func TestATerminalRequestCoversOnlyWhatItAsked(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-asked", Repository: "novox/app",
Seat: "git", Path: "modules/app", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
other := onTrunk("build-asked", "agent/sudo", "git", "modules/app", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, other); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another repository under a terminal request's id was taken in: %v", err)
}
elsewhere := onTrunk("build-asked", "novox/app", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, elsewhere); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another path under a terminal request's id was taken in: %v", err)
}
}
// The serving controller, and everything it starts, is never the terminal: a build asked in it reads as asked
// through the mesh even when no verb and no caller is named.
func TestTheServingControllerIsNeverTheTerminal(t *testing.T) {
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
t.Setenv(servedVar, "")
if !startedAtTheTerminal() {
t.Fatal("a process started by hand is not the terminal")
}
markServed()
if startedAtTheTerminal() {
t.Fatal("the serving controller reads as the terminal")
}
child := exec.Command(os.Args[0], "-test.run=^$")
child.Env = os.Environ()
if !slices.Contains(child.Env, servedVar+"=1") {
t.Fatal("what the serving controller starts does not carry its mark")
}
}
// The forge applies one rule to a branch: the rule named for it, else the first glob that covers it. A stronger
// rule later in the list is not what guards the branch, and is not read as if it were (the confirmation review).
@@ -120,9 +369,16 @@ func TestTheRuleJudgedIsTheOneTheForgeApplies(t *testing.T) {
}
// An id the forge could not give keeps the id already recorded.
// **The operator's Approve of a reviewed group registers a moved module from its new repository** (novox/hq ADR 0420 §7,
// issue 1025). sudo is built from the catalogue; a merge into the repository of node-engine's module, announced by
// mesh-delivery with the ask it merged on, brings its build from there. Accepted with the operator's Approve of the
// merged head; refused with no kept merge, a warrant for other heads, a Decline, a person who is not the operator, and
// through a verb.
func TestAnUnknownIdentityKeepsTheRecordedOne(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 100); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 0); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "sudo"); id != 100 {
t.Fatalf("the recorded id became %d", id)
}
}
cmd/mesh-controller/build_test.go
+116 -43
View File
@@ -1,13 +1,12 @@
package main
import (
"bytes"
"reflect"
"encoding/json"
"errors"
"strings"
"testing"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -16,12 +15,126 @@ import (
// the module registered with its source as the seat and path when the request said so — never the
// URL. A definition naming an installation is recorded and not registered; a failure is recorded
// and said.
func TestABuildHeardIsRecordedAndRegistered(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "shop", "version": "3"})
m, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, link.BuildResult{
ID: "b-1", Repository: "http://forge.internal:20000/novox/shop.git", Path: "modules/shop",
Ref: "main", On: "anchor", Commit: "abcdef0123", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/shop"},
}))
if err != nil {
t.Fatal(err)
}
if m.Module != "shop" {
t.Fatalf("registered %q", m.Module)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if _, held := shelf["shop"]; !held {
t.Fatal("the module a heard build produced is not in the catalogue")
}
src, err := open.inventory.SourceOf(ctx, "shop")
if err != nil || src.Seat != "git" || src.Repository != "novox/shop" || src.BuiltFrom != "abcdef0123" {
t.Fatalf("the source is the seat and the path, never the URL: %+v %v", src, err)
}
builds, err := open.inventory.Builds(ctx, "shop", 5)
if err != nil || len(builds) != 1 || builds[0].ID != "b-1" {
t.Fatalf("the build is not recorded once: %v %v", builds, err)
}
named, _ := json.Marshal(map[string]any{"module": "idp", "version": "1", "resources": []any{
map[string]any{"id": "server", "type": "container", "image": "x@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}}}})
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{
ID: "b-2", Repository: "/r", On: "anchor", Commit: "0123456789", Manifest: named})
if err == nil || !strings.Contains(err.Error(), "does not register it") {
t.Fatalf("a definition naming an installation was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["idp"].Module != "" {
t.Fatal("the refused module was registered anyway")
}
if builds, _ := open.inventory.Builds(ctx, "idp", 5); len(builds) != 1 {
t.Fatalf("the refused build was not recorded: %v", builds)
}
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{ID: "b-3", Repository: "/r", On: "anchor", Failed: "no compiler"})
if err == nil || !strings.Contains(err.Error(), "no compiler") {
t.Fatalf("a failure is said in the builder's words: %v", err)
}
}
// novox/hq 04-ISSUES/215: a build asked at a commit is recorded as built from that commit, and the
// module keeps following the branch it followed — a new one, the default branch.
func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "unifi", "version": "1"})
result := func(id, ref, commit string) link.BuildResult {
return link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/unifi", Ref: ref, On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result("b-1", "main", "1111111aaaa"))); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, result("b-2", "9c97a8a", "9c97a8a1d2c3")); err != nil {
t.Fatal(err)
}
src, err := open.inventory.SourceOf(ctx, "unifi")
if err != nil {
t.Fatal(err)
}
if src.Ref != "main" || src.BuiltFrom != "9c97a8a1d2c3" {
t.Errorf("after a build at a commit the module follows %q, built from %q; want main, 9c97a8a1d2c3", src.Ref, src.BuiltFrom)
}
// One new to the catalogue, first built at a commit, follows the default branch.
other, _ := json.Marshal(map[string]any{"module": "letta", "version": "1"})
r := result("b-3", "deadbeef", "deadbeefcafe")
r.Manifest, r.Path = other, "modules/letta"
if _, _, err := takeIn(ctx, open.inventory, r); err != nil {
t.Fatal(err)
}
if src, _ := open.inventory.SourceOf(ctx, "letta"); src.Ref != "" {
t.Errorf("a module first built at a commit follows %q, want the default branch", src.Ref)
}
}
// novox/hq 04-ISSUES/219: an older request heard after a newer one is recorded and not registered,
// so a push sends what the newer request built.
func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
result := func(asked time.Time, image string) link.BuildResult {
manifest, _ := json.Marshal(map[string]any{"module": "postgres", "version": image})
return link.BuildResult{ID: link.NewBuildID(asked), Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result(newer, "4bcd5f73"))); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
if !errors.Is(err, inventory.ErrSuperseded) {
t.Fatalf("the older request's outcome was taken in as current: %v", err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "4bcd5f73" {
t.Errorf("postgres is %q; want the newer request's 4bcd5f73", got)
}
if builds, _ := open.inventory.Builds(ctx, "postgres", 5); len(builds) != 2 {
t.Errorf("the late build was not recorded: %v", builds)
}
}
func TestABuildIDSaysWhenItWasAsked(t *testing.T) {
at := time.Date(2026, 10, 3, 21, 51, 57, 392539762, time.UTC)
@@ -37,43 +150,3 @@ func TestABuildIDSaysWhenItWasAsked(t *testing.T) {
}
}
}
// novox/hq issue 1204: builds says the source fingerprint the builder recorded, the one tool that reads it.
func TestBuildsSaysTheSourceFingerprint(t *testing.T) {
at := time.Date(2026, 10, 19, 10, 40, 0, 0, time.UTC)
built := inventory.Build{ID: "build-1", Module: "mailu", Commit: "b4527d9091e972f28cb365623d9280045e7741a3", On: "ace",
Repository: "novox/mesh-catalog", At: at, ReuseKey: "r1:abc",
SourceFingerprint: "src1:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}
// 1204/AC1
var out bytes.Buffer
printBuild(&out, built)
lines := strings.Split(strings.TrimRight(out.String(), "\n"), "\n")
if last := lines[len(lines)-1]; last != " source "+built.SourceFingerprint {
t.Errorf("last line %q; want the source line after the reuse key:\n%s", last, out.String())
}
if !strings.Contains(lines[len(lines)-2], "reuse key r1:abc") {
t.Errorf("the source line does not follow the reuse key:\n%s", out.String())
}
// 1204/AC2
without := built
without.SourceFingerprint = ""
var plain bytes.Buffer
printBuild(&plain, without)
if strings.Contains(plain.String(), "source") {
t.Errorf("a build with no fingerprint says a source line:\n%s", plain.String())
}
if want := strings.Join(lines[:len(lines)-1], "\n") + "\n"; plain.String() != want {
t.Errorf("the other lines changed:\n%s\nwant\n%s", plain.String(), want)
}
}
// 1215/AC4: what the builder said of the inputs is what the controller keeps of the build.
func TestTheControllerKeepsTheInputsABuilderSaid(t *testing.T) {
in := &sdkdelivery.SourceInputs{Module: "mailu", Tree: "t", Toolchains: map[string]string{"code": "ts sha256:aa 01"}}
kept := buildFrom(link.BuildResult{ID: "build-1", Repository: "r", Commit: "c", SourceFingerprint: "src1:x", SourceInputs: in})
if !reflect.DeepEqual(kept.SourceInputs, in) {
t.Errorf("the controller keeps %+v, the builder said %+v", kept.SourceInputs, in)
}
}
cmd/mesh-controller/bus_step.go
+1 -1
View File
@@ -11,7 +11,7 @@ import (
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
cmd/mesh-controller/bus_users_closed_test.go
-92
View File
@@ -1,92 +0,0 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **Every user's slow consumer is S9's, not the controller's alone** (novox/hq ADR 0357, issue 270). On
// 2026-10-07 the bus closed two clients on the laptop as slow consumers and nothing raised a condition:
// the controller heard only its own connection. Said by the bus module and read back, a user the bus
// closes is one condition naming the module and its machine, raised at the first look and cleared an hour
// after the last closure.
func TestAUserTheBusClosedAsASlowConsumerIsOneConditionNamingItsModuleAndMachine(t *testing.T) {
closed := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
now := closed.Add(20 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
advisories := link.BusClosures([]link.BusClosure{
{User: "laptop.claude-code", CID: 10, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Hour),
Closed: closed.Add(-10 * time.Minute)},
{User: "laptop.claude-code", CID: 44, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Minute),
Closed: closed},
{User: "anchor.desk", CID: 7, Reason: "Maximum Message Payload Exceeded", Opened: closed.Add(-time.Hour),
Closed: closed},
}, []string{link.AdvisorySlowConsumer, link.AdvisorySlowConsumer, link.AdvisoryConnectionClosed}, closed.Add(-time.Hour))
look := func() {
t.Helper()
f := &signalFacts{now: now, host: "anchor", advisories: advisories}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
}
look()
slow, found, err := conditions.ReadOne(t.Context(), store, "bus.laptop.claude-code.slow-consumer")
if err != nil || !found {
t.Fatalf("no slow-consumer condition for the user: %v %v", found, err)
}
if slow.Subject.Machine != "laptop" || slow.Observations != 2 || !slow.LastObserved.Equal(closed) {
t.Fatalf("the slow consumer is said on %q, %d closure(s), last %s: %+v", slow.Subject.Machine, slow.Observations,
slow.LastObserved, slow)
}
if !strings.Contains(slow.Summary, "claude-code on laptop") || strings.Contains(slow.Summary, "laptop.claude-code") {
t.Fatalf("the summary does not name the module in words, or carries the dotted user: %q", slow.Summary)
}
if !strings.Contains(slow.Headline, "Claude-code on laptop") {
t.Fatalf("the headline is %q", slow.Headline)
}
cut, found, err := conditions.ReadOne(t.Context(), store, "bus.anchor.desk.connection-closed")
if err != nil || !found || cut.Subject.Machine != "anchor" || !strings.Contains(cut.Summary, "maximum message payload exceeded") {
t.Fatalf("the user cut off for breaking the bus's rules: %v %v %+v", found, err, cut)
}
// An hour after the newest closure without another, both clear.
now = closed.Add(time.Hour + time.Minute)
look()
for _, key := range []string{"bus.laptop.claude-code.slow-consumer", "bus.anchor.desk.connection-closed"} {
if _, found, _ := k.Get(t.Context(), key); found {
t.Fatalf("%s is still open an hour after the last closure", key)
}
}
}
// **A read of the bus module's events that fails does not blind S9** (review of #326): the controller's own
// signals are still said, and the failed read is a condition of its own.
func TestABusModuleReadThatFailsKeepsTheControllersOwnSignals(t *testing.T) {
at := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
f := &signalFacts{now: at.Add(time.Minute), host: "anchor",
advisories: []link.Advisory{{Kind: link.AdvisorySlowConsumer, ID: "controller", Said: "the controller fell behind",
First: at, Last: at, Count: 1}},
busFaultsErr: errors.New("nats: timeout")}
keys := map[string]bool{}
for _, o := range watchAdvisories(f) {
keys[o.Key()] = true
}
if !keys["bus.controller.slow-consumer"] || !keys["bus.bus-module.bus-faults-unread"] {
t.Fatalf("observed %v: the controller's own slow consumer and the unread events, both", keys)
}
for _, row := range signalsTable {
if row.Row == "S9" && row.needs(f) != nil {
t.Fatal("the S9 row went blind for a read of the bus module's events")
}
}
}
cmd/mesh-controller/bus_waits_integration_test.go
-70
View File
@@ -1,70 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
func TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds(t *testing.T) {
now := time.Date(2026, 10, 8, 18, 30, 0, 0, time.UTC)
b := aBusOnAnchor()
walk := func(id, repo, note string, updated time.Time) inventory.Plan {
return inventory.Plan{ID: id, Repository: repo, Commit: "c0ffee001122", State: inventory.PlanRolling, Tier: 0,
Tiers: [][]string{{"mesh-host"}}, Modules: map[string]*inventory.PlanModule{"mesh-host": {}}, Note: note,
Updated: updated}
}
for _, c := range []struct {
name string
bus busPending
plans []inventory.Plan
first map[string]time.Time
want []string
since time.Time
}{
{"a walk refused for the bus", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-27*time.Minute))}, nil,
[]string{"plan-1"}, now.Add(-27 * time.Minute)},
{"refused earlier than its last save, as this controller saw it", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-5*time.Minute))},
map[string]time.Time{"plan-1": now.Add(-28 * time.Minute)}, []string{"plan-1"}, now.Add(-28 * time.Minute)},
{"a walk refused for another reason", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", "tier 0: the build seat is paused — tried again", now)}, nil,
nil, time.Time{}},
{"refused for an older bus build than the one held now", func() busPending { o := b; o.to = "99999999cccc"; return o }(),
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"the bus already runs the build held: its step started", func() busPending {
o := aBusOnAnchor()
o.from = map[string]string{"anchor": o.to}
return o
}(), []inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"a walk waiting for its delivery's word asks no send", b, func() []inventory.Plan {
p := walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)
p.Delivery = &inventory.PlanDelivery{Awaits: "mesh-delivery"}
return []inventory.Plan{p}
}(), nil, nil, time.Time{}},
} {
t.Run(c.name, func(t *testing.T) {
got := busWaitsOf(c.plans, c.bus, func(id string) time.Time { return c.first[id] })
var ids []string
for _, w := range got.waits {
ids = append(ids, w.plan)
}
if strings.Join(ids, ",") != strings.Join(c.want, ",") {
t.Fatalf("held for the bus: %v, want %v", ids, c.want)
}
if len(c.want) > 0 {
if !got.waits[0].since.Equal(c.since) {
t.Errorf("waiting since %s, want %s", got.waits[0].since, c.since)
}
if strings.Join(got.machines, ",") != "anchor" || got.to != b.to || got.module != "nats" {
t.Errorf("behind %+v", got)
}
}
})
}
}
cmd/mesh-controller/bus_waits_test.go
+60
View File
@@ -7,6 +7,7 @@ import (
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 336: a send held because it would replace the bus outside its planned step waits for a
@@ -26,6 +27,65 @@ func refusedNote(b busPending, tier int) string {
return "tier " + string(rune('0'+tier)) + ": " + errBusWaits.Error() + ": " + held + " — tried again"
}
func TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds(t *testing.T) {
now := time.Date(2026, 10, 8, 18, 30, 0, 0, time.UTC)
b := aBusOnAnchor()
walk := func(id, repo, note string, updated time.Time) inventory.Plan {
return inventory.Plan{ID: id, Repository: repo, Commit: "c0ffee001122", State: inventory.PlanRolling, Tier: 0,
Tiers: [][]string{{"mesh-host"}}, Modules: map[string]*inventory.PlanModule{"mesh-host": {}}, Note: note,
Updated: updated}
}
for _, c := range []struct {
name string
bus busPending
plans []inventory.Plan
first map[string]time.Time
want []string
since time.Time
}{
{"a walk refused for the bus", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-27*time.Minute))}, nil,
[]string{"plan-1"}, now.Add(-27 * time.Minute)},
{"refused earlier than its last save, as this controller saw it", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-5*time.Minute))},
map[string]time.Time{"plan-1": now.Add(-28 * time.Minute)}, []string{"plan-1"}, now.Add(-28 * time.Minute)},
{"a walk refused for another reason", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", "tier 0: the build seat is paused — tried again", now)}, nil,
nil, time.Time{}},
{"refused for an older bus build than the one held now", func() busPending { o := b; o.to = "99999999cccc"; return o }(),
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"the bus already runs the build held: its step started", func() busPending {
o := aBusOnAnchor()
o.from = map[string]string{"anchor": o.to}
return o
}(), []inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"a walk waiting for its delivery's word asks no send", b, func() []inventory.Plan {
p := walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)
p.Delivery = &inventory.PlanDelivery{Awaits: "mesh-delivery"}
return []inventory.Plan{p}
}(), nil, nil, time.Time{}},
} {
t.Run(c.name, func(t *testing.T) {
got := busWaitsOf(c.plans, c.bus, func(id string) time.Time { return c.first[id] })
var ids []string
for _, w := range got.waits {
ids = append(ids, w.plan)
}
if strings.Join(ids, ",") != strings.Join(c.want, ",") {
t.Fatalf("held for the bus: %v, want %v", ids, c.want)
}
if len(c.want) > 0 {
if !got.waits[0].since.Equal(c.since) {
t.Errorf("waiting since %s, want %s", got.waits[0].since, c.since)
}
if strings.Join(got.machines, ",") != "anchor" || got.to != b.to || got.module != "nats" {
t.Errorf("behind %+v", got)
}
}
})
}
}
// **Said at once, not as lateness, and cleared when the step starts**: the walk held only for the bus raises
// the bus's condition on the first tick, before any tier bound, naming what waits, the bus build from and to,
// since when and `bus upgrade`, for the operator; S3 says nothing of it even past its bound; and the
cmd/mesh-controller/busobjects.go
+5 -42
View File
@@ -49,12 +49,10 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus
// by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28).
consumers, records, err := moduleConsumersAndRecords(ctx, inv)
consumers, err := moduleConsumers(ctx, inv)
if err != nil {
return nil, err
}
// And a grant of a restricted stream that composes nothing, said on every raise and send (novox/hq ADR 0376 §2).
sayWithheld(records, " ")
// And the work queues of seats that name their caller or their kind, with each holder's worker
// (novox/hq ADR 0259 §3): an ask queues until the router takes it, a channel's work until that kind
// takes it.
@@ -62,19 +60,9 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err != nil {
return nil, err
}
// And every module's own stream (novox/hq ADR 0363), from the catalogue, before the consumers on it.
modStreams, err := inv.DeclaredStreams(ctx)
if err != nil {
return nil, err
}
// Every one tried, and every failure named: one module's consumer the bus refuses is no reason
// the modules after it in the list hear nothing (novox/hq issue 208, where this runs on each send).
var failed []error
for _, s := range broker.ModuleStreams(modStreams) {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the module stream %s: %w", s.Name, err))
}
}
for _, s := range trafficStreams {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the work queue %s: %w", s.Name, err))
@@ -85,30 +73,11 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
failed = append(failed, fmt.Errorf("the worker %s on %s: %w", c.Name, c.Stream, err))
}
}
restricted := map[string]bool{}
for _, s := range modStreams {
restricted[s.Name] = s.Restricted
}
for _, c := range consumers {
if err := broker.EnsureHeard(r, c.Consumer, restricted[c.Consumer.Stream]); err != nil {
if err := r.EnsureConsumer(c.Consumer); err != nil {
failed = append(failed, fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err))
}
}
// **And a consumer no longer granted loses what it reads through** (novox/hq ADR 0363 §6): every consumer
// on a module's stream the composition no longer asks for is removed, on this very send.
if p, prunes := r.(broker.Pruner); prunes && len(modStreams) > 0 {
var wanted []broker.Consumer
for _, c := range consumers {
wanted = append(wanted, c.Consumer)
}
removed, err := broker.PruneHeard(p, modStreams, wanted)
if err != nil {
failed = append(failed, err)
}
for _, gone := range removed {
fmt.Printf(" removed the consumer %s: its module is no longer granted that stream (novox/hq ADR 0363)\n", gone)
}
}
if len(failed) > 0 {
return nil, errors.Join(failed...)
}
@@ -167,21 +136,15 @@ func unassertedObservation(err error) conditions.Observation {
// moduleConsumers is every module's durable consumer, from the records the user list is composed from.
func moduleConsumers(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, error) {
consumers, _, err := moduleConsumersAndRecords(ctx, inv)
return consumers, err
}
// moduleConsumersAndRecords is moduleConsumers with the records they were composed from.
func moduleConsumersAndRecords(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, broker.Records, error) {
records, err := inv.BusRecords(ctx)
if err != nil {
return nil, broker.Records{}, err
return nil, err
}
users, err := broker.Users(records)
if err != nil {
return nil, broker.Records{}, err
return nil, err
}
return broker.ConsumersOf(users), records, nil
return broker.ConsumersOf(users), nil
}
// seatTrafficObjects is the work queues and workers of seats that name their caller or their kind, from
cmd/mesh-controller/busobjects_test.go
+1 -3
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
@@ -9,8 +7,8 @@ import (
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
cmd/mesh-controller/busy.go
-103
View File
@@ -1,103 +0,0 @@
package main
import (
"fmt"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A first-node gate waits for a busy container (novox/hq ADR 0335 §4).
//
// When a machine reports, on what the gate sent it, that a container of a judged module was left running
// because it said busy, the judging has not started: the build is not yet what runs there. The walk says it
// waits for a busy container on that machine, counts no pass and no bound, and starts the judging — its
// Since — at the report that no longer lists the container waiting. The wait is the node-engine's, bounded
// at 15 minutes; the controller's own bound (busyGateBound past the engine's) is for an engine that stopped
// reporting while it waited, so no walk waits for ever. An engine older than the field never lists one, and
// its gate starts as before.
// busyGateSlack is how long past the engine's bound the controller still waits for the report that says the
// container replaced: a pass of the engine (its asks every 20 seconds) and the report's way here.
var busyGateSlack = 5 * time.Minute
// busyEngineBound is the node-engine's bound on a wait (novox/hq ADR 0335 §3), counted again here on this
// controller's clock from when it first read the wait.
const busyEngineBound = 15 * time.Minute
// busyGateEnds is when the gate stops waiting: the engine's own word on its bound, or this controller's count of
// it from when it first read the wait, whichever is sooner, and the slack — so neither a clock ahead on the
// machine nor a wrong word from it holds a walk longer.
func busyGateEnds(b *inventory.GateBusy) time.Time {
until := b.Seen.Add(busyEngineBound)
if !b.Until.IsZero() && b.Until.Before(until) {
until = b.Until
}
return until.Add(busyGateSlack)
}
// busyWait decides, before a judging, whether the gate waits for a busy container on a machine it judges:
// true while it waits, the gate saying so. False when nothing waits, or no longer: a wait that ended moves the
// gate's Since to its end. Pure but for the gate it is handed.
func busyWait(g *inventory.PlanGate, pairs []judged, f gateFacts, now time.Time) bool {
if g.Busy != nil && g.Busy.Ended != nil {
return false
}
for _, j := range pairs {
b, waits := busyOn(j, f)
if !waits {
continue
}
if g.Busy == nil {
g.Busy = &inventory.GateBusy{Machine: j.node, Module: j.module, Container: b.Container, Since: b.Since,
Seen: now}
}
if g.Busy.Seen.IsZero() {
g.Busy.Seen = now
}
g.Busy.Until, g.Busy.Said = b.Until, b.Said
if now.After(busyGateEnds(g.Busy)) {
// The engine said it would have replaced it by now: the judging starts, and finds the module not
// yet healthy while the machine still lists it waiting (busyStill), until the gate's own bound.
end := now
g.Busy.Ended, g.Busy.PastBound = &end, true
g.Since, g.Passes, g.LastPass, g.Healthy, g.HealthyAt, g.Applied = &end, 0, nil, nil, nil, nil
return false
}
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, g.Busy.Words(), nil
g.Read(now, false, g.Busy.Words())
return true
}
if g.Busy == nil {
return false
}
// Replaced: the judging starts at the report that said so.
end := now
if r, said := f.reports[g.Busy.Machine]; said && r.At != nil && r.At.After(g.Busy.Since) && r.At.Before(now) {
end = r.At.UTC()
}
g.Busy.Ended = &end
g.Since, g.Passes, g.LastPass, g.Last, g.Healthy, g.HealthyAt, g.Applied = &end, 0, nil, "", nil, nil, nil
return false
}
// busyOn is the container of a judged module its machine last reported, on what the gate sent it, as left
// running because it said busy.
func busyOn(j judged, f gateFacts) (inventory.BusyWait, bool) {
r, said := f.reports[j.node]
if !said || r.At == nil || !f.reportedOn(j.node, r) {
return inventory.BusyWait{}, false
}
return inventory.WaitingFor(r.Busy, j.module)
}
// busyStill is a judged module whose container its machine still lists waiting once the gate stopped waiting
// for it: not yet what was sent, so not healthy, whatever its health says of the container still running.
func busyStill(j judged, f gateFacts) (string, bool) {
b, waits := busyOn(j, f)
if !waits {
return "", false
}
return fmt.Sprintf("%s still runs its busy container %s, past its node-engine's bound of %s",
j.node, b.Container, b.Until.UTC().Format(time.RFC3339)), true
}
cmd/mesh-controller/busy_test.go
-148
View File
@@ -1,148 +0,0 @@
package main
import (
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A walk whose module's container waits as busy on its first machine says so, and starts that machine's
// first-machine gate only once the machine reports the container replaced (novox/hq ADR 0335 §4).
func TestAGateWaitsForABusyContainerAndStartsWhenItIsReplaced(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent,
Sent: map[string]inventory.SentDeclaration{"novox": {Digest: "d1", Sequence: 7}}}
pairs := []judged{{module: "build-agent", node: "novox"}, {module: "mesh-controller", node: "novox"}}
reported := sent.Add(20 * time.Second)
waiting := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: reported, Until: reported.Add(15 * time.Minute),
Said: "busy: novox/mesh-controller (check of #280), in hand since 00:58:00Z"}}
facts := func(at time.Time, busy []inventory.BusyWait) gateFacts {
// As judgeMoves hands it: the report held against the gate's own send.
return gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"novox": {Node: "novox",
Outcome: inventory.OutcomeApplied, At: &at, Declared: "d1", ReportedSequence: 7, Busy: busy}}}
}
now := reported.Add(time.Minute)
if !busyWait(g, pairs, facts(reported, waiting), now) {
t.Fatal("a gate whose module's container waits as busy judged it")
}
if !g.Since.Equal(sent) || g.Busy == nil || g.Busy.Machine != "novox" || g.Busy.Ended != nil {
t.Fatalf("the gate does not keep the wait: %+v %+v", g.Since, g.Busy)
}
for _, said := range []string{gateLine(g), g.Last} {
if !strings.Contains(said, "waits for a busy container on novox") {
t.Errorf("the walk does not say it waits for a busy container there: %s", said)
}
}
if !strings.Contains(g.Last, "check of #280") {
t.Errorf("the wait does not say what the container holds: %s", g.Last)
}
// Ten minutes on, still busy, within the engine's bound: still waiting, nothing judged, no bound run.
now = reported.Add(10 * time.Minute)
if !busyWait(g, pairs, facts(reported.Add(9*time.Minute), waiting), now) || g.Passes != 0 {
t.Fatalf("the gate stopped waiting within the bound: %+v", g)
}
// The machine reports it replaced: the judging starts at that report.
replacedAt := reported.Add(12 * time.Minute)
now = replacedAt.Add(5 * time.Second)
replaced := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: "replaced-when-idle", Since: reported, Until: reported.Add(15 * time.Minute)}}
if busyWait(g, pairs, facts(replacedAt, replaced), now) {
t.Fatal("the gate still waits after the container was replaced")
}
if g.Since == nil || !g.Since.Equal(replacedAt) || g.Busy.Ended == nil || !g.Busy.Ended.Equal(replacedAt) {
t.Fatalf("the gate does not start at the report that said it replaced: since %v, %+v", g.Since, g.Busy)
}
if strings.Contains(gateLine(g), "waits for a busy container") {
t.Errorf("a gate judging still says it waits: %s", gateLine(g))
}
// And a later report listing it waiting again does not hold a gate that started.
if busyWait(g, pairs, facts(now, waiting), now.Add(time.Minute)) {
t.Error("a gate that started waits again")
}
}
// An engine that stops reporting while it waits holds no walk for ever: past the engine's bound and the slack,
// the judging starts, and finds the module not yet healthy while its container is still listed waiting.
func TestAGateWaitsNoLongerThanTheEnginesBound(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent}
pairs := []judged{{module: "build-agent", node: "novox"}}
waiting := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: sent, Until: sent.Add(15 * time.Minute)}}
f := gateFacts{reports: map[string]inventory.Reported{"novox": {Node: "novox", Outcome: inventory.OutcomeApplied,
At: &sent, Current: true, Busy: waiting}}}
now := sent.Add(15*time.Minute + busyGateSlack + time.Second)
if busyWait(g, pairs, f, now) {
t.Fatal("a gate waited past the engine's bound and the slack")
}
if !g.Busy.PastBound || !g.Since.Equal(now) {
t.Fatalf("the gate does not start, said past the bound: %+v %v", g.Busy, g.Since)
}
if why, still := busyStill(pairs[0], f); !still || !strings.Contains(why, "past its node-engine's bound") {
t.Fatalf("a module still running its busy container reads as what was sent: %q", why)
}
}
// A report from an engine older than the field lists nothing: the gate starts at the send, as before; and a
// report not about the gate's send says nothing of it.
func TestAGateOfAnEngineThatNeverWaitsStartsAsBefore(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"ace"}, Since: &sent,
Sent: map[string]inventory.SentDeclaration{"ace": {Digest: "d2", Sequence: 9}}}
pairs := []judged{{module: "build-agent", node: "ace"}}
at := sent.Add(time.Minute)
older := gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"ace": {Node: "ace", Outcome: inventory.OutcomeApplied,
At: &at, Declared: "d2", ReportedSequence: 9}}}
if busyWait(g, pairs, older, at) || g.Busy != nil || !g.Since.Equal(sent) {
t.Fatalf("a gate on an engine that never waits did not start at the send: %+v %v", g.Busy, g.Since)
}
elsewhere := gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"ace": {Node: "ace", Outcome: inventory.OutcomeApplied,
At: &at, Declared: "d1", ReportedSequence: 8, Busy: []inventory.BusyWait{{ID: "build-agent.server",
State: inventory.BusyWaiting, Until: at.Add(time.Hour)}}}}}
if busyWait(g, pairs, elsewhere, at) {
t.Fatal("a wait reported about an earlier declaration held the gate")
}
}
// The wait is bounded on this controller's clock: an engine whose clock runs an hour ahead, saying it waits until
// then, holds the gate no longer than 15 minutes and the slack from when the controller first read the wait.
func TestAGateWaitIsBoundedOnTheControllersClock(t *testing.T) {
seen := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &seen}
pairs := []judged{{module: "build-agent", node: "novox"}}
ahead := seen.Add(time.Hour)
f := gateFacts{reports: map[string]inventory.Reported{"novox": {Node: "novox", Outcome: inventory.OutcomeApplied,
At: &seen, Current: true, Busy: []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: ahead, Until: ahead.Add(15 * time.Minute)}}}}}
if !busyWait(g, pairs, f, seen) || !g.Busy.Seen.Equal(seen) {
t.Fatalf("the gate did not start waiting, seen now: %+v", g.Busy)
}
if !busyWait(g, pairs, f, seen.Add(15*time.Minute+busyGateSlack-time.Second)) {
t.Fatal("the gate stopped waiting within the bound on its own clock")
}
if busyWait(g, pairs, f, seen.Add(15*time.Minute+busyGateSlack+time.Second)) || !g.Busy.PastBound {
t.Fatalf("an engine's later word held the gate past the controller's bound: %+v", g.Busy)
}
}
// `plan` says, for a machine whose node-engine is older than the field, each container whose busy is not sent.
func TestAPlanSaysABusyAnOlderEngineIsNotSent(t *testing.T) {
modules := []catalogue.Manifest{{Module: "build-agent", Resources: []map[string]any{
{"id": "server", "type": "container", "busy": map[string]any{"command": "mesh-builder busy"}},
{"id": "agent-env", "type": "file"}}}}
lines := busyNotSent(modules, false)
if len(lines) != 1 || !strings.Contains(lines[0], "build-agent.server declares busy") ||
!strings.Contains(lines[0], "replaced at once") {
t.Fatalf("the plan does not say the busy not sent: %v", lines)
}
if lines := busyNotSent(modules, true); len(lines) != 0 {
t.Fatalf("a busy sent is said as not sent: %v", lines)
}
}
cmd/mesh-controller/calls_verb_integration_test.go
-36
View File
@@ -1,36 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"strings"
"testing"
)
func TestCallsIsServedAndSaysWhatItKeeps(t *testing.T) {
handlers, behind, err := seatToolHandlers()
if err != nil || len(behind) != 0 {
t.Fatalf("%v %v", behind, err)
}
calls, ok := handlers["calls"]
if !ok {
t.Fatal("calls is not served")
}
if _, err := calls(context.Background(), json.RawMessage(`{"node":"anchor"}`)); err == nil ||
!strings.Contains(err.Error(), `"node"`) {
t.Errorf("calls took an argument it does not declare: %v", err)
}
if _, err := calls(context.Background(), json.RawMessage(`{"call":"call-0-0"}`)); err == nil ||
!strings.Contains(err.Error(), "not across a restart") {
t.Errorf("an unknown call was not said plainly: %v", err)
}
got, err := calls(context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if _, listed := got.(map[string]any)["calls"]; !listed {
t.Errorf("calls answered %v", got)
}
}
cmd/mesh-controller/calls_verb_test.go
+28
View File
@@ -1,6 +1,9 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
@@ -49,3 +52,28 @@ func TestAPushAnswersBeforeItSends(t *testing.T) {
}
// `calls` is served, takes a call's id and nothing else, and says plainly when it holds no such call.
func TestCallsIsServedAndSaysWhatItKeeps(t *testing.T) {
handlers, behind, err := seatToolHandlers()
if err != nil || len(behind) != 0 {
t.Fatalf("%v %v", behind, err)
}
calls, ok := handlers["calls"]
if !ok {
t.Fatal("calls is not served")
}
if _, err := calls(context.Background(), json.RawMessage(`{"node":"anchor"}`)); err == nil ||
!strings.Contains(err.Error(), `"node"`) {
t.Errorf("calls took an argument it does not declare: %v", err)
}
if _, err := calls(context.Background(), json.RawMessage(`{"call":"call-0-0"}`)); err == nil ||
!strings.Contains(err.Error(), "not across a restart") {
t.Errorf("an unknown call was not said plainly: %v", err)
}
got, err := calls(context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if _, listed := got.(map[string]any)["calls"]; !listed {
t.Errorf("calls answered %v", got)
}
}
cmd/mesh-controller/chain.go
-177
View File
@@ -1,177 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"time"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The chain on the bus (novox/hq ADR 0408, design 47 milestone 5): builds, rollouts and assignments are keys of the
// bus's key-value store, written by the controller that holds the lease, and the controller's store keeps none of them.
// chainHolds is whether this process may write the chain now: the lease's gate, the one every act passes.
func chainHolds(ctx context.Context) error {
_, err := theLease.epoch(ctx)
return err
}
// openTheChain opens the chain's buckets on the bus: the serving controller's connection, or one dialled for a command.
func openTheChain(ctx context.Context) (*chain.Chain, error) {
js, err := aBus()
if err != nil {
return nil, err
}
// Not closed: a borrowed connection closes nothing, and a dialled one lasts as long as the command.
return chain.OnTheBus(ctx, js.Conn(), chainHolds)
}
// attachTheChain gives the inventory of the serving controller the chain on its connection.
func attachTheChain(ctx context.Context, inv *inventory.Inventory, server *link.Server) error {
bus, ok := server.Bus().(link.OverNATS)
if !ok {
return errors.New("the chain lives on the bus, and this control plane is not on it")
}
c, err := chain.OnTheBus(ctx, bus.Conn, chainHolds)
if err != nil {
return err
}
inv.UseChain(c)
inv.RequireLayout()
// And the builds laid out under their modules, once, before they are read (issue 1121) — **before this returns**, so
// nothing the start goes on to read meets a chain not laid out yet: on 2026-10-18 the layout ran beside the start, a
// read of the start was refused, and the controller exited and was started again every five seconds. The layout of
// keys laid out before skips them, so this takes seconds; one that fails is tried again beside the serving, loudly.
if err := inv.LayBuildsOut(ctx, func(format string, args ...any) { fmt.Printf("chain: "+format+"\n", args...) }); err != nil {
fmt.Printf("chain: the builds are not laid out under their modules yet, and none is read until they are: %v\n", err)
go layBuildsOutSoon(ctx, inv)
}
// And the requests the move of delivery 1 found never handed over, asked again once (askAgain), once it is serving.
go askAgainSoon(ctx, inv)
// And what delivery 1 recorded of what each machine was sent, copied onto the keys of the assignments (CopySentBuilds).
go copySentBuildsSoon(ctx, inv)
return nil
}
// askAgain asks once more for each build request the move found never handed over (novox/hq issue 981 note 7 Q5): the
// requests a merge could not ask for, which a person's hand `build` would otherwise have to repeat. **At most once**: a
// request is claimed before it is asked and settled after, so a crash between loses the ask, which is where it stood.
func askAgain(ctx context.Context, inv *inventory.Inventory) {
pending, err := inv.RequestsToAskAgain(ctx)
if err != nil {
fmt.Printf("chain: the requests never handed over cannot be read: %v\n", err)
return
}
for _, r := range pending {
if claimed, err := inv.ClaimToAskAgain(ctx, r.ID); err != nil || !claimed {
if err != nil {
fmt.Printf("chain: %s cannot be claimed to be asked again: %v\n", r.ID, err)
}
continue
}
id, err := askABuild(ctx, buildSource{Repository: r.Repository, Seat: r.Seat}, r.Path, r.Ref)
if err != nil {
if serr := inv.AskedAgainAs(ctx, r.ID, "not asked again: "+err.Error()); serr != nil {
fmt.Printf("chain: %s was not asked again, and that could not be kept: %v\n", r.ID, serr)
}
fmt.Printf("chain: %s was never handed over and cannot be asked again: %v\n", r.ID, err)
continue
}
recordBuildRequest(ctx, inv, inventory.BuildRequest{ID: id, Repository: r.Repository, Seat: r.Seat, Path: r.Path,
Ref: r.Ref, Commit: r.Commit, For: r.For})
if err := inv.AskedAgainAs(ctx, r.ID, id); err != nil {
fmt.Printf("chain: %s was asked again as %s, and that could not be kept: %v\n", r.ID, id, err)
}
fmt.Printf("chain: %s, never handed over, is asked again as %s\n", r.ID, id)
}
}
// askAgainSoon waits for the controller to be serving, which an ask needs, and then asks again.
func askAgainSoon(ctx context.Context, inv *inventory.Inventory) {
select {
case <-ctx.Done():
case <-time.After(time.Minute):
askAgain(ctx, inv)
}
}
// kindSentBuildsNotCopied is the condition of the one-time copy of node.sent_builds onto the assignments' keys that has not
// succeeded: until it has, a module an assignment pulled in may be missing from what a machine is read to run.
const kindSentBuildsNotCopied = "sent-builds-not-copied"
// copySentBuildsSoon copies what the store recorded of what each machine was sent onto the keys of the assignments, as soon
// as this controller holds the lease (the chain refuses a write before), and tries again every half minute while it fails,
// each time it fails saying so as an urgent condition until it has worked. It is idempotent: CopySentBuilds overwrites no key
// that names a commit and empties each column it has copied.
func copySentBuildsSoon(ctx context.Context, inv *inventory.Inventory) {
failed := false
for {
_, err := inv.CopySentBuilds(ctx)
o := conditions.Observation{Scope: conditions.ScopeBuild, ID: kindSentBuildsNotCopied, Token: kindSentBuildsNotCopied,
Kind: kindSentBuildsNotCopied, Severity: conditions.Urgent, Resolver: conditions.ResolverOperator,
Source: "chain", Summary: "what each machine was sent is not copied onto its assignments: "}
if err != nil {
fmt.Printf("chain: what the machines were sent is not copied onto the assignments: %v\n", err)
if keeper := conditionsFrom; keeper != nil && !failed {
o.Summary += err.Error()
if _, oerr := keeper.Observe(ctx, o); oerr == nil {
failed = true
}
}
} else {
if keeper := conditionsFrom; keeper != nil && failed {
_, _ = keeper.Clear(ctx, o.Key(), "copied")
}
return
}
select {
case <-ctx.Done():
return
case <-time.After(chainMoveRetry):
}
}
}
// chainMoveRetry is how long the copy waits before it tries again.
const chainMoveRetry = 30 * time.Second
// kindBuildsNotLaidOut is the condition of the layout of the builds under their modules that has not succeeded: until it
// has, no build is read, and a send that needs one is refused with the reason.
const kindBuildsNotLaidOut = "builds-not-laid-out"
// layBuildsOutSoon lays the builds out under their modules as soon as this controller holds the lease (the chain refuses a
// write before), and tries again every half minute while it fails, loudly: each failure is printed, and the first raises an
// urgent condition until it has worked. It is idempotent (LayBuildsOut).
func layBuildsOutSoon(ctx context.Context, inv *inventory.Inventory) {
failed := false
for {
err := inv.LayBuildsOut(ctx, func(format string, args ...any) { fmt.Printf("chain: "+format+"\n", args...) })
o := conditions.Observation{Scope: conditions.ScopeBuild, ID: kindBuildsNotLaidOut, Token: kindBuildsNotLaidOut,
Kind: kindBuildsNotLaidOut, Severity: conditions.Urgent, Resolver: conditions.ResolverOperator,
Source: "chain", Summary: "the builds are not laid out under their modules, so none is read: "}
if err != nil {
fmt.Printf("chain: the builds are not laid out under their modules, and none is read until they are: %v\n", err)
if keeper := conditionsFrom; keeper != nil && !failed {
o.Summary += err.Error()
if _, oerr := keeper.Observe(ctx, o); oerr == nil {
failed = true
}
}
} else {
if keeper := conditionsFrom; keeper != nil && failed {
_, _ = keeper.Clear(ctx, o.Key(), "laid out")
}
return
}
select {
case <-ctx.Done():
return
case <-time.After(chainMoveRetry):
}
}
}
cmd/mesh-controller/chain_integration_test.go
-245
View File
@@ -1,245 +0,0 @@
//go:build integration
package main
import (
"context"
"fmt"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
"github.com/novox/mesh-controller/internal/link"
)
// **Acceptance 10 of design 47: one read of an assignment names the commit, the build and the rollout that put it there,
// and one read of the rollout lists the assignments it changed** (novox/hq ADR 0408, issue 1095). It enters by the live
// trigger — a merge heard, its batch cut into a plan when the window closed, its build answered through the build seat's
// path, the plan advanced — and writes no key and no row by hand. Only the machine's end of the send is the test's: the send
// is recorded as the controller records one it made, and the gate's facts are the store's.
func TestOneReadOfAnAssignmentNamesItsCommitBuildAndRolloutAndOneReadOfTheRolloutListsItsAssignments(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksRecorded(t)
ctx := t.Context()
inv := open.inventory
if _, err := assign(ctx, open, "anchor", "notes"); err != nil {
t.Fatal(err)
}
var sent [][]string
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
sent = append(sent, append([]string(nil), names...))
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
for _, node := range names {
digest := fmt.Sprintf("d-%s-%d", node, len(sent))
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), digest,
map[string]string{"notes": current["notes"].Commit}); err != nil {
return nil, err
}
}
return names, nil
}
t.Cleanup(func() { sendRollout = wasSend })
wasGather := gatherGateFacts
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
f := gateFacts{now: time.Now(), reports: map[string]inventory.Reported{}, engines: map[string]string{},
rolledBack: map[string][]lease.Rollback{}, served: map[string]served{}}
reports, err := open.inventory.LastReports(ctx)
if err != nil {
return f, err
}
for _, r := range reports {
f.reports[r.Node] = r
}
return f, nil
}
t.Cleanup(func() { gatherGateFacts = wasGather })
// A merge of notes is heard, and its window closes: the batch is cut into the walk.
hear(t, open, catalogueMerge("c-notes", "notes", t0), t0.Add(time.Second))
cutAt(t, open, t0.Add(time.Second+90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walk(s) after the cut", len(ws))
}
rollout := ws[0].ID
// The walk asks its first tier (at the cut, or at its next step); the build seat answers, and the controller takes the
// outcome in as it does.
asking, err := inv.PlanByID(ctx, rollout)
if err != nil {
t.Fatal(err)
}
for try := 0; try < 3 && (asking.Modules["notes"] == nil || asking.Modules["notes"].Build == ""); try++ {
advancePlans(ctx, open)
if asking, err = inv.PlanByID(ctx, rollout); err != nil {
t.Fatal(err)
}
}
buildID := asking.Modules["notes"].Build
if buildID == "" || len(*asked) != 1 || !strings.HasSuffix((*asked)[0], "#"+buildID) {
t.Fatalf("the walk asked %v, and notes' build is %q", *asked, buildID)
}
digest := "sha256:" + strings.Repeat("a", 64)
result := outcome(buildID, "modules/notes", "c-notes", "notes", "")
result.ReuseKey = "rk-notes"
result.Made = []link.MadeArtifact{{Name: "notes", Kind: "image", Reference: "artifact-store://notes/notes@" + digest}}
if err := (builds{inv: inv, open: open}).Built(ctx, result); err != nil {
t.Fatal(err)
}
// The walk sends the tier to its first machine.
for try := 0; try < 3 && len(sent) == 0; try++ {
advancePlans(ctx, open)
}
if !slices.Equal(slices.Concat(sent...), []string{"anchor"}) {
t.Fatalf("sent %v", sent)
}
// One read of the assignment.
on, err := inv.AssignmentsOn(ctx, "anchor")
a, ok := assignmentOf(on, "notes")
if err != nil || !ok {
t.Fatalf("the assignments of anchor: %+v %v", on, err)
}
// Beside it a module pulled in by the assignment, which no send of this walk put anywhere.
if wireguard, pulled := assignmentOf(on, "mesh-wireguard"); pulled && (wireguard.Build != "" || wireguard.Rollout != "") {
t.Fatalf("a module this rollout did not send names a build or a rollout: %+v", wireguard)
}
if a.Module != "notes" || a.Commit != "c-notes" || a.Build != buildID || a.Rollout != rollout || a.Artifact != digest ||
a.Since.IsZero() || a.Assigned.IsZero() || !a.Stands() {
t.Fatalf("the assignment names %+v, not commit c-notes, build %s, artifact %s, rollout %s", a, buildID, digest, rollout)
}
// One read of the rollout.
p, err := inv.PlanByID(ctx, rollout)
if err != nil {
t.Fatal(err)
}
wantChange := chain.Change{Node: "anchor", Module: "notes", After: chain.Ref{Build: buildID, Commit: "c-notes"}}
if !slices.Contains(p.Changed, wantChange) || !slices.Contains(p.Builds, buildID) || len(p.Steps) != 1 ||
p.Steps[0].Node != "anchor" || p.Steps[0].Sent == nil {
t.Fatalf("the rollout lists changed %+v, builds %v, steps %+v; want %+v", p.Changed, p.Builds, p.Steps, wantChange)
}
// And one read of the build names the same commit, with what it published and its reuse key.
built, found, err := inv.BuildByID(ctx, buildID)
if err != nil || !found || built.Commit != "c-notes" || built.Module != "notes" {
t.Fatalf("the build: %+v %v %v", built, found, err)
}
lately, err := inv.Builds(ctx, "notes", 1)
if err != nil || len(lately) != 1 || lately[0].ReuseKey != "rk-notes" || lately[0].Published == nil ||
lately[0].Published.Artifact != digest {
t.Fatalf("what the build published: %+v %v", lately, err)
}
// The verbs say the links.
if lines := changedLines(p); len(lines) != 2 || !strings.Contains(lines[1], "anchor/notes") ||
!strings.Contains(lines[1], buildID) {
t.Fatalf("`plans` says %v", lines)
}
if lines := runLines(on); len(lines) != 2 || !strings.Contains(lines[1], buildID) || !strings.Contains(lines[1], rollout) {
t.Fatalf("`node show` says %v", lines)
}
}
// A machine sent again what it runs says nothing new, and a push by hand with no rollout fills what is missing and blanks
// nothing (issue 1095).
func TestASendOfWhatAMachineRunsAlreadyChangesNoLinkOfItsAssignment(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: "notes", Version: "1"})
if _, err := assign(ctx, open, "anchor", "notes"); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-1", Module: "notes", Commit: "c1", Repository: "novox/notes"}); err != nil {
t.Fatal(err)
}
node := nodeID(t, open, "anchor")
inRollout := inventory.WithRollout(ctx, "plan-1", map[string]inventory.RolloutModule{"notes": {Commit: "c1", Build: "build-1"}})
if err := inv.RecordSent(inRollout, node, "d1", map[string]string{"notes": "c1"}); err != nil {
t.Fatal(err)
}
// Sent again by hand, with no rollout: the assignment still names the rollout that put the build there.
if err := inv.RecordSent(ctx, node, "d2", map[string]string{"notes": "c1"}); err != nil {
t.Fatal(err)
}
on, _ := inv.AssignmentsOn(ctx, "anchor")
if a, _ := assignmentOf(on, "notes"); a.Rollout != "plan-1" || a.Build != "build-1" || a.Commit != "c1" {
t.Fatalf("the assignment names %+v", on)
}
// A declaration sent by hand with no builds known makes what the machine runs unknown: a time on the key, no commit.
if err := inv.RecordSent(ctx, node, "d3", nil); err != nil {
t.Fatal(err)
}
if on, _ = inv.AssignmentsOn(ctx, "anchor"); runOf(on, "notes").Commit != "" || runOf(on, "notes").SentUnknown.IsZero() {
t.Fatalf("a send of unknown builds left %+v", on)
}
// A module sent a commit no build is kept of is named by its commit alone.
if err := inv.RecordSent(ctx, node, "d4", map[string]string{"notes": "c2"}); err != nil {
t.Fatal(err)
}
if on, _ = inv.AssignmentsOn(ctx, "anchor"); runOf(on, "notes").Commit != "c2" || runOf(on, "notes").Build != "" ||
runOf(on, "notes").Rollout != "" {
t.Fatalf("the assignment after a push of c2 by hand names %+v", on)
}
}
// assignmentOf is one module's assignment among those of a machine: assigning a module assigns what it requires beside it.
func assignmentOf(on []chain.Assignment, module string) (chain.Assignment, bool) {
for _, a := range on {
if a.Module == module {
return a, true
}
}
return chain.Assignment{}, false
}
func runOf(on []chain.Assignment, module string) chain.Assignment {
a, _ := assignmentOf(on, module)
return a
}
// **An assignment waiting for its build is one key of the assignments bucket** (ADR 0261, ADR 0408, issue 1094): entered
// through the `assign` verb for a module whose build is in flight, one read of `<node>.<module>` says state pending with the
// build it waits for, the build's outcome through the build seat's path makes the key stand with the wait carried as applied,
// and the table of pending assignments is read by nobody. No key is written by hand.
func TestAnAssignmentWaitingForItsBuildIsOneKeyOfTheAssignmentsBucket(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
asksWithPaths(t)
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatalf("an assignment while its build runs was refused: %v", err)
}
waiting, found, err := inv.AssignmentValue(ctx, "laptop", "sensors")
if err != nil || !found || waiting.State != chain.StatePending || waiting.Stands() || !waiting.Assigned.IsZero() ||
waiting.Pending == nil || waiting.Pending.State != chain.PendingWaiting || waiting.Pending.Build != "b-modules/sensors" ||
!strings.HasPrefix(waiting.Pending.ID, "p-") {
t.Fatalf("one read of the key says %+v (found %v, %v)", waiting, found, err)
}
if got := assignedTo(t, open, "laptop"); slices.Contains(got, "sensors") {
t.Fatalf("a module only waiting is assigned: %v", got)
}
if err := (builds{inv: inv, open: open}).Built(ctx, outcome("b-modules/sensors", "modules/sensors", "3da80a4b00aa", "sensors", "")); err != nil {
t.Fatal(err)
}
made, found, err := inv.AssignmentValue(ctx, "laptop", "sensors")
if err != nil || !found || !made.Stands() || made.State != "" || made.Assigned.IsZero() || made.Pending == nil ||
made.Pending.State != chain.PendingApplied || made.Pending.ID != waiting.Pending.ID {
t.Fatalf("one read of the key after the build says %+v (found %v, %v)", made, found, err)
}
if !slices.Contains(assignedTo(t, open, "laptop"), "sensors") {
t.Fatal("the build registered sensors and it was not assigned")
}
}
cmd/mesh-controller/chain_links.go
-121
View File
@@ -1,121 +0,0 @@
package main
import (
"context"
"fmt"
"slices"
"time"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/inventory"
)
// The links a rollout makes (novox/hq ADR 0408 §1, design 47 acceptance 10, issue 1095): the send of a plan's tier puts a
// build on a machine; the assignment of the module there names the build, its commit and the rollout (written by the record of
// the send, inventory.RecordSent, from the context a send is made in), and the rollout lists the assignments it changed with the
// build before and after.
// rolloutModules is what a plan sends of each module: the commit and the build of it.
func rolloutModules(p *inventory.Plan) map[string]inventory.RolloutModule {
out := make(map[string]inventory.RolloutModule, len(p.Modules))
for m, s := range p.Modules {
if s != nil && s.Commit != "" {
out[m] = inventory.RolloutModule{Commit: s.Commit, Build: s.Build}
}
}
return out
}
// sendingAs is the context a plan's send is made in: what is recorded of the send names this rollout.
func sendingAs(ctx context.Context, p *inventory.Plan) context.Context {
return inventory.WithRollout(ctx, p.ID, rolloutModules(p))
}
// ranBefore is what each module on a machine ran before a send, as the chain names it: its assignment's build and commit,
// else the commit the machine was last sent for it. Read before the send, which writes the new ones.
func ranBefore(ctx context.Context, inv *inventory.Inventory, node string) map[string]chain.Ref {
was := map[string]chain.Ref{}
if sent, known, err := inv.SentBuilds(ctx, node); err == nil && known {
for m, commit := range sent {
was[m] = chain.Ref{Commit: commit}
}
}
if ran, err := inv.AssignmentRuns(ctx, node); err == nil {
for m, r := range ran {
was[m] = r
}
}
return was
}
// noteChanges writes into a plan that a send to a machine changed the assignments of the modules it carried: the build and
// commit before and after, the builds the rollout carried, and the step taken. A change noted already for the machine and the
// module is replaced, so a send made again says it once.
func noteChanges(p *inventory.Plan, node string, modules []string, was map[string]chain.Ref, at time.Time) {
for _, m := range modules {
s := p.Modules[m]
if s == nil {
continue
}
change := chain.Change{Node: node, Module: m, Before: was[m], After: chain.Ref{Build: s.Build, Commit: s.Commit}}
if at := slices.IndexFunc(p.Changed, func(c chain.Change) bool { return c.Node == node && c.Module == m }); at >= 0 {
change.Before = p.Changed[at].Before
p.Changed[at] = change
} else {
p.Changed = append(p.Changed, change)
}
if s.Build != "" && !slices.Contains(p.Builds, s.Build) {
p.Builds = append(p.Builds, s.Build)
}
}
if !slices.ContainsFunc(p.Steps, func(s chain.Step) bool { return s.Tier == p.Tier && s.Node == node }) {
sent := at.UTC()
p.Steps = append(p.Steps, chain.Step{Tier: p.Tier, Node: node, Sent: &sent})
}
}
// refSaid is a build and the commit it was made from as a person reads them; "nothing" for a module that ran none.
func refSaid(r chain.Ref) string {
switch {
case r.Build == "" && r.Commit == "":
return "nothing"
case r.Build == "":
return "commit " + short(r.Commit)
}
return r.Build + " (" + short(r.Commit) + ")"
}
// changedLines say the assignments a rollout changed: the machine, the module, and the build before and after.
func changedLines(p inventory.Plan) []string {
if len(p.Changed) == 0 {
return nil
}
out := []string{" changed:"}
for _, c := range p.Changed {
out = append(out, fmt.Sprintf(" %s/%s %s → %s", c.Node, c.Module, refSaid(c.Before), refSaid(c.After)))
}
return out
}
// runLines say what each module assigned to a machine runs there, as its assignment names it: the build, the commit, the
// rollout that put it there and since when.
func runLines(assigned []chain.Assignment) []string {
var out []string
for _, a := range assigned {
if a.Build == "" && a.Commit == "" {
continue
}
line := fmt.Sprintf(" %-22s %s", a.Module, refSaid(chain.Ref{Build: a.Build, Commit: a.Commit}))
if a.Rollout != "" {
line += ", sent by " + a.Rollout
}
if !a.Since.IsZero() {
line += " at " + a.Since.Local().Format("2006-01-02 15:04")
}
out = append(out, line)
}
if len(out) == 0 {
return nil
}
return append([]string{" what they run, as the chain names it:"}, out...)
}
cmd/mesh-controller/chain_links_test.go
-69
View File
@@ -1,69 +0,0 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/inventory"
)
// **A rollout lists the assignments it changed, each with the build before and after** (ADR 0408 §1, issue 1095): one
// change for a machine and a module however often the send is noted, the build before kept from the first, the builds the
// rollout carried once each, and one step for a machine and a tier.
func TestARolloutListsTheAssignmentsItChangedWithTheBuildBeforeAndAfter(t *testing.T) {
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
p := inventory.Plan{ID: "plan-1", Tier: 1, Modules: map[string]*inventory.PlanModule{
"notes": {Commit: "c2", Build: "b2"}, "wiki": {Commit: "c2w", Build: "b2w"}, "nothing": nil}}
was := map[string]chain.Ref{"notes": {Build: "b1", Commit: "c1"}, "wiki": {Commit: "c1w"}}
noteChanges(&p, "anchor", []string{"notes", "wiki", "nothing"}, was, at)
noteChanges(&p, "laptop", []string{"notes"}, nil, at)
// The send noted again, after what was sent had become what the machine runs: the first before stands.
noteChanges(&p, "anchor", []string{"notes"}, map[string]chain.Ref{"notes": {Build: "b2", Commit: "c2"}}, at.Add(time.Minute))
want := []chain.Change{
{Node: "anchor", Module: "notes", Before: chain.Ref{Build: "b1", Commit: "c1"}, After: chain.Ref{Build: "b2", Commit: "c2"}},
{Node: "anchor", Module: "wiki", Before: chain.Ref{Commit: "c1w"}, After: chain.Ref{Build: "b2w", Commit: "c2w"}},
{Node: "laptop", Module: "notes", After: chain.Ref{Build: "b2", Commit: "c2"}},
}
if len(p.Changed) != len(want) {
t.Fatalf("changed %+v", p.Changed)
}
for k := range want {
if p.Changed[k] != want[k] {
t.Errorf("change %d is %+v, want %+v", k, p.Changed[k], want[k])
}
}
if len(p.Builds) != 2 || p.Builds[0] != "b2" || p.Builds[1] != "b2w" {
t.Errorf("builds %v", p.Builds)
}
if len(p.Steps) != 2 || p.Steps[0].Node != "anchor" || p.Steps[0].Tier != 1 || p.Steps[0].Sent == nil || p.Steps[1].Node != "laptop" {
t.Errorf("steps %+v", p.Steps)
}
lines := changedLines(p)
if len(lines) != 4 || !strings.Contains(lines[1], "anchor/notes b1 (c1) → b2 (c2)") ||
!strings.Contains(lines[3], "laptop/notes nothing → b2 (c2)") {
t.Errorf("changed lines %q", lines)
}
if got := changedLines(inventory.Plan{}); got != nil {
t.Errorf("a plan that changed nothing says %q", got)
}
}
func TestWhatAMachineRunsIsSaidAsItsAssignmentNamesIt(t *testing.T) {
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
lines := runLines([]chain.Assignment{
{Node: "anchor", Module: "notes", Build: "b2", Commit: "c2c2c2c2c2c2", Rollout: "plan-1", Since: at},
{Node: "anchor", Module: "web", Commit: "c3"},
{Node: "anchor", Module: "never-sent"},
})
if len(lines) != 3 || !strings.Contains(lines[1], "notes") || !strings.Contains(lines[1], "b2 (c2c2c2c2)") ||
!strings.Contains(lines[1], "sent by plan-1") || !strings.Contains(lines[2], "web") || !strings.Contains(lines[2], "commit c3") {
t.Fatalf("lines %q", lines)
}
if runLines([]chain.Assignment{{Node: "a", Module: "m"}}) != nil {
t.Fatal("a module never sent is said to run something")
}
}
cmd/mesh-controller/chain_no_table_integration_test.go
-85
View File
@@ -1,85 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// **Acceptance 12 of design 47: no table in the controller's store holds builds, rollouts or assignments** (novox/hq ADR
// 0408, issue 1111). A delivery is driven through the live trigger — a merge heard, its batch cut into a plan, the plan's
// build asked and answered, the assignment made — until a build, a rollout and an assignment exist; they are read from the
// chain, and then the store's own catalog is read: none of the tables of the chain is in it, nor the function that raised the generation from the assignment table; the table that stays, stays.
func TestNoTableOfTheControllersStoreHoldsBuildsRolloutsOrAssignments(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksRecorded(t)
ctx := t.Context()
inv := open.inventory
if _, err := assign(ctx, open, "anchor", "notes"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("c-notes", "notes", t0), t0.Add(time.Second))
cutAt(t, open, t0.Add(91*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walk(s) after the cut", len(ws))
}
rollout := ws[0].ID
asking, err := inv.PlanByID(ctx, rollout)
if err != nil {
t.Fatal(err)
}
for try := 0; try < 3 && (asking.Modules["notes"] == nil || asking.Modules["notes"].Build == ""); try++ {
advancePlans(ctx, open)
if asking, err = inv.PlanByID(ctx, rollout); err != nil {
t.Fatal(err)
}
}
buildID := asking.Modules["notes"].Build
if buildID == "" || len(*asked) != 1 {
t.Fatalf("the walk asked %v, and notes' build is %q", *asked, buildID)
}
digest := "sha256:" + strings.Repeat("b", 64)
result := outcome(buildID, "modules/notes", "c-notes", "notes", "")
result.Made = []link.MadeArtifact{{Name: "notes", Kind: "image", Reference: "artifact-store://notes/notes@" + digest}}
if err := (builds{inv: inv, open: open}).Built(ctx, result); err != nil {
t.Fatal(err)
}
// What the live path made is in the chain.
if built, found, err := inv.BuildByID(ctx, buildID); err != nil || !found || built.Commit != "c-notes" {
t.Fatalf("the build is not read from the chain: %+v %v %v", built, found, err)
}
if p, err := inv.PlanByID(ctx, rollout); err != nil || p.ID != rollout {
t.Fatalf("the rollout is not read from the chain: %+v %v", p, err)
}
if on, err := inv.AssignmentsOn(ctx, "anchor"); err != nil {
t.Fatal(err)
} else if _, ok := assignmentOf(on, "notes"); !ok {
t.Fatalf("the assignment is not read from the chain: %+v", on)
}
// And the catalog of the store holds none of them.
have := map[string]bool{}
for _, row := range inv.QueryForTest(t, `select table_name::text from information_schema.tables where table_schema = 'public'`) {
have[strings.Trim(row[0], `"`)] = true
}
for _, table := range []string{"build", "build_request", "release_plan", "batched_merge", "assignment", "pending_assignment", "chain_moved"} {
if have[table] {
t.Errorf("the table %s is in the controller's store", table)
}
}
for _, table := range []string{"build_gate", "assignment_generation"} {
if !have[table] {
t.Errorf("the table %s, which stays, is gone", table)
}
}
if rows := inv.QueryForTest(t, `select count(*) from pg_proc where proname = 'raise_assignment_generation'`); rows[0][0] != "0" {
t.Errorf("the function raise_assignment_generation is in the store: %v", rows)
}
}
cmd/mesh-controller/changeplan.go
+1 -6
View File
@@ -7,7 +7,7 @@ import (
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -73,11 +73,6 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
"(issue 280)", busUpgradeNeeded, name, busUpgradeVerb, strings.Join(e.On, ", ")))
case name == "nats" || catalogue.ProvidesBus(e.Manifest):
p.Steps = append(p.Steps, "a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send")
case waits && len(e.On) > 0 && (u.From == catalogue.FromModule || u.From == catalogue.FromData):
// Judged per machine by the walk, against the build each was last sent (novox/hq ADR 0377).
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s) — unless on a machine "+
"its move recreates none of its containers and moves none of its data, which the walk judges and then sends",
name, orNone(u.From)))
case waits && len(e.On) > 0:
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s)", name,
orNone(u.From)))
cmd/mesh-controller/changeplan_test.go
+1 -1
View File
@@ -4,7 +4,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
cmd/mesh-controller/check.go
+10 -62
View File
@@ -10,8 +10,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/catalogue"
)
// moduleCheck judges manifests where they are written, with no mesh (novox/hq ADR 0037, issue 148).
@@ -48,24 +47,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
shelf := catalogue.Shelf{}
faulted := map[string]bool{}
failed := 0
// The list of predefined keys at the root of the repository above each manifest (novox/hq ADR 0348 §7), read
// once per repository: a manifest naming a key is judged against the list its own tree holds.
lists := map[string]catalogue.PredefinedList{}
listOf := func(path string) (catalogue.PredefinedList, error) {
root := catalogue.RepositoryRoot(filepath.Dir(path))
if root == "" {
return nil, nil
}
if list, read := lists[root]; read {
return list, nil
}
list, err := catalogue.ReadPredefined(root)
if err != nil {
return nil, err
}
lists[root] = list
return list, nil
}
for _, path := range paths {
raw, err := os.ReadFile(path)
if err != nil {
@@ -73,18 +54,7 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed++
continue
}
list, err := listOf(path)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
continue
}
m, err := catalogue.ParseManifestWith(raw, list)
if err == nil {
// Each template's text read from the tree beside the manifest, and judged as inline content is
// (novox/hq ADR 0349 §2): a missing template is refused by name.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
@@ -105,30 +75,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed += len(wrong)
faulted[m.Module] = true
}
// A module that declares a setting names the keys of its mergeable files (novox/hq ADR 0326 §3,
// issue 574): judged where a definition is written, as at registration, never where a stored
// build is read again.
if wrong := catalogue.WrittenSettingsProblems(m); len(wrong) > 0 {
for _, p := range wrong {
fmt.Fprintf(out, "%s: %s\n", path, p)
}
failed += len(wrong)
faulted[m.Module] = true
}
// A module lives whole in one repository (novox/hq ADR 0420 §1, issue 1019): when the check is told the
// repository it runs for (MESH_CHECK_REPOSITORY, set by the build's check), an artifact building from
// another is refused, as the merge gate refuses it. A run by hand names no repository and judges none.
if repo := os.Getenv("MESH_CHECK_REPOSITORY"); repo != "" && m.Build != nil {
for _, a := range m.Build.Artifacts {
if a.Context != nil && a.Context.Repository != "" && !sameRepository(a.Context.Repository,
link.SourceMoved{Owner: ownerOf(repo), Repo: repoOf(repo)}) {
fmt.Fprintf(out, "%s: %s builds artifact %s from %s: a module lives whole in one repository "+
"(novox/hq ADR 0420 §1)\n", path, m.Module, a.Name, a.Context.Repository)
failed++
faulted[m.Module] = true
}
}
}
if named := catalogue.InstallationProblems(m); len(named) > 0 {
for _, p := range named {
fmt.Fprintf(out, "%s: %s\n", path, p)
@@ -206,6 +152,14 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
}
}
// **A module that uses several libraries is warned** (novox/hq ADR 0328 §4): an image that drops added groups when
// it changes user reaches only the first. Printed, never counted.
for _, name := range names {
for _, w := range catalogue.LibraryWarnings(shelf[name]) {
fmt.Fprintf(out, "warning: %s\n", w)
}
}
// **A file that asks for a setting and does not say whether it is trusted counts as trusted** (novox/hq issue
// 339): listed and counted, never refused, so an author can opt out a file nothing trusts.
unsaid := 0
@@ -259,12 +213,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
checks = append(checks, fmt.Sprintf("%v by %s", r["id"], catalogue.HealthWords(h)))
}
}
// And its own runtime's, where it runs as its own account (novox/hq ADR 0366).
if own, has := catalogue.OwnRuntimeHealth(m); has {
if h, _, _ := catalogue.ReadHealth(own); h.Kind != "" {
checks = append(checks, fmt.Sprintf("%v by %s", own["id"], catalogue.HealthWords(h)))
}
}
if len(checks) > 0 {
fmt.Fprintf(out, ", ready: %s", strings.Join(checks, "; "))
}
cmd/mesh-controller/check_builds.go
-87
View File
@@ -1,87 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"strconv"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
)
// A pull request's check builds go when it closes (novox/hq ADR 0350 §5): the build seat keeps them in the artifact
// store's scratch namespace for the walk of its merge to take. **Let go of where the moment is heard**: the
// controller ends the walk answering a merge — after that walk took what it reuses — and the delivery's owner, which
// hears the forge say a pull request closed unmerged, asks the `forget-checks` verb; both let go through the store,
// as the controller's own collection does. The build seat's holders let go of anything a week old on their own,
// so a word lost here leaves a week's builds, never more.
// checkBuildsIn is the scratch namespace as the controller reaches the artifact store.
func checkBuildsIn(ctx context.Context, inv *inventory.Inventory) (builder.Scratch, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, errors.New("the artifact store is not on the mesh's network")
}
return builder.Registry{Address: address}, nil
}
// forgetChecks lets go of a pull request's check builds and answers what it did.
var forgetChecks = func(ctx context.Context, inv *inventory.Inventory, pull builder.PullRef) (string, error) {
scratch, err := checkBuildsIn(ctx, inv)
if err != nil {
return "", err
}
n, err := scratch.Forget(ctx, pull)
if err != nil {
return "", fmt.Errorf("%s: %d check(s)' builds let go of, and not all: %w", pull, n, err)
}
return fmt.Sprintf("%s: %d check(s)' builds let go of", pull, n), nil
}
// forgetChecksOfWalk lets go, on the save that ends a walk, of the check builds of every pull request whose merge
// it answers: the walk took what it reuses. Said, never fatal.
func forgetChecksOfWalk(ctx context.Context, inv *inventory.Inventory, p inventory.Plan) {
if p.WalkSaid != inventory.WalkEnded || p.Delivery == nil {
return
}
for _, pull := range reuseAskOf(&p).Pulls {
said, err := forgetChecks(ctx, inv, builder.PullRef{Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number})
if err != nil {
fmt.Fprintf(os.Stderr, "%s: its check builds stay until the week's sweep: %v\n", p.ID, err)
continue
}
fmt.Printf("%s: %s\n", p.ID, said)
}
}
// forgetChecksCommand is `forget-checks <owner> <repo> <number>`: the verb the delivery's owner asks when the forge
// says a pull request closed unmerged (novox/hq ADR 0350 §5).
func forgetChecksCommand(ctx context.Context, args []string) error {
if len(args) != 3 {
return errors.New("forget-checks <owner> <repo> <number>")
}
number, err := strconv.Atoi(args[2])
if err != nil || number <= 0 {
return fmt.Errorf("%q is not a pull request's number", args[2])
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
said, err := forgetChecks(ctx, open.inventory, builder.PullRef{Owner: args[0], Repo: args[1], Number: number})
if err != nil {
return err
}
fmt.Println(said)
return nil
}
cmd/mesh-controller/check_builds_integration_test.go
-102
View File
@@ -1,102 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func TestOnlyAWalksAskTakesAChecksBuildAndADryRunNever(t *testing.T) {
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}, {Repository: "http://forge.invalid/novox/web.git", Number: 3},
{Repository: "novox/shop", Number: 7}, {Repository: "novox/shop"}}}}
asked := reuseAskOf(p)
if want := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}, {Owner: "novox", Repo: "web", Number: 3}}; !reflect.DeepEqual(asked.Pulls, want) {
t.Fatalf("the walk names %+v", asked.Pulls)
}
walk := withReuse(t.Context(), asked)
if reuseAsked(walk, false) != asked {
t.Fatal("a walk's ask does not carry its leave")
}
if reuseAsked(walk, true) != nil {
t.Fatal("a dry run carries a leave to take a check's build")
}
if reuseAsked(t.Context(), false) != nil {
t.Fatal("an ask that is not a walk's carries a leave")
}
}
func TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
gateEvery = 0
digest := "sha256:" + strings.Repeat("c", 64)
reused := func(id, reference string) link.BuildResult {
manifest := mustJSON(t, catalogue.Manifest{Module: "app", Version: "c3", Resources: []map[string]any{
{"id": "svc", "type": "container", "name": "app", "image": reference}}})
return link.BuildResult{ID: id, Repository: "novox/mesh-catalog", Path: "modules/app", Ref: "main", On: "anchor",
Commit: "c3", Manifest: manifest, Trunk: "main", OnTrunk: true, Branches: []string{"main"},
Made: []link.MadeArtifact{{Name: "server", Kind: "image", Reference: reference}},
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"},
Reused: "reused from the check of novox/mesh-catalog#9 at head abcdefabcdef (merged into trunk 123456123456): server " + digest,
ReusedFrom: &link.ReusedFrom{Pull: link.PullRef{Owner: "novox", Repo: "mesh-catalog", Number: 9},
Head: "abcdefabcdef", Trunk: "123456123456", Key: "r1:" + strings.Repeat("1", 64)}}
}
if _, _, err := takeIn(ctx, inv, reused(link.NewBuildID(time.Now()), "store.internal:5000/scratch/app/server@"+digest)); err == nil ||
!strings.Contains(err.Error(), "scratch") {
t.Fatalf("a build in the scratch namespace was registered: %v", err)
}
id := link.NewBuildID(time.Now().Add(time.Second))
result := reused(id, "store.internal:5000/app/server@"+digest)
if _, _, err := takeIn(ctx, inv, result); err != nil {
t.Fatal(err)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil || current["app"].Commit != "c3" {
t.Fatalf("the reused build is not what app is registered at: %+v %v", current["app"], err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(shelf["app"].Resources)
if !strings.Contains(string(raw), "app/server@"+digest) {
t.Fatalf("app is registered as %s, not by the check's digest in its own namespace", raw)
}
// The walk asked it: the walk keeps what its build did, and its gate judges the registered build.
asked := time.Now().UTC()
p := g.plan(t)
p.Commit = "c3"
p.Modules["app"] = &inventory.PlanModule{State: "asked", AskedAt: &asked, Build: id}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
planBuiltSaying(ctx, g.open, "app", "c3", "", asked, id, result.MadeWords())
p = g.plan(t)
if m := p.Modules["app"]; m.State != "built" || !strings.HasPrefix(m.Made, "reused from the check of novox/mesh-catalog#9") {
t.Fatalf("the walk keeps %+v", m)
}
for i := 0; i < 6; i++ {
advancePlans(ctx, g.open)
}
p = g.plan(t)
gate := p.Modules["app"].Gate
if gate == nil || gate.To != "c3" || gate.Verdict != inventory.GatePassed {
t.Fatalf("the gate judged %+v, not the reused build", gate)
}
if sent, _, _ := inv.SentBuilds(ctx, "anchor"); sent["app"] != "c3" {
t.Fatalf("the first machine was sent %v, not the reused build", sent)
}
}
cmd/mesh-controller/check_builds_test.go
-193
View File
@@ -1,193 +0,0 @@
package main
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// One build per change (novox/hq ADR 0350), as the controller asks it and takes it in.
// The check of a change is asked to build what a merge of it would move, in the walk's order: a module of the
// repository checked from the merge, one standing on it from its own repository as the walk builds it, and one
// that reads the repository through a context not at all. A change that moves no module builds nothing.
func TestACheckIsAskedToBuildWhatTheMergeWouldMove(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "app"}, Source: inventory.Source{Repository: "novox/shop", Seat: "git", Path: "modules/app"}},
{Manifest: catalogue.Manifest{Module: "web"}, Source: inventory.Source{Repository: "novox/web", Seat: "git", Ref: "main"}},
{Manifest: catalogue.Manifest{Module: "docs"}, Source: inventory.Source{Repository: "novox/docs", Seat: "git"}},
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Base: "main", Commit: "abc"}
scope := checkScope{Reach: mergeReach{Packaging: entries[2:], Plan: inventory.Plan{Tiers: [][]string{{"app", "docs"}, {"web"}}}}}
clone := func(s inventory.Source) (string, error) { return "http://forge.invalid/" + s.Repository + ".git", nil }
got := checkBuildsOf(p, scope, entries, clone)
want := []link.CheckBuild{
{Module: "app", Path: "modules/app", Merged: true, Tier: 0},
{Module: "docs", Tier: 0, Skip: "its build reads novox/shop through a context, which a check cannot hand the merge"},
{Module: "web", Repository: "http://forge.invalid/novox/web.git", Ref: "main", Tier: 1},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("the check is asked to build\n%+v\nwant\n%+v", got, want)
}
if got := checkBuildsOf(p, checkScope{}, entries, clone); got != nil {
t.Fatalf("a change that moves no module asks for builds: %+v", got)
}
}
// A walk's ask carries its leave to take a check's build, naming the pull requests of the merges it answers; a dry
// run never does, and neither does an ask that is not a walk's.
// A walk ended lets go of its merges' check builds — only once it ended, after it took what it reuses.
func TestAWalkEndedLetsGoOfItsMergesCheckBuilds(t *testing.T) {
var forgot []builder.PullRef
was := forgetChecks
forgetChecks = func(_ context.Context, _ *inventory.Inventory, pull builder.PullRef) (string, error) {
forgot = append(forgot, pull)
return "", nil
}
t.Cleanup(func() { forgetChecks = was })
p := inventory.Plan{ID: "plan-1", Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}}}}
p.WalkSaid = inventory.WalkStarted
forgetChecksOfWalk(t.Context(), nil, p)
p.WalkSaid = ""
forgetChecksOfWalk(t.Context(), nil, p)
if len(forgot) != 0 {
t.Fatalf("a walk not ended let go of %v, which it may still take", forgot)
}
p.WalkSaid = inventory.WalkEnded
forgetChecksOfWalk(t.Context(), nil, p)
if want := []builder.PullRef{{Owner: "novox", Repo: "shop", Number: 7}}; !reflect.DeepEqual(forgot, want) {
t.Fatalf("the walk ended let go of %v", forgot)
}
}
// A build in the scratch namespace is never registered: only the walk's copy in the module's own namespace is.
func TestABuildInTheScratchNamespaceIsNeverRegistered(t *testing.T) {
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: "store.internal:5000/app/server@sha256:aa"}}); got != "" {
t.Fatalf("a module's own build reads as a scratch build: %s", got)
}
scratch := "store.internal:5000/scratch/app/server@sha256:aa"
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: scratch}}); got != scratch {
t.Fatalf("a scratch build was not found: %q", got)
}
if got := inScratch([]link.MadeArtifact{{Name: "look", Reference: "http://store.internal:5000/v2/scratch/app/look/blobs/sha256:bb"}}); got == "" {
t.Fatal("a scratch archive was not found")
}
}
// The walk says per module what it did: "reused from …" or "built: <why>" — and a build seat from before says
// neither, which is said.
func TestAWalksBuildSaysWhetherItReusedOrBuilt(t *testing.T) {
if got := (link.BuildResult{Reused: "reused from the check of novox/shop#7"}).MadeWords(); got != "reused from the check of novox/shop#7" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{BuiltWhy: "no check built it"}).MadeWords(); got != "built: no check built it" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{}).MadeWords(); !strings.Contains(got, "predates ADR 0350") {
t.Fatalf("said %q", got)
}
}
// A walk's module the check's build stood in for is registered by the digest the check built — the copy in the
// module's namespace — and the walk keeps "reused from …" for it; its first-node gate judges that registered
// build as it judges one the walk built. A scratch reference is refused. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
// A walk's leave names the builds its own pull requests' verdicts made at the head each merge merged — never an
// earlier head's — read once per walk, and none when they cannot be read: the builder takes nothing a verdict does
// not name (novox/hq ADR 0350 §2, issue 703).
func TestAWalksLeaveNamesTheBuildsItsPullRequestsVerdictsMade(t *testing.T) {
was := checkedBuildsOf
forgetWalkVerdicts()
t.Cleanup(func() { checkedBuildsOf = was; forgetWalkVerdicts() })
var asked [][]link.PullRef
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
atHead := link.CheckedBuild{Pull: pull, Check: "build-2", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:2"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{atHead, earlier}, nil
}
pulls := []link.PullRef{pull}
heads := map[link.PullRef]string{pull: "bbbbbbbbbbbb0123456789"}
for range 3 {
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, heads); !reflect.DeepEqual(got.Builds,
[]link.CheckedBuild{atHead}) {
t.Fatalf("the leave names %+v, not only the merged head's build", got.Builds)
}
}
if !reflect.DeepEqual(asked, [][]link.PullRef{pulls}) {
t.Fatalf("the verdicts were read %d times for one walk: %v", len(asked), asked)
}
// A merge whose head was not said takes nothing.
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, nil); got.Builds != nil {
t.Fatalf("a merge of no said head names %+v", got.Builds)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := reuseWithVerdicts(t.Context(), "plan-2", &link.ReuseAsk{Pulls: pulls}, heads); got.Builds != nil {
t.Fatalf("verdicts that could not be read named %+v", got.Builds)
}
}
// A check is handed only the builds earlier checks of its own pull request at its own head made: another pull
// request's build, or an earlier head's, is never kept (novox/hq issue 703). None when they cannot be read.
func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
was := checkedBuildsOf
t.Cleanup(func() { checkedBuildsOf = was })
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
mine := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-0", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
another := link.CheckedBuild{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 8}, Check: "build-9",
Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
var asked [][]link.PullRef
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{mine, earlier, another}, nil
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Commit: "bbbbbbbbbbbb0123456789"}
builds := []link.CheckBuild{{Module: "app"}}
if got := namedForTheCheck(t.Context(), p, builds); !reflect.DeepEqual(got, []link.CheckedBuild{mine}) ||
!reflect.DeepEqual(asked, [][]link.PullRef{{pull}}) {
t.Fatalf("the check is handed %+v, read for %v", got, asked)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := namedForTheCheck(t.Context(), p, builds); got != nil {
t.Fatalf("verdicts that could not be read named %+v", got)
}
}
func forgetWalkVerdicts() {
walkVerdicts.Lock()
defer walkVerdicts.Unlock()
walkVerdicts.read = map[string]verdictsRead{}
}
// A walk's merged heads are the forge's word for each merge; a merge announced without its head, or a pull request
// merged at two heads in one walk, has none, and its walk takes no check's build (novox/hq issue 703).
func TestAWalksMergedHeadsAreTheForgesWord(t *testing.T) {
merge := func(number int, head string) inventory.PlanMerge {
return inventory.PlanMerge{Repository: "novox/shop", Number: number, Head: head}
}
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
merge(7, "aaaaaaaaaaaa1"), merge(8, ""), merge(9, "bbbbbbbbbbbb"), merge(9, "cccccccccccc")}}}
heads := mergedHeadsOf(p)
shop := func(n int) link.PullRef { return link.PullRef{Owner: "novox", Repo: "shop", Number: n} }
if heads[shop(7)] != "aaaaaaaaaaaa1" || heads[shop(8)] != "" || heads[shop(9)] != "" {
t.Fatalf("merged heads %v", heads)
}
}
cmd/mesh-controller/check_here.go
+1 -1
View File
@@ -11,8 +11,8 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
cmd/mesh-controller/check_pulls_test.go
-44
View File
@@ -1,44 +0,0 @@
package main
import (
"reflect"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// 1212/AC1 and AC2: the check request names the pulls bucket of each module claiming the delivery seat, in the order
// of the claimants' names.
func TestACheckNamesThePullsBucketsOfTheSeatsClaimants(t *testing.T) {
for name, c := range map[string]struct {
entries []inventory.Entry
want []string
}{
"only the gate": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), namedOnly("mesh-delivery")}, []string{"mesh-delivery-gate_pulls"}},
"only today's": {[]inventory.Entry{claimingTheSeat("mesh-delivery"), namedOnly("gitea")}, []string{"mesh-delivery_pulls"}},
"both, name order": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls", "mesh-delivery-gate_pulls"}},
"none": {[]inventory.Entry{namedOnly("gitea")}, nil},
} {
if got := pullsBucketsOf(c.entries); !reflect.DeepEqual(got, c.want) {
t.Errorf("%s: %v, want %v", name, got, c.want)
}
}
}
// 1212/AC1 and AC2: the check a request carries names those buckets (the wiring, not only the helper).
func TestTheCheckRequestCarriesThePullsBuckets(t *testing.T) {
for name, c := range map[string]struct {
entries []inventory.Entry
want []string
}{
"only the gate": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate")}, []string{"mesh-delivery-gate_pulls"}},
"only today's": {[]inventory.Entry{claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls"}},
"both, name order": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls", "mesh-delivery-gate_pulls"}},
} {
got := checkOf(link.PullUpdated{Owner: "novox", Repo: "a", Number: 7}, checkScope{}, c.entries, nil, nil, nil, nil)
if !reflect.DeepEqual(got.Pulls, c.want) || got.Owner != "novox" || got.Number != 7 {
t.Errorf("%s: %+v, want pulls %v", name, got, c.want)
}
}
}
cmd/mesh-controller/check_siblings.go
-165
View File
@@ -1,165 +0,0 @@
package main
import (
"context"
"fmt"
"regexp"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382, issue 867).
//
// A mesh-host change and a mesh-controller change that need each other could not both pass their own merge checks:
// each was read beside the other's repository at the commit the mesh runs, which holds neither change. So the
// controller asks mesh-delivery's `groups` once for every pull request's check it asks (§1), and names, in the ask,
// the group's other open pull requests into a base of the same name, in a repository the build seat reads beside the
// check — mesh-lab never — each with its whole head. The build seat reads those repositories at the sibling's merge (§2).
// A pull request for which mesh-delivery answered and named no sibling is asked exactly as before (§7).
// wholeCommit is a commit as the forge says a head: forty hexadecimal characters.
var wholeCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
// besideRepositories are the repositories (lower case, owner/repo) a check reads beside its own, by the directory each
// is found under — the core's, as besideOf clones them, and mesh-sdk beside the controller; never mesh-lab, which is
// always read at its trunk.
func besideRepositories(entries []inventory.Entry) map[string]string {
read := map[string]string{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
read[nameOf(e.Source.Repository)] = dir
if dir == "mesh-controller" && e.Source.Seat != "" {
read[nameOf(siblingOf(e.Source.Repository, "mesh-sdk"))] = "mesh-sdk"
}
}
delete(read, "")
return read
}
// nameOf is a recorded repository as owner/repo in lower case, empty when it names no owner.
func nameOf(repository string) string {
owner, repo := sourceOwner(repository), sourceName(repository)
if owner == "" || repo == "" {
return ""
}
return strings.ToLower(owner + "/" + repo)
}
// siblingsFor is the pull request's siblings, as mesh-delivery's groups say them (ADR 0382 §1): the other members of
// the group its head branch names that are open, unmerged and into a base of its own base's name, in a repository the
// check reads beside it (beside: the directories the ask already names). The second answer is why they could not be
// learned, empty when they could: the group's members carry no head or base — a mesh-delivery from before the rule.
func siblingsFor(p link.PullUpdated, entries []inventory.Entry, beside map[string]link.CheckedOut,
lines []groupsLine) ([]link.CheckSibling, string) {
own := strings.ToLower(p.Owner + "/" + p.Repo)
var group *groupsLine
if p.Head == "" {
// A recheck (mesh-delivery's `recheck` through `delivery check`) carries no head branch name: the group is the one
// whose members hold this very pull request, by repository and number.
learnable := false
for i := range lines {
for _, m := range lines[i].Details {
learnable = true
if p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
group = &lines[i]
}
}
}
if group == nil {
for _, l := range lines {
if len(l.Details) == 0 && len(l.Members) > 0 && !learnable {
return nil, "mesh-delivery named the groups' members without their pull requests (a mesh-delivery from before novox/hq ADR 0382): " +
"the group of a pull request announced without its head branch cannot be found"
}
}
return nil, "" // a lone pull request
}
} else {
for i := range lines {
if lines[i].ID == p.Head {
group = &lines[i]
}
}
if group == nil {
return nil, ""
}
}
if len(group.Details) == 0 && len(group.Members) > 0 {
return nil, "mesh-delivery named the group's members without their heads and bases (a mesh-delivery from before novox/hq ADR 0382)"
}
read := besideRepositories(entries)
var siblings []link.CheckSibling
for _, m := range group.Details {
repository := strings.ToLower(m.Repository)
if repository == own {
continue
}
if !m.Open {
continue
}
if m.Head == "" || m.Base == "" {
return nil, "mesh-delivery named " + m.ID + " without its head or base (a mesh-delivery from before novox/hq ADR 0382)"
}
dir, readBeside := read[repository]
if m.Base != p.Base || !readBeside || dir == "mesh-lab" {
continue
}
if _, there := beside[dir]; !there {
continue
}
head := strings.ToLower(m.Head)
if !wholeCommit.MatchString(head) {
return nil, fmt.Sprintf("mesh-delivery named %s with a head that is not a whole commit (%q)", m.ID, m.Head)
}
siblings = append(siblings, link.CheckSibling{Repository: m.Repository, Number: m.Number, Head: head, Base: m.Base, Dir: dir})
}
sort.Slice(siblings, func(i, j int) bool { return siblings[i].Dir < siblings[j].Dir })
return siblings, ""
}
// withSiblings asks mesh-delivery's `groups` once and says in the request the pull request's siblings, or why they
// could not be learned (ADR 0382 §1): a check never waits on mesh-delivery past groupAskWithin and never fails for it,
// so a fix to mesh-delivery is still checked without it.
func withSiblings(ctx context.Context, request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry) {
lines, err := groupsAsked(ctx, deliverySeatHeld(entries))
siblingsFrom(request, p, entries, lines, err)
}
// siblingsFrom is withSiblings from mesh-delivery's answer already had.
func siblingsFrom(request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry, lines []groupsLine, err error) {
if request.Check == nil {
return
}
if err != nil {
request.Check.SiblingsUnknown = fmt.Sprintf("mesh-delivery did not say the group's pull requests (%v): "+
"the check reads no siblings", err)
return
}
request.Check.Siblings, request.Check.SiblingsUnknown = siblingsFor(p, entries, request.Check.Beside, lines)
}
// composedMember says the pull request is a member of a delivery group whose gate is the composed check's (novox/hq
// ADR 0403 clauses 1 and 7): the group has two members or more and mesh-delivery says so on its line. The group is
// found as siblingsFor finds it, by its head branch name, or, for a recheck that carries none, by the pull request's
// repository and number — not by siblings, which name only what the check reads beside it.
func composedMember(p link.PullUpdated, lines []groupsLine) bool {
own := strings.ToLower(p.Owner + "/" + p.Repo)
for _, l := range lines {
found := p.Head != "" && l.ID == p.Head
for _, m := range l.Details {
if p.Head == "" && p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
found = true
}
}
if found {
return l.ComposedGate && max(len(l.Details), len(l.Members)) >= 2
}
}
return false
}
cmd/mesh-controller/check_siblings_test.go
-248
View File
@@ -1,248 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382 §1, §7, issue 867): the
// controller asks mesh-delivery's `groups` once for every check it asks, and names in the ask the group's other open
// pull requests into a base of the same name, in a repository the build seat reads beside it — mesh-lab never.
const (
headHost = "1111111111111111111111111111111111111111"
headController = "2222222222222222222222222222222222222222"
headSDK = "3333333333333333333333333333333333333333"
headLab = "4444444444444444444444444444444444444444"
headHq = "5555555555555555555555555555555555555555"
)
func theCoreOnTheGitSeat() []inventory.Entry {
onSeat := func(module, repository string) inventory.Entry {
e := fromRepo(module, repository, "")
e.Source.Seat = "git"
return e
}
nats := onSeat("nats", "novox/mesh-catalog")
nats.Source.Path = "modules/nats"
held := catalogue.Manifest{Module: "mesh-delivery", Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}}
return []inventory.Entry{onSeat("mesh-controller", "novox/mesh-controller"), onSeat("mesh-host", "novox/mesh-host"),
onSeat("node-tools", "novox/mesh-tools"), nats, {Manifest: held, On: []string{"anchor"}}}
}
func besideTheCore() map[string]link.CheckedOut {
return map[string]link.CheckedOut{"mesh-controller": {Repository: "c"}, "mesh-host": {Repository: "h"},
"mesh-tools": {Repository: "t"}, "mesh-catalog": {Repository: "k"}, "mesh-sdk": {Repository: "s"}, "mesh-lab": {Repository: "l"}}
}
func member(repository string, number int, head, base string, open bool) memberLine {
return memberLine{ID: repository + "@" + head[:12], Repository: repository, State: "checked", Number: number, Head: head,
Base: base, Open: open}
}
func group(id string, members ...memberLine) groupsLine {
line := groupsLine{ID: id, Details: members}
for _, m := range members {
line.Members = append(line.Members, m.ID+" ("+m.State+")")
}
return line
}
func TestASiblingIsTheGroupsOtherOpenPullRequestIntoABaseOfTheSameName(t *testing.T) {
entries := theCoreOnTheGitSeat()
pull := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
for _, c := range []struct {
what string
lines []groupsLine
want []string
unknown string
}{
{"the other member, open, same base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
[]string{"mesh-controller=novox/mesh-controller#346@" + headController + " onto main"}, ""},
{"a member into a base of another name is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "release", true))},
nil, ""},
{"a member not open (merged or closed) is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", false))},
nil, ""},
{"mesh-lab is never one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-lab", 5, headLab, "main", true))},
nil, ""},
{"a repository the check reads nothing of is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 9, headHq, "main", true))},
nil, ""},
{"mesh-sdk, read beside the controller, is one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-sdk", 7, headSDK, "main", true))},
[]string{"mesh-sdk=novox/mesh-sdk#7@" + headSDK + " onto main"}, ""},
{"another group is none of its business", []groupsLine{group("feat/y",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
nil, ""},
{"a group of its own alone has none", []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))},
nil, ""},
{"an older mesh-delivery names no heads", []groupsLine{{ID: "feat/x",
Members: []string{"novox/mesh-host@111111111111 (checked)", "novox/mesh-controller@222222222222 (checked)"}}},
nil, "without their heads and bases"},
{"a member without its base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "", true))},
nil, "without its head or base"},
{"a head that is no commit", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController[:12]+"zzzzzzzzzzzzzzzzzzzzzzzzzzzz", "main", true))},
nil, "not a whole commit"},
} {
got, unknown := siblingsFor(pull, entries, besideTheCore(), c.lines)
var said []string
for _, s := range got {
said = append(said, s.Dir+"="+s.Repository+"#"+itoa(s.Number)+"@"+s.Head+" onto "+s.Base)
}
if strings.Join(said, ";") != strings.Join(c.want, ";") {
t.Errorf("%s: siblings %v, wanted %v", c.what, said, c.want)
}
if (c.unknown == "") != (unknown == "") || !strings.Contains(unknown, c.unknown) {
t.Errorf("%s: could not learn %q, wanted %q", c.what, unknown, c.unknown)
}
}
// A pull request into a release branch has siblings into the release branch of the same name (ADR 0358 §1).
release := pull
release.Base = "feat/57"
lines := []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "feat/57", true),
member("novox/mesh-controller", 346, headController, "feat/57", true), member("novox/mesh-tools", 8, headLab, "main", true))}
if got, _ := siblingsFor(release, entries, besideTheCore(), lines); len(got) != 1 || got[0].Dir != "mesh-controller" {
t.Errorf("into a release branch: %+v", got)
}
// A recheck announces the pull request without its head branch: its group is the one holding it (repository, number).
pull.Head = ""
got, unknown := siblingsFor(pull, entries, besideTheCore(), lines)
if len(got) != 1 || got[0].Dir != "mesh-tools" || unknown != "" {
t.Errorf("a recheck of a group member: %+v %q", got, unknown)
}
// A pull request in no group, rechecked, is alone: no sibling, and nothing said.
pull.Number = 999
if got, unknown := siblingsFor(pull, entries, besideTheCore(), lines); got != nil || unknown != "" {
t.Errorf("a lone pull request: %+v %q", got, unknown)
}
if got, unknown := siblingsFor(pull, entries, besideTheCore(), nil); got != nil || unknown != "" {
t.Errorf("no groups at all: %+v %q", got, unknown)
}
// A mesh-delivery that names members without their pull requests cannot find the group of a recheck: it says so.
older := []groupsLine{{ID: "feat/x", Members: []string{"novox/mesh-host@111111111111 (checked)"}}}
pull.Number = 114
if got, unknown := siblingsFor(pull, entries, besideTheCore(), older); got != nil || unknown == "" {
t.Errorf("an older mesh-delivery: %+v %q", got, unknown)
}
}
func itoa(n int) string { b, _ := json.Marshal(n); return string(b) }
// Every check asks mesh-delivery's `groups` once; the ask names the sibling's repository, head and base.
func TestEveryCheckAsksMeshDeliveryOnceAndNamesItsSiblings(t *testing.T) {
entries := theCoreOnTheGitSeat()
asked := 0
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
asked++
if !held {
t.Error("the seat is held, yet it was asked as not held")
}
return []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true),
member("novox/mesh-controller", 346, headController, "main", true))}, nil
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if asked != 1 {
t.Fatalf("mesh-delivery was asked %d times for one check", asked)
}
want := link.CheckSibling{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", Dir: "mesh-controller"}
if got := request.Check.Siblings; len(got) != 1 || got[0] != want || request.Check.SiblingsUnknown != "" {
t.Fatalf("the ask names siblings %+v, unknown %q", got, request.Check.SiblingsUnknown)
}
body, _ := json.Marshal(request.Check)
if !strings.Contains(string(body), `"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"`+headController+
`","base":"main","dir":"mesh-controller"}]`) {
t.Fatalf("on the wire: %s", body)
}
}
// When mesh-delivery does not answer in time (or is not held, or answered without the fields), the check reads no
// sibling and its request says it could not learn them, so a fix to mesh-delivery is still checked without it.
func TestWhenMeshDeliveryDoesNotAnswerTheCheckReadsNoSiblingAndSaysSo(t *testing.T) {
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, c := range []struct {
what string
err error
held bool
}{{"it did not answer", context.DeadlineExceeded, true}, {"nothing serves it", errors.New("nothing serves mesh-delivery.groups"), true},
{"no holder on record", errors.New("no holder of mesh-delivery is on record to say it"), false}} {
entries := theCoreOnTheGitSeat()
if !c.held {
entries = entries[:4]
}
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
if held != c.held {
t.Errorf("%s: held %v", c.what, held)
}
return nil, c.err
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if len(request.Check.Siblings) != 0 || !strings.Contains(request.Check.SiblingsUnknown, c.err.Error()) ||
!strings.Contains(request.Check.SiblingsUnknown, "reads no siblings") {
t.Errorf("%s: siblings %+v, unknown %q", c.what, request.Check.Siblings, request.Check.SiblingsUnknown)
}
}
}
// **A pull request for which mesh-delivery answered and named no sibling is asked exactly as before** (ADR 0382 §7):
// byte for byte the request it was without the rule.
func TestAPullRequestWithNoSiblingIsAskedByteForByteAsBefore(t *testing.T) {
entries := theCoreOnTheGitSeat()
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, answer := range [][]groupsLine{nil, {group("feat/other", member("novox/mesh-controller", 346, headController, "main", true))},
{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 3, headHq, "main", true))}} {
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, nil }
build := func() link.BuildRequest {
return link.BuildRequest{ID: "build-1", Repository: "r", Ref: headHost, Check: &link.CheckRequest{Owner: "novox",
Repo: "mesh-host", Number: 114, Base: "main", Beside: besideTheCore(), Modules: []string{"mesh-host"}, Judge: "validator"}}
}
before, _ := json.Marshal(build())
after := build()
withSiblings(t.Context(), &after, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
got, _ := json.Marshal(after)
if string(before) != string(got) {
t.Fatalf("the request changed:\nbefore %s\nafter %s", before, got)
}
}
}
// What the build seat's verdict says of its siblings reaches the forge's holder in the controller's `checked`.
func TestTheVerdictCarriesTheSiblingsAndWhereTheJudgeCameFrom(t *testing.T) {
read := []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", BaseCommit: headHost}}
c := checkedOf(link.BuildResult{ID: "b", Ref: headHost, Checked: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114},
Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok", Siblings: read, JudgeFrom: "sibling novox/mesh-controller#346 merge abc",
Validator: "the change's own validator"}})
if len(c.Siblings) != 1 || c.Siblings[0] != read[0] || c.JudgeFrom == "" || c.Validator == "" {
t.Fatalf("the verdict says %+v", c)
}
body, _ := json.Marshal(c)
for _, want := range []string{`"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"` + headController +
`","base":"main","base_commit":"` + headHost + `"}]`, `"judge_from":"sibling novox/mesh-controller#346 merge abc"`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the verdict lacks %s:\n%s", want, body)
}
}
unknown := checkedOf(link.BuildResult{ID: "b", Check: &link.CheckOutcome{Verdict: "pass", SiblingsUnknown: "mesh-delivery did not answer"}})
if body, _ := json.Marshal(unknown); !strings.Contains(string(body), `"siblings_unknown":"mesh-delivery did not answer"`) ||
strings.Contains(string(body), `"siblings":`) {
t.Fatalf("the verdict that could not learn them says %s", body)
}
}
cmd/mesh-controller/check_test.go
+1 -38
View File
@@ -8,8 +8,8 @@ import (
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/beside"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The check anybody can run is the check registration runs (novox/hq issue 148, ADR 0037): a manifest
@@ -122,40 +122,3 @@ func TestASeatAnotherModuleDeclaresIsANoteOverSomeManifests(t *testing.T) {
t.Fatalf("the note was not said:\n%s", out.String())
}
}
// **A module lives whole in one repository** (novox/hq ADR 0420 §1, issue 1019): `module check`, told the
// repository it runs for, refuses an artifact whose build context names another; run by hand, it does not know.
func TestModuleCheckRefusesAnArtifactContextNamingAnotherRepository(t *testing.T) {
dir := t.TempDir()
write := func(name, context string) string {
path := filepath.Join(dir, name+".json")
raw := `{"module":"` + name + `","version":"1","build":{"artifacts":[{"name":"server","kind":"image","from":"Dockerfile"` + context + `}]}}`
if err := os.WriteFile(path, []byte(raw), 0o600); err != nil {
t.Fatal(err)
}
return path
}
other := write("other", `,"context":{"seat":"git","repository":"novox/mesh-controller","ref":"main"}`)
own := write("own", `,"context":{"seat":"git","repository":"novox/mesh-catalog","ref":"main"}`)
none := write("none", ``)
t.Setenv("MESH_CHECK_REPOSITORY", "novox/mesh-catalog")
var out bytes.Buffer
if err := moduleCheck([]string{other}, &out); err == nil {
t.Fatalf("a context naming another repository passed:\n%s", out.String())
}
if !strings.Contains(out.String(), "other builds artifact server from novox/mesh-controller: a module lives whole in one repository (novox/hq ADR 0420 §1)") {
t.Errorf("the refusal is not in the words of the gate:\n%s", out.String())
}
for _, path := range []string{own, none} {
out.Reset()
if err := moduleCheck([]string{path}, &out); err != nil {
t.Errorf("%s was refused: %v\n%s", path, err, out.String())
}
}
t.Setenv("MESH_CHECK_REPOSITORY", "")
out.Reset()
if err := moduleCheck([]string{other}, &out); err != nil {
t.Errorf("a local run, with no repository named, refused it: %v\n%s", err, out.String())
}
}
cmd/mesh-controller/checks.go
+40 -242
View File
@@ -4,8 +4,6 @@ import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"path"
"regexp"
"runtime/debug"
@@ -14,8 +12,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -75,7 +72,7 @@ type checkScope struct {
// Mesh says the repository is the mesh's: modules are built from it on some branch, its owner is the
// core's, or the change adds a module to it.
Mesh bool
// Judge is who judges the gate (link.JudgeSelf, or the running controller).
// Judge is who judges the gate (link.JudgeSelf, link.JudgeValidator, or the running controller).
Judge string
// From is a module built from the repository, for how the mesh clones it; nil when none is.
From *inventory.Entry
@@ -101,6 +98,10 @@ func pullScope(p link.PullUpdated, entries []inventory.Entry, read map[string][]
switch e.Manifest.Module {
case "mesh-controller":
s.Judge = link.JudgeSelf
case "mesh-host":
if s.Judge == "" {
s.Judge = link.JudgeValidator
}
}
}
for _, d := range r.Added {
@@ -139,116 +140,63 @@ func sourceOwner(repository string) string {
// PullUpdated decides a pull request's merge check, and asks for it when there is something to run.
func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
_, err := f.pullUpdated(ctx, p, os.Stdout)
return err
}
// pullUpdated is PullUpdated saying what it did to w. It returns the id of the check it asked of the build seat, or
// "" when it said a verdict itself and asked nothing. A command that answers JSON on standard output (`delivery
// check`) gives it standard error, so its output stays the document alone (novox/hq issue 1311).
func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Writer) (string, error) {
inv := f.open.inventory
entries, err := inv.Catalogued(ctx)
if err != nil {
return "", err
return err
}
read, err := readForPlanning(ctx, inv)
if err != nil {
return "", err
return err
}
edges, err := inv.Dependencies(ctx)
if err != nil {
return "", err
return err
}
scope := pullScope(p, entries, read, edges)
// **A cycle of edges fails the check, naming every module and edge** (novox/hq ADR 0419 clause 6), a lone pull
// request's as a group's composed check is: no order exists for what it changes.
alone := orderMember{ID: p.Owner + "/" + p.Repo + "@" + twelve(p.Commit), Repository: p.Owner + "/" + p.Repo, Base: p.Base,
Head: p.Commit, Number: p.Number, Paths: p.Paths, PathsTruncated: p.PathsTruncated, Removed: p.Removed, BaseSHA: p.BaseSHA}
if failing, cyclic, err := cycleVerdict(ctx, "", []orderMember{alone},
map[string]orderReach{alone.ID: {Moved: scope.Modules}}, edges, entries); err != nil {
return "", err
} else if cyclic {
failing.Commit, failing.Trunk = p.Commit, twelve(p.BaseSHA)
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s\n", p.Owner, p.Repo, p.Number, p.Commit, failing.Summary)
sayChecked(ctx, w, failing)
return "", nil
}
// **A member of a group whose gate is the composed check's runs its repository's check alone** (novox/hq ADR 0403
// clause 1): mesh-delivery is asked once, and says so on the group's line. Told nothing, the pull request is
// checked alone, gate included (clause 6).
lines, groupsErr := groupsAsked(ctx, deliverySeatHeld(entries))
composed := groupsErr == nil && composedMember(p, lines)
// The change plan of the commit at hand (ADR 0238): what a merge of it would build and send, posted
// with the verdict whatever the verdict is.
plan := changePlanOf(p.Owner+"/"+p.Repo, p.Base, p.Commit, scope.Reach, entries, func(module string) (inventory.Upgrade, bool) {
u, err := inv.UpgradeOf(ctx, module)
return u, err == nil
})
fmt.Fprint(w, planText(plan))
fmt.Print(planText(plan))
direct := directVerdict(p, plan)
if composed {
scope.Modules, scope.Dependents, scope.New, scope.Manifests = nil, nil, nil, nil
composedGate(&direct)
}
switch {
case !scope.gated() && !scope.Mesh:
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
fmt.Printf("%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched)
sayChecked(ctx, w, direct)
return "", nil
sayChecked(ctx, direct)
return nil
case !scope.gated() && p.MergeCheckSaid && !p.MergeCheck:
direct.RepoCheck = &link.CheckLayer{Verdict: "warning", Summary: noMergeCheck}
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s; %s\n", p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched, noMergeCheck)
sayChecked(ctx, w, direct)
return "", nil
fmt.Printf("%s/%s#%d (%.8s): %s; %s\n", p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched, noMergeCheck)
sayChecked(ctx, direct)
return nil
}
request, err := checkRequestFor(ctx, f.open, p, scope, entries)
if err != nil {
return "", err
return err
}
request.Check.Plan = &plan
request.Check.Composed = composed
// **The group's other open pull requests are read beside it** (novox/hq ADR 0382 §1).
siblingsFrom(&request, p, entries, lines, groupsErr)
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
return "", err
return err
}
defer ask.Close()
if err := ask.Ask(ctx, request); err != nil {
return "", err
return err
}
// **A newer request replaces the one still waiting, and a newer key stops the one running** (novox/hq ADR 0406 §3, §4,
// issue 974): the check queue keeps one request a pull request, and a running check watches its key. Nothing is asked
// of the forge here (issue 726).
what := "its own merge-check.sh alone: " + noModuleTouched
if composed {
what = "its own merge-check.sh alone: the gate is its group's composed check's"
}
if scope.gated() {
what = fmt.Sprintf("the gate over %s (a merge would build %d module(s) in %d tier(s))",
strings.Join(append(append([]string{}, scope.Modules...), prefixedAll("new:", scope.New)...), ", "),
scope.Width, scope.Tiers)
}
fmt.Fprintf(w, "%s/%s#%d (%.8s): asked %s to check it before it merges — %s — as %s\n", p.Owner, p.Repo, p.Number,
fmt.Printf("%s/%s#%d (%.8s): asked %s to check it before it merges — %s — as %s\n", p.Owner, p.Repo, p.Number,
p.Commit, seat, what, request.ID)
return request.ID, nil
}
// composedWords is what a group member's own verdict says of its gate layer (novox/hq ADR 0403 clause 1).
const composedWords = "judged by its group's composed check"
// gateComposed is a gate layer's verdict when no gate was run for the member alone: never a pass. Mesh-delivery reads it
// as the composed check's to judge, and the forge's holder sets no mesh/merge-gate from it.
const gateComposed = "composed"
// composedGate says a group member's own verdict has no gate of its own: its gate layer, and the verdict that is
// the gate's, read "composed" in place of a pass.
func composedGate(c *link.Checked) {
c.Verdict, c.Summary = gateComposed, composedWords
c.Gate = &link.CheckLayer{Verdict: gateComposed, Summary: composedWords}
return nil
}
// directVerdict is the verdict of a change no build judges — it touches no module, and either the repository is not
@@ -277,7 +225,7 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
for _, e := range entries {
shelf[e.Manifest.Module] = e.Manifest
}
world, err := restOfTheMesh(ctx, open, shelf, "")
world, err := theRestOfTheMesh(ctx, open.inventory, shelf, "")
if err != nil {
return link.BuildRequest{}, err
}
@@ -300,35 +248,7 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
if err != nil {
return link.BuildRequest{}, err
}
beside, err := besideOf(entries, func(module string) string { return current[module].Commit }, clone)
if err != nil {
return link.BuildRequest{}, err
}
builds := checkBuildsOf(p, scope, entries, clone)
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
Ref: p.Commit,
Held: heldBy(ctx),
Seats: seatBases(ctx),
Source: sourceOnSeat(source),
Check: checkOf(p, scope, entries, beside, builds, namedForTheCheck(ctx, p, builds), runningBy(ctx)),
}, nil
}
// checkOf is the check a request carries: the pull request, what the change touches, and the buckets of the pull
// requests' keys of the modules claiming the delivery seat (novox/hq ADR 0437 §3, issue 1212).
func checkOf(p link.PullUpdated, scope checkScope, entries []inventory.Entry, beside map[string]link.CheckedOut,
builds []link.CheckBuild, named []link.CheckedBuild, running map[string]string) *link.CheckRequest {
return &link.CheckRequest{Pulls: pullsBucketsOf(entries), Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Trunk: p.BaseSHA, Ready: p.Ready, Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents,
New: scope.New, Manifests: scope.Manifests, Judge: scope.Judge, Builds: builds, Named: named, Running: running}
}
// besideOf is what a check reads beside its own repository, by the directory it is found under: each core repository
// by the module the mesh builds from it, at besideRefs. running is the commit the mesh runs of a module.
func besideOf(entries []inventory.Entry, running func(module string) string,
clone func(inventory.Source) (string, error)) (map[string]link.CheckedOut, error) {
// Beside it, at what the mesh runs: each core repository by the module the mesh builds from it.
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
@@ -337,12 +257,11 @@ func besideOf(entries []inventory.Entry, running func(module string) string,
}
url, err := clone(e.Source)
if err != nil {
return nil, err
return link.BuildRequest{}, err
}
refs := besideRefs(dir, running(e.Manifest.Module))
refs := besideRefs(dir, current[e.Manifest.Module].Commit)
beside[dir] = link.CheckedOut{Repository: url, Ref: refs[dir]}
if dir == "mesh-controller" {
// The controller's main stays its main: it judges only a check the running controller predates.
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
@@ -354,98 +273,17 @@ func besideOf(entries []inventory.Entry, running func(module string) string,
}
}
}
return beside, nil
}
// trunkOf is the branch a module's source follows: main when it records none, or records a commit.
func trunkOf(s inventory.Source) string {
if t := strings.TrimSpace(followedBranch(s.Ref)); t != "" {
return t
}
return "main"
}
// sourceName is the name of a recorded repository, a path on the git seat or a URL: novox/mesh-host → mesh-host.
func sourceName(repository string) string {
parts := strings.Split(strings.Trim(strings.TrimSuffix(repository, ".git"), "/"), "/")
if len(parts) < 2 {
return ""
}
return parts[len(parts)-1]
}
// namedForTheCheck are the builds earlier checks of the same pull request at the same head named, read once for the
// check (novox/hq issue 703): asked again because only the trunk moved, the check keeps what they built of a key
// still the same. Another pull request's build, or one of an earlier head, is never kept. None when they cannot be
// read, and the check builds, as before.
func namedForTheCheck(ctx context.Context, p link.PullUpdated, builds []link.CheckBuild) []link.CheckedBuild {
if len(builds) == 0 || p.Commit == "" {
return nil
}
pull := link.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number}
read, err := checkedBuildsOf(ctx, []link.PullRef{pull})
if err != nil {
fmt.Printf(" the earlier checks' builds of %s/%s#%d could not be read, so the check builds: %v\n",
p.Owner, p.Repo, p.Number, err)
// empty-on-error: said above; a check handed no named build builds every module, as before issue 703
return nil
}
var named []link.CheckedBuild
for _, b := range read {
if b.Pull == pull && link.SameHead(b.Head, p.Commit) {
named = append(named, b)
}
}
return named
}
// checkBuildsOf are the modules a merge of the change would build, in the walk's order, as the check builds them
// for the walk to take (novox/hq ADR 0350 §1): a module built from the repository checked, at the branch it merges
// into, from the merge the check makes; any other — a module standing on one — from its own repository at the
// branch it follows, as the walk builds it. A module whose build reads the repository checked through a context,
// or that the merge deletes, cannot be built as the walk would: said, and nothing after it is built. A change that
// moves no module builds nothing.
func checkBuildsOf(p link.PullUpdated, scope checkScope, entries []inventory.Entry,
clone func(inventory.Source) (string, error)) []link.CheckBuild {
if len(scope.Reach.Plan.Tiers) == 0 {
return nil
}
m := link.SourceMoved{Owner: p.Owner, Repo: p.Repo, Base: p.Base, CloneURL: p.CloneURL}
byName := map[string]inventory.Entry{}
for _, e := range entries {
byName[e.Manifest.Module] = e
}
skip := map[string]string{}
for _, e := range scope.Reach.Packaging {
skip[e.Manifest.Module] = "its build reads " + p.Owner + "/" + p.Repo + " through a context, which a check " +
"cannot hand the merge"
}
for _, e := range scope.Reach.Deleted {
skip[e.Manifest.Module] = "the merge deletes it"
}
var out []link.CheckBuild
for tier, names := range scope.Reach.Plan.Tiers {
for _, name := range names {
e, known := byName[name]
b := link.CheckBuild{Module: name, Tier: tier, Skip: skip[name]}
switch {
case !known:
b.Skip = "the catalogue does not hold it"
case b.Skip != "":
case sourceIs(e.Source, m):
b.Merged, b.Path = true, e.Source.Path
default:
url, err := clone(e.Source)
if err != nil {
b.Skip = "its repository cannot be cloned: " + err.Error()
break
}
b.Repository, b.Path, b.Ref = url, e.Source.Path, followedBranch(e.Source.Ref)
}
out = append(out, b)
}
}
return out
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
Ref: p.Commit,
Held: heldBy(ctx),
Seats: seatBases(ctx),
Source: sourceOnSeat(source),
Check: &link.CheckRequest{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents, New: scope.New,
Manifests: scope.Manifests, Judge: scope.Judge},
}, nil
}
// besideRefs is the ref each repository is cloned at beside a check, by the directory it is found under:
@@ -527,14 +365,7 @@ const maxCheckReport = 60 << 10
// checked says a merge check's verdict as the controller's `checked`. Nothing is recorded or
// registered: a check builds nothing (issue 240's rule for a dry run, kept for a check).
func checked(ctx context.Context, result link.BuildResult) {
// **A check stopped by its key says nothing** (novox/hq ADR 0406 §4, issue 974, review 2): it stopped because the head or
// the trunk commit moved on, and the newer request will judge. Said as an error it would name the live head — red
// statuses on the forge, a delivery refused — and written to the key it would stand in the newer request's way.
if result.Failed == link.SupersededByKey {
fmt.Printf("%s: stopped, its pull request's key moved on; the newer check will judge\n", result.ID)
return
}
sayChecked(ctx, os.Stdout, checkedOf(result))
sayChecked(ctx, checkedOf(result))
}
// checkedOf is what a check's outcome says: each layer, and an error — never a pass — for a check that
@@ -551,9 +382,6 @@ func checkedOf(result link.BuildResult) link.Checked {
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and the commits of the
// repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
c.Built = result.Check.Built
c.Validator = result.Check.Validator
c.Siblings, c.SiblingsUnknown, c.JudgeFrom = result.Check.Siblings, result.Check.SiblingsUnknown, result.Check.JudgeFrom
if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
@@ -571,12 +399,6 @@ func checkedOf(result link.BuildResult) link.Checked {
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
c.RepoCheck = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
}
// **A group member's own check has no gate of its own** (novox/hq ADR 0403 clause 1): what ran was the repository's
// check alone, and its gate layer is "composed", never a pass. A failure or an error stays what it was.
if result.Checked != nil && result.Checked.Composed && result.Check != nil && c.Gate != nil &&
(c.Gate.Verdict == "pass" || c.Gate.Verdict == "warning") {
composedGate(&c)
}
if result.Checked != nil && c.Gate != nil && len(c.Gate.Modules) == 0 {
c.Gate.Modules = append(append([]string{}, result.Checked.Modules...), prefixedAll("new:", result.Checked.New)...)
}
@@ -607,20 +429,6 @@ func checkedOf(result link.BuildResult) link.Checked {
l.Verdict = "error"
}
}
// **A check run again says so** (novox/hq issue 594). The summaries are the lines the pull request
// shows, clipped by the forge to a few words, so they get only a short marker after the verdict's own
// words; the report gets the whole sentence. Layers copied, so the build's own outcome is left as it came.
if mark := link.RunMarker(result.Delivery); mark != "" {
c.Summary += " " + mark
for _, l := range []**link.CheckLayer{&c.Gate, &c.RepoCheck} {
if *l != nil {
layer := **l
layer.Summary += " " + mark
*l = &layer
}
}
c.Report = link.Restarted(result.Delivery) + "\n\n" + c.Report
}
if len(c.Report) > maxCheckReport {
c.Report = "…" + c.Report[len(c.Report)-maxCheckReport:]
}
@@ -650,7 +458,7 @@ func prefixedAll(prefix string, items []string) []string {
}
// sayChecked says a merge check's verdict on the bus, where the forge's holder hears it.
func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
func sayChecked(ctx context.Context, c link.Checked) {
repo := "none"
if c.RepoCheck != nil {
repo = strings.ToUpper(c.RepoCheck.Verdict) + " — " + c.RepoCheck.Summary
@@ -659,7 +467,7 @@ func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
if c.Trunk != "" {
judged = " merged into trunk " + c.Trunk
}
fmt.Fprintf(w, "%s: %s/%s#%d at %.8s%s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
fmt.Printf("%s: %s/%s#%d at %.8s%s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, judged, orSomewhere(c.On), strings.ToUpper(c.Verdict), c.Summary, repo)
if checkEvents == nil {
return
@@ -671,16 +479,6 @@ func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
stating, stop := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer stop()
if err := checkEvents.PublishSeatEvent(stating, link.MeshControllerSeat, link.KeyChecked, body); err != nil {
fmt.Fprintf(w, "%s: the verdict could not be said, so the pull request is not told it: %v\n", c.ID, err)
fmt.Printf("%s: the verdict could not be said, so the pull request is not told it: %v\n", c.ID, err)
}
}
// pullsBucketsOf are the buckets of the pull requests' keys a check names for the build agent: one per module
// claiming the delivery seat, in the order of their names (novox/hq ADR 0437 §3, issue 1212).
func pullsBucketsOf(entries []inventory.Entry) []string {
var out []string
for _, m := range claimantsOf(entries, catalogue.DeliverySeat) {
out = append(out, broker.BucketName(m, "pulls"))
}
return out
}
cmd/mesh-controller/checks_integration_test.go
-56
View File
@@ -1,56 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
func TestACheckStoppedByItsKeyIsAnnouncedAsNoVerdict(t *testing.T) {
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
legacy, err := js.Conn().JetStream()
if err != nil {
t.Fatal(err)
}
was := checkEvents
checkEvents = link.OverNATS{Conn: js.Conn(), JS: legacy}
t.Cleanup(func() { checkEvents = was })
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeyChecked), heard)
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 3}
checked(t.Context(), link.BuildResult{ID: "stopped", Ref: "abc", Checked: asked, Failed: link.SupersededByKey})
checked(t.Context(), link.BuildResult{ID: "broken", Ref: "abc", Checked: asked, Failed: "the facts snapshot cannot be read"})
select {
case m := <-heard:
var c link.Checked
_ = json.Unmarshal(m.Data, &c)
if c.ID != "broken" || c.Verdict != "error" {
t.Fatalf("the first verdict said is %+v; want only the broken check's error", c)
}
case <-time.After(5 * time.Second):
t.Fatal("nothing was said")
}
select {
case m := <-heard:
t.Fatalf("a second verdict was said: %s", m.Data)
case <-time.After(300 * time.Millisecond):
}
}
cmd/mesh-controller/checks_queue_test.go
-97
View File
@@ -1,97 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"testing"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Acceptance 5 of milestone 2 (novox/hq ADR 0406 §3, issue 974), live, with the store and a real bus: two merges into one
// repository within a minute, with two open pull requests, leave each pull request with exactly one queued check, by the
// stream's rule. Each merge moves the trunk, and mesh-delivery asks the controller to check both open pull requests again
// against it (`delivery check`, which is the controller's own path for a head announced); nothing moves a request, nothing
// asks the forge, and the queue is read back whole.
func TestTwoMergesWithinAMinuteLeaveEachOpenPullRequestOneQueuedCheck(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
holdTheDeliverySeat(t, open)
was := askDeliveryOwner
t.Cleanup(func() { askDeliveryOwner = was })
askDeliveryOwner = func(_ context.Context, _ *nats.Conn, verb string, _ map[string]any) (json.RawMessage, error) {
if verb != "groups" {
t.Fatalf("asked the owner %s", verb)
}
return json.RawMessage(`[]`), nil
}
// The repository the two pull requests are into, as a module the mesh builds from it.
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
heads := map[int]string{7: "7777777777777777777777777777777777777777", 8: "8888888888888888888888888888888888888888"}
check := func(number int, trunk string) {
t.Helper()
p := orderMember{ID: fmt.Sprintf("novox/mesh-controller@%.12s", heads[number]), Repository: "novox/mesh-controller",
Base: "main", Head: heads[number], Number: number, Paths: []string{"main.go"}, ModuleDirsSaid: true,
BaseSHA: trunk, Ready: number == 8}.pull()
p.MergeCheck, p.MergeCheckSaid = true, true
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
}
// Announced against the trunk as it stands, then asked again after each of two merges.
for _, trunk := range []string{"1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222",
"3333333333333333333333333333333333333333"} {
check(7, trunk)
check(8, trunk)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
seen := map[int]string{}
for _, a := range q.Asks {
c := a.Request.Check
if c == nil {
t.Fatalf("%s is no check", a.ID)
}
if _, twice := seen[c.Number]; twice {
t.Errorf("pull request %d has two queued checks", c.Number)
}
seen[c.Number] = c.Trunk
if a.State != link.AskWaiting {
t.Errorf("%s is %s", a.ID, a.State)
}
if want := "3333333333333333333333333333333333333333"; c.Trunk != want {
t.Errorf("pull request %d is queued against %.8s; want the newest trunk %.8s", c.Number, c.Trunk, want)
}
}
if len(seen) != 2 {
t.Fatalf("queued checks for %v; want one for each of the two open pull requests", seen)
}
// By the streams' own count: the ready one on its stream, the other on its.
for stream, want := range map[string]uint64{broker.ChecksReadyStream: 1, broker.ChecksOtherStream: 1} {
info, err := js.Context().StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != want {
t.Errorf("%s holds %d request(s); want %d", stream, info.State.Msgs, want)
}
}
}
cmd/mesh-controller/checks_test.go
+2 -5
View File
@@ -53,8 +53,8 @@ func TestThePullRequestIsMappedOntoTheModuleGraph(t *testing.T) {
"", "modules/newmod", "modules/newmod/module.json", true, ""},
{"the controller judges itself", pull("novox", "mesh-controller", "main", []string{"cmd/x.go"}),
"mesh-controller", "", "module.json", true, "self"},
{"the node-engine is judged by the controller the mesh runs, as any module is (issue 1391)",
pull("novox", "mesh-host", "main", []string{"validate/v.go"}), "mesh-host", "", "module.json", true, ""},
{"the node-engine is judged with its validator", pull("novox", "mesh-host", "main", []string{"validate/v.go"}),
"mesh-host", "", "module.json", true, "validator"},
{"the core's owner, no module: the mesh's, its own check alone", pull("novox", "hq", "main", []string{"README.md"}),
"", "", "", true, ""},
{"a branch nothing is built from: the mesh's repository, no module", pull("novox", "photos", "master",
@@ -288,6 +288,3 @@ func TestARecheckedHeadsDirectVerdictNamesItsTrunk(t *testing.T) {
t.Fatalf("a rechecked head's direct verdict names trunk %q", c.Trunk)
}
}
// A check stopped by its pull request's key moving on says no verdict (issue 974, review 2): an error said for it would name the
// live head, and the newer request is the one that judges. A check that failed for any other reason still says its error.
cmd/mesh-controller/compose_callers_test.go
-63
View File
@@ -1,63 +0,0 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"path/filepath"
"sort"
"strings"
"testing"
)
// 1388: no caller composes through the controller's own composition any more. The only functions left that call it are the
// composition's own, which issue 1389 deletes with it; this fails the moment any other function reaches for it again.
func TestNoCallerComposesThroughTheControllersOwnComposition(t *testing.T) {
legacy := map[string]bool{"planFor": true, "declarationWith": true, "composedAndValidated": true, "generators": true,
"renderingFor": true, "theRestOfTheMesh": true, "whoResolves": true}
// The composition's own pairs, until 1389: "file: caller calls callee".
allowed := map[string]bool{
"facts.go: composedAndValidated calls declarationWith": true,
"facts.go: composedAndValidated calls planFor": true,
"network.go: generators calls whoResolves": true,
"network.go: whoResolves calls planFor": true,
"plan.go: declarationWith calls renderingFor": true,
"plan.go: grantsFor calls planFor": true,
"plan.go: planFor calls theRestOfTheMesh": true,
}
files, err := filepath.Glob("*.go")
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
var found []string
for _, f := range files {
if strings.HasSuffix(f, "_test.go") {
continue
}
file, err := parser.ParseFile(fset, f, nil, 0)
if err != nil {
t.Fatal(err)
}
for _, d := range file.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok || fn.Body == nil {
continue
}
ast.Inspect(fn.Body, func(n ast.Node) bool {
if c, ok := n.(*ast.CallExpr); ok {
if id, ok := c.Fun.(*ast.Ident); ok && legacy[id.Name] {
found = append(found, f+": "+fn.Name.Name+" calls "+id.Name)
}
}
return true
})
}
}
sort.Strings(found)
for _, pair := range found {
if !allowed[pair] {
t.Errorf("%s: it composes through the controller's own composition, not mesh-sdk's", pair)
}
}
}
cmd/mesh-controller/compose_equal_integration_test.go
-310
View File
@@ -1,310 +0,0 @@
//go:build integration
package main
import (
"context"
"flag"
"fmt"
"os"
"sort"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/compose"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
)
// 1384: nothing shows the controller's composition and mesh-sdk's compose every node alike on the live facts. This test is
// that proof (novox/hq ADR 0453 clause 3): it reads the facts snapshot MESH_FACTS names (the live one in every check and
// every try, the build seat gives it), else one captured in testdata/facts-equal.json (taken with `facts show` and
// scrubbed as the snapshot is), else the snapshot of a small mesh of its own; raises it into throwaway stores as the merge
// gate does; and composes every machine both ways.
//
// It runs only when named (a try with this test's name as the tests pattern); a whole-suite run skips it.
//
// (a) Foreseeing on the fresh store: the same error or none, the same validator problems, LeftOut and its reasons, Withheld,
// Unbound, Unplaced, HandedOnly and the own secrets named foreseen; and the ports mesh-sdk names foreseen are the ones the
// controller allocates when it sends.
// (b) the send: on two stores raised from the same snapshot, the controller's Allocating composition of every machine, and the
// adapter's compose, perform and compose again; the same ports, pair credentials and own secrets were made, by key.
// (c) after (b), Reading on one store, both ways: the declaration byte for byte.
//
// It sends nothing and opens no store but the throwaway ones. A difference names the machine, the comparison and the first
// field that differs; a machine both paths refuse alike is said, not failed. The test is deleted with the controller's
// composition (issue 1389).
func TestEveryMachineComposesAlikeBothWays(t *testing.T) {
// It is slow (it composes every machine of the live mesh several times each way) and is deleted with the controller's
// composition (issue 1389), so it runs only when asked for by name: a try or a `go test -run` that names it. A merge check
// runs every test and does not.
if run := flag.Lookup("test.run"); run == nil || !strings.Contains(run.Value.String(), "TestEveryMachineComposesAlikeBothWays") {
t.Skip("runs only when named: a try with the tests pattern TestEveryMachineComposesAlikeBothWays")
}
ctx := t.Context()
admin := os.Getenv("MESH_TEST_POSTGRES")
if admin == "" {
t.Skip("no throwaway PostgreSQL (MESH_TEST_POSTGRES) to raise the mesh in")
}
started := time.Now()
phase := func(what string) { t.Logf("[%s] %s", time.Since(started).Round(time.Second), what) }
f, source := equalityFacts(t)
if len(f.Machines) < 1 {
t.Fatalf("the facts (%s) hold no machine, so nothing was composed and nothing is proved", source)
}
t.Logf("the facts of %s (%s): %d machine(s)", f.Taken.Format("2006-01-02 15:04:05"), source, len(f.Machines))
shelf, err := shelfOfFacts(f)
if err != nil {
t.Fatal(err)
}
sources := sourcesOfFacts(f)
// Three stores, raised from the same snapshot, one after the other: (a) both ways on a fresh store, (b) and (c) the
// controller's sends and readings, (b) mesh-sdk's sends. They cannot run together: throwawayStores finds its databases by
// the process environment (and a store opened later finds them there too), so a second one raised at the same time takes
// the first one's connection from under it (the live try: database gate_inventory_... does not exist).
var (
oldForeseen, sdkForeseen = map[string]composedBoth{}, map[string]composedBoth{}
foreseenPorts = map[string][]string{}
readOld, readSDK = map[string]composedBoth{}, map[string]composedBoth{}
oldMade, sdkMade made
oldRefused = map[string]string{}
sendRefused = map[string]string{}
errA, errOld, errSDK error
)
func() { // (a) Foreseeing on the fresh store.
phase("(a) starts: raising a fresh store")
fresh, drop, err := throwawayStores(ctx, admin)
if err != nil {
errA = err
return
}
defer drop()
if _, err := raiseFromFacts(ctx, fresh, f, shelf, sources); err != nil {
errA = err
return
}
// mesh-sdk's composition first: it writes nothing, where the controller's makes every pair credential on the way. The
// Input is read once for all of its machines.
phase("(a) the store is raised; mesh-sdk's Foreseeing starts")
cached := withInputCache(ctx)
for _, m := range f.Machines {
in, err := inputFor(cached, fresh)
if err != nil {
errA = fmt.Errorf("%s: the adapter cannot read the mesh: %w", m.Name, err)
return
}
d, problems, cerr := compose.ComposeIn(in.Input, m.Name, compose.Foreseeing)
if cerr == nil {
foreseenPorts[m.Name] = append([]string(nil), d.ForeseenPorts...)
sort.Strings(foreseenPorts[m.Name])
}
sdkForeseen[m.Name] = capturedFrom(sendableOf(d), problems, cerr)
}
phase("(a) mesh-sdk's Foreseeing of every machine")
gens, gensErr := generators(ctx, fresh)
for _, m := range f.Machines {
if gensErr != nil {
oldForeseen[m.Name] = composedBoth{Err: gensErr.Error()}
continue
}
d, problems, err := composedAndValidated(ctx, fresh, m.Name, gens, Foreseeing)
oldForeseen[m.Name] = capturedFrom(d, problems, err)
}
phase("(a) the controller's Foreseeing of every machine")
}()
if errA != nil {
t.Fatal(errA)
}
func() { // (b) and (c): the controller's send of every machine, then both ways of reading.
phase("(b) starts: the controller's send, raising a store")
oldStore, drop, err := throwawayStores(ctx, admin)
if err != nil {
errOld = err
return
}
defer drop()
if _, err := raiseFromFacts(ctx, oldStore, f, shelf, sources); err != nil {
errOld = err
return
}
oldGens, oldGensErr := generators(ctx, oldStore)
for _, m := range f.Machines {
if oldGensErr != nil {
oldRefused[m.Name] = oldGensErr.Error()
continue
}
if _, _, err := composedAndValidated(ctx, oldStore, m.Name, oldGens, Allocating); err != nil {
oldRefused[m.Name] = err.Error()
}
}
phase("(b) the controller's send of every machine")
if oldMade, err = madeIn(ctx, oldStore); err != nil {
errOld = err
return
}
cached := withInputCache(ctx)
for _, m := range f.Machines {
if oldGensErr != nil {
readOld[m.Name] = composedBoth{Err: oldGensErr.Error()}
} else {
d, problems, err := composedAndValidated(ctx, oldStore, m.Name, oldGens, Reading)
readOld[m.Name] = capturedFrom(d, problems, err)
}
d, problems, err := composeQuestion(cached, oldStore, m.Name, compose.Reading, true)
readSDK[m.Name] = capturedFrom(d, problems, err)
}
phase("(c) both ways of reading every machine after the controller's send")
}()
if errOld != nil {
t.Fatal(errOld)
}
func() { // (b) mesh-sdk's send of every machine.
phase("(b) starts: mesh-sdk's send, raising a store")
sdkStore, drop, err := throwawayStores(ctx, admin)
if err != nil {
errSDK = err
return
}
defer drop()
if _, err := raiseFromFacts(ctx, sdkStore, f, shelf, sources); err != nil {
errSDK = err
return
}
for _, m := range f.Machines {
if _, err := composeSend(ctx, sdkStore, m.Name); err != nil {
sendRefused[m.Name] = err.Error()
}
}
phase("(b) mesh-sdk's send of every machine")
sdkMade, errSDK = madeIn(ctx, sdkStore)
}()
if errSDK != nil {
t.Fatal(errSDK)
}
var failures []string
for _, m := range f.Machines {
failures = append(failures, bothWaysDiffer(m.Name, "(a) Foreseeing", oldForeseen[m.Name], sdkForeseen[m.Name], false)...)
failures = append(failures, bothWaysDiffer(m.Name, "(c) Reading after the send", readOld[m.Name], readSDK[m.Name], true)...)
}
// **The send must compose every machine both ways.** A machine either send refuses is no proof of anything: what the
// controller made on the way to refusing it is no send's, and a refusal alike says nothing about what a send of it makes.
// So it fails here, naming the machine and both refusals (a refusal alike in (a), the Foreseeing on a store that holds no
// secret a send made, is said and stays). A pass proves that every machine of the facts was sent and compared.
refused := map[string]bool{}
for _, m := range f.Machines {
why, oldNo := oldRefused[m.Name]
sdkWhy, sdkNo := sendRefused[m.Name]
if !oldNo && !sdkNo {
continue
}
refused[m.Name] = true
failures = append(failures, fmt.Sprintf("%s, (b) the send: it is not composed both ways, so nothing of it is compared: "+
"the controller's refusal %s, mesh-sdk's %s", m.Name, quoted(oneLine(why)), quoted(oneLine(sdkWhy))))
}
failures = append(failures, madeDiffer("(b) the send", oldMade, sdkMade, refused)...)
// (a)'s ports: those mesh-sdk names foreseen, for each machine, are the ones the controller's send allocated to it.
for _, m := range f.Machines {
if refused[m.Name] {
continue
}
var allocated []string
for _, p := range oldMade.Ports {
if node, rest, ok := strings.Cut(p, " "); ok && node == m.Name {
module, port, _ := strings.Cut(strings.SplitN(rest, "->", 2)[0], ":")
allocated = append(allocated, module+":"+port)
}
}
sort.Strings(allocated)
// A port the snapshot already holds is not allocated by a send, and one a module is given is not allocated either: the
// ones mesh-sdk names must all be among those the send allocated.
for _, want := range foreseenPorts[m.Name] {
found := false
for _, got := range allocated {
found = found || got == want
}
if !found {
failures = append(failures, fmt.Sprintf("%s, (a) Foreseeing: mesh-sdk names the port %s foreseen and the "+
"controller's send allocates none for it", m.Name, want))
}
}
}
for machine, why := range sendRefused {
t.Logf("%s: mesh-sdk's send of it is refused: %s", machine, why)
}
for machine, c := range oldForeseen {
if c.Err != "" {
t.Logf("%s: the controller refuses it, and mesh-sdk alike: %s", machine, oneLine(c.Err))
}
}
if len(failures) > 0 {
t.Fatalf("the two compositions differ on %d point(s):\n %s", len(failures), strings.Join(failures, "\n "))
}
t.Logf("%d machine(s) were sent and compose alike both ways", len(f.Machines))
}
// equalityFacts is the snapshot the comparison composes: MESH_FACTS, else the one captured in testdata, else the facts of a
// mesh of its own (the same shape, small). Source says which.
func equalityFacts(t *testing.T) (snapshot.Facts, string) {
t.Helper()
ctx := t.Context()
if path := os.Getenv("MESH_FACTS"); path != "" {
f, err := readFacts(ctx, path)
if err != nil {
t.Fatalf("MESH_FACTS names %s and it is not a snapshot: %v", path, err)
}
return f, "MESH_FACTS " + path
}
if _, err := os.Stat("testdata/facts-equal.json"); err == nil {
f, err := readFacts(ctx, "testdata/facts-equal.json")
if err != nil {
t.Fatal(err)
}
return f, "captured in testdata/facts-equal.json"
}
open, _ := aMeshWithSecrets(t)
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
return f, "gathered from a mesh of the test's own"
}
// madeIn is what a store holds that a send makes, by key and never by value.
func madeIn(ctx context.Context, open *stores) (made, error) {
var out made
inv := open.inventory
nodes, err := inv.Nodes(ctx)
if err != nil {
return out, err
}
for _, n := range nodes {
ports, err := inv.PortsFor(ctx, n.Name)
if err != nil {
return out, err
}
for _, p := range ports {
out.Ports = append(out.Ports, fmt.Sprintf("%s %s:%d->%d", n.Name, p.Module, p.Wanted, p.Machine))
}
rows, err := inv.SecretsFrom(ctx, n.Name)
if err != nil {
return out, err
}
for _, s := range rows {
out.Pairs = append(out.Pairs, fmt.Sprintf("%s %s on %s from %s local=%s", s.Name, s.ConsumerModule, s.Consumer, n.Name, s.Local))
}
}
own, err := inv.OwnSecretRows(ctx)
if err != nil {
return out, err
}
for _, r := range own {
out.OwnSecrets = append(out.OwnSecrets, fmt.Sprintf("%s %s/%s (%s)", r.Node, r.Module, r.Name, r.Origin))
}
sort.Strings(out.Ports)
sort.Strings(out.Pairs)
sort.Strings(out.OwnSecrets)
return out, nil
}
cmd/mesh-controller/compose_equal_test.go
-341
View File
@@ -1,341 +0,0 @@
package main
import (
"bytes"
"encoding/json"
"fmt"
"reflect"
"sort"
"strings"
"testing"
)
// The comparison of the controller's composition and mesh-sdk's (novox/hq ADR 0453 clause 3, design 47 R4b, issue 1384),
// the part of it that needs no store. The test that raises a mesh and composes every machine both ways is
// compose_equal_integration_test.go; both are deleted with the controller's own composition (issue 1389).
// composedBoth is what one composition of a machine said, in the terms the two paths are compared in.
type composedBoth struct {
Err string
Problems []string
LeftOut map[string]string
Withheld []string
Unbound []string
Unplaced []string
Handed []string
// Foreseen are the own secrets the composition names foreseen, as `module/name`.
Foreseen []string
Body []byte
}
// capturedFrom is a sendable, its validator problems and the error that came with it, as compared.
func capturedFrom(d sendable, problems []string, err error) composedBoth {
if err != nil {
return composedBoth{Err: err.Error()}
}
c := composedBoth{Problems: append([]string(nil), problems...), LeftOut: d.leftOutWhy,
Unplaced: append([]string(nil), d.unplaced...), Handed: append([]string(nil), d.handedOnly...),
Foreseen: append([]string(nil), d.foreseen...)}
for _, o := range d.withheld {
c.Withheld = append(c.Withheld, o.String())
}
for _, u := range d.unbound {
c.Unbound = append(c.Unbound, u.String())
}
sort.Strings(c.Problems)
sort.Strings(c.Withheld)
sort.Strings(c.Unbound)
sort.Strings(c.Unplaced)
sort.Strings(c.Handed)
sort.Strings(c.Foreseen)
c.Body, _ = d.Body()
return c
}
// bothWaysDiffer says how two compositions of one machine differ, one line for each field that does, naming the machine and
// the comparison. A machine both refuse alike is no difference. withBody compares the declaration byte for byte and names
// the first resource that differs.
func bothWaysDiffer(machine, comparison string, old, sdk composedBoth, withBody bool) []string {
var out []string
say := func(field string, a, b any) {
out = append(out, fmt.Sprintf("%s, %s: %s differs: the controller's %v, mesh-sdk's %v", machine, comparison, field, a, b))
}
if old.Err != sdk.Err {
say("the refusal", quoted(old.Err), quoted(sdk.Err))
return out
}
if old.Err != "" {
return nil
}
for _, f := range []struct {
name string
old, sdk []string
}{
{"the validator's problems", old.Problems, sdk.Problems},
{"Withheld", old.Withheld, sdk.Withheld},
{"Unbound", old.Unbound, sdk.Unbound},
{"Unplaced", old.Unplaced, sdk.Unplaced},
{"HandedOnly", old.Handed, sdk.Handed},
{"the own secrets named foreseen", old.Foreseen, sdk.Foreseen},
} {
if !sameStrings(f.old, f.sdk) {
say(f.name, f.old, f.sdk)
}
}
if !sameLeftOut(old.LeftOut, sdk.LeftOut) {
say("LeftOut and its reasons", old.LeftOut, sdk.LeftOut)
}
if withBody && !bytes.Equal(old.Body, sdk.Body) {
out = append(out, fmt.Sprintf("%s, %s: the declaration differs byte for byte; %s", machine, comparison,
firstDifferingResource(old.Body, sdk.Body)))
}
return out
}
func quoted(s string) string {
if s == "" {
return "none"
}
return fmt.Sprintf("%q", s)
}
func sameStrings(a, b []string) bool {
if len(a) == 0 && len(b) == 0 {
return true
}
return reflect.DeepEqual(a, b)
}
func sameLeftOut(a, b map[string]string) bool {
if len(a) == 0 && len(b) == 0 {
return true
}
return reflect.DeepEqual(a, b)
}
// firstDifferingResource names the first resource of two declaration bodies that is not the same, and the first field of it.
func firstDifferingResource(a, b []byte) string {
var x, y struct {
Resources []map[string]any `json:"resources"`
}
if json.Unmarshal(a, &x) != nil || json.Unmarshal(b, &y) != nil {
return "a body is not a declaration"
}
byID := func(rs []map[string]any) map[string]map[string]any {
out := map[string]map[string]any{}
for _, r := range rs {
out[fmt.Sprint(r["id"])] = r
}
return out
}
ids := map[string]bool{}
left, right := byID(x.Resources), byID(y.Resources)
for id := range left {
ids[id] = true
}
for id := range right {
ids[id] = true
}
var order []string
for id := range ids {
order = append(order, id)
}
sort.Strings(order)
for _, id := range order {
l, inLeft := left[id]
r, inRight := right[id]
switch {
case !inRight:
return fmt.Sprintf("resource %s is only in the controller's", id)
case !inLeft:
return fmt.Sprintf("resource %s is only in mesh-sdk's", id)
}
var fields []string
for k := range l {
fields = append(fields, k)
}
for k := range r {
if _, have := l[k]; !have {
fields = append(fields, k)
}
}
sort.Strings(fields)
for _, k := range fields {
if !reflect.DeepEqual(l[k], r[k]) {
return fmt.Sprintf("resource %s, field %s: the controller's %v, mesh-sdk's %v", id, k, l[k], r[k])
}
}
}
if !reflect.DeepEqual(left, right) {
return "the envelopes differ"
}
return "the resources are the same; the envelope differs"
}
// made is what a send made in a store, by key and never by value: the machine ports given, the pair credentials and the
// own secrets.
type made struct {
Ports []string
Pairs []string
OwnSecrets []string
}
// madeDiffer says how what two sends made differs, by entry: for each kind, the entries only the controller made, the ones only
// mesh-sdk made, and the ones both made with a different value. Never a whole list. The entries of a machine in refused (a
// machine whose send is refused on either side, which sends nothing) are left out of both.
func madeDiffer(comparison string, old, sdk made, refused map[string]bool) []string {
var out []string
for _, f := range []struct {
name string
old, sdk []string
split func(string) (key, value string)
nodes func(string) []string
}{
{"the ports allocated", old.Ports, sdk.Ports, splitAt("->"), firstWord},
{"the pair credentials made", old.Pairs, sdk.Pairs, func(l string) (string, string) { return l, "" }, pairNodes},
{"the own secrets made", old.OwnSecrets, sdk.OwnSecrets, splitAtLast(" ("), firstWord},
} {
left, right := entriesOf(f.old, f.split, f.nodes, refused), entriesOf(f.sdk, f.split, f.nodes, refused)
var onlyOld, onlySDK, different []string
for key, v := range left {
if w, both := right[key]; !both {
onlyOld = append(onlyOld, key)
} else if w != v {
different = append(different, fmt.Sprintf("%s (the controller's %s, mesh-sdk's %s)", key, v, w))
}
}
for key := range right {
if _, both := left[key]; !both {
onlySDK = append(onlySDK, key)
}
}
sort.Strings(onlyOld)
sort.Strings(onlySDK)
sort.Strings(different)
var said []string
if len(onlyOld) > 0 {
said = append(said, "made only by the controller: "+strings.Join(onlyOld, ", "))
}
if len(onlySDK) > 0 {
said = append(said, "made only by mesh-sdk: "+strings.Join(onlySDK, ", "))
}
if len(different) > 0 {
said = append(said, "made by both with a different value: "+strings.Join(different, ", "))
}
if len(said) > 0 {
out = append(out, fmt.Sprintf("%s: %s differ: %s", comparison, f.name, strings.Join(said, "; ")))
}
}
return out
}
// entriesOf is the entries of a list by key, those of a machine in skip left out.
func entriesOf(lines []string, split func(string) (string, string), nodes func(string) []string, skip map[string]bool) map[string]string {
out := map[string]string{}
for _, l := range lines {
skipped := false
for _, n := range nodes(l) {
skipped = skipped || skip[n]
}
if skipped {
continue
}
k, v := split(l)
out[k] = v
}
return out
}
func splitAt(sep string) func(string) (string, string) {
return func(l string) (string, string) {
k, v, _ := strings.Cut(l, sep)
return k, v
}
}
func splitAtLast(sep string) func(string) (string, string) {
return func(l string) (string, string) {
if i := strings.LastIndex(l, sep); i >= 0 {
return l[:i], l[i+1:]
}
return l, ""
}
}
func firstWord(l string) []string {
f := strings.Fields(l)
if len(f) == 0 {
return nil
}
return f[:1]
}
// pairNodes are the consumer and the provider of a pair credential line ("name module on consumer from provider local=...").
func pairNodes(l string) []string {
f := strings.Fields(l)
var out []string
for i := 0; i+1 < len(f); i++ {
if f[i] == "on" || f[i] == "from" {
out = append(out, f[i+1])
}
}
return out
}
// 1384/AC3: the comparison itself. A composition that differs by one setting's worth is named, with its machine and the
// resource and field that first differ; two that agree, and two that refuse alike, are not differences. It failed first,
// before bothWaysDiffer read the body: a changed setting went unseen.
func TestTheComparisonNamesTheMachineWhoseCompositionDiffers(t *testing.T) {
body := func(port string) []byte {
raw, err := json.Marshal(map[string]any{"declaration": 1, "resources": []map[string]any{
{"id": "files.config", "type": "file", "content": "port=" + port},
{"id": "files.dir", "type": "directory"}}})
if err != nil {
t.Fatal(err)
}
return raw
}
same := composedBoth{Body: body("8080"), Foreseen: []string{"files/admin"}}
if got := bothWaysDiffer("laptop", "Reading", same, same, true); len(got) != 0 {
t.Fatalf("a composition differs from itself: %v", got)
}
changed := composedBoth{Body: body("8081"), Foreseen: []string{"files/admin"}}
got := bothWaysDiffer("laptop", "Reading", same, changed, true)
if len(got) != 1 || !strings.Contains(got[0], "laptop, Reading") ||
!strings.Contains(got[0], "resource files.config, field content") ||
!strings.Contains(got[0], "port=8080") || !strings.Contains(got[0], "port=8081") {
t.Fatalf("the changed setting is not named with its machine, resource and field: %v", got)
}
// A field of the lists.
other := composedBoth{Body: body("8080"), Foreseen: []string{"files/admin", "files/token"}}
if got := bothWaysDiffer("anchor", "Foreseeing", same, other, false); len(got) != 1 ||
!strings.Contains(got[0], "anchor, Foreseeing: the own secrets named foreseen differs") {
t.Fatalf("a secret named foreseen on one path alone is not named: %v", got)
}
// Refused alike is said, not failed; refused differently is a difference.
a, b := composedBoth{Err: "files on laptop needs a credential"}, composedBoth{Err: "files on laptop needs a credential"}
if got := bothWaysDiffer("laptop", "Foreseeing", a, b, true); len(got) != 0 {
t.Fatalf("two compositions that refuse a machine alike differ: %v", got)
}
if got := bothWaysDiffer("laptop", "Foreseeing", a, composedBoth{Body: body("1")}, true); len(got) != 1 ||
!strings.Contains(got[0], "the refusal differs") {
t.Fatalf("a machine one path refuses and the other composes is not named: %v", got)
}
// And what a send made: only the entries that differ, per machine, never the whole lists.
long := made{Ports: []string{"anchor files:80->8080", "anchor web:81->8081", "laptop db:5432->5432"},
OwnSecrets: []string{"anchor files/admin (made)", "laptop db/root (made)"}}
other2 := made{Ports: []string{"anchor files:80->8080", "anchor web:81->9091", "laptop db:5432->5432", "laptop cache:6379->6379"},
OwnSecrets: []string{"anchor files/admin (made)", "laptop db/root (accepted)"}}
got = madeDiffer("the send", long, other2, nil)
if len(got) != 2 || !strings.Contains(got[0], "made only by mesh-sdk: laptop cache:6379") ||
!strings.Contains(got[0], "anchor web:81 (the controller's 8081, mesh-sdk's 9091)") ||
strings.Contains(got[0], "anchor files:80") || strings.Contains(got[0], "laptop db:5432") ||
!strings.Contains(got[1], "laptop db/root (the controller's (made), mesh-sdk's (accepted))") ||
strings.Contains(got[1], "anchor files/admin") {
t.Fatalf("the send's differences are not only the differing entries: %v", got)
}
// A machine refused on either side is left out of both.
if got := madeDiffer("the send", long, other2, map[string]bool{"anchor": true, "laptop": true}); len(got) != 0 {
t.Fatalf("a refused machine is compared: %v", got)
}
}
cmd/mesh-controller/compose_fallbacks_test.go
-114
View File
@@ -1,114 +0,0 @@
package main
import (
"encoding/base64"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/broker"
)
// aCoreInput is an Input of one machine that takes a core module whose credential the vault has not boxed: the composition
// stands in a vault box for it, and the machine holds the controller's own seal of it.
func aCoreInput(t *testing.T) (*meshInput, string) {
t.Helper()
raw := make([]byte, 32)
raw[0] = 9
key := base64.StdEncoding.EncodeToString(raw)
client := catalogue.Manifest{Module: "mesh-provisioner-client", Version: "1",
Claims: []catalogue.Claim{{Name: broker.NodeProvisionerSeat, Scope: catalogue.ScopeNode}},
OwnSecrets: catalogue.OwnSecrets{broker.ProvisionerSecret: {Path: "/var/lib/mesh-provisioner-client/broker"}}}
return &meshInput{Input: compose.Input{
Catalogue: map[string]catalogue.Manifest{client.Module: client},
Nodes: []compose.Node{{Name: "laptop", Assigned: []string{client.Module}, SealingKey: key, Stated: true,
Contract: compose.VaultBoxContract}},
OwnSecrets: []compose.OwnSecretRow{{Node: "laptop", Module: client.Module, Name: broker.ProvisionerSecret,
Origin: "made", SealedTo: key, Sealed: "SEALED-BY-THE-CONTROLLER"}},
}}, client.Module
}
// 1385 (the lead's answer 1): where the vault is silent and no box is kept, the declaration carries the controller's own seal
// for the core credential, as ADR 0396 has it, and stands in for nothing. It failed first: the composition's declaration
// carried the stand-in box, and a send was refused.
func TestACoreCredentialTheVaultDidNotBoxIsSealedByTheController(t *testing.T) {
in, module := aCoreInput(t)
d, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
if len(d.CoreBoxesWanted) != 1 || len(stillForeseen(d)) == 0 {
t.Fatalf("the composition does not stand in for the vault's box: %v", d.CoreBoxesWanted)
}
sealed, _, err := sealedByTheController(in, "laptop", d, []string{module})
if err != nil {
t.Fatal(err)
}
if left := stillForeseen(sealed); len(left) != 0 {
t.Fatalf("the declaration still stands in for %v", left)
}
var file map[string]any
for _, r := range sealed.Resources {
if r["id"] == module+"."+catalogue.NeedID(broker.ProvisionerSecret) {
file = r
}
}
if file == nil || file["sealed"] != "SEALED-BY-THE-CONTROLLER" || file[catalogue.VaultSealedField] != nil {
t.Fatalf("the core credential is carried as %v: the controller's own seal, not the vault's box", file)
}
// And a module the vault did answer for is left as it is.
other, _, err := sealedByTheController(in, "laptop", d, []string{"another-module"})
if err != nil {
t.Fatal(err)
}
if len(stillForeseen(other)) == 0 {
t.Fatal("a module the vault answered for was sealed by the controller")
}
}
// 1383 (the lead's answer 4): the machine that runs the bus is refused where no user has a credential, with busUserList's
// words, and a machine that does not run the bus is not. It failed first: the Input stood a list in and the bus node composed.
func TestTheMachineThatRunsTheBusIsRefusedWhereNoUserHasACredential(t *testing.T) {
bus := catalogue.Manifest{Module: "nats", Version: "1", BusUsers: "/etc/nats/users.conf",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}}}
other := catalogue.Manifest{Module: "files", Version: "1"}
why := errors.New("this machine runs the bus and not one user has a credential, so the composed list would refuse every connection in the mesh")
in := &meshInput{busRefused: why, Input: compose.Input{
Catalogue: map[string]catalogue.Manifest{"nats": bus, "files": other},
Nodes: []compose.Node{{Name: "anchor", Assigned: []string{"nats"}}, {Name: "laptop", Assigned: []string{"files"}}},
}}
if err := refusedBus(in, "anchor"); err == nil || !strings.Contains(err.Error(), "not one user has a credential") {
t.Fatalf("the machine that runs the bus is not refused: %v", err)
}
if err := refusedBus(in, "laptop"); err != nil {
t.Fatalf("a machine that does not run the bus is refused: %v", err)
}
if err := refusedBus(&meshInput{Input: in.Input}, "anchor"); err != nil {
t.Fatalf("a bus with users is refused: %v", err)
}
}
// The controller's seal replaces the stand-in box whatever path the resource carries: the path is the manifest's with the
// machine's words filled in, not the manifest's own. It failed first, with the path compared (the live try: "composed twice and
// still names ... asked of the vault by the next send").
func TestTheControllersSealReplacesTheBoxWhateverPathTheFileCarries(t *testing.T) {
in, module := aCoreInput(t)
d, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
for _, r := range d.Resources {
if _, box := r[catalogue.VaultSealedField]; box {
r["path"] = "/home/someone/.var/lib/mesh-provisioner-client/broker"
}
}
sealed, _, err := sealedByTheController(in, "laptop", d, []string{module})
if err != nil {
t.Fatal(err)
}
if left := stillForeseen(sealed); len(left) != 0 {
t.Fatalf("the declaration still stands in for %v", left)
}
}
cmd/mesh-controller/compose_input.go
-436
View File
@@ -1,436 +0,0 @@
package main
import (
"context"
"os"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/inventory"
)
// The controller's adapter from its stores to mesh-sdk's composition input (novox/hq ADR 0453 clause 3, design 47 R4b,
// issues 1382 and 1383).
//
// **The adapter reads, and perform writes.** readComposeInput asks the stores once for everything the composition reads
// and returns compose.Input, which the composition never changes; it makes, seals, allocates and records nothing. A
// pair credential, a port, an own secret or a core box that the next send would make is not made here: the composition
// names it foreseen, and compose_perform.go makes it for a send, then reads the stores again and composes again. A
// question (status, facts, probes, the merge gate's reading) never performs.
//
// **The Input is built once per operation.** A verb that composes every node reads the stores once and hands the Input to
// each node's composition of it.
// composeInputBuilt is called once for each Input the adapter builds: the seam a test counts the reads of an operation by
// (issue 1382, criterion 5).
var composeInputBuilt = func() {}
// readComposeInput reads the stores once and returns what the composition needs. ctx carries what a send keeps (the
// recorded modules kept at the build their machine runs, recorded_kept.go): the same contexts every send composes under.
func readComposeInput(ctx context.Context, open *stores) (*meshInput, error) {
in, shape, err := readMeshAndNodes(ctx, open)
if err != nil {
return nil, err
}
// What each node resolves to, with no credential given (the settings are read before it): the modules whose settings are read, and the
// needs whose pair credentials are asked of the stores. Nothing is written by it.
unread, err := readSettings(ctx, open, &in)
if err != nil {
return nil, err
}
plans := map[string]compose.Resolved{}
for _, n := range in.Nodes {
if shape.refused[n.Name] != nil {
continue // refused for what it keeps: nothing of it is read further
}
resolved, err := compose.Resolve(in, n.Name)
switch {
case compose.IsUnresolvable(err):
// Their set does not compose: nothing of theirs is read further, as the controller never read a thing for a
// node that did not resolve.
continue
case err != nil:
return nil, err
}
plans[n.Name] = resolved
}
if err := unreadSettings(unread, plans); err != nil {
return nil, err
}
var busRefused error
wouldMake, err := readCredentialsAndBuilds(ctx, open, &in, plans, shape, &busRefused)
if err != nil {
return nil, err
}
composeInputBuilt()
return &meshInput{Input: in, wouldMake: wouldMake, busRefused: busRefused, refused: shape.refused}, nil
}
// meshInput is the Input an operation composes over, and the pair credentials its reading found none held for: the ones the
// next send makes, which perform makes.
type meshInput struct {
compose.Input
wouldMake []inventory.PairCredential
// busRefused is why the bus's user list cannot be composed (no user has a credential): the machine that runs the bus is
// refused with it, as the controller's own composition refused it, and no other machine is.
busRefused error
// refused is, by node, why that machine alone cannot be composed (nodeRefusal).
refused map[string]error
}
// nodeOf is the node of that name in an Input.
func nodeOf(in compose.Input, name string) (compose.Node, bool) {
for _, n := range in.Nodes {
if n.Name == name {
return n, true
}
}
return compose.Node{}, false
}
// inputShape is what the first half of the reading found that the second half needs.
type inputShape struct {
sent map[string]map[string]string // by node: module to the commit last sent
kept map[string]map[string]string // by node: module to the commit kept
current map[string]inventory.CurrentBuild
// refused is, by node, why a recorded module cannot be kept at the build the machine runs (keepRecorded's refusal, in
// its words): that machine alone is refused with it, as planFor refused it, and nothing else of the mesh is.
refused map[string]error
}
// readMeshAndNodes is the mesh-wide and per-node half of the Input (issue 1382).
func readMeshAndNodes(ctx context.Context, open *stores) (compose.Input, inputShape, error) {
inv := open.inventory
var in compose.Input
shape := inputShape{sent: map[string]map[string]string{}, kept: map[string]map[string]string{}, refused: map[string]error{}}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return in, shape, err
}
in.Catalogue = shelf
if in.Holdings, err = inv.Holdings(ctx); err != nil {
return in, shape, err
}
if in.Licences, err = readLicences(ctx, open); err != nil {
return in, shape, err
}
nodes, err := inv.Nodes(ctx)
if err != nil {
return in, shape, err
}
places, err := inv.Overlays(ctx)
if err != nil {
return in, shape, err
}
placed := map[string]inventory.Overlay{}
for _, p := range places {
placed[p.Name] = p
}
if shape.current, err = inv.CurrentBuilds(ctx); err != nil {
return in, shape, err
}
// The range the mesh allocates from and what its internal names end in, read from the environment here and nowhere else.
in.OverlayCIDR = strings.TrimSpace(os.Getenv(OverlayCIDRVar))
in.InternalSuffix = overlay.Suffix()
if in.Tunnels, err = readTunnels(ctx, inv); err != nil {
return in, shape, err
}
if in.Adopted, err = readAdopted(ctx, inv); err != nil {
return in, shape, err
}
carried, err := inv.CarriedPeers(ctx)
if err != nil {
return in, shape, err
}
for _, c := range carried {
in.CarriedPeers = append(in.CarriedPeers, compose.CarriedPeer{
PublicKey: c.PublicKey, Address: c.Address, EnrolledAs: c.EnrolledAs, Named: c.Named})
}
if in.Joining, err = inv.NodesWithALiveToken(ctx); err != nil {
return in, shape, err
}
keepsSecrets := false
for _, m := range shelf {
keepsSecrets = keepsSecrets || m.Keeps != ""
}
for _, n := range nodes {
var refusal error
node, err := readNode(ctx, open, n, placed[n.Name], &refusal)
if err != nil {
return in, shape, err
}
if refusal != nil {
shape.refused[n.Name] = refusal
}
shape.sent[n.Name], shape.kept[n.Name] = node.SentBuilds, node.KeptBuilds
in.Nodes = append(in.Nodes, node)
}
if keepsSecrets {
if in.OperatorExport, err = inv.OperatorExport(ctx); err != nil {
return in, shape, err
}
}
return in, shape, nil
}
// readNode is one node of the Input: everything the composition reads about it, in the order the controller read it.
func readNode(ctx context.Context, open *stores, n inventory.Node, place inventory.Overlay,
refused *error) (compose.Node, error) {
inv := open.inventory
out := compose.Node{
Name: n.Name,
Place: compose.Place{Key: place.Key, Endpoint: place.Endpoint, Site: place.Site, Hub: place.Hub,
Address: place.Address},
Account: n.Account, AccountHome: n.AccountHome, AgentAccount: n.AgentAccount, AgentAccountHome: n.AgentAccountHome,
Adopted: n.Adopted,
}
var err error
if out.Assigned, err = inv.Assigned(ctx, n.Name); err != nil {
return out, err
}
if out.Capabilities, err = inv.ProfileOf(ctx, n.Name); err != nil {
return out, err
}
if out.PublicDomain, err = inv.PublicDomainOf(ctx, n.Name); err != nil {
return out, err
}
if out.Pins, err = inv.PinsFor(ctx, n.Name); err != nil {
return out, err
}
if out.Bound, err = inv.BindingsFor(ctx, n.Name); err != nil {
return out, err
}
given, err := inv.PortsFor(ctx, n.Name)
if err != nil {
return out, err
}
for _, a := range given {
if out.Ports == nil {
out.Ports = map[string]map[int]int{}
}
if out.Ports[a.Module] == nil {
out.Ports[a.Module] = map[int]int{}
}
out.Ports[a.Module][a.Wanted] = a.Machine
}
if out.SealingKey, err = inv.SealingKeyOf(ctx, n.Name); err != nil {
return out, err
}
if n.Adopted {
if out.Taken, err = inv.Taken(ctx, n.Name); err != nil {
return out, err
}
}
if out.OutwardLinks, err = inv.OutwardLinksOf(ctx, n.Name); err != nil {
return out, err
}
account, had, err := inv.MachineOf(ctx, n.Name)
if err != nil {
return out, err
}
if had && account.Machine != nil {
out.NetworksSaid = true
for _, a := range account.Machine.Networks {
out.Networks = append(out.Networks, catalogue.AttachedNetwork{Link: a.Link, Network: a.Network})
}
}
if out.Sequence, err = inv.Sequence(ctx, n.ID); err != nil {
return out, err
}
if out.Epoch, err = inv.SentEpoch(ctx, n.ID); err != nil {
return out, err
}
if out.Generation, err = inv.SentGeneration(ctx, n.ID); err != nil {
return out, err
}
if out.ReadsModules, err = inv.ReadsModules(ctx, n.ID); err != nil {
return out, err
}
if out.SentBuilds, out.SentKnown, err = inv.SentBuilds(ctx, n.Name); err != nil {
return out, err
}
runs, err := inv.AssignmentRuns(ctx, n.Name)
if err != nil {
return out, err
}
if len(runs) > 0 {
out.Runs = make(map[string]compose.Run, len(runs))
for module, ref := range runs {
out.Runs[module] = compose.Run{Commit: ref.Commit, Build: ref.Build}
}
}
// A recorded module kept at the build this machine runs, in the context a send composes under (recorded_kept.go): its
// manifest and the commit, applied to this node's own resolution only.
// What cannot be kept refuses this machine alone (keepRecorded's refusal, as planFor returned it).
kept, err := recordedKept(ctx, open, n.Name)
if err != nil {
*refused = err
} else if len(kept) > 0 {
out.KeptBuilds = kept
if out.Kept, err = keptManifests(ctx, open, n.Name, kept); err != nil {
*refused = err
out.KeptBuilds, out.Kept = nil, nil
}
}
stated, has, err := inv.HealthOf(ctx, n.Name)
if err != nil {
return out, err
}
out.Stated = has
if has {
out.Contract = stated.Contract
}
return out, nil
}
// keptManifests is the manifest of each module kept at the build a machine runs: keepRecorded's reads, and its refusals in its
// words, without putting any of it into a shelf (that is the node's own Kept, applied by the composition).
func keptManifests(ctx context.Context, open *stores, node string, kept map[string]string) (map[string]catalogue.Manifest, error) {
// applyKept puts each manifest into the shelf it is given and refuses a build that cannot be read; the shelf here is a
// scratch one, whose entries are the kept manifests.
scratch := map[string]catalogue.Manifest{}
if err := applyKept(ctx, open, node, kept, scratch); err != nil {
return nil, err
}
return scratch, nil
}
func readTunnels(ctx context.Context, inv *inventory.Inventory) (map[string]compose.FoundTunnel, error) {
found, err := inv.Tunnels(ctx)
if err != nil {
return nil, err
}
if len(found) == 0 {
return nil, nil
}
out := make(map[string]compose.FoundTunnel, len(found))
for name, t := range found {
out[name] = compose.FoundTunnel{Tunnel: tunnelOf(t.Tunnel), NodeAdopted: t.NodeAdopted}
}
return out, nil
}
func readAdopted(ctx context.Context, inv *inventory.Inventory) (*compose.AdoptedTunnel, error) {
tunnel, hub, adopted, err := inv.AdoptedTunnel(ctx)
if err != nil {
return nil, err
}
if hub == "" {
return nil, nil
}
return &compose.AdoptedTunnel{Tunnel: tunnelOf(tunnel), Hub: hub, Adopted: adopted}, nil
}
func tunnelOf(t inventory.Tunnel) compose.Tunnel {
out := compose.Tunnel{Interface: t.Interface, Unit: t.Unit, Config: t.Config, Port: t.Port, MTU: t.MTU,
Address: t.Address, Range: t.Range, PublicKey: t.PublicKey}
for _, p := range t.Peers {
out.Peers = append(out.Peers, compose.TunnelPeer{PublicKey: p.PublicKey, Address: p.Address})
}
return out
}
// readLicences is every licence the mesh holds, with who uses it and the key sealed to each holder: given even where empty,
// so a holder the mesh holds no key for composes as that, and not as a key that was not asked for.
func readLicences(ctx context.Context, open *stores) ([]compose.Licence, error) {
held, err := open.Licences(ctx)
if err != nil {
return nil, err
}
all, err := held.All(ctx)
if err != nil {
return nil, err
}
var out []compose.Licence
for _, one := range all {
l := compose.Licence{Name: one.Name, Serves: one.Serves, Keys: map[compose.Holder]string{}}
holders, err := held.HoldersOf(ctx, one.Name)
if err != nil {
return nil, err
}
for _, h := range holders {
holder := compose.Holder{Node: h.Node, Module: h.Module}
l.Holders = append(l.Holders, holder)
key, err := held.KeyFor(ctx, one.Name, h.Node, h.Module)
if err != nil {
return nil, err
}
l.Keys[holder] = key
}
node, module, err := held.ManagerOf(ctx, one.Name)
if err != nil {
return nil, err
}
l.Manager = compose.Holder{Node: node, Module: module}
out = append(out, l)
}
return out, nil
}
// readSettings is each node's settings layers for every module that has any, read BEFORE any node is resolved: the first pass
// resolves a node whose set takes a provider's served facts from the node that runs it, with that node's settings on them
// (servedOnNode), so a mesh whose provider says ${setting:domain} resolves only with the settings in the Input. A module's
// settings that fail to read are returned in unread, by node and module, and are an error only for a module a node resolves to
// (unreadSettings): the controller read none for a module nothing runs.
func readSettings(ctx context.Context, open *stores, in *compose.Input) (unread map[string]map[string]error, err error) {
have, err := open.inventory.ModulesWithSettings(ctx)
if err != nil {
return nil, err
}
names := make([]string, 0, len(have))
for module := range have {
if _, onShelf := in.Catalogue[module]; onShelf {
names = append(names, module)
}
}
sort.Strings(names)
for i := range in.Nodes {
n := &in.Nodes[i]
for _, module := range names {
layers, err := open.inventory.SettingsFor(ctx, n.Name, module)
if err != nil {
if unread == nil {
unread = map[string]map[string]error{}
}
if unread[n.Name] == nil {
unread[n.Name] = map[string]error{}
}
unread[n.Name][module] = err
continue
}
if len(layers) == 0 {
continue
}
if n.Settings == nil {
n.Settings = map[string][]catalogue.Layer{}
}
n.Settings[module] = layers
}
}
return unread, nil
}
// unreadSettings is the error of a module's settings that did not read, for a module the node resolves to.
func unreadSettings(unread map[string]map[string]error, plans map[string]compose.Resolved) error {
names := make([]string, 0, len(plans))
for node := range plans {
names = append(names, node)
}
sort.Strings(names)
for _, node := range names {
for _, m := range plans[node].Plan.Modules {
if err := unread[node][m.Module]; err != nil {
return err
}
}
}
return nil
}
cmd/mesh-controller/compose_input_credentials.go
-285
View File
@@ -1,285 +0,0 @@
package main
import (
"context"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/inventory"
)
// The second half of the adapter (issue 1383): the credential and build fields of the Input, read and never made.
// readCredentialsAndBuilds fills what the first half left: the credentials, the own secrets, the vault's boxes, the bus,
// each node's serving key and certificate, and the builds.
func readCredentialsAndBuilds(ctx context.Context, open *stores, in *compose.Input, plans map[string]compose.Resolved,
shape inputShape, busRefused *error) ([]inventory.PairCredential, error) {
if err := readIdentity(ctx, open, in); err != nil {
return nil, err
}
wouldMake, err := readCredentials(ctx, open, in, plans, busRefused)
if err != nil {
return nil, err
}
return wouldMake, readBuilds(ctx, open, in, plans, shape)
}
// readIdentity is the mesh's authority and each node's serving key and the certificate kept for it, for a mesh in which a
// module asks for a certificate. It establishes and issues nothing: a certificate not kept yet is composed with a stand-in, and
// issued by the send (certificateFor).
func readIdentity(ctx context.Context, open *stores, in *compose.Input) error {
needs := false
for _, m := range in.Catalogue {
needs = needs || m.Certificate != nil
}
if !needs {
return nil
}
ident, err := open.Identity(ctx)
if err != nil {
return err
}
if certificate, known, err := ident.KnownAuthority(ctx); err != nil {
return err
} else if known {
in.Authority = certificate
}
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return err
}
ids := map[string]string{}
for _, n := range nodes {
ids[n.Name] = n.ID
}
for i := range in.Nodes {
n := &in.Nodes[i]
if n.ServingKey, err = ident.ServingKeyOf(ctx, ids[n.Name]); err != nil {
return err
}
if n.ServingKey == "" {
continue
}
if certificate, known, err := ident.KeptCertificate(ctx, overlay.InternalName(n.Name), n.ServingKey); err != nil {
return err
} else if known {
n.Certificate = certificate
}
}
return nil
}
// readCredentials is how each pair credential a node takes from elsewhere stands, the credentials made for each provider's
// consumers, the own secrets, the vault's boxes and the bus. It reads and never makes: a credential a send would make is
// left out (the composition's would-make), a credential only a person gives and nobody gave is not-given, and one the send
// would be refused is refused with the controller's words.
func readCredentials(ctx context.Context, open *stores, in *compose.Input, plans map[string]compose.Resolved,
busRefused *error) ([]inventory.PairCredential, error) {
var wouldMake []inventory.PairCredential
inv := open.inventory
shelf := in.Catalogue
nodeOf := map[string]compose.Node{}
for _, n := range in.Nodes {
nodeOf[n.Name] = n
}
names := make([]string, 0, len(plans))
for name := range plans {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
resolved := plans[name]
nodeShelf := shelfWith(shelf, nodeOf[name].Kept)
for _, need := range resolved.Plan.Needs {
if need.ByRecord {
continue
}
p := inventory.PairCredential{Name: need.Name, Consumer: name, ConsumerModule: need.For,
Provider: need.From, Local: need.Local}
if module, outside := issuedOutsideBy(nodeShelf, need); outside {
p.Kind, p.ProviderModule = inventory.PairOutside, module
} else if need.SharedOwn != "" {
p.Kind, p.Own = inventory.PairShared, need.SharedOwn
p.ProviderModule = sharedProviderModule(resolved.Plan, shelf, need)
}
state, err := inv.PairCredentialState(ctx, p)
if err != nil {
return nil, err
}
c := compose.Credential{Provision: need.Name, Consumer: name, ConsumerModule: need.For,
Provider: need.From, Local: need.Local}
switch state.State {
case inventory.PairHeld:
c.State, c.Sealed = compose.CredentialHeld, state.Sealed
case inventory.PairNotGiven:
c.State, c.Why = compose.CredentialNotGiven, state.Why
case inventory.PairRefused:
c.State, c.Why = compose.CredentialRefused, state.Why
default:
// Left out: a need with no entry is the credential the next send makes, and perform makes it.
wouldMake = append(wouldMake, p)
continue
}
in.Credentials = append(in.Credentials, c)
}
}
// The credentials made for each provider's consumers, so the grants it must create are read from them.
for _, n := range in.Nodes {
rows, err := inv.SecretsFrom(ctx, n.Name)
if err != nil {
return nil, err
}
for _, s := range rows {
in.Issued = append(in.Issued, compose.Issued{Provision: s.Name, Consumer: s.Consumer,
ConsumerModule: s.ConsumerModule, Local: s.Local, Provider: n.Name, ForProvider: s.ForProvider})
}
}
own, err := inv.OwnSecretRows(ctx)
if err != nil {
return nil, err
}
for _, r := range own {
in.OwnSecrets = append(in.OwnSecrets, compose.OwnSecretRow{Node: r.Node, Module: r.Module, Name: r.Name,
Origin: r.Origin, SealedTo: r.SealedTo, Sealed: r.Sealed})
}
for _, n := range in.Nodes {
boxes, err := inv.KeptCoreBoxes(ctx, n.Name)
if err != nil {
return nil, err
}
for module, byName := range boxes {
for name, b := range byName {
in.KeptCoreBoxes = append(in.KeptCoreBoxes, compose.CoreBox{Node: n.Name, Module: module, Name: name,
Generation: b.Generation, SealedTo: b.SealedTo, Box: b.Box})
}
}
}
sort.Slice(in.KeptCoreBoxes, func(i, j int) bool {
a, b := in.KeptCoreBoxes[i], in.KeptCoreBoxes[j]
if a.Node != b.Node {
return a.Node < b.Node
}
if a.Module != b.Module {
return a.Module < b.Module
}
return a.Name < b.Name
})
// The bus: its user list as the controller composes it, for the machine that runs it, and who the mesh has minted a
// credential for.
var bus []catalogue.Manifest
for _, m := range shelf {
if m.BusUsers != "" && m.ClaimsSeat("mesh-broker") {
bus = append(bus, m)
}
}
if len(bus) > 0 {
list, missing, err := busUserList(ctx, inv, bus)
if err != nil && !strings.Contains(err.Error(), "not one user has a credential") {
return nil, err
}
if err != nil {
// The machine that runs the bus is refused, as the controller's own composition refused it: a list that
// refuses every connection in the mesh is never composed (refusedBus); the other machines compose.
*busRefused = err
}
in.BusUsers, in.BusMissing = list, missing
}
if in.BusMinted, err = inv.BusMinted(ctx); err != nil {
return nil, err
}
if in.BusMemberships, err = inv.BusMemberships(ctx); err != nil {
return nil, err
}
return wouldMake, nil
}
// readBuilds is the build half of the Input: every repository the mesh built, each module's current build, and the build made
// from a commit of a module for the machines whose node-engine reads them.
func readBuilds(ctx context.Context, open *stores, in *compose.Input, plans map[string]compose.Resolved, shape inputShape) error {
inv := open.inventory
held, err := inv.Held(ctx)
if err != nil {
return err
}
for repository := range held {
in.Built = append(in.Built, repository)
}
sort.Strings(in.Built)
in.CurrentBuilds = make(map[string]string, len(shape.current))
for module, b := range shape.current {
in.CurrentBuilds[module] = b.Commit
}
// The build made from each commit a declaration may carry, for the machines that read a build per module.
for _, n := range in.Nodes {
if !n.ReadsModules {
continue
}
resolved, ok := plans[n.Name]
if !ok {
continue
}
for _, m := range resolved.Plan.Modules {
for _, commit := range []string{shape.current[m.Module].Commit, shape.kept[n.Name][m.Module], shape.sent[n.Name][m.Module]} {
if commit == "" {
continue
}
if _, done := in.BuildIDs[m.Module][commit]; done {
continue
}
build, err := inv.BuildOf(ctx, m.Module, commit)
if err != nil {
return err
}
if in.BuildIDs == nil {
in.BuildIDs = map[string]map[string]string{}
}
if in.BuildIDs[m.Module] == nil {
in.BuildIDs[m.Module] = map[string]string{}
}
in.BuildIDs[m.Module][commit] = build
}
}
}
return nil
}
// shelfWith is the shelf with each module a node keeps at the build it runs put in place of the current one: a copy.
func shelfWith(shelf map[string]catalogue.Manifest, kept map[string]catalogue.Manifest) map[string]catalogue.Manifest {
if len(kept) == 0 {
return shelf
}
out := make(map[string]catalogue.Manifest, len(shelf)+len(kept))
for name, m := range shelf {
out[name] = m
}
for name, m := range kept {
out[name] = m
}
return out
}
// sharedProviderModule is which module answers a shared credential on the providing node: the one in the plan, else the first
// the catalogue names (providerModuleOf, in an order that does not move between calls).
func sharedProviderModule(plan catalogue.Resolution, shelf map[string]catalogue.Manifest, n catalogue.Needed) string {
for _, m := range plan.Modules {
if _, shared := m.SharedCredentialOf(n.Name); shared {
return m.Module
}
}
names := make([]string, 0, len(shelf))
for name := range shelf {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
if _, shared := shelf[name].SharedCredentialOf(n.Name); shared {
return name
}
}
return ""
}
cmd/mesh-controller/compose_input_credentials_integration_test.go
-354
View File
@@ -1,354 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"fmt"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/inventory"
)
// tableState is every table's rows and every sequence's last value, as the adapter's reads must leave them.
func tableState(t *testing.T, inv *inventory.Inventory) map[string]string {
t.Helper()
out := map[string]string{}
for _, row := range inv.QueryForTest(t, `select table_name from information_schema.tables
where table_schema = 'public' and table_type = 'BASE TABLE' order by table_name`) {
name := strings.Trim(row[0], `"`)
count := inv.QueryForTest(t, fmt.Sprintf(`select count(*)::text from %q`, name))
out["rows:"+name] = fmt.Sprint(count)
digest := inv.QueryForTest(t, fmt.Sprintf(`select coalesce(md5(string_agg(t::text, '|' order by t::text)), '') from %q t`, name))
out["content:"+name] = fmt.Sprint(digest)
}
for _, row := range inv.QueryForTest(t, `select sequencename, coalesce(last_value::text, '') from pg_sequences
where schemaname = 'public' order by sequencename`) {
out["sequence:"+row[0]] = row[1]
}
return out
}
// 1382/AC4 and 1383/AC6: building the Input writes nothing. The store's rows, their contents and its sequences are the same
// before and after, on a mesh whose credentials are not made, whose ports are not given and which keeps a recorded build.
func TestBuildingTheInputWritesNothing(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
// A consumer whose credential nothing has made yet, on a mesh that already composed.
register(t, open, catalogue.Manifest{Module: "reports", Version: "1", Requires: []string{"s3-bucket"},
Resources: []map[string]any{{"id": "config", "type": "file", "path": "/etc/reports/config.json", "mode": "0600",
"content": "{}", "merge": "json"}}})
if _, err := inv.Assign(ctx, "laptop", "reports"); err != nil {
t.Fatal(err)
}
before := tableState(t, inv)
if _, err := readComposeInput(ctx, open); err != nil {
t.Fatal(err)
}
after := tableState(t, inv)
var moved []string
for k, v := range after {
if before[k] != v {
moved = append(moved, k)
}
}
sort.Strings(moved)
if len(moved) > 0 {
t.Fatalf("building the Input wrote: %v", moved)
}
}
// a consumer of the s3-bucket aMeshWithSecrets' provider offers, with a file the credential goes into.
func aConsumerOfTheBucket(name string) catalogue.Manifest {
return catalogue.Manifest{Module: name, Version: "1", Requires: []string{"s3-bucket"},
Secrets: map[string]string{"s3-bucket": "/etc/" + name + "/s3"},
Resources: []map[string]any{{"id": "config", "type": "file", "path": "/etc/" + name + "/config.json", "mode": "0600",
"content": "{}", "merge": "json"}}}
}
// 1383/AC1: of a provider's three consumers, two have their credentials made and one has not: Issued holds the two rows, with
// the copy sealed to the provider, the third is a credential the next send makes, and no row was inserted for it (planFor
// inserted it, plan.go: SecretFor in every mode).
func TestAPairCredentialNoRowHoldsIsAWouldMakeAndNoRowIsInserted(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
register(t, open, aConsumerOfTheBucket("reports"))
if _, err := inv.Assign(ctx, "laptop", "reports"); err != nil {
t.Fatal(err)
}
// The controller's own planning makes the credentials of files and reports.
if _, _, err := planFor(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
register(t, open, aConsumerOfTheBucket("metrics"))
if _, err := inv.Assign(ctx, "laptop", "metrics"); err != nil {
t.Fatal(err)
}
before := tableState(t, inv)
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
var rows []compose.Issued
for _, r := range in.Issued {
if r.Provider == "anchor" && r.Provision == "s3-bucket" {
rows = append(rows, r)
if r.ForProvider == "" {
t.Errorf("the row of %s carries no copy sealed to the provider", r.ConsumerModule)
}
}
}
if len(rows) != 2 {
t.Fatalf("Issued holds %d row(s) for anchor's s3-bucket, want files and reports: %+v", len(rows), rows)
}
var would []string
for _, p := range in.wouldMake {
would = append(would, p.ConsumerModule)
}
if len(would) != 1 || would[0] != "metrics" {
t.Fatalf("the credentials the next send makes are %v, want metrics", would)
}
for _, c := range in.Credentials {
if c.ConsumerModule == "metrics" {
t.Errorf("a credential nobody holds has an entry: %+v", c)
}
}
after := tableState(t, inv)
for k, v := range after {
if before[k] != v {
t.Errorf("reading the Input moved %s", k)
}
}
}
// 1383/AC2: a credential only a person gives that nobody gave is not given, and so is one given under a key the machine no
// longer has; a credential given by a person for the pair and sealed to a key one end no longer has is refused: in the words
// plan.go and secrets.go use.
func TestACredentialOnlyAPersonGivesIsNotGivenAndAGivenOneUnderAnOldKeyIsSaidSo(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: "indexers", Version: "1", Provides: []catalogue.Offer{{
Name: "indexer-api", Scope: catalogue.ScopeMesh, Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedOutside}}}})
register(t, open, catalogue.Manifest{Module: "grabber", Version: "1", Requires: []string{"indexer-api"},
Secrets: map[string]string{"indexer-api": "/etc/grabber/key"}})
register(t, open, aConsumerOfTheBucket("ledger"))
register(t, open, catalogue.Manifest{Module: "objects", Version: "1", Provides: catalogue.FromAnywhere("s3-bucket")})
for _, a := range [][2]string{{"anchor", "indexers"}, {"laptop", "grabber"}, {"anchor", "objects"}, {"laptop", "ledger"}} {
if _, err := inv.Assign(ctx, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
stateOf := func(provision, consumerModule string) (compose.Credential, bool) {
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
for _, c := range in.Credentials {
if c.Provision == provision && c.ConsumerModule == consumerModule {
return c, true
}
}
return compose.Credential{}, false
}
c, found := stateOf("indexer-api", "grabber")
if !found || c.State != compose.CredentialNotGiven || !strings.Contains(c.Why, "nobody gave it") {
t.Fatalf("a credential nobody gave reads as %+v", c)
}
if err := inv.AcceptSecretForPair(ctx, "indexer-api", "laptop", "grabber", "anchor", "", "the-key"); err != nil {
t.Fatal(err)
}
if c, found := stateOf("indexer-api", "grabber"); !found || c.State != compose.CredentialHeld || c.Sealed == "" {
t.Fatalf("a credential a person gave reads as %+v", c)
}
// The consumer's machine generated a key of its own since.
if err := inv.RecordSealingKey(ctx, nodeID(t, open, "laptop"), aPublicKey(t)); err != nil {
t.Fatal(err)
}
if c, found := stateOf("indexer-api", "grabber"); !found || c.State != compose.CredentialNotGiven ||
!strings.Contains(c.Why, "a sealing key one end no longer has") {
t.Fatalf("a credential given under a key the machine no longer has reads as %+v", c)
}
// A credential for a pair, given by a person, under a key one end has since replaced: the mesh cannot make another.
if err := inv.RecordSealingKey(ctx, nodeID(t, open, "laptop"), aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := inv.AcceptSecretForPair(ctx, "s3-bucket", "laptop", "ledger", "anchor", "", "the-bucket-key"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSealingKey(ctx, nodeID(t, open, "laptop"), aPublicKey(t)); err != nil {
t.Fatal(err)
}
if c, found := stateOf("s3-bucket", "ledger"); !found || c.State != compose.CredentialRefused ||
!strings.Contains(c.Why, "was accepted from a person, and a sealing key at one end has changed since") {
t.Fatalf("a credential given for the pair under an old key reads as %+v", c)
}
}
// 1383/AC3: the own secrets carry their origin, the key they are sealed to and the sealed value; the vault's kept boxes, the
// licences (with their keys given, so an empty key means the mesh holds none), the users the bus has minted and the
// memberships equal what the inventory answers.
func TestTheAdaptersInputCarriesTheOwnSecretsTheBoxesTheLicencesAndTheBus(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
key, err := inv.SealingKeyOf(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
sealed, err := inv.SecretForModule(ctx, "anchor", "objects", "admin")
if err != nil {
t.Fatal(err)
}
box := json.RawMessage(`{"v":1,"eph":"e","iv":"i","tag":"t","ct":"c"}`)
if err := inv.KeepCoreBox(ctx, "anchor", inventory.CoreBox{Module: "objects", Name: "admin", Generation: 4,
SealedTo: key, Box: box}); err != nil {
t.Fatal(err)
}
if err := inv.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.objects", Kind: "module", Node: "anchor",
Module: "objects"}, "a-password"); err != nil {
t.Fatal(err)
}
if err := inv.PutBusMembership(ctx, "laptop", "sealed-membership"); err != nil {
t.Fatal(err)
}
held, err := open.Licences(ctx)
if err != nil {
t.Fatal(err)
}
if err := held.Add(ctx, "claude", "static", map[string]any{"model": "x"}); err != nil {
t.Fatal(err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
var own *compose.OwnSecretRow
for i := range in.OwnSecrets {
if r := &in.OwnSecrets[i]; r.Node == "anchor" && r.Module == "objects" && r.Name == "admin" {
own = r
}
}
if own == nil || own.Origin != "made" || own.SealedTo != key || own.Sealed != sealed {
t.Errorf("the own secret reads as %+v; the inventory holds %q sealed to %q", own, sealed, key)
}
var kept *compose.CoreBox
for i := range in.KeptCoreBoxes {
if b := &in.KeptCoreBoxes[i]; b.Node == "anchor" && b.Module == "objects" && b.Name == "admin" {
kept = b
}
}
if kept == nil || kept.Generation != 4 || kept.SealedTo != key || string(kept.Box) != string(box) {
t.Errorf("the kept box reads as %+v", kept)
}
if got := strings.Join(in.BusMinted, ","); !strings.Contains(got, "anchor.objects") {
t.Errorf("BusMinted is %v", in.BusMinted)
}
if in.BusMemberships["laptop"] != "sealed-membership" {
t.Errorf("BusMemberships is %v", in.BusMemberships)
}
if len(in.Licences) != 1 || in.Licences[0].Name != "claude" || in.Licences[0].Keys == nil {
t.Errorf("Licences read as %+v", in.Licences)
}
}
// 1383/AC4: the bus's user list is what busUserList gives for the machine that runs the bus, and the Input carries none when
// no module of the shelf runs the bus.
func TestTheBusUsersAreWhatTheControllerComposesForTheMachineThatRunsTheBus(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
if in.BusUsers != "" || len(in.BusMissing) != 0 {
t.Fatalf("a mesh whose shelf runs no bus carries a bus user list: %q %v", in.BusUsers, in.BusMissing)
}
bus := catalogue.Manifest{Module: "nats", Version: "1", BusUsers: "/etc/nats/users.conf",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}}}
register(t, open, bus)
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "controller", Kind: "controller"}, "pw"); err != nil {
t.Fatal(err)
}
in, err = readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
list, missing, _ := busUserList(ctx, open.inventory, []catalogue.Manifest{bus})
if in.BusUsers != list || fmt.Sprint(in.BusMissing) != fmt.Sprint(missing) {
t.Errorf("the Input carries %q (%v); busUserList gives %q (%v)", in.BusUsers, in.BusMissing, list, missing)
}
}
// 1383/AC5: Built, CurrentBuilds and BuildIDs equal the inventory's Held, CurrentBuilds and BuildOf, and the serving key and
// the certificate are the machine's.
func TestTheAdaptersInputCarriesTheBuildsAndTheMachinesKey(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
held, err := inv.Held(ctx)
if err != nil {
t.Fatal(err)
}
var wantBuilt []string
for repository := range held {
wantBuilt = append(wantBuilt, repository)
}
sort.Strings(wantBuilt)
if fmt.Sprint(in.Built) != fmt.Sprint(wantBuilt) && !(len(in.Built) == 0 && len(wantBuilt) == 0) {
t.Errorf("Built is %v, the inventory's %v", in.Built, wantBuilt)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
t.Fatal(err)
}
for module, b := range current {
if in.CurrentBuilds[module] != b.Commit {
t.Errorf("CurrentBuilds[%s] is %q, the inventory's %q", module, in.CurrentBuilds[module], b.Commit)
}
}
for module, byCommit := range in.BuildIDs {
for commit, build := range byCommit {
if want, _ := inv.BuildOf(ctx, module, commit); want != build {
t.Errorf("BuildIDs[%s][%s] is %q, the inventory's %q", module, commit, build, want)
}
}
}
}
// The lead's answer 4: where the mesh holds no credential for any bus user, the machine that runs the bus is refused with
// busUserList's words (the controller's composition refused it), the Input still builds, and another machine composes.
func TestTheBusMachineIsRefusedWhereNoUserHasACredentialAndTheOthersCompose(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
bus := catalogue.Manifest{Module: "nats", Version: "1", BusUsers: "/etc/nats/users.conf",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}}}
register(t, open, bus)
register(t, open, aWebModule())
if _, err := assign(ctx, open, "anchor", "nats"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "web"); err != nil {
t.Fatal(err)
}
if _, _, err := composeQuestion(ctx, open, "anchor", compose.Foreseeing, false); err == nil ||
!strings.Contains(err.Error(), "not one user has a credential") {
t.Fatalf("the bus machine is composed: %v", err)
}
if _, _, err := composeQuestion(ctx, open, "laptop", compose.Foreseeing, false); err != nil {
t.Fatalf("a machine that does not run the bus is refused: %v", err)
}
}
cmd/mesh-controller/compose_input_integration_test.go
-329
View File
@@ -1,329 +0,0 @@
//go:build integration
package main
import (
"reflect"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/link"
)
// 1382: the adapter's Input names, for every node, what the inventory answers for it, each compared with the read named
// beside the field in go/compose/input.go (novox/hq ADR 0453 clause 3).
func TestTheAdaptersInputNamesWhatTheInventoryAnswersForEachNode(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
if err := inv.SetPublicDomain(ctx, "anchor", "example.test"); err != nil {
t.Fatal(err)
}
if _, err := inv.PortFor(ctx, "laptop", "files", 8080, false); err != nil {
t.Fatal(err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(in.Catalogue, shelf) {
t.Error("the Input's Catalogue is not the shelf the inventory holds")
}
nodes, err := inv.Nodes(ctx)
if err != nil {
t.Fatal(err)
}
if len(in.Nodes) != len(nodes) {
t.Fatalf("the Input names %d node(s), the inventory has %d", len(in.Nodes), len(nodes))
}
places, err := inv.Overlays(ctx)
if err != nil {
t.Fatal(err)
}
for _, record := range nodes {
n, ok := nodeOf(in.Input, record.Name)
if !ok {
t.Fatalf("%s is not in the Input", record.Name)
}
for _, p := range places {
if p.Name == record.Name {
if want := (compose.Place{Key: p.Key, Endpoint: p.Endpoint, Site: p.Site, Hub: p.Hub, Address: p.Address}); n.Place != want {
t.Errorf("%s: Place is %+v, the inventory's %+v", record.Name, n.Place, want)
}
}
}
assigned, _ := inv.Assigned(ctx, record.Name)
if !reflect.DeepEqual(n.Assigned, assigned) {
t.Errorf("%s: Assigned is %v, the inventory's %v", record.Name, n.Assigned, assigned)
}
caps, _ := inv.ProfileOf(ctx, record.Name)
if !reflect.DeepEqual(n.Capabilities, caps) {
t.Errorf("%s: Capabilities are %v, the inventory's %v", record.Name, n.Capabilities, caps)
}
domain, _ := inv.PublicDomainOf(ctx, record.Name)
if n.PublicDomain != domain {
t.Errorf("%s: PublicDomain is %q, the inventory's %q", record.Name, n.PublicDomain, domain)
}
if n.Account != record.Account || n.AccountHome != record.AccountHome || n.AgentAccount != record.AgentAccount ||
n.AgentAccountHome != record.AgentAccountHome || n.Adopted != record.Adopted {
t.Errorf("%s: the accounts or the mode differ from the inventory's: %+v", record.Name, n)
}
pins, _ := inv.PinsFor(ctx, record.Name)
if len(pins) != len(n.Pins) || (len(pins) > 0 && !reflect.DeepEqual(n.Pins, pins)) {
t.Errorf("%s: Pins are %v, the inventory's %v", record.Name, n.Pins, pins)
}
bound, _ := inv.BindingsFor(ctx, record.Name)
if len(bound) != len(n.Bound) || (len(bound) > 0 && !reflect.DeepEqual(n.Bound, bound)) {
t.Errorf("%s: Bound is %v, the inventory's %v", record.Name, n.Bound, bound)
}
key, _ := inv.SealingKeyOf(ctx, record.Name)
if n.SealingKey != key {
t.Errorf("%s: SealingKey is %q, the inventory's %q", record.Name, n.SealingKey, key)
}
ports, _ := inv.PortsFor(ctx, record.Name)
want := map[string]map[int]int{}
for _, a := range ports {
if want[a.Module] == nil {
want[a.Module] = map[int]int{}
}
want[a.Module][a.Wanted] = a.Machine
}
if len(want) != len(n.Ports) || (len(want) > 0 && !reflect.DeepEqual(n.Ports, want)) {
t.Errorf("%s: Ports are %v, the inventory's %v", record.Name, n.Ports, want)
}
// The settings of every module it resolves to, layer for layer.
resolved, err := compose.Resolve(in.Input, record.Name)
if err != nil {
continue
}
for _, m := range resolved.Plan.Modules {
layers, err := inv.SettingsFor(ctx, record.Name, m.Module)
if err != nil {
t.Fatal(err)
}
if len(layers) != len(n.Settings[m.Module]) || (len(layers) > 0 && !reflect.DeepEqual(n.Settings[m.Module], layers)) {
t.Errorf("%s: the settings of %s are %v, the inventory's %v", record.Name, m.Module, n.Settings[m.Module], layers)
}
}
}
if domain := nodeDomain(in.Input, "anchor"); domain != "example.test" {
t.Errorf("anchor's PublicDomain is %q", domain)
}
if got := nodePorts(in.Input, "laptop"); got["files"][8080] == 0 {
t.Errorf("laptop's port for files:8080 is not in the Input: %v", got)
}
}
func nodeDomain(in compose.Input, name string) string {
n, _ := nodeOf(in, name)
return n.PublicDomain
}
func nodePorts(in compose.Input, name string) map[string]map[int]int {
n, _ := nodeOf(in, name)
return n.Ports
}
// 1382/AC2: a node keeping a recorded module at the build it runs has that module in Kept and KeptBuilds, and the Input's
// Catalogue is the current shelf, not the kept manifest (R4a review round 1, finding 1).
func TestANodeKeepingARecordedBuildHasItInKeptAndTheCatalogueStaysCurrent(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
start := time.Now().Add(-time.Hour)
image := "registry.invalid:5000/postgres/server@sha256:" + strings.Repeat("a", 64)
for _, b := range []link.BuildResult{
aContainerBuild(t, "postgres", "c1111111", catalogue.PolicyRecord, start, map[string][2]string{"server": {image, ""}}),
} {
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, b)); err != nil {
t.Fatal(err)
}
}
if _, err := inv.Assign(ctx, "anchor", "postgres"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"postgres": "c1111111"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, inv, aContainerBuild(t, "postgres", "c2222222", catalogue.PolicyRecord, start.Add(time.Minute),
map[string][2]string{"server": {image, "runtime"}})); err != nil {
t.Fatal(err)
}
// A send that is not a person's push keeps the build the machine runs.
in, err := readComposeInput(keepingRecorded(ctx), open)
if err != nil {
t.Fatal(err)
}
anchor, _ := nodeOf(in.Input, "anchor")
if anchor.KeptBuilds["postgres"] != "c1111111" {
t.Fatalf("anchor keeps %v, want postgres at c1111111", anchor.KeptBuilds)
}
kept, found := anchor.Kept["postgres"]
if !found {
t.Fatalf("the kept manifest of postgres is not in anchor's Kept: %v", anchor.Kept)
}
current, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(in.Catalogue["postgres"], current["postgres"]) {
t.Error("the Input's Catalogue is not the current shelf")
}
if reflect.DeepEqual(kept, current["postgres"]) {
t.Error("the kept manifest is the current one: nothing is kept")
}
if laptop, _ := nodeOf(in.Input, "laptop"); len(laptop.Kept) != 0 || len(laptop.KeptBuilds) != 0 {
t.Errorf("laptop keeps what only anchor runs: %v", laptop.KeptBuilds)
}
// A person's push keeps nothing.
pushed, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
if a, _ := nodeOf(pushed.Input, "anchor"); len(a.Kept) != 0 || len(a.KeptBuilds) != 0 {
t.Errorf("a person's push keeps %v", a.KeptBuilds)
}
}
// 1382/AC3: the seats on record, the tunnels, the adopted tunnel, its carried peers and the nodes joining equal the
// inventory's answers; the range and the suffix are read from the environment once, here.
func TestTheAdaptersInputCarriesTheNetworkAsTheInventoryHoldsIt(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
t.Setenv(OverlayCIDRVar, "10.99.0.0/16")
t.Setenv("MESH_INTERNAL_SUFFIX", "mesh.test")
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
holdings, err := inv.Holdings(ctx)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(in.Holdings, holdings) {
t.Errorf("Holdings are %v, the inventory's %v", in.Holdings, holdings)
}
tunnels, err := inv.Tunnels(ctx)
if err != nil {
t.Fatal(err)
}
if len(tunnels) != len(in.Tunnels) {
t.Errorf("Tunnels are %v, the inventory's %v", in.Tunnels, tunnels)
}
if _, hub, _, err := inv.AdoptedTunnel(ctx); err != nil {
t.Fatal(err)
} else if (hub == "") != (in.Adopted == nil) {
t.Errorf("Adopted is %v, the inventory's hub is %q", in.Adopted, hub)
}
carried, err := inv.CarriedPeers(ctx)
if err != nil {
t.Fatal(err)
}
if len(carried) != len(in.CarriedPeers) {
t.Errorf("CarriedPeers are %v, the inventory's %v", in.CarriedPeers, carried)
}
joining, err := inv.NodesWithALiveToken(ctx)
if err != nil {
t.Fatal(err)
}
if len(joining) != len(in.Joining) {
t.Errorf("Joining is %v, the inventory's %v", in.Joining, joining)
}
if in.OverlayCIDR != "10.99.0.0/16" || in.InternalSuffix != "mesh.test" {
t.Errorf("the range and the suffix are %q and %q", in.OverlayCIDR, in.InternalSuffix)
}
// Read once: what the environment says afterwards moves nothing already read.
t.Setenv(OverlayCIDRVar, "10.1.0.0/16")
if in.OverlayCIDR != "10.99.0.0/16" {
t.Error("the range moved with the environment")
}
}
// The lead's equality finding (build 1792487777757012513): a provider on one machine serves a fact that says ${setting:domain},
// the setting is set for it, and a consumer on another machine takes that fact. The first pass of the adapter resolved every
// node with no settings in the Input, so the served fact could not be settled and the whole Input failed. The settings are in
// the Input before any node is resolved, and the Input builds, with the same plan for the consumer as the controller's own
// composition. It failed first, with the finding's words.
func TestAServedFactThatSaysASettingIsResolvedWithTheSettingInTheInput(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "step-ca", Version: "1",
Provides: []catalogue.Offer{{Name: "acme-ca", Scope: catalogue.ScopeMesh}},
Serves: map[string]map[string]any{"acme-ca": {"domain": "${setting:domain}"}}})
register(t, open, catalogue.Manifest{Module: "route-proxy", Version: "1", Requires: []string{"acme-ca"}})
// The setting is there before the machines are assigned: an assignment resolves the mesh, and a served fact that says
// ${setting:domain} resolves only with it.
if err := open.inventory.KeepSettings(ctx, "", "step-ca", map[string]any{"domain": "example.test"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "step-ca"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "route-proxy"); err != nil {
t.Fatal(err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatalf("the Input cannot be built: %v", err)
}
if anchor, _ := nodeOf(in.Input, "anchor"); len(anchor.Settings["step-ca"]) == 0 {
t.Fatalf("the Input does not carry the setting of the provider: %v", anchor.Settings)
}
oldPlan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
newPlan, _, err := resolveNode(withInputCache(ctx), open, "laptop")
if err != nil {
t.Fatalf("the consumer does not resolve: %v", err)
}
if !reflect.DeepEqual(oldPlan.Needs, newPlan.Needs) {
t.Errorf("the consumer's needs are %+v, the controller's own %+v", newPlan.Needs, oldPlan.Needs)
}
t.Logf("a served fact that says a setting resolves with the setting in the Input: the Input builds and the consumer's needs equal planFor's")
}
// A setting that nothing sets, in a fact a provider serves, is not the adapter's to hide: the controller's own composition
// refuses it as well (theRestOfTheMesh returns servedOnNode's error), and the adapter refuses with the same words rather than
// building an Input that composes a placeholder.
func TestAServedFactThatSaysAnUnsetSettingIsRefusedAsTheControllerRefusesIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "step-ca", Version: "1",
Provides: []catalogue.Offer{{Name: "acme-ca", Scope: catalogue.ScopeMesh}},
Serves: map[string]map[string]any{"acme-ca": {"domain": "${setting:domain}"}}})
register(t, open, catalogue.Manifest{Module: "route-proxy", Version: "1", Requires: []string{"acme-ca"}})
// Assigned with the setting there, then the setting is cleared: the mesh the controller holds when someone clears it.
if err := open.inventory.KeepSettings(ctx, "", "step-ca", map[string]any{"domain": "example.test"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "step-ca"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "route-proxy"); err != nil {
t.Fatal(err)
}
if err := open.inventory.ClearSettings(ctx, "", "step-ca"); err != nil {
t.Fatal(err)
}
_, _, oldErr := planFor(ctx, open, "laptop")
_, newErr := readComposeInput(ctx, open)
if (oldErr == nil) != (newErr == nil) {
t.Fatalf("the controller's composition says %v, the adapter says %v", oldErr, newErr)
}
if oldErr != nil && !strings.Contains(newErr.Error(), "nothing sets \"domain\"") {
t.Errorf("the adapter's words are %q, the controller's %q", newErr, oldErr)
}
t.Logf("an unset setting in a served fact is refused alike: the controller says %v, the adapter says %v", oldErr != nil, newErr != nil)
}
cmd/mesh-controller/compose_once_integration_test.go
-44
View File
@@ -1,44 +0,0 @@
//go:build integration
package main
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/compose"
)
// 1382/AC5: composing every node of one operation builds the Input once; without an operation's cache, once for each.
func TestComposingEveryNodeOfAnOperationBuildsTheInputOnce(t *testing.T) {
open, _ := aMeshWithSecrets(t)
built := 0
was := composeInputBuilt
composeInputBuilt = func() { built++ }
t.Cleanup(func() { composeInputBuilt = was })
ctx := withInputCache(t.Context())
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
t.Fatal(err)
}
for _, n := range nodes {
if _, _, err := resolveNode(ctx, open, n.Name); err != nil && !unresolvable(err) {
t.Fatal(err)
}
if _, _, err := composeQuestion(ctx, open, n.Name, compose.Reading, false); err != nil && !unresolvable(err) {
t.Fatal(err)
}
}
if _, err := generatorsOf(ctx, open); err != nil {
t.Fatal(err)
}
if built != 1 {
t.Fatalf("an operation over %d node(s) built the Input %d time(s)", len(nodes), built)
}
built = 0
if _, _, err := resolveNode(t.Context(), open, nodes[0].Name); err != nil && !unresolvable(err) {
t.Fatal(err)
}
if built != 1 {
t.Fatalf("a composition with no cache built the Input %d time(s)", built)
}
}
cmd/mesh-controller/compose_perform.go
-275
View File
@@ -1,275 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"io"
"os"
"slices"
"sort"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/inventory"
)
// perform makes what a composition named foreseen (novox/hq ADR 0453 clause 3, design 47 R4b, issue 1385).
//
// **The composition makes nothing; a send does.** mesh-sdk's composition over an Input names what the next send would make
// (the ports it would assign, the credentials it would seal, the core boxes it would ask the vault for, the seats it would
// pass on, the certificate it would issue) and stands in for each. A send composes, performs, reads the stores again and
// composes again (send in compose_sdk.go), and refuses to send a declaration that still names anything foreseen. A question
// (status, facts, probes, the merge gate's reading) never reaches this file.
//
// What is made is exactly what the controller's own composition made on the way to a declaration it was going to send:
// - every pair credential no row holds, for every machine that resolves, since planning one node resolved them all;
// - the seats the mesh passes on, with the hand act (not on a throwaway mesh, whose hand act has nowhere to go);
// - the machine ports the node's modules were given none for;
// - the node's own secrets, by the controller's own rules (a bus credential nobody issued was refused by the composition);
// - the core credentials, asked of the vault every send, so a rotated one reaches the machine;
// - the certificate of a machine that was never certified.
//
// perform returns whether it made or recorded anything: the reason the caller reads the stores again.
func perform(ctx context.Context, open *stores, in *meshInput, node string, d compose.Declared) (bool, error) {
did := false
inv := open.inventory
// The seats passed on: recorded in the store with the controller's own hand act (ADR 0449 §4, issue 1317).
if len(d.Passes.Passes) > 0 {
var w io.Writer = os.Stderr
if lw, ok := ctx.Value(leftOutKey{}).(io.Writer); ok {
w = lw
}
for _, line := range d.Passes.Said {
fmt.Fprintln(w, line)
}
sp := seatPassingOn(inv, w)
for _, p := range d.Passes.Passes {
moved, err := inv.PassSeat(ctx, p.Seat.Name, p.Seat.Scope, p.From, p.To)
if err != nil {
// Said, and the record stays as it was: a composition is never refused for it.
fmt.Fprintf(w, "%s could not be passed from %s on %s to %s on %s: %v\n", p.Seat.Name,
p.From.Module, p.From.Node, p.To.Module, p.To.Node, err)
continue
}
if !moved {
continue // another composition passed it first, and wrote the act
}
did = true
if inv.Throwaway() {
continue
}
if err := sp.act(ctx, p); err != nil {
fmt.Fprintf(w, "%s passed from %s on %s to %s on %s, and the hand act could NOT be recorded: %v\n",
p.Seat.Name, p.From.Module, p.From.Node, p.To.Module, p.To.Node, err)
}
}
}
// Every pair credential the stores hold none for: made once and kept, and sealed to both ends before it was ever written
// down. The error is the one planFor met: the machine that needs it, and why none could be made.
for _, p := range in.wouldMake {
if err := makePair(ctx, inv, p); err != nil {
return did, fmt.Errorf("%s on %s needs %s from %s and no credential could be made for it: %w",
p.ConsumerModule, p.Consumer, p.Name, p.Provider, err)
}
did = true
}
// The ports this node's modules were given none for.
if len(d.ForeseenPorts) > 0 {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return did, err
}
for _, one := range d.ForeseenPorts {
module, port, ok := strings.Cut(one, ":")
wanted, perr := strconv.Atoi(port)
if !ok || perr != nil {
return did, fmt.Errorf("the composition named a port the controller cannot read: %q", one)
}
fixed := false
for _, l := range nodeShelf(in, shelf, node)[module].Listens {
if l.Port == wanted {
fixed = l.Fixed
}
}
if _, err := inv.PortFor(ctx, node, module, wanted, fixed); err != nil {
return did, fmt.Errorf("%s needs %d reachable on %s and it could not be assigned: %w", module, wanted, node, err)
}
did = true
}
}
// The node's own secrets: a module's superuser password, an administrator, an account. The composition refused what the
// send refuses (a bus credential nobody issued, a given value under an old key).
for _, one := range d.StoodIn {
module, name, ok := strings.Cut(one, "/")
if !ok || strings.Contains(one, " ") {
continue // a grant's sealed copy, the certificate or the bus's list: made below or by nobody
}
if _, err := inv.SecretForModule(ctx, node, module, name); err != nil {
return did, err
}
did = true
}
// The certificate of a machine that was never certified: issued and kept, with the authority when there is none yet.
for _, one := range d.StoodIn {
if strings.HasPrefix(one, "the certificate and the authority of ") {
if _, _, err := certificateFor(ctx, open, node); err != nil {
return did, err
}
did = true
break
}
}
return did, nil
}
// makePair makes one pair credential by the controller's own read of it: the credential for the pair, or the provider's one
// credential sealed to this consumer too. A credential only a person gives is never made (it is not a would-make).
func makePair(ctx context.Context, inv *inventory.Inventory, p inventory.PairCredential) error {
var err error
switch p.Kind {
case inventory.PairShared:
_, err = inv.SharedSecretFor(ctx, p.Name, p.Consumer, p.ConsumerModule, p.Provider, p.ProviderModule, p.Local, p.Own)
case inventory.PairPlain:
_, err = inv.SecretFor(ctx, p.Name, p.Consumer, p.ConsumerModule, p.Provider, p.Local)
default:
return errors.New("a credential only a person gives is not made")
}
if inventory.IsNotGiven(err) {
return nil
}
return err
}
// stillForeseen says what a composed declaration still stands in for: the lists the composition names, and the stand-in itself
// anywhere in the body (a pair credential's stand-in is not in any list). A declaration that has any is never sent.
func stillForeseen(d compose.Declared) []string {
var left []string
left = append(left, d.Foreseen...)
left = append(left, d.StoodIn...)
for _, p := range d.ForeseenPorts {
left = append(left, "the port "+p)
}
for _, c := range d.CoreBoxesWanted {
left = append(left, "the vault's box for "+c.Module+"/"+c.Name)
}
for _, p := range d.Passes.Passes {
left = append(left, "the pass of "+p.Seat.Name+" to "+p.To.Module+" on "+p.To.Node)
}
if body, err := d.Body(); err == nil {
for _, marker := range []string{catalogue.ForeseenSealed, "asked of the vault by the next send", compose.ForeseenCertificate,
compose.ForeseenBusUsers} {
if strings.Contains(string(body), marker) {
left = append(left, "a stand-in: "+marker)
}
}
}
sort.Strings(left)
return slicesCompact(left)
}
func slicesCompact(in []string) []string {
var out []string
for _, s := range in {
if len(out) == 0 || out[len(out)-1] != s {
out = append(out, s)
}
}
return out
}
// nodeShelf is the shelf as one node composes its own set: the current one with each module it keeps at the build it runs put
// in place of the current.
func nodeShelf(in *meshInput, shelf map[string]catalogue.Manifest, node string) map[string]catalogue.Manifest {
n, _ := nodeOf(in.Input, node)
return shelfWith(shelf, n.Kept)
}
// prepareSend is what a send does before it composes, every send and whatever the composition would have named: the
// controller's own composition did it on the way, so a module's file that names an own secret composes (mesh-sdk's
// composition can stand in for none: a file naming a secret not made yet is refused in every mode), and the core's credentials
// are asked of the vault on every send, so a rotated one reaches the machine and a silent vault is said (core_box.go).
//
// - Each own secret of a module the node runs (not one handed to the provisioner, not one issued outside the mesh, not one a
// person gives) that no row holds is made, unless the composition would refuse it (a bus credential nobody issued): that
// refusal is the composition's to say, in its words.
// - The core's credentials are asked of the vault, the boxes newly answered are kept, and the modules the vault did not answer
// for with no box kept are returned: the controller seals those itself until the vault holds them (ADR 0396).
func prepareSend(ctx context.Context, open *stores, node string) ([]string, error) {
in, err := inputFor(ctx, open)
if err != nil {
return nil, err
}
if in.refused[node] != nil {
return nil, nil // empty-on-error: refused alone for what it keeps; the composition that follows says so, in its words
}
resolved, err := compose.Resolve(in.Input, node)
if err != nil {
return nil, nil // empty-on-error: it does not compose for its own reasons; the composition that follows says them
}
n, _ := nodeOf(in.Input, node)
hands := n.Stated && n.Contract >= compose.HandedContract
handing := catalogue.Rendering{Hands: hands, Provisioner: provisionerAccount(resolved.Plan)}
inv := open.inventory
did := false
if n.SealingKey != "" {
held := map[string]bool{}
for _, r := range in.OwnSecrets {
if r.Node == node {
held[r.Module+"/"+r.Name] = true
}
}
for _, m := range resolved.Plan.Modules {
if handing.Handing(m) {
continue
}
plain := m.OwnSecrets.Plain()
names := make([]string, 0, len(plain))
for name := range plain {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
if plain[name].IssuedBy == catalogue.IssuedOutside || held[m.Module+"/"+name] {
continue
}
switch name {
case "broker":
if !slices.Contains(in.BusMinted, node+"."+m.Module) {
continue
}
case compose.ProvisionerSecret:
if !slices.Contains(in.BusMinted, "provisioner."+node) {
continue
}
}
if _, err := inv.SecretForModule(ctx, node, m.Module, name); err == nil {
did = true
}
}
}
}
var byController []string
if n.Stated && n.Contract >= compose.VaultBoxContract {
boxes, err := coreBoxesFor(ctx, inv, node, resolved.Plan.Modules, func(m catalogue.Manifest) bool {
return handing.Handing(m)
}, Allocating)
if err != nil {
return nil, err
}
did = did || len(boxes.keep) > 0
byController = boxes.byController
}
if did {
// What was made is in the stores now: the composition reads them afresh.
forgetInput(ctx)
}
return byController, nil
}
cmd/mesh-controller/compose_perform_integration_test.go
-197
View File
@@ -1,197 +0,0 @@
//go:build integration
package main
import (
"errors"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/link"
)
func aWebModule() catalogue.Manifest {
return catalogue.Manifest{Module: "web", Version: "1",
Listens: []catalogue.Listening{{Port: 8080, From: catalogue.FromEverywhere}},
OwnSecrets: catalogue.OwnSecrets{"admin": {Path: "/etc/web/admin"}},
Resources: []map[string]any{
{"id": "server", "type": "container", "name": "web", "ports": []any{"8080"},
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)},
}}
}
// 1385/AC1: a port the composition names foreseen is allocated with the inventory's PortFor for that module and port, a second
// composition names it no more, and the number is the one the controller's own Allocating composition gives in the same store.
func TestPerformAllocatesAForeseenPortAndTheNumberIsTheControllers(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aWebModule())
if _, err := assign(ctx, open, "laptop", "web"); err != nil {
t.Fatal(err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
d, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
if len(d.ForeseenPorts) != 1 || d.ForeseenPorts[0] != "web:8080" {
t.Fatalf("the composition names the ports %v foreseen, want web:8080", d.ForeseenPorts)
}
did, err := perform(ctx, open, in, "laptop", d)
if err != nil || !did {
t.Fatalf("perform did %v: %v", did, err)
}
held, err := open.inventory.PortsFor(ctx, "laptop")
if err != nil || len(held) != 1 || held[0].Module != "web" || held[0].Wanted != 8080 {
t.Fatalf("the port perform allocated reads %+v: %v", held, err)
}
in, err = readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
if again, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing); err != nil || len(again.ForeseenPorts) != 0 {
t.Fatalf("a second composition names %v foreseen: %v", again.ForeseenPorts, err)
}
// The number is the one the controller allocates in the same store.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
old, err := declarationFor(ctx, open, "laptop", plan, settings)
if err != nil {
t.Fatal(err)
}
after, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
if got, want := portsOf(after.Resources), portsOf(old.Resources); got != want {
t.Fatalf("the declaration publishes %s; the controller's allocation publishes %s", got, want)
}
}
func portsOf(resources []map[string]any) string {
var out []string
for _, r := range resources {
if r["type"] == "container" {
out = append(out, fmt.Sprint(r["name"], r["ports"]))
}
}
return strings.Join(out, ";")
}
// 1385/AC2: an own secret named foreseen is made with SecretForModule, and a pair credential the Input names would-make is
// made as planFor chose; both exist after the send's composition and a second composition names neither.
func TestPerformMakesTheOwnSecretsAndThePairCredentialsTheCompositionNamed(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
register(t, open, aWebModule())
register(t, open, aConsumerOfTheBucket("reports"))
for _, m := range []string{"web", "reports"} {
if _, err := assign(ctx, open, "laptop", m); err != nil {
t.Fatal(err)
}
}
before, err := madeIn(ctx, open)
if err != nil {
t.Fatal(err)
}
declared, err := composeSend(ctx, open, "laptop")
if err != nil {
t.Fatalf("the send is refused: %v", err)
}
if len(declared.foreseen) != 0 {
t.Fatalf("the declaration still names %v foreseen", declared.foreseen)
}
after, err := madeIn(ctx, open)
if err != nil {
t.Fatal(err)
}
madeNow := func(have, was []string) []string {
seen := map[string]bool{}
for _, s := range was {
seen[s] = true
}
var out []string
for _, s := range have {
if !seen[s] {
out = append(out, s)
}
}
return out
}
if got := madeNow(after.OwnSecrets, before.OwnSecrets); len(got) != 1 || !strings.Contains(got[0], "web/admin") {
t.Errorf("the own secrets made are %v, want laptop web/admin", got)
}
if got := madeNow(after.Pairs, before.Pairs); len(got) != 1 || !strings.Contains(got[0], "reports") {
t.Errorf("the pair credentials made are %v, want the one of reports", got)
}
// And the credential is the one a second composition reads as held.
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
if len(in.wouldMake) != 0 {
t.Errorf("after the send the next send would still make %v", in.wouldMake)
}
if _, err := inv.SecretsFrom(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
// 1385/AC3 (the lead's answer 1): with the vault unreachable and no box of the credential kept, the send carries the
// controller's own seal for it (ADR 0396), says so as a condition, and keeps no box.
func TestAVaultThatIsSilentSealsTheSendByTheControllerWhereNoBoxIsKept(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
vault.silent = errors.New("no responders")
declared, err := composeSend(t.Context(), open, "laptop")
if err != nil {
t.Fatalf("the send is refused: %v", err)
}
f := clientsBroker(declared.Resources)
if f == nil || f["sealed"] == nil || f[catalogue.VaultSealedField] != nil {
t.Fatalf("the core credential is composed as %v: the controller's own seal, until the vault holds it", f)
}
if len(declared.foreseen) != 0 {
t.Fatalf("the declaration still names %v foreseen", declared.foreseen)
}
if c, found := vaultSilence(t, "laptop"); !found || !strings.Contains(c.Summary, "sealed by the controller until the vault holds it") {
t.Fatalf("no condition says why: %+v", c)
}
kept, err := open.inventory.KeptCoreBoxes(t.Context(), "laptop")
if err != nil {
t.Fatal(err)
}
if len(kept["mesh-provisioner-client"]) != 0 {
t.Fatalf("a box was kept for a credential the vault never boxed: %v", kept)
}
}
// 1385/AC6: perform is idempotent: a second send's composition over the stores it left makes and records nothing.
func TestAFinishedSendMakesNothingTheSecondTime(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
register(t, open, aWebModule())
if _, err := assign(ctx, open, "laptop", "web"); err != nil {
t.Fatal(err)
}
if _, err := composeSend(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
before := tableState(t, open.inventory)
if _, err := composeSend(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
for k, v := range tableState(t, open.inventory) {
if before[k] != v {
t.Errorf("a second send of the same machine moved %s", k)
}
}
}
cmd/mesh-controller/compose_sdk.go
-482
View File
@@ -1,482 +0,0 @@
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"os"
"slices"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/validate"
"github.com/novox/mesh-controller/internal/inventory"
)
// The controller's composition on mesh-sdk's (novox/hq ADR 0453 clause 3, design 47 R4b, issues 1386 to 1388): every
// caller that planned or composed a node reads the Input once for its operation (compose_input.go), asks the SDK to
// compose over it, and, when it sends, performs what the composition names foreseen (compose_perform.go) and composes again.
// inputCacheKey carries an operation's Input in its context, so the planning of every node of one verb reads the stores once.
type inputCacheKey struct{}
type inputCache struct {
mu sync.Mutex
by map[string]*meshInput
}
// withInputCache is ctx for an operation: the Input read for it is kept until a send changes what it read. A context that
// carries none reads the stores for each composition, which is correct and slow.
func withInputCache(ctx context.Context) context.Context {
if _, ok := ctx.Value(inputCacheKey{}).(*inputCache); ok {
return ctx
}
return context.WithValue(ctx, inputCacheKey{}, &inputCache{by: map[string]*meshInput{}})
}
// freshInput is ctx for one composition whose Input is read afresh and kept only until the composition makes something: a
// send that is composed after another one made and recorded things (a push's machines, one after the other).
func freshInput(ctx context.Context) context.Context {
return context.WithValue(ctx, inputCacheKey{}, &inputCache{by: map[string]*meshInput{}})
}
// keptSignature is what the Input read depends on in ctx: which recorded modules each machine keeps at the build it runs
// (recorded_kept.go), so an Input read for a person's push is never taken for one read for a gated send.
func keptSignature(ctx context.Context) string {
skip, on := keptExcept(ctx)
var names []string
for m := range skip {
names = append(names, m)
}
sort.Strings(names)
every, _ := ctx.Value(keepEveryBuildKey{}).(string)
settings, _ := ctx.Value(keepForSettingsKey{}).(string)
return fmt.Sprintf("kept=%v;except=%v;every=%s;settings=%s", on, names, every, settings)
}
// inputFor is the Input for ctx's operation.
func inputFor(ctx context.Context, open *stores) (*meshInput, error) {
cache, ok := ctx.Value(inputCacheKey{}).(*inputCache)
if !ok {
return readComposeInput(ctx, open)
}
sig := keptSignature(ctx)
cache.mu.Lock()
defer cache.mu.Unlock()
if in := cache.by[sig]; in != nil {
return in, nil
}
in, err := readComposeInput(ctx, open)
if err != nil {
return nil, err
}
cache.by[sig] = in
return in, nil
}
// forgetInput drops what the operation read, because a send made or recorded something it read.
func forgetInput(ctx context.Context) {
if cache, ok := ctx.Value(inputCacheKey{}).(*inputCache); ok {
cache.mu.Lock()
cache.by = map[string]*meshInput{}
cache.mu.Unlock()
}
}
// resolveNode works out everything a node should run, from what was assigned to it (planFor on the SDK): over the Input,
// writing nothing. A failure to compose the node's own modules is marked notResolvable, as planFor marked it; every other
// failure is returned as it is.
func resolveNode(ctx context.Context, open *stores, node string) (catalogue.Resolution, catalogue.SettingsBy, error) {
in, err := inputFor(ctx, open)
if err != nil {
return catalogue.Resolution{}, nil, err
}
if err := in.refused[node]; err != nil {
return catalogue.Resolution{}, nil, err
}
resolved, err := compose.Resolve(in.Input, node)
if err != nil {
if compose.IsUnresolvable(err) {
return catalogue.Resolution{}, nil, notResolvable{err}
}
return catalogue.Resolution{}, nil, err
}
logUnheld(node, resolved.Plan.Unheld)
return resolved.Plan, resolved.Settings, nil
}
// generatorsOf is what every node's declaration is composed with, over the whole mesh (generators on the SDK).
func generatorsOf(ctx context.Context, open *stores) (map[string]catalogue.Generator, error) {
in, err := inputFor(ctx, open)
if err != nil {
return nil, err
}
return compose.Generators(in.Input)
}
// whoResolvesOn is the machines whose resolution answers a requirement, and why the others did not (whoResolves over the
// Input): by what a module provides, not by its name. A machine whose set does not compose is in refused, said; one the mesh
// could not be asked about is an error, never a refusal (novox/hq 04-ISSUES/152).
func whoResolvesOn(ctx context.Context, open *stores, requirement string) (map[string]bool, map[string]string, error) {
in, err := inputFor(ctx, open)
if err != nil {
return nil, nil, err
}
on := map[string]bool{}
refused := map[string]string{}
for _, n := range in.sortedNodes() {
if err := in.refused[n.Name]; err != nil {
return nil, nil, fmt.Errorf("whether %s answers %q cannot be read: %w", n.Name, requirement, err)
}
resolved, err := compose.Resolve(in.Input, n.Name)
switch {
case compose.IsUnresolvable(err):
refused[n.Name] = err.Error()
continue
case err != nil:
return nil, nil, fmt.Errorf("whether %s answers %q cannot be read: %w", n.Name, requirement, err)
}
for _, m := range resolved.Plan.Modules {
for _, offered := range m.Offers() {
if offered == requirement {
on[n.Name] = true
}
}
}
}
return on, refused, nil
}
// sortedNodes is the Input's nodes by name, the order the controller reads them in.
func (in *meshInput) sortedNodes() []compose.Node {
out := append([]compose.Node(nil), in.Nodes...)
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
return out
}
// composeQuestion composes a node's declaration as a question asks it: nothing is made, and what a send would make is a
// stand-in (Foreseeing) or left out (Reading). It also answers what the node-engine's validator refuses of it. The lines the
// composition found are said where the controller says its own (standard error, or the writer an act gave).
//
// withOrder says the declaration carries the order it was last sent under (composedAndValidated's, which a comparison with the
// last send needs); without it the order is none, as declarationWith left it for the sender to stamp.
func composeQuestion(ctx context.Context, open *stores, node string, mode compose.Mode, withOrder bool) (sendable, []string, error) {
in, err := inputFor(ctx, open)
if err != nil {
return sendable{}, nil, err
}
if err := in.refused[node]; err != nil {
return sendable{}, nil, err
}
d, problems, err := compose.ComposeIn(in.Input, node, mode)
if err != nil {
return sendable{}, nil, markUnresolvable(err)
}
if err := refusedBus(in, node); err != nil {
return sendable{}, nil, err
}
sayFound(ctx, d)
out := sendableOf(d)
if !withOrder {
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
}
return out, problems, nil
}
// refusedBus is why the node cannot be composed because it runs the bus and the bus's user list cannot be: no user has a
// credential, so the list would refuse every connection in the mesh (busUserList). The controller's own composition refused that
// machine with those words; no other machine is refused for it. nil when nothing refuses the node.
func refusedBus(in *meshInput, node string) error {
if in.busRefused == nil {
return nil
}
resolved, err := compose.Resolve(in.Input, node)
if err != nil {
return nil // it does not compose for its own reasons, which are said
}
for _, m := range resolved.Plan.Modules {
if m.BusUsers != "" && m.ClaimsSeat("mesh-broker") {
return in.busRefused
}
}
return nil
}
// markUnresolvable is an error of a node whose own set does not compose, as planFor marked it.
func markUnresolvable(err error) error {
if compose.IsUnresolvable(err) {
return notResolvable{err}
}
return err
}
// composeSend composes a node's declaration to send it, as the Allocating composition did: composed over what is read,
// then what it named foreseen is made (perform), the stores are read again and it is composed again. A declaration that
// still names anything foreseen is refused, said, and never sent.
func composeSend(ctx context.Context, open *stores, node string) (sendable, error) {
declared, _, err := composeSendValidated(ctx, open, node)
return declared, err
}
// composeSendValidated is composeSend and what the node-engine's validator refuses of the declaration it composed, for the
// merge gate's two passes (composedAndValidated in its Allocating mode).
func composeSendValidated(ctx context.Context, open *stores, node string) (sendable, []string, error) {
// The core modules whose credentials the controller sealed itself because the vault was silent and no box was kept
// (ADR 0396): the composition stands in a vault box for each, and a send carries the controller's own seal instead.
byController, err := prepareSend(ctx, open, node)
if err != nil {
return sendable{}, nil, err
}
for attempt := 0; ; attempt++ {
in, err := inputFor(ctx, open)
if err != nil {
return sendable{}, nil, err
}
if err := in.refused[node]; err != nil {
return sendable{}, nil, err
}
d, problems, err := compose.ComposeIn(in.Input, node, compose.Foreseeing)
if err != nil {
return sendable{}, nil, markUnresolvable(err)
}
if err := refusedBus(in, node); err != nil {
return sendable{}, nil, err
}
if len(byController) > 0 {
if d, problems, err = sealedByTheController(in, node, d, byController); err != nil {
return sendable{}, nil, err
}
}
left := stillForeseen(d)
resolved, rerr := compose.Resolve(in.Input, node)
if rerr != nil {
return sendable{}, nil, markUnresolvable(rerr)
}
if len(resolved.Foreseen) > 0 {
left = append(left, resolved.Foreseen...)
}
if len(left) == 0 && len(in.wouldMake) == 0 {
sayFound(ctx, d)
out := sendableOf(d)
// The sender stamps the order it allots (order.stamp); a composition for a send carries none of its own.
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
here, _ := nodeOf(in.Input, node)
out.Builds = buildsOf(in.Input, here, resolved.Plan, d.LeftOutWhy)
out.Bindings = boundToData(resolved.Plan, d.LeftOutWhy)
if len(d.Filled) > 0 {
if out.plans, err = handedPlans(in, node, resolved); err != nil {
return sendable{}, nil, err
}
}
return out, problems, nil
}
if attempt > 0 {
return sendable{}, nil, fmt.Errorf("%s was composed twice and still names what only a send makes (%s): nothing is sent "+
"that stands in for a value", node, strings.Join(left, "; "))
}
if _, err := perform(ctx, open, in, node, d); err != nil {
return sendable{}, nil, err
}
// What was made is in the stores now: the next read is a fresh one.
forgetInput(ctx)
}
}
// restOfTheMesh is what every other machine holds and offers (theRestOfTheMesh over the Input), for a question: the lines it
// finds are said where the controller says its own, and a seat it would pass on is passed by the next send (perform), not by
// the question. shelf, when given, replaces the Input's catalogue: a caller composing a change's shelf.
func restOfTheMesh(ctx context.Context, open *stores, shelf map[string]catalogue.Manifest, exclude string) (catalogue.World, error) {
in, err := inputFor(ctx, open)
if err != nil {
return catalogue.World{}, err
}
input := in.Input
if shelf != nil {
input.Catalogue = shelf
}
world, passes, left, err := compose.TheRestOfTheMesh(input, exclude)
if err != nil {
return catalogue.World{}, err
}
var w io.Writer = os.Stderr
if lw, ok := ctx.Value(leftOutKey{}).(io.Writer); ok {
w = lw
}
for _, l := range left {
fmt.Fprintln(w, l.String())
}
for _, line := range passes.Said {
fmt.Fprintln(w, line)
}
return world, nil
}
// restOfTheMeshOf is restOfTheMesh for a caller that holds the inventory alone: the other stores the Input reads are opened for
// the call and let go of again.
func restOfTheMeshOf(ctx context.Context, inv *inventory.Inventory, shelf map[string]catalogue.Manifest, exclude string) (catalogue.World, error) {
open := &stores{inventory: inv}
defer func() {
if open.licences != nil {
open.licences.Close()
}
if open.identity != nil {
open.identity.Close()
}
}()
return restOfTheMesh(ctx, open, shelf, exclude)
}
// sayFound says the lines a composition found while composing, once each: a machine left out of the rest of the mesh, a seat
// that two claimants could take, the bus's users left out for want of a credential.
func sayFound(ctx context.Context, d compose.Declared) {
if len(d.Said) == 0 {
return
}
var w io.Writer = os.Stderr
if lw, ok := ctx.Value(leftOutKey{}).(io.Writer); ok {
w = lw
}
for _, line := range d.Said {
fmt.Fprintln(w, line)
}
}
// sendableOf is a composed declaration as the controller's sends know it.
func sendableOf(d compose.Declared) sendable {
out := sendable{Resources: d.Resources, Sequence: d.Sequence, Epoch: d.Epoch, Generation: d.Generation,
ReadsModules: d.ReadsModules, moduleOf: d.ModuleOf, buildIDs: d.BuildIDs, modules: d.Modules,
localNetwork: d.LocalNetwork, Received: d.Received, Mesh: d.Mesh, BusUsers: d.BusUsers,
LeftOut: d.LeftOut, leftOutWhy: d.LeftOutWhy, withheld: d.Withheld, unbound: d.Unbound, foreseen: d.Foreseen,
unplaced: d.Unplaced, handedOnly: d.HandedOnly, Filled: d.Filled, Provisioner: d.Provisioner,
requiredBy: d.RequiredBy}
if d.Adoption != nil {
out.Adoption = &adoptionEnvelope{Taken: d.Adoption.Taken, Untaken: d.Adoption.Untaken}
}
return out
}
// buildsOf is the build of each module a declaration for the node carries, as a send records it (novox/hq issue 259): the
// module's current build for each module in it, the build it was last sent for one left out of it, and a module kept at the
// build the machine runs carried at that one (ADR 0245). Never nil.
func buildsOf(in compose.Input, node compose.Node, plan catalogue.Resolution, leftOut map[string]string) map[string]string {
before := node.SentBuilds
if !node.SentKnown {
before = nil
}
out := map[string]string{}
for _, m := range plan.Modules {
if _, left := leftOut[m.Module]; left {
if was, kept := before[m.Module]; kept {
out[m.Module] = was
}
continue
}
out[m.Module] = in.CurrentBuilds[m.Module]
}
for m, was := range node.KeptBuilds {
if _, carried := out[m]; carried {
out[m] = was
}
}
return out
}
// handedPlans is the plan of each module handed to the provisioner, which the assignments mirror keeps with the send (design
// 57 P6): the composition's declaration does not carry it, so it is composed from the same rendering again.
func handedPlans(in *meshInput, node string, resolved compose.Resolved) (map[string]catalogue.Plan, error) {
rendered, err := compose.Render(in.Input, node, compose.Foreseeing)
if err != nil {
return nil, err
}
gens, err := compose.Generators(in.Input)
if err != nil {
return nil, err
}
with := rendered.Rendering
with.Generators = gens
composed, err := resolved.Plan.Compose(with)
if err != nil {
return nil, err
}
return composed.Plans, nil
}
// sealedByTheController is the declaration with the controller's own seal in place of the vault's box for each core module the
// vault did not answer for and no box was kept for (core_box.go, ADR 0396): the file carries "sealed", as the controller's
// composition sent it, until the vault holds the credential. What the composition stood in for the box is replaced and no longer
// wanted; the validator is run again over the body.
func sealedByTheController(in *meshInput, node string, d compose.Declared, modules []string) (compose.Declared, []string, error) {
rendered, err := compose.Render(in.Input, node, compose.Foreseeing)
if err != nil {
return d, nil, err
}
stood, _ := compactBox(catalogue.ForeseenVaultBox)
var wanted []compose.CoreBoxWanted
var foreseen []string
for _, w := range d.CoreBoxesWanted {
if !slices.Contains(modules, w.Module) {
wanted = append(wanted, w)
continue
}
sealed := rendered.Rendering.Needed[w.Module][w.Name]
if sealed == "" {
return d, nil, fmt.Errorf("%s/%s on %s has no seal of the controller's to carry until the vault holds it", w.Module, w.Name, node)
}
for _, r := range d.Resources {
box, isBox := r[catalogue.VaultSealedField]
// By the resource's id alone: its path is the manifest's with the machine's words filled in, and so is not the
// manifest's own.
if !isBox || r["id"] != w.Module+"."+catalogue.NeedID(w.Name) {
continue
}
if had, err := compactBox(box); err != nil || had != stood {
continue
}
delete(r, catalogue.VaultSealedField)
r["sealed"] = sealed
}
}
// A stand-in box that is left is said by its resource, never sent and never guessed at.
for _, r := range d.Resources {
if box, has := r[catalogue.VaultSealedField]; has {
if had, err := compactBox(box); err == nil && had == stood {
if id, _ := r["id"].(string); slices.ContainsFunc(modules, func(m string) bool { return strings.HasPrefix(id, m+".") }) {
return d, nil, fmt.Errorf("%s on %s still carries a stand-in for the vault's box, and the controller has "+
"no seal of its own for it", id, node)
}
}
}
}
d.CoreBoxesWanted = wanted
for _, f := range d.Foreseen {
module, _, _ := strings.Cut(f, "/")
if !slices.Contains(modules, module) {
foreseen = append(foreseen, f)
}
}
d.Foreseen = foreseen
body, err := d.Body()
if err != nil {
return d, nil, err
}
return d, validate.Declaration(body), nil
}
// compactBox is a value as the compact JSON it marshals to, so a box is compared by what it says and not by the type that
// holds it.
func compactBox(v any) (string, error) {
raw, err := json.Marshal(v)
if err != nil {
return "", err
}
var out bytes.Buffer
if err := json.Compact(&out, raw); err != nil {
return "", err
}
return out.String(), nil
}
cmd/mesh-controller/compose_sdk_test.go
-123
View File
@@ -1,123 +0,0 @@
package main
import (
"context"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/inventory"
)
// A declaration as the SDK composes it is the one the controller sends, byte for byte, whatever it carries: the body is the
// envelope's only marshaller on either side (design 47 R4b, issue 1386).
func TestASendableIsTheComposedDeclarationByteForByte(t *testing.T) {
d := compose.Declared{
Resources: []map[string]any{{"id": "files.config", "type": "file"}, {"id": "mesh.hosts", "type": "file"}},
Sequence: 7, Epoch: 3, Generation: 9, ReadsModules: true,
ModuleOf: map[string]string{"files.config": "files"}, BuildIDs: map[string]string{"files": "b-1"},
Adoption: &compose.Adoption{Taken: []string{"files"}, Untaken: map[string][]string{"nats": {"nats.server"}}},
Filled: []string{"/etc/files/filled.json"}, Provisioner: "mesh-provisioner",
LeftOut: []string{"cache"}, LeftOutWhy: map[string]string{"cache": "a setting cannot compose"},
}
want, err := d.Body()
if err != nil {
t.Fatal(err)
}
got, err := sendableOf(d).Body()
if err != nil {
t.Fatal(err)
}
if string(got) != string(want) {
t.Fatalf("the controller would send\n%s\nand the composition made\n%s", got, want)
}
if empty, err := sendableOf(compose.Declared{}).Body(); err != nil || !strings.Contains(string(empty), `"owns_nothing":true`) {
t.Fatalf("an empty declaration is not deliberate: %s %v", empty, err)
}
}
// 1386: what a send records as carried is what the controller's own carriedOn recorded.
func TestWhatASendRecordsAsCarriedIsWhatTheControllerRecorded(t *testing.T) {
plan := catalogue.Resolution{Modules: []catalogue.Manifest{{Module: "postgres"}, {Module: "files"}, {Module: "cache"}}}
current := map[string]inventory.CurrentBuild{"postgres": {Commit: "p2"}, "files": {Commit: "f2"}, "cache": {Commit: "c2"}}
leftOut := map[string]string{"cache": "a setting cannot compose"}
for name, c := range map[string]struct {
sent map[string]string
known bool
kept map[string]string
}{
"nothing sent": {},
"sent and kept": {sent: map[string]string{"postgres": "p1", "cache": "c1", "files": "f1"}, known: true, kept: map[string]string{"postgres": "p1"}},
"sent but not known": {sent: map[string]string{"cache": "c1"}},
"kept a module not in": {known: true, kept: map[string]string{"mail": "m1"}},
} {
var names []string
for _, m := range plan.Modules {
names = append(names, m.Module)
}
before := c.sent
if !c.known {
before = nil
}
want := carriedBuilds(names, leftOut, current, before)
for m, was := range c.kept {
if _, carried := want[m]; carried {
want[m] = was
}
}
in := compose.Input{CurrentBuilds: map[string]string{"postgres": "p2", "files": "f2", "cache": "c2"}}
node := compose.Node{SentBuilds: c.sent, SentKnown: c.known, KeptBuilds: c.kept}
if got := buildsOf(in, node, plan, leftOut); !reflect.DeepEqual(got, want) {
t.Errorf("%s: a send records %v, the controller recorded %v", name, got, want)
}
}
}
// 1385: a declaration that stands in for a value anywhere is still foreseen, in a list the composition names or in the body
// alone (a pair credential's stand-in is in no list); one that stands in for nothing is not. It failed first, before the body
// was scanned.
func TestADeclarationThatStandsInForAValueIsStillForeseen(t *testing.T) {
clean := compose.Declared{Resources: []map[string]any{{"id": "files.config", "type": "file", "content": "ok"}}}
if left := stillForeseen(clean); len(left) != 0 {
t.Fatalf("a declaration that stands in for nothing is foreseen: %v", left)
}
for name, d := range map[string]compose.Declared{
"an own secret": {StoodIn: []string{"files/admin"}},
"a port": {ForeseenPorts: []string{"files:80"}},
"a core box": {CoreBoxesWanted: []compose.CoreBoxWanted{{Node: "anchor", Module: "mesh-vault", Name: "root"}}},
"a pass of a seat": {Passes: compose.Passes{Passes: []catalogue.SeatPass{{Seat: catalogue.Seat{Name: "mesh-store"}}}}},
"a pair credential": {Resources: []map[string]any{{"id": "files.needs-db", "sealed": catalogue.ForeseenSealed}}},
"the certificate": {Resources: []map[string]any{{"id": "proxy.cert", "content": compose.ForeseenCertificate}}},
"the bus's list": {Resources: []map[string]any{{"id": "nats.users", "content": compose.ForeseenBusUsers}}},
"the vault's box": {Resources: []map[string]any{{"id": "x", "box": catalogue.ForeseenVaultBox}}},
} {
if left := stillForeseen(d); len(left) == 0 {
t.Errorf("%s: a declaration that stands in for it is not foreseen", name)
}
}
}
// 1386: an Input read for a person's push is never the one read for a send that keeps recorded builds, and the cache is
// emptied by what a send made.
func TestAnInputIsCachedPerWhatItKeepsAndForgottenAfterASend(t *testing.T) {
ctx := withInputCache(context.Background())
plain := keptSignature(ctx)
kept := keptSignature(keepingRecorded(ctx))
bus := keptSignature(keepingRecorded(ctx, "nats"))
every := keptSignature(keepingEveryBuild(ctx, "anchor"))
for name, sig := range map[string]string{"a push": plain, "a kept send": kept, "the bus step": bus, "the grants step": every} {
for other, o := range map[string]string{"a push": plain, "a kept send": kept, "the bus step": bus, "the grants step": every} {
if name != other && sig == o {
t.Errorf("%s and %s read the same Input", name, other)
}
}
}
cache := ctx.Value(inputCacheKey{}).(*inputCache)
cache.by[plain] = &meshInput{}
forgetInput(ctx)
if len(cache.by) != 0 {
t.Fatal("what a send made left the Input it read in the cache")
}
}
cmd/mesh-controller/composed_member_check_test.go
-68
View File
@@ -1,68 +0,0 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A member of a group whose gate is the composed check's is asked of the build seat with no module: its repository's
// check alone (novox/hq ADR 0403 clause 1). With no word from mesh-delivery, or one that does not say the gate is
// composed, it is checked alone with its gate (clause 6).
func askedForMember(t *testing.T, answer []groupsLine, answerErr error) link.CheckRequest {
t.Helper()
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
was := groupsAsked
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, answerErr }
t.Cleanup(func() { groupsAsked = was })
p := link.PullUpdated{Owner: "novox", Repo: "mesh-controller", Number: 346, Base: "main", Head: "feat/x", Commit: headController,
Paths: []string{"main.go"}, ModuleDirsSaid: true, MergeCheck: true, MergeCheckSaid: true}
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil || len(q.Asks) != 1 {
t.Fatalf("asks %d: %v", len(q.Asks), err)
}
return *q.Asks[0].Request.Check
}
func TestAComposedMembersCheckIsItsRepositoryChecksAlone(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
line.ComposedGate = true
c := askedForMember(t, []groupsLine{line}, nil)
if len(c.Modules) != 0 || len(c.New) != 0 || len(c.Dependents) != 0 || !c.Composed {
t.Fatalf("the member's check is asked as %+v; want no module and composed", c)
}
}
func TestAMemberOfAGroupOfAnOlderMeshDeliveryKeepsItsOwnGate(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
c := askedForMember(t, []groupsLine{line}, nil)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed {
t.Fatalf("an older mesh-delivery's member is asked as %+v; want its own gate", c)
}
}
func TestAMemberWhoseGroupCannotBeLearnedIsCheckedAloneWithItsGate(t *testing.T) {
c := askedForMember(t, nil, context.DeadlineExceeded)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed ||
!strings.Contains(c.SiblingsUnknown, "reads no siblings") {
t.Fatalf("the member is asked as %+v; want its gate and why its siblings are unknown", c)
}
}
cmd/mesh-controller/composed_verdict_test.go
-136
View File
@@ -1,136 +0,0 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A delivery group's gate is the composed check's, and its members' own checks run their repository's check alone
// (novox/hq ADR 0403 clauses 1 and 7). Pure: no store, no bus.
// A member of a group whose line says composed_gate is a composed member; one whose group says nothing of it (an older
// mesh-delivery), one of a group of a single pull request and a lone pull request are not.
func TestAMemberOfAComposedGroupIsKnownByTheGroupsLine(t *testing.T) {
two := func(composed bool) groupsLine {
l := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))
l.ComposedGate = composed
return l
}
p := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
recheck := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Commit: headHost}
for name, c := range map[string]struct {
p link.PullUpdated
lines []groupsLine
want bool
}{
"composed": {p, []groupsLine{two(true)}, true},
"a recheck, by its number": {recheck, []groupsLine{two(true)}, true},
"an older mesh-delivery": {p, []groupsLine{two(false)}, false},
"another group": {p, []groupsLine{group("feat/y", member("novox/mesh-host", 1, headHost, "main", true))}, false},
"no group": {p, nil, false},
} {
if got := composedMember(c.p, c.lines); got != c.want {
t.Errorf("%s: composed %v, want %v", name, got, c.want)
}
}
alone := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))
alone.ComposedGate = true
if composedMember(p, []groupsLine{alone}) {
t.Error("a group of a single pull request is composed")
}
}
// A composed member's verdict never says its gate passed: the gate layer, and the verdict that is the gate's, read
// "composed"; a failure or an error stays what it was.
func TestAComposedMembersVerdictHasNoPassingGate(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114, Composed: true}
pass := link.BuildResult{ID: "b", Ref: headHost, Checked: asked, Check: &link.CheckOutcome{Verdict: "pass", Summary: "no module touched",
Gate: &link.CheckLayer{Verdict: "pass", Summary: "no module touched"}, RepoCheck: &link.CheckLayer{Verdict: "pass", Summary: "ok"}}}
c := checkedOf(pass)
if c.Gate == nil || c.Gate.Verdict != "composed" || c.Verdict != "composed" || c.RepoCheck == nil || c.RepoCheck.Verdict != "pass" {
t.Fatalf("the member's verdict says %+v gate %+v", c, c.Gate)
}
asked.Composed = false
if c := checkedOf(pass); c.Gate.Verdict != "pass" || c.Verdict != "pass" {
t.Fatalf("a member of no composed group has its own gate: %+v", c.Gate)
}
asked.Composed = true
failing := pass
failing.Check = &link.CheckOutcome{Verdict: "fail", Summary: "conflict", Gate: &link.CheckLayer{Verdict: "fail", Summary: "conflict"},
RepoCheck: &link.CheckLayer{Verdict: "fail", Summary: "conflict"}}
if c := checkedOf(failing); c.Gate.Verdict != "fail" || c.Verdict != "fail" {
t.Fatalf("a failure was read as composed: %+v", c.Gate)
}
broken := link.BuildResult{ID: "b", Ref: headHost, Checked: asked, Failed: "the facts snapshot cannot be read"}
if c := checkedOf(broken); c.Verdict != "error" || c.Gate.Verdict != "error" {
t.Fatalf("an error was read as composed: %+v", c)
}
}
// The composed verdict on the bus carries every member, each with its trunk commit, for the forge's holder to set each
// member's mesh/merge-gate from it.
func TestTheComposedVerdictCarriesEveryMember(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114, Group: "feat/x",
Members: []link.GroupMember{{Owner: "novox", Repo: "mesh-controller", Number: 346, Ref: headController}}}
c := checkedOf(link.BuildResult{ID: "g", Ref: headHost, Checked: asked, Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok",
Trunk: "aaaaaaaaaaaa", Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-controller", Commit: headController[:12], Trunk: "bbbbbbbbbbbb"}}}})
body, _ := json.Marshal(c)
for _, want := range []string{`"group":"feat/x"`, `"members":[{"owner":"novox","repo":"mesh-host","number":114,"commit":"` + headHost + `","trunk":"aaaaaaaaaaaa"}`,
`{"owner":"novox","repo":"mesh-controller","number":346,"commit":"` + headController + `","trunk":"bbbbbbbbbbbb"}`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the composed verdict lacks %s:\n%s", want, body)
}
}
}
// Members whose manifests order their modules in a cycle fail the composed check, naming every module and edge, and no
// ask is made of the build seat.
func TestAGroupInACycleFailsItsComposedCheckWithoutABuild(t *testing.T) {
members := []orderMember{
{ID: "novox/x@" + headHost[:12], Repository: "novox/x", Base: "main", Head: headHost, Number: 7, Paths: []string{"module.json"}},
{ID: "novox/y@" + headController[:12], Repository: "novox/y", Base: "main", Head: headController, Number: 9, Paths: []string{"module.json"}},
}
reach := map[string]orderReach{members[0].ID: {Moved: []string{"mx"}}, members[1].ID: {Moved: []string{"my"}}}
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
manifestAt = func(_ context.Context, _, repo, _, ref string) (catalogue.Manifest, bool, error) {
if ref == "main" {
return catalogue.Manifest{}, false, nil // what the pull requests merge into holds no such manifest
}
if repo == "x" {
return catalogue.Manifest{Module: "mx", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "a"}}}},
Invokes: []string{"seat:s.b"}}, true, nil
}
return catalogue.Manifest{Module: "my", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "b"}}}},
Invokes: []string{"seat:s.a"}}, true, nil
}
c, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, []inventory.Edge(nil), nil)
if err != nil || !cyclic {
t.Fatalf("no cycle found: %v", err)
}
if c.Verdict != "fail" || c.Group != "feat/cycle" || len(c.Members) != 2 || c.Gate == nil || c.Gate.Verdict != "fail" {
t.Fatalf("the verdict is %+v", c)
}
for _, w := range []string{"mx", "my", "s.a", "s.b"} {
if !strings.Contains(c.Summary, w) {
t.Errorf("the cycle does not name %s: %s", w, c.Summary)
}
}
// Members with no edge between them are no cycle.
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
manifestAt = func(context.Context, string, string, string, string) (catalogue.Manifest, bool, error) {
return catalogue.Manifest{}, false, nil
}
if _, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, nil, nil); err != nil || cyclic {
t.Fatalf("a cycle where there is none: %v %v", cyclic, err)
}
}
cmd/mesh-controller/condition_looks_test.go
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (

Some files were not shown because too many files have changed in this diff Show More