Compare commits

..
2 Commits
Author SHA1 Message Date
jochen 52bcef159d A settings proposal carries every key with its whole value in its Details (issue 383)
mesh/delivery-group group fix/383-an-ask-reads-at-a-glance checking: composed check out of date: novox/mesh-controller@52bcef159d95 was judged against trunk 6f…
mesh/merge-gate out of date: judged against trunk 6f1a05366c1d, the trunk is now 8ddf8b700558; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 6f1a05366c1d, the trunk is now 8ddf8b700558; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
The review of #304: with more than three keys the ask showed only the names of the rest, a long value was
cut without a word, and the Details held no key, so Approve bound values the operator never saw. The
Details now list every key uncut — masked, and whole in WholeDetails for a channel that proves who
answers — before who proposed it and the fingerprint; the glance says what it cut. Vendors mesh-sdk at
f474544 and captures its conformance events there.
2026-10-13 17:04:48 +02:00
jochen 35232c8534 A settings proposal reads at a glance, its values whole where the sender is proven (issues 383 and 639)
mesh/merge-gate head 35232c853483 merged into trunk 6f1a05366c1d — pass: builds build-agent, mesh-controller, route-proxy → ace, g14, novox, shanks; no bus…
mesh/repo-check head 35232c853483 merged into trunk 6f1a05366c1d — pass: its merge-check.sh passed
mesh/delivery-group group fix/383-an-ask-reads-at-a-glance checking: composed check out of date: novox/mesh-catalog@ea1dfdce5bc7 was judged against trunk 5f491…
mesh/delivery superseded: a newer head of the same pull request
The operator could not read the settings proposals on the phone: the change, the fingerprint and a how-to
paragraph in one wall of plain text, and the values masked as ‹path› and ‹address›, so the approval was
blind.

The ask is now a summary of what changes where ("Changes 3 setting(s) of mounts on shanks; approved, the
layer is set at once and shanks takes it at its next push."), then at most three points, one key each (more
keys: the first two and the names of the rest). Who proposed it, the fingerprint and how to read it whole
are its Details, which the router notes after the answers. The change is carried twice: under the content
rule, and whole (asks.Ask.Whole), shown only on a channel that proves who answers, a value shaped like a
secret withheld in both (from the earlier work on issue 383, #202).

Vendors mesh-sdk at 9588871 (fix/639-an-ask-reads-at-a-glance: Message.Note, asks.Glance), and captures its
conformance events at that commit.
2026-10-13 16:33:52 +02:00
1217 changed files with 45540 additions and 113006 deletions
+3 -7
View File
@@ -95,7 +95,7 @@ PROXY_IMAGE ?= mesh-route-proxy:$(VERSION)
PROXY_DEV_TAG ?= mesh-route-proxy:development
proxy-image:
docker build --build-arg GO_BASE=$(GO_BASE) -f modules/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
docker build --build-arg GO_BASE=$(GO_BASE) -f examples/route-proxy/Dockerfile -t $(PROXY_IMAGE) -t $(PROXY_DEV_TAG) .
@echo
@docker image inspect $(PROXY_IMAGE) --format 'built {{.RepoTags}} {{.Size}} bytes'
@@ -110,18 +110,14 @@ proxy-image:
# after formatting never ran through it. The script is the one list of steps; this target only gives it
# a database (MESH_TEST_POSTGRES, exported above).
#
# **Two levels** (issue 895): the unit level, `go test ./...` untagged, timed and held to a bound; and the integration
# level, `-tags integration`, the tests that start a bus, a store or another process. Each is said with its time.
#
# **Packages in parallel, under the race detector, each test on a bus of its own** (internal/testbus),
# as the script runs them: a bus per test, of the release the mesh runs, makes the suite the same in any
# order, and the timeout bounds a hang to a failure with a stack, never a stalled gate.
check: postgres
@sh merge-check.sh ; status=$$? ; docker rm -f $(PG_CONTAINER) >/dev/null 2>&1 || true ; exit $$status
# The unit level alone (untagged: no bus, no store, no other process), which a developer runs continuously. The tests
# that need a bus or the store carry `//go:build integration` and run in `check`, not here; without a database the
# store's tests would skip rather than fail anyway.
# Without a database the store's tests skip rather than fail, so this is the honest subset and not
# the gate.
test:
go test -timeout 15m ./...
-6
View File
@@ -1,6 +0,0 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
-29
View File
@@ -1,29 +0,0 @@
#!/bin/sh
# The unit tests of every Go module of its own below this repository's root, run from the root, for merge-check.sh
# and for the test that tries it on a tree (merge_check_nested_test.go). Its own file so that the test needs no
# particular merge-check.sh: the check that judges a change to merge-check.sh is main's.
set -eu
cgo="${cgo:-0}"
race="${race:-}"
# nested_modules vets and tests every Go module of its own below the root, `modules/<name>/go.mod` (novox/hq ADR 0420
# §5, issue 1025): `./...` stops at a nested go.mod, so a module that lives whole in this repository would have its
# unit tests run by nobody. None is a pass. Each runs in its own directory, outside the vendor directory this
# repository's own packages are built from; a nested module vendors what the forge holds (its own `vendor/`, so the
# check runs offline), as this repository's root module does (issue 1019). A failing module fails the check; the others are still run.
nested_modules() {
nested=0
for gomod in modules/*/go.mod; do
[ -e "$gomod" ] || continue
dir=$(dirname "$gomod")
echo "nested module $dir:"
if ! (cd "$dir" && GOFLAGS= CGO_ENABLED=0 go vet ./... &&
GOFLAGS= CGO_ENABLED="${cgo:-0}" go test ${race:-} -count=1 -timeout 30m ./...); then
echo "nested module $dir FAILED"
nested=1
fi
done
return $nested
}
nested_modules
-146
View File
@@ -1,146 +0,0 @@
package main
import (
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"time"
)
// **The build agent says whether it is busy, and being asked drains it** (novox/hq ADR 0335 §1, §5). Its
// container declares `busy` as `mesh-builder busy`: the node-engine runs it inside the container before it
// replaces it. It exits 0 while a build is in hand and 1 when none is; and from the first ask the agent takes
// no new build request, so its answer ends once the build in hand ends — a replaced builder finishes the check
// it runs, instead of the build queue handing it out again from the start (issue 594).
//
// Two marks in the container's own temporary directory, as the ready mark (ready.go): a new container starts
// without them, and a restarted one removes them before it takes work again.
//
// - in hand: written by the agent when it claims a request, before it reads whether it is drained; removed
// when the build ends.
// - drained: written by the ask, before it reads whether a build is in hand.
//
// Each side writes its own mark before it reads the other's, so an ask racing a fetch sees the build in hand,
// or the agent sees the drain and hands the request back; never neither.
func inHandMark() string { return filepath.Join(os.TempDir(), "mesh-builder.in-hand") }
func drainMark() string { return filepath.Join(os.TempDir(), "mesh-builder.drained") }
// The drain lapses when no ask has come for drainLapse: the node-engine asks every 20 seconds for as long as it
// really waits, so an agent whose replacement was withdrawn — the declaration put back, the walk stopped —
// takes work again rather than staying drained for ever (review of ADR 0335). And an agent drained for longer
// than the node-engine's bound and the slack says it is not ready: the replacement it was drained for never
// came, which is a fault to see.
const (
drainLapse = 2 * time.Minute
drainTooLong = 15*time.Minute + 5*time.Minute
)
// drainedAt is when the drain began and when it was last asked, from its mark; false when there is no mark.
func drainedAt() (began, asked time.Time, ok bool) {
info, err := os.Stat(drainMark())
if err != nil {
return time.Time{}, time.Time{}, false
}
asked = info.ModTime()
began = asked
if raw, err := os.ReadFile(drainMark()); err == nil {
if t, err := time.Parse(time.RFC3339, strings.TrimSpace(string(raw))); err == nil && !t.After(asked) {
began = t
}
}
return began, asked, true
}
// drained says the agent was asked whether it is busy within drainLapse: it takes no new build request.
func drained() bool { return drainedSince(time.Now()) }
func drainedSince(now time.Time) bool {
_, asked, ok := drainedAt()
return ok && now.Sub(asked) < drainLapse
}
// drainedTooLong is the health check's second half: nil unless the agent has been drained, asked all along, for
// longer than its node-engine's bound and the slack.
func drainedTooLong(now time.Time) error {
began, asked, ok := drainedAt()
if !ok || now.Sub(asked) >= drainLapse || now.Sub(began) <= drainTooLong {
return nil
}
return fmt.Errorf("drained since %s and still asked: the replacement its node-engine waits for has not come "+
"within %s (novox/hq ADR 0335)", began.UTC().Format(time.RFC3339), drainTooLong)
}
// claim marks a request in hand, unless the agent is drained: then nothing is in hand, and the request goes
// back to the queue.
func claim() bool {
// Made, never written over: one build at a time, and a mark already there is a build in hand.
f, err := os.OpenFile(inHandMark(), os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0o644)
if err != nil {
// Unmarked, an ask would say idle while this build runs: the build is not taken.
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
_, err = f.WriteString(time.Now().UTC().Format(time.RFC3339) + "\n")
if closeErr := f.Close(); err == nil {
err = closeErr
}
if err != nil {
_ = os.Remove(inHandMark())
fmt.Fprintf(os.Stderr, "cannot mark a build in hand (%s), so it is handed back: %v\n", inHandMark(), err)
return false
}
if drained() {
_ = os.Remove(inHandMark())
fmt.Fprintln(os.Stderr, "asked whether busy, so this agent takes no new build: handed back")
return false
}
return true
}
// holding says what is in hand, for the ask to print: written once the request is read.
func holding(what string) {
_ = os.WriteFile(inHandMark(), []byte(time.Now().UTC().Format(time.RFC3339)+" "+what+"\n"), 0o644)
}
// released ends the build in hand.
func released() {
if err := os.Remove(inHandMark()); err != nil && !errors.Is(err, os.ErrNotExist) {
fmt.Fprintf(os.Stderr, "cannot remove the mark of the build in hand (%s): %v\n", inHandMark(), err)
}
}
// undrained clears both marks at the start of a holder: a container started again is asked afresh.
func undrained() error {
for _, mark := range []string{drainMark(), inHandMark()} {
if err := os.Remove(mark); err != nil && !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("cannot remove %s: %w", mark, err)
}
}
return nil
}
// busy is the ask: it drains the agent, then says whether a build is in hand. Nil — exit 0 — while one is.
func busy() (string, error) {
// The mark keeps when the drain began while it is asked within drainLapse, and is written again on every
// ask: its time is the last ask.
began := time.Now().UTC()
if was, _, ok := drainedAt(); ok && drained() {
began = was.UTC()
}
if err := os.WriteFile(drainMark(), []byte(began.Format(time.RFC3339)+"\n"), 0o644); err != nil {
// Not drained, it could take the next build: said, and answered busy only for what is in hand now.
fmt.Fprintf(os.Stderr, "cannot mark this agent drained (%s): %v\n", drainMark(), err)
}
held, err := os.ReadFile(inHandMark())
if err != nil {
return "", errors.New("idle: no build in hand, and this agent takes no new one")
}
since, what, _ := strings.Cut(strings.TrimSpace(string(held)), " ")
if what == "" {
what = "a build"
}
return fmt.Sprintf("busy: %s, in hand since %s; taking no new build", what, since), nil
}
-162
View File
@@ -1,162 +0,0 @@
package main
import (
"os"
"strings"
"sync"
"testing"
"time"
)
// `mesh-builder busy` answers 0 while a build is in hand and non-zero when idle, and after its first ask the
// agent takes no new build request (novox/hq ADR 0335 §5).
func TestBusyAnswersWhileABuildIsInHandAndDrainsTheAgent(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("an agent never asked is drained")
}
// A build is claimed and runs.
if !claim() {
t.Fatal("an agent never asked refused a build")
}
holding("novox/mesh-controller (check-280)")
said, err := busy()
if err != nil || !strings.Contains(said, "novox/mesh-controller (check-280)") {
t.Fatalf("an agent with a build in hand did not answer busy with what it holds: %q %v", said, err)
}
// From that ask on it takes nothing new: the fetch is not made, and one fetched is handed back.
if !drained() {
t.Fatal("the ask did not drain the agent")
}
if claim() {
t.Fatal("a drained agent took a new build")
}
// The build in hand is still in hand after the refused claim, and busy goes on saying so.
if _, err := busy(); err != nil {
t.Fatalf("a refused claim ended the build in hand: %v", err)
}
// It ends: idle, and still drained.
released()
if _, err := busy(); err == nil {
t.Fatal("an agent with nothing in hand answered busy")
}
if claim() {
t.Fatal("a drained agent took a build after its last one ended")
}
// Started again, it is asked afresh.
if err := undrained(); err != nil {
t.Fatal(err)
}
if drained() || !claim() {
t.Fatal("a holder started again is still drained")
}
released()
}
// An ask racing a fetch sees the build in hand, or the agent sees the drain and hands the request back: never
// neither, which would replace the container under a build just taken.
func TestAnAskRacingAFetchNeverMissesTheBuild(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
for i := 0; i < 200; i++ {
if err := undrained(); err != nil {
t.Fatal(err)
}
var claimed bool
var askErr error
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); claimed = claim() }()
go func() { defer wg.Done(); _, askErr = busy() }()
wg.Wait()
if claimed && askErr != nil {
t.Fatalf("round %d: the agent took a build and the ask said idle", i)
}
released()
}
}
// A drain no longer asked lapses: an agent whose replacement was withdrawn after the first ask (a put-back, a
// revert, a stopped walk) takes work again after drainLapse, rather than staying drained with its health green.
func TestADrainNoLongerAskedLapses(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if !drained() || claim() {
t.Fatal("an agent just asked takes work")
}
// The last ask was three minutes ago.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if drained() {
t.Fatal("a drain no longer asked for three minutes still holds")
}
if !claim() {
t.Fatal("an agent whose drain lapsed takes no work")
}
released()
}
// An agent drained, and asked all along, for longer than its node-engine's bound and the slack is not ready: the
// replacement it was drained for never came. One asked again after a lapse starts a new drain.
func TestAnAgentDrainedTooLongIsNotReady(t *testing.T) {
t.Setenv("TMPDIR", t.TempDir())
if err := undrained(); err != nil {
t.Fatal(err)
}
taking("mesh-build-machine")
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
now := time.Now()
if err := drainedTooLong(now); err != nil {
t.Fatalf("an agent drained a moment ago is not ready: %v", err)
}
// The drain began 25 minutes ago, and the node-engine asked a moment ago.
began := now.Add(-25 * time.Minute).UTC().Format(time.RFC3339)
if err := os.WriteFile(drainMark(), []byte(began+"\n"), 0o644); err != nil {
t.Fatal(err)
}
err := drainedTooLong(time.Now())
if err == nil || !strings.Contains(err.Error(), "has not come") {
t.Fatalf("an agent drained for 25 minutes reads ready: %v", err)
}
// Each ask keeps when the drain began.
if _, err := busy(); err == nil {
t.Fatal("an idle agent answered busy")
}
if err := drainedTooLong(time.Now()); err == nil {
t.Fatal("an ask started the drain again")
}
// The asks stopped: the drain lapsed, and so did the fault.
old := time.Now().Add(-3 * time.Minute)
if err := os.Chtimes(drainMark(), old, old); err != nil {
t.Fatal(err)
}
if err := drainedTooLong(time.Now()); err != nil {
t.Fatalf("a lapsed drain still reads not ready: %v", err)
}
}
// An ask this build does not know exits non-zero: an older image asked `busy` reads idle, not busy for the
// node-engine's whole bound.
func TestAnUnknownCommandFails(t *testing.T) {
was := os.Args
t.Cleanup(func() { os.Args = was })
os.Args = []string{"mesh-builder", "some-later-word"}
if err := run(); err == nil {
t.Fatal("an unknown command exited 0")
}
os.Args = []string{"mesh-builder", "help"}
if err := run(); err != nil {
t.Fatalf("help failed: %v", err)
}
}
-38
View File
@@ -1,38 +0,0 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/link"
)
// The build seat hands the check the judge cache the build agent declares (MESH_JUDGE_CACHE), never a path of its own.
func TestTheBuildSeatHandsTheCheckTheJudgeCacheItDeclares(t *testing.T) {
t.Setenv("MESH_JUDGE_CACHE", "/var/lib/mesh/build-agent/judge-cache")
if got := judgeCache(); got != "/var/lib/mesh/build-agent/judge-cache" {
t.Fatalf("the judge cache the manifest declares is read as %q", got)
}
t.Setenv("MESH_JUDGE_CACHE", "")
if got := judgeCache(); got != "" {
t.Fatalf("with no judge cache declared, %q is used", got)
}
}
// The siblings a check is asked to read reach the build seat's spec and, as read, the outcome (novox/hq ADR 0382).
func TestTheSiblingsAnAskNamesReachTheSpecAndTheOutcome(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main",
Siblings: []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", Dir: "mesh-controller"}},
SiblingsUnknown: "why"}})
if len(spec.Siblings) != 1 || spec.Siblings[0] != (builder.Sibling{Repository: "novox/mesh-controller", Number: 346,
Head: "a", Base: "main", Dir: "mesh-controller"}) || spec.SiblingsUnknown != "why" {
t.Fatalf("the spec reads %+v", spec)
}
out := siblingsOut([]builder.JudgedSibling{{Repository: "novox/mesh-controller", Number: 346, Head: "a", Base: "main", BaseCommit: "b"}})
if len(out) != 1 || out[0].BaseCommit != "b" || out[0].Number != 346 {
t.Fatalf("the outcome reads %+v", out)
}
if plain := checkSpecOf(link.BuildRequest{Check: &link.CheckRequest{Owner: "o", Repo: "r"}}); plain.Siblings != nil {
t.Fatalf("a request that names none has siblings %+v", plain.Siblings)
}
}
+1 -32
View File
@@ -14,8 +14,8 @@ import (
"github.com/nats-io/nats.go/micro"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
@@ -41,9 +41,6 @@ type holder struct {
say func(link.HolderState) error
// remove runs what a kill needs outside the build: the containers left behind.
remove builder.Runner
// scratch is where check builds are kept (novox/hq ADR 0350 §5): what the week's sweep lets go of. Nil where
// this machine publishes to no artifact store.
scratch builder.Scratch
mu sync.Mutex
paused bool
@@ -134,34 +131,6 @@ func (h *holder) stateWhilePaused(ctx context.Context, every time.Duration) {
}
}
// expireChecks lets go of every check build older than a week (novox/hq ADR 0350 §5), at start and every while
// after. Every holder sweeps the one store; what one let go of, another finds gone.
func (h *holder) expireChecks(ctx context.Context, every time.Duration, now func() time.Time) {
sweep := func() {
if h.scratch == nil {
return
}
n, err := h.scratch.Expire(ctx, builder.CheckBuildsKept, now())
switch {
case err != nil:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d let go of, and not all: %v\n", builder.CheckBuildsKept, n, err)
case n > 0:
fmt.Fprintf(os.Stderr, "check builds older than %s: %d check(s)' builds let go of\n", builder.CheckBuildsKept, n)
}
}
sweep()
tick := time.NewTicker(every)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
sweep()
}
}
}
// begin records the build this machine took, with the cancel that ends it.
func (h *holder) begin(request link.BuildRequest, cancel context.CancelFunc) *running {
r := &running{request: request, started: time.Now(), cancel: cancel, done: make(chan struct{})}
+12 -207
View File
@@ -26,8 +26,6 @@ import (
"os/signal"
"sort"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
@@ -77,12 +75,6 @@ digest of their own configuration. The result is printed as JSON.
exits 0 once the agent running in this container is bound to its build work and pulls it, and
1 before: its container's health check.
mesh-builder busy
exits 0 while the agent running in this container has a build in hand, and 1 when it has none;
from the first ask the agent takes no new build: its container's busy command, asked before the
container is replaced (novox/hq ADR 0335).
`
func run() error {
@@ -94,41 +86,17 @@ func run() error {
case "build":
return buildOnce(context.Background(), os.Args[2:])
case "ready":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work, and
// once it has been drained longer than its node-engine would wait to replace it (ADR 0335).
if err := ready(); err != nil {
return err
}
return drainedTooLong(time.Now())
case "busy":
// The container's busy command (novox/hq ADR 0335): drains the agent, and exits non-zero once
// nothing is in hand.
said, err := busy()
if err != nil {
return err
}
fmt.Println(said)
return nil
case "help", "-h", "--help":
// The container's health check (novox/hq issue 544): exits non-zero until the agent takes work.
return ready()
default:
fmt.Print(usage)
return nil
default:
// **An ask this build does not know fails** (novox/hq ADR 0335): an image older than `busy`, asked
// it, must read as idle — exiting 0 would hold its container for the node-engine's whole bound.
fmt.Fprint(os.Stderr, usage)
return fmt.Errorf("%q is not a command this mesh-builder knows", os.Args[1])
}
}
if err := notReady(); err != nil {
return err
}
// **The builder's own version is part of every reuse key** (novox/hq ADR 0350 §1): a builder that builds
// differently never takes a build another one made.
builder.Version = builder.SelfVersion(version)
if err := undrained(); err != nil {
return err
}
credential, err := brokerFrom()
if err != nil {
return err
@@ -181,13 +149,8 @@ func run() error {
// **What an earlier holder here left goes before anything is taken** (novox/hq issue 306): a holder
// recreated mid-check left the check's store and bus running, and the ask went to another machine.
h.removeLeftBehind()
// Paused or drained, it takes nothing new (novox/hq ADR 0219, ADR 0335): a drained agent finishes the build
// in hand and then says idle, and its container is replaced between builds.
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{
Paused: func() bool { return h.Paused() || drained() },
Taking: func() { taking(seat) },
Claim: claim,
})
machine := link.MachineOverNATSWith(js, on, seat, link.MachineOptions{Paused: h.Paused,
Taking: func() { taking(seat) }})
defer machine.Close()
// The seat's verbs, on this machine's subjects. Only the seat that declares them: the retired
@@ -211,13 +174,8 @@ func run() error {
fmt.Fprintf(os.Stderr, "building for the mesh, publishing to %s\n", registry)
publisher := builder.Registry{Address: registry, Run: builder.Command}
// Check builds are let go of after a week (novox/hq ADR 0350 §5).
h.scratch = publisher
go h.expireChecks(ctx, time.Hour, time.Now)
return machine.Take(ctx, func(ctx context.Context, work link.Build) {
defer released()
holding(work.Request().Repository + " (" + work.Request().ID + ")")
answer(ctx, publisher, on, workspace, work, h)
})
}
@@ -274,12 +232,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// role's own events under the build's id (novox/hq ADR 0157) — so whoever asked, and anybody
// watching, reads the same lines this container's log holds, live, and after the fact from the
// stream. Said first, before anything runs, so a build that hangs is one that visibly started.
// One line at a time: a check's builds for its merge say their lines while its tests say theirs (novox/hq
// issue 716).
var saying sync.Mutex
say := func(step, message string) {
saying.Lock()
defer saying.Unlock()
fmt.Fprintf(os.Stderr, " [%s] %s\n", step, message)
work.Say(step, message)
if mine != nil && step != "run" && step != "output" {
@@ -291,26 +244,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if err := work.Began(ctx); err != nil {
fmt.Fprintf(os.Stderr, "cannot say a build started: %v\n", err)
}
// **A running check watches its pull request's key** (novox/hq ADR 0406 §4, issue 974): when the head or the trunk
// commit in it moves, the check is for a state nobody wants any more, and ends itself — nothing is asked of the forge
// (issue 726). A group's check, a try and a build watch nothing.
var superseded atomic.Bool
if w, ok := work.(interface {
WatchPull(context.Context, func(link.PullKey)) (func(), error)
}); ok && request.Check != nil {
stopWatching, err := w.WatchPull(building, func(k link.PullKey) {
superseded.Store(true)
say("superseded", fmt.Sprintf("the pull request's key moved to %.12s on %.12s while this checked %.12s on %.12s: stopped",
k.Head, k.Trunk, request.Ref, request.Check.Trunk))
cancel()
})
if err != nil {
fmt.Fprintf(os.Stderr, "the key of %s/%s#%d could not be watched, so its check runs on if the key moves: %v\n",
request.Check.Owner, request.Check.Repo, request.Check.Number, err)
} else {
defer stopWatching()
}
}
result := link.BuildResult{
ID: request.ID, Repository: request.Repository, Path: request.Path,
@@ -329,7 +262,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if err == nil {
say("packages", saysRegistry(npmrc))
}
registryHere, isRegistry := publisher.(builder.Registry)
var built builder.Result
if request.Check != nil {
// **A pull request's merge check, not a build** (novox/hq to-be 45 §9): nothing is built,
@@ -339,37 +271,17 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if r, ok := publisher.(builder.Registry); ok {
registry = r.Address
}
spec := checkSpecOf(request)
// The judge cache the build agent declares as its own data, never in the workspace: only judge builds and
// mesh-lab's replays mount it, never a check's containers, so no pull request's script can write what a judge
// is built with or kept as, or what the replays run (novox/hq ADR 0358 §6, issue 709).
spec.JudgeCache = judgeCache()
// What the merge would move, built by the check for the walk to take (novox/hq ADR 0350 §1): into the
// scratch namespace. **Never with the package registry's credential**: this is code nobody has approved
// yet, and a recipe could read the credential it is handed (issue 462). A module that resolves packages
// at build time has no reuse key anyway, so nothing is lost by refusing it here.
if isRegistry && len(spec.Builds) > 0 {
spec.BuildWith = &builder.CheckBuilding{Publish: publisher, Scratch: registryHere, Held: request.Held,
Running: request.Check.Running, Seats: request.Seats, Named: namedOf(request.Check.Named)}
}
var v builder.CheckVerdict
v, err = builder.Check(building, builder.Command, spec, workspace, registry, forgeFrom(), say)
v, err = builder.Check(building, builder.Command, checkSpecOf(request), workspace, registry, forgeFrom(), say)
if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built),
Validator: v.Validator, Siblings: siblingsOut(v.Siblings), SiblingsUnknown: v.SiblingsUnknown, JudgeFrom: v.JudgeFrom}
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk})
}
// A branch tried before its pull request reads its outcome from this log, whole (novox/hq issue 686).
if request.Check.Try {
if line, err := json.Marshal(triedOf(v)); err == nil {
say(link.StepTried, string(line))
}
}
}
} else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a
@@ -377,22 +289,9 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
// after a clone that then fails at npm ci.
// Every container it starts is labelled with its id, so a kill finds what outlived the
// docker client (ADR 0219).
if request.Reuse != nil && isRegistry && !request.DryRun {
// **A walk's build takes the check's build of the same key** (novox/hq ADR 0350 §2).
reuse := builder.Reuse{Scratch: registryHere}
for _, p := range request.Reuse.Pulls {
reuse.Pulls = append(reuse.Pulls, builder.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number})
}
// Only what a verdict names is taken (ADR 0350 §2).
reuse.Named = namedOf(request.Reuse.Builds)
built, err = builder.BuildReusing(building, builder.Labelled(builder.Command, request.ID), publisher, reuse,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
} else {
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
built, err = builder.Build(building, builder.Labelled(builder.Command, request.ID), publisher,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
}
// What it copied into the store, whatever became of the build (novox/hq ADR 0257).
result.Mirrored = built.Mirrored
@@ -402,11 +301,7 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
if mine != nil {
killed = h.returned(mine) && err != nil
}
if superseded.Load() && err != nil {
// Ended by its own key moving on (ADR 0406 §4): said as the check's failure, as a kill is.
result.Failed = link.SupersededByKey
say("failed", link.SupersededByKey)
} else if killed {
if killed {
// **Killed by hand is the outcome, whatever the build was doing** (novox/hq ADR 0219): the
// error it ended with is the kill's consequence, not a fault of the source.
result.Failed = link.KilledByHand
@@ -432,20 +327,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
}
result.Against = built.Against
result.SourceFingerprint = built.Source
result.SourceInputs = built.SourceInputs
// What the walk did instead of building, or why it built (novox/hq ADR 0350 §3, §4).
result.ReuseKey = built.ReuseKey
if built.Reused != nil {
f := built.Reused.From
result.Reused = built.Reused.Words()
result.ReusedFrom = &link.ReusedFrom{Pull: link.PullRef{Owner: f.Pull.Owner, Repo: f.Pull.Repo,
Number: f.Pull.Number}, Head: f.Head, Trunk: f.Trunk, Check: f.Check, Key: f.Key}
} else if request.Reuse != nil {
result.BuiltWhy = built.Why
if result.BuiltWhy == "" {
result.BuiltWhy = "it has no artifact to build"
}
}
result.Trunk, result.OnTrunk, result.Branches = built.Trunk, built.OnTrunk, built.Branches
for _, r := range built.Read {
result.Read = append(result.Read, link.ReadRepository{Repository: r.Repository, Ref: r.Ref})
@@ -453,11 +334,6 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
for _, s := range built.Sources {
result.Sources = append(result.Sources, link.BuildSource{Repository: s.Repository, Ref: s.Ref, Paths: s.Paths})
}
// The list of predefined keys the manifest was judged against, for the controller to publish
// (novox/hq ADR 0348 §7).
if p := built.Predefined; p != nil {
result.Predefined = &link.PredefinedList{Keys: p.Keys, Generation: p.Generation}
}
say("built", built.Manifest.Module+" from "+short(built.Commit))
}
}
@@ -497,19 +373,10 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
spec := builder.CheckSpec{ID: request.ID, Repository: request.Repository, Ref: request.Ref,
Owner: c.Owner, Repo: c.Repo, Number: c.Number, Paths: c.Paths, Beside: map[string]builder.Beside{},
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, SiblingsUnknown: c.SiblingsUnknown}
for _, s := range c.Siblings {
spec.Siblings = append(spec.Siblings, builder.Sibling{Repository: s.Repository, Number: s.Number, Head: s.Head,
Base: s.Base, Dir: s.Dir})
}
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held)}
for dir, b := range c.Beside {
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref}
}
for _, b := range c.Builds {
spec.Builds = append(spec.Builds, builder.CheckBuildSpec{Module: b.Module, Repository: b.Repository, Path: b.Path,
Ref: b.Ref, Merged: b.Merged, Tier: b.Tier, Skip: b.Skip})
}
for _, m := range c.Members {
spec.Group = append(spec.Group, builder.GroupHead{Owner: m.Owner, Repo: m.Repo, Repository: m.Repository,
Ref: m.Ref, Paths: m.Paths, Base: m.Base})
@@ -517,68 +384,6 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
return spec
}
// siblingsOut is the siblings a check read as the outcome carries them (novox/hq ADR 0382 §4).
func siblingsOut(read []builder.JudgedSibling) []link.CheckSibling {
var out []link.CheckSibling
for _, s := range read {
out = append(out, link.CheckSibling{Repository: s.Repository, Number: s.Number, Head: s.Head, Base: s.Base,
BaseCommit: s.BaseCommit})
}
return out
}
// judgeCache is the judge cache the build agent's manifest declares (MESH_JUDGE_CACHE, novox/hq ADR 0358 §6): a
// directory mounted at the same path on the host, so a judge build's mounts name what the agent sees. Empty when the
// agent declares none, and every judge is then built cold under its check's root.
func judgeCache() string { return os.Getenv("MESH_JUDGE_CACHE") }
// triedOf is a try's outcome as its log line says it (novox/hq issue 686): the repository's own check's verdict,
// or the check's where that did not run — a head that does not merge — and the lines naming what failed.
func triedOf(v builder.CheckVerdict) link.Tried {
t := link.Tried{Verdict: v.Verdict, Summary: v.Summary, Took: v.Took.Round(time.Second).String(),
Head: v.Head, Trunk: v.Trunk}
if v.Repo != nil && (v.Gate == nil || v.Gate.Verdict == "pass") {
t.Verdict, t.Summary = v.Repo.Verdict, v.Repo.Summary
}
if v.Repo != nil && v.Repo.Failed != "" {
t.Failed = strings.Split(v.Repo.Failed, "\n")
}
if t.Verdict == "" {
t.Verdict = "error"
}
return t
}
// namedOf are the builds the seat's verdicts name, as the builder believes a scratch record by them (novox/hq ADR
// 0350 §2, issue 703): each module's key with every artifact's digest.
func namedOf(builds []link.CheckedBuild) []builder.NamedBuild {
var out []builder.NamedBuild
for _, b := range builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
out = append(out, named)
}
return out
}
// checkBuiltOf is a check's builds for its merge as the outcome carries them (novox/hq ADR 0350 §1).
func checkBuiltOf(built []builder.CheckBuilt) []link.CheckBuilt {
var out []link.CheckBuilt
for _, b := range built {
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String(),
Kept: b.Kept, Same: b.Same, SameAs: b.SameAs}
for _, m := range b.Made {
c.Made = append(c.Made, link.MadeArtifact{Name: m.Name, Kind: m.Kind, Reference: m.Reference})
}
out = append(out, c)
}
return out
}
// layerOf is one layer of a check as the outcome carries it.
func layerOf(l *builder.Layer) *link.CheckLayer {
if l == nil {
-1
View File
@@ -16,7 +16,6 @@ func TestBuilderDiagnosticsStayOffStdout(t *testing.T) {
"version)": true, // --version
`"stopping")`: true, // the loop.s shutdown line
"usage)": true, // --help text, for a human
"said)": true, // busy: the answer its node-engine reads (novox/hq ADR 0335)
}
for _, file := range []string{"once.go", "main.go"} {
src, err := os.ReadFile(file)
-31
View File
@@ -1,31 +0,0 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// A try's outcome, as the build seat says it in the build's log (novox/hq issue 686): the repository's own
// check's verdict, how long it took, and each line naming what failed.
func TestATrysOutcomeNamesWhatFailed(t *testing.T) {
tried := triedOf(builder.CheckVerdict{Verdict: "pass", Summary: "the change touches no module", Took: 303 * time.Second,
Head: "c0ffee", Trunk: "ca3f43c", Gate: &builder.Layer{Verdict: "pass"},
Repo: &builder.Layer{Verdict: "fail", Summary: "its merge-check.sh failed: TestX",
Failed: "--- FAIL: TestX (0.01s)\n x_test.go:12: got 1, want 2"}})
if tried.Verdict != "fail" || tried.Took != "5m3s" || len(tried.Failed) != 2 || tried.Trunk != "ca3f43c" ||
tried.Summary != "its merge-check.sh failed: TestX" {
t.Fatalf("a failed try: %+v", tried)
}
// A head that does not merge fails both layers: the check's own word.
conflict := triedOf(builder.CheckVerdict{Verdict: "fail", Summary: "does not merge", Gate: &builder.Layer{Verdict: "fail"},
Repo: &builder.Layer{Verdict: "fail", Summary: "does not merge"}})
if conflict.Verdict != "fail" || conflict.Summary != "does not merge" {
t.Fatalf("a try that does not merge: %+v", conflict)
}
// No verdict is an error, never a pass.
if none := triedOf(builder.CheckVerdict{}); none.Verdict != "error" {
t.Fatalf("a try with no verdict: %+v", none)
}
}
+2 -4
View File
@@ -6,7 +6,7 @@ import (
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
@@ -55,8 +55,6 @@ func offeredOutsideBy(ctx context.Context, provision string) (string, error) {
// its next send would, and the machines that could not be resolved, with why.
func consumersFrom(ctx context.Context, open *stores, provider, providerModule, provision string) (
[]inventory.PairEnd, []string, error) {
// One reading of the mesh for every machine this looks at.
ctx = withInputCache(ctx)
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, nil, err
@@ -64,7 +62,7 @@ func consumersFrom(ctx context.Context, open *stores, provider, providerModule,
var out []inventory.PairEnd
var unresolved []string
for _, n := range nodes {
plan, _, err := resolveNode(ctx, open, n.Name)
plan, _, err := planFor(ctx, open, n.Name)
if err != nil {
unresolved = append(unresolved, fmt.Sprintf("%s: %s", n.Name, oneLine(err.Error())))
continue
+2 -5
View File
@@ -26,11 +26,8 @@ import (
//
// - **The serving controller** takes it before it does anything else — before it asserts the bus's
// objects, which are the controller's to write — waiting while another holds it, and renews it.
// Another holder seen, or the key's age passed with no renewal made, is the lease lost: the gate
// closes at once and the process exits, so its service manager restarts it as a candidate (serve,
// in push.go). A renewal the bus does not answer in time is not a loss (novox/hq issue 477): the
// lease is kept, no new act starts once half the key's age passed unrenewed, and the next renewal
// made opens the gate again.
// A renewal refused or failed is the lease lost: the gate closes at once and the process exits, so
// its service manager restarts it as a candidate (serve, in push.go).
// - **A command run at a shell** — `push` in the installer, the lab, a person repairing a mesh whose
// controller is down (issue 201) — acts **under the holder's epoch** when a controller holds the
// lease: it is the same mesh's word, composed and sent under the store's hold of each machine like
+7 -13
View File
@@ -10,7 +10,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
@@ -314,7 +314,7 @@ type afterAct struct {
// after the write, which can only find more unmet than the closure would: a seat said held there is
// held in the closure too.
func resolvedAfter(ctx context.Context, open *stores, shelf map[string]catalogue.Manifest, node string) afterAct {
plan, _, err := resolveNode(ctx, open, node)
plan, _, err := planFor(ctx, open, node)
if err == nil {
out := afterAct{plan: plan, unheld: plan.Unheld}
for _, m := range plan.Modules {
@@ -355,8 +355,6 @@ func splitModules(field string) []string {
// Nothing here can fail the act it reports on. A mesh that cannot be read is worth saying and is
// not a reason to claim the assignment did not happen — it did.
func blockedElsewhere(ctx context.Context, open *stores, except string) string {
// One reading of the mesh for every machine this looks at.
ctx = withInputCache(ctx)
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return "\n\nThe rest of the mesh could not be checked: " + err.Error()
@@ -366,7 +364,7 @@ func blockedElsewhere(ctx context.Context, open *stores, except string) string {
if n.Name == except {
continue
}
if _, _, err := resolveNode(ctx, open, n.Name); err != nil {
if _, _, err := planFor(ctx, open, n.Name); err != nil {
blocked[n.Name] = err.Error()
}
}
@@ -393,10 +391,10 @@ func blockedElsewhere(ctx context.Context, open *stores, except string) string {
}
// issueOnAssign gives a newly assigned module its bus credential, the way `module issue` does, and
// says what it did in one line. Nothing for a module that declares no broker secret and runs no
// provisioner client; nothing for one whose user is already minted (a credential is rotated on
// purpose, never by re-assigning); and when the bus cannot be reached from here, the line names the
// verb and the push that would refuse the module until it is run — never a silent placeholder (novox/hq issue 203).
// says what it did in one line. Nothing for a module that declares no broker secret; nothing for one
// whose user is already minted (a credential is rotated on purpose, never by re-assigning); and when
// the bus cannot be reached from here, the line names the verb and the push that would refuse the
// module until it is run — never a silent placeholder (novox/hq issue 203).
func issueOnAssign(ctx context.Context, open *stores, node, module string) string {
inv := open.inventory
shelf, err := inv.Catalogue(ctx)
@@ -408,10 +406,6 @@ func issueOnAssign(ctx context.Context, open *stores, node, module string) strin
return ""
}
user := broker.Principal{Kind: broker.KindModule, Node: node, Module: module}.Username()
if _, reads := m.OwnSecrets["broker"]; !reads {
// A module running the provisioner client alone is minted the client's account (issue 917).
user = broker.Principal{Kind: broker.KindProvisioner, Node: node, Module: module}.Username()
}
if _, minted, err := inv.BusUserHash(ctx, user); err != nil || minted {
return ""
}
+1 -3
View File
@@ -1,12 +1,10 @@
//go:build integration
package main
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
)
// What an assignment says about the machines it was not about.
@@ -1,228 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
)
func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aStore())
if _, err := assign(ctx, open, "anchor", "distribution"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5101}}); err != nil {
t.Fatal(err)
}
// A module the mesh built, recorded by digest and path, running on the other machine.
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "app", Commit: "abc",
Made: []inventory.Artifact{{Name: "server", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "app/server@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "app", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-app",
"image": catalogue.ArtifactStoreScheme + "app/server@" + aDigest}}})
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
// The runtime's trust is the runtime's module's to write (novox/hq ADR 0222): a stand-in for it
// asks where this machine reaches the store, as the docker module does.
register(t, open, catalogue.Manifest{Module: "runtime", Version: "1",
Resources: []map[string]any{{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json",
"into": "json", "content": `{"insecure-registries": ["${seat:mesh-artifact-store:reach}"]}` + "\n"}}})
if _, err := assign(ctx, open, "laptop", "runtime"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
if err != nil || !found || node != "anchor" || port != "5101" {
t.Fatalf("the store is found on %q:%q (%v, %v); the node put it on 5101", node, port, found, err)
}
var trust string
for _, r := range composed(t, open, "laptop").Resources {
if r["path"] == "/etc/docker/daemon.json" {
trust, _ = r["content"].(string)
}
if r["id"] == "app.server" && r["image"] != "anchor.internal:5101/app/server@"+aDigest {
t.Errorf("the image the mesh built is fetched as %v", r["image"])
}
}
if !strings.Contains(trust, "anchor.internal:5101") || strings.Contains(trust, ":5000") {
t.Fatalf("the runtime is told to trust %q; the node put the store on 5101", trust)
}
// And a replay to the catalogue says where the store is now.
announced, err := following{open: open}.Announceable(ctx)
if err != nil {
t.Fatal(err)
}
if len(announced) != 1 || announced[0].Made[0].Reference != "anchor.internal:5101/app/server@"+aDigest {
t.Fatalf("the replay announces %+v", announced)
}
}
func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "controller", Kind: catalogue.ArtifactBundle,
Reference: "https://registry.example/mesh-controller/controller.tar.gz", Digest: aDigest}})
if err != nil {
t.Fatal(err)
}
register(t, open, control)
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}, "image": "pg@" + aDigest}}})
register(t, open, catalogue.Manifest{Module: "lavinmq", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5671, From: catalogue.FromMesh},
{Port: 5672, From: catalogue.FromMesh}},
Guards: []int{15672},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}, "image": "mq@" + aDigest}}})
// The control plane's own bus user is the installer's, seeded at genesis before the controller
// runs (SeedBusUser); without it a push now refuses the credential nobody issued (issue 203).
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.mesh-controller",
Kind: inventory.BusController, Node: "anchor", Module: "mesh-controller"}, "bootstrap"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The store's module is registered and given its port, and NOT assigned: the state genesis
// leaves a given-port node in before the foundation is adopted as modules (04-ISSUES/085).
if err := open.inventory.SetSettings(ctx, "anchor", "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 6852}}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "lavinmq"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "lavinmq",
map[string]any{catalogue.PortsSetting: map[string]any{"5672": 5679}}); err != nil {
t.Fatal(err)
}
var env map[string]any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "mesh-controller.controller" {
env, _ = r["env"].(map[string]any)
}
}
if env == nil {
t.Fatal("the control plane's process is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
// Neither given nor assigned by the mesh: the manifest's own number is NOT the answer,
// because the sealed value beside it carries the port genesis wrote (finding F3).
"MESH_BROKER_ADDRESS_PORT": "",
"MESH_BROKER_MANAGEMENT_PORT": "",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node says %q", key, env[key], want)
}
}
}
func aLoneNode(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t)
licences.ForTest(t)
open, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(open.Close)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
}
}
record, err := open.inventory.AddNode(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
reported, _ := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}}})
var profile map[string]any
_ = json.Unmarshal(reported, &profile)
if err := open.inventory.RecordProfile(t.Context(), record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
return open
}
func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
open := aLoneNode(t)
ctx := t.Context()
register(t, open, aStore())
register(t, open, catalogue.Manifest{Module: "builder", Version: "1",
Requires: []string{catalogue.ArtifactStoreProvision},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-builder",
"image": "registry.example/mesh-builder@" + aDigest}}})
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "postgres", Commit: "abc",
Made: []inventory.Artifact{{Name: "runtime", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-postgres",
"image": catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}}})
for _, module := range []string{"distribution", "builder", "postgres"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5100}}); err != nil {
t.Fatal(err)
}
var image any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "postgres.runtime" {
image = r["image"]
}
}
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
+214 -1
View File
@@ -2,10 +2,13 @@ package main
import (
"encoding/json"
"os"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/licences"
"github.com/novox/mesh-controller/internal/link"
)
@@ -83,12 +86,150 @@ func aStore() catalogue.Manifest {
// **The trust a machine writes for the store, and the address every built image is fetched
// through, say the port the node gave the store** — not the catalogue's number.
func TestTheRegistryTrustAndEveryImageFollowThePortTheNodeGaveTheStore(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aStore())
if _, err := assign(ctx, open, "anchor", "distribution"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5101}}); err != nil {
t.Fatal(err)
}
// A module the mesh built, recorded by digest and path, running on the other machine.
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "app", Commit: "abc",
Made: []inventory.Artifact{{Name: "server", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "app/server@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "app", Version: "1",
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-app",
"image": catalogue.ArtifactStoreScheme + "app/server@" + aDigest}}})
if _, err := assign(ctx, open, "laptop", "app"); err != nil {
t.Fatal(err)
}
// The runtime's trust is the runtime's module's to write (novox/hq ADR 0222): a stand-in for it
// asks where this machine reaches the store, as the docker module does.
register(t, open, catalogue.Manifest{Module: "runtime", Version: "1",
Resources: []map[string]any{{"id": "daemon", "type": "file", "path": "/etc/docker/daemon.json",
"into": "json", "content": `{"insecure-registries": ["${seat:mesh-artifact-store:reach}"]}` + "\n"}}})
if _, err := assign(ctx, open, "laptop", "runtime"); err != nil {
t.Fatal(err)
}
on := map[string]bool{"anchor": true, "laptop": true}
node, port, found, err := artifactStoreOnNetwork(ctx, open.inventory, on)
if err != nil || !found || node != "anchor" || port != "5101" {
t.Fatalf("the store is found on %q:%q (%v, %v); the node put it on 5101", node, port, found, err)
}
var trust string
for _, r := range composed(t, open, "laptop").Resources {
if r["path"] == "/etc/docker/daemon.json" {
trust, _ = r["content"].(string)
}
if r["id"] == "app.server" && r["image"] != "anchor.internal:5101/app/server@"+aDigest {
t.Errorf("the image the mesh built is fetched as %v", r["image"])
}
}
if !strings.Contains(trust, "anchor.internal:5101") || strings.Contains(trust, ":5000") {
t.Fatalf("the runtime is told to trust %q; the node put the store on 5101", trust)
}
// And a replay to the catalogue says where the store is now.
announced, err := following{open}.Announceable(ctx)
if err != nil {
t.Fatal(err)
}
if len(announced) != 1 || announced[0].Made[0].Reference != "anchor.internal:5101/app/server@"+aDigest {
t.Fatalf("the replay announces %+v", announced)
}
}
// **The control plane's own connections say the port the node gave the store and the broker.**
//
// Composed from the control plane's own manifest against a real inventory: the store's module is
// given 6852 on this node the way genesis or an operator gives it, and the control plane's
// process is told so beside the sealed connection genesis wrote.
func TestTheControlPlaneIsToldWhereTheNodePutTheStoreAndTheBroker(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
raw, err := os.ReadFile("../../module.json")
if err != nil {
t.Fatal(err)
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
t.Fatal(err)
}
control, err := withSeatPorts(m).Resolve([]catalogue.Built{{Name: "controller", Kind: catalogue.ArtifactBundle,
Reference: "https://registry.example/mesh-controller/controller.tar.gz", Digest: aDigest}})
if err != nil {
t.Fatal(err)
}
register(t, open, control)
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-store", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5432, From: catalogue.FromMesh}},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-store",
"ports": []any{"5432:5432"}, "image": "pg@" + aDigest}}})
register(t, open, catalogue.Manifest{Module: "lavinmq", Version: "1",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}},
Listens: []catalogue.Listening{{Port: 5671, From: catalogue.FromMesh},
{Port: 5672, From: catalogue.FromMesh}},
Guards: []int{15672},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-broker",
"ports": []any{"5671:5671", "5672:5672", "127.0.0.1:15672:15672"}, "image": "mq@" + aDigest}}})
// The control plane's own bus user is the installer's, seeded at genesis before the controller
// runs (SeedBusUser); without it a push now refuses the credential nobody issued (issue 203).
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.mesh-controller",
Kind: inventory.BusController, Node: "anchor", Module: "mesh-controller"}, "bootstrap"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "mesh-controller"); err != nil {
t.Fatal(err)
}
// The store's module is registered and given its port, and NOT assigned: the state genesis
// leaves a given-port node in before the foundation is adopted as modules (04-ISSUES/085).
if err := open.inventory.SetSettings(ctx, "anchor", "postgres",
map[string]any{catalogue.PortsSetting: map[string]any{"5432": 6852}}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "lavinmq"); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSettings(ctx, "anchor", "lavinmq",
map[string]any{catalogue.PortsSetting: map[string]any{"5672": 5679}}); err != nil {
t.Fatal(err)
}
var env map[string]any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "mesh-controller.controller" {
env, _ = r["env"].(map[string]any)
}
}
if env == nil {
t.Fatal("the control plane's process is not in its own node's declaration")
}
for key, want := range map[string]string{
"MESH_STORE_INVENTORY_PORT": "6852",
"MESH_STORE_IDENTITY_PORT": "6852",
"MESH_STORE_LICENCES_PORT": "6852",
"MESH_BROKER_AMQP_PORT": "5679",
// Neither given nor assigned by the mesh: the manifest's own number is NOT the answer,
// because the sealed value beside it carries the port genesis wrote (finding F3).
"MESH_BROKER_ADDRESS_PORT": "",
"MESH_BROKER_MANAGEMENT_PORT": "",
} {
if env[key] != want {
t.Errorf("the control plane is told %s=%v; the node says %q", key, env[key], want)
}
}
}
// withSeatPorts is the control plane's manifest with the seat placeholders in its environment —
// added here until module.json carries them (the manifest lands one commit after the code that
@@ -133,7 +274,79 @@ func withSeatPorts(m catalogue.Manifest) catalogue.Manifest {
// aLoneNode is one capable machine with nothing placed on any network — the control-node during
// genesis, before the "network" step, which is after the store, the broker, the vault and the
// catalogue have each been built and pushed (finding F2).
func aLoneNode(t *testing.T) *stores {
t.Helper()
inventory.ForTest(t)
licences.ForTest(t)
open, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(open.Close)
for _, m := range provided {
if err := open.inventory.Provide(t.Context(), m); err != nil {
t.Fatal(err)
}
}
record, err := open.inventory.AddNode(t.Context(), "anchor")
if err != nil {
t.Fatal(err)
}
reported, _ := json.Marshal(map[string]any{"capabilities": []map[string]any{
{"name": "container-runtime", "present": true}}})
var profile map[string]any
_ = json.Unmarshal(reported, &profile)
if err := open.inventory.RecordProfile(t.Context(), record.ID, profile); err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(t.Context(), record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
return open
}
// **Before the network exists, the store's own node reaches it by loopback** — never refused,
// never handed the scheme: a genesis pushes the store, the broker, the vault and the catalogue to
// a node on no network, and builds the catalogue on a base it must be able to pull.
func TestOnANodeWithNoNetworkTheStoreIsReachedByLoopback(t *testing.T) {
open := aLoneNode(t)
ctx := t.Context()
register(t, open, aStore())
register(t, open, catalogue.Manifest{Module: "builder", Version: "1",
Requires: []string{catalogue.ArtifactStoreProvision},
Resources: []map[string]any{{"id": "server", "type": "container", "name": "mesh-builder",
"image": "registry.example/mesh-builder@" + aDigest}}})
if err := open.inventory.RecordBuild(ctx, inventory.Build{
ID: "b1", Repository: "r", Module: "postgres", Commit: "abc",
Made: []inventory.Artifact{{Name: "runtime", Kind: "image",
Reference: catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}},
}); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "postgres", Version: "1",
Resources: []map[string]any{{"id": "runtime", "type": "container", "name": "mesh-postgres",
"image": catalogue.ArtifactStoreScheme + "postgres/runtime@" + aDigest}}})
for _, module := range []string{"distribution", "builder", "postgres"} {
if _, err := assign(ctx, open, "anchor", module); err != nil {
t.Fatal(err)
}
}
if err := open.inventory.SetSettings(ctx, "anchor", "distribution",
map[string]any{catalogue.PortsSetting: map[string]any{"5000": 5100}}); err != nil {
t.Fatal(err)
}
var image any
for _, r := range composed(t, open, "anchor").Resources {
if r["id"] == "postgres.runtime" {
image = r["image"]
}
}
if image != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("on the store's own node, off any network, the image is fetched as %v", image)
}
held := heldBy(ctx)
if got := held["postgres/runtime"]; got != "127.0.0.1:5100/postgres/runtime@"+aDigest {
t.Fatalf("a builder beside the store is handed the base %q", got)
}
}
+2 -4
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
@@ -12,10 +10,10 @@ import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/overlay"
)
// novox/hq ADR 0100: taking a module is its cutover; converging a node is one act, previewed, and
+12 -10
View File
@@ -12,9 +12,9 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// A node is adopted or converged (novox/hq ADR 0100), and it is said to be adopted wherever the
@@ -243,7 +243,7 @@ func take(ctx context.Context, open *stores, node, module string, opts takeOptio
return "", err
}
if !slices.Contains(assigned, module) {
if plan, _, err := resolveNode(ctx, open, node); err == nil {
if plan, _, err := planFor(ctx, open, node); err == nil {
if why, runs := plan.Because[module]; runs {
return "", fmt.Errorf("%w: %s runs on %s because %s — assign it to %s to take it",
inventory.ErrNotAssigned, module, node, why, node)
@@ -424,9 +424,8 @@ func comparisonOf(module string, c comparison, opts takeOptions) (preview string
}
// And its settings on this machine, composed against its definition (rule 1, rule 6).
for _, layer := range c.layers {
said := layer.Whole()
keys := make([]string, 0, len(said))
for k := range said {
keys := make([]string, 0, len(layer.Values))
for k := range layer.Values {
keys = append(keys, k)
}
sort.Strings(keys)
@@ -649,7 +648,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
if err != nil {
return "", err
}
plan, _, err := resolveNode(ctx, open, node)
plan, settings, err := planFor(ctx, open, node)
if err != nil {
return "", err
}
@@ -706,8 +705,11 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
filter)
}
// What the filter is derived from: the rendering of this machine as a question reads it, with the private network.
with, err := renderingOf(ctx, open, node)
gens, err := generators(ctx, open)
if err != nil {
return "", err
}
with, _, err := renderingFor(ctx, open, node, plan, settings, gens, Reading)
if err != nil {
return "", err
}
@@ -773,7 +775,7 @@ func converge(ctx context.Context, open *stores, node string, yes bool, digest s
if _, err := inv.Assign(ctx, node, filter); err != nil {
return "", err
}
if _, _, err := resolveNode(ctx, open, node); err != nil {
if _, _, err := planFor(ctx, open, node); err != nil {
_ = inv.Unassign(ctx, node, filter)
return "", fmt.Errorf("%s cannot run %s, so it was not converged: %w", node, filter, err)
}
@@ -1,161 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
snapshot "git.novox.be/novox/mesh-sdk/go/facts"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-host/rootsearch"
)
func TestTheSelfCheckSaysAnAgentAccountThatCanBecomeRoot(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.NodeByName(ctx, n); err != nil {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAccount(ctx, n, "ops", ""); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
found = onlyMachine(found, "anchor")
if len(found) != 1 || found[0].Machine != "anchor" || found[0].Severity != conditions.Urgent ||
!strings.Contains(found[0].Said, "not judged") {
t.Fatalf("a named agent account with no verdict: %+v", found)
}
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if w.Headline == "" || w.Needs == "" || w.Resolved == "" {
t.Errorf("the condition has no plain words: %+v", w)
}
healthy := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: link.StateHealthy, Root: link.RootNever, Account: "agent"}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{healthy}}); err != nil {
t.Fatal(err)
}
if found, err = probeAgentAccounts(ctx, d); err != nil || len(onlyMachine(found, "anchor")) != 0 {
t.Fatalf("a judged agent account still fails: %+v %v", found, err)
}
if named, confined, why, err := agentConfined(ctx, inv, "anchor", time.Now()); err != nil || !named || !confined {
t.Fatalf("agentConfined on anchor: %v %v %q %v", named, confined, why, err)
}
if named, _, why, err := agentConfined(ctx, inv, "laptop", time.Now()); err != nil || named ||
!strings.Contains(why, "operator account") {
t.Fatalf("agentConfined on a machine naming none: %v %q %v", named, why, err)
}
}
func TestTheFactsCarryTheAgentAccountAsAPseudonym(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
if err := open.inventory.SetAccount(ctx, "anchor", "keeper", ""); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetAgentAccount(ctx, "anchor", "warden", ""); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
body, _ := f.Encode()
if strings.Contains(string(body), "warden") {
t.Error("the agent account's name is in the snapshot")
}
m, ok := f.Machine(snapshot.Pseudonym("machine", "anchor"))
if !ok || m.AgentAccount != snapshot.Pseudonym("account", "warden") || m.Account == m.AgentAccount {
t.Fatalf("the anchor's agent account reads as %q (operator %q)", m.AgentAccount, m.Account)
}
}
func TestASearchStillRunningAfterARestartIsNotUrgent(t *testing.T) {
if searchQuietFor != rootsearch.Quiet {
t.Fatalf("the quiet is %s and the node-engine's %s: they are one value", searchQuietFor, rootsearch.Quiet)
}
// A daily search that finishes within the quiet never leaves the account unjudged between two of them.
if rootsearch.FreshFor < rootsearch.Every+rootsearch.Quiet {
t.Fatalf("a complete search judges for %s, less than a day's search (%s) and the quiet (%s)",
rootsearch.FreshFor, rootsearch.Every, rootsearch.Quiet)
}
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.NodeByName(ctx, "anchor"); err != nil {
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
say := func(state, reason string) []conditions.Observation {
t.Helper()
// The engine's since is always now: it was just restarted.
v := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: state, Reason: reason, Root: link.RootNever,
Account: "agent", Since: time.Now()}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{v}}); err != nil {
t.Fatal(err)
}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
return onlyMachine(found, "anchor")
}
running := link.ReasonRootPending + ": the search for setuid programs, started at 19:21, has not finished yet"
healthy := func() {
t.Helper()
if found := say(link.StateHealthy, ""); len(found) != 0 {
t.Fatalf("a healthy verdict raised: %+v", found)
}
}
if found := say(link.StateUnknown, running); len(found) != 0 {
t.Fatalf("a search first seen now was raised: %+v", found)
}
if _, confined, why, _ := agentConfined(ctx, inv, "anchor", time.Now()); confined || !strings.Contains(why, "not judged") {
t.Fatalf("an account whose search runs was read as confined: %q", why)
}
// The engine restarted again and again, each statement's own since fresh: the controller's clock runs on.
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err)
}
healthy() // a complete verdict forgets when the waiting began …
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err) // … and this restart loop began past the bound
}
if found := say(link.StateUnknown, running); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search pending past the bound, by the controller's clock, was not urgent: %+v", found)
}
healthy()
incomplete := rootsearch.ReasonIncomplete + ": the search for setuid programs did not finish (last tried at " +
"19:23: timeout); it is tried again later"
if found := say(link.StateUnknown, incomplete); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search that did not finish was not urgent: %+v", found)
}
if found := say(link.StateUnhealthy, link.ReasonRoot+": in the group docker; "+running); len(found) != 1 ||
found[0].Severity != conditions.Urgent {
t.Fatalf("a way to root found while the search runs was not urgent: %+v", found)
}
}
+146 -1
View File
@@ -1,12 +1,14 @@
package main
import (
"github.com/novox/mesh-host/rootsearch"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
snapshot "github.com/novox/mesh-controller/internal/facts"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -63,6 +65,55 @@ func TestAnAgentAccountIsConfinedOnlyOnAHealthyVerdictJudgedForRoot(t *testing.T
// DA raises an urgent condition, with plain words, on a machine whose agent account is not judged unable to
// become root; a machine that names none is not its to judge.
func TestTheSelfCheckSaysAnAgentAccountThatCanBecomeRoot(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.NodeByName(ctx, n); err != nil {
if _, err := inv.AddNode(ctx, n); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAccount(ctx, n, "ops", ""); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
found = onlyMachine(found, "anchor")
if len(found) != 1 || found[0].Machine != "anchor" || found[0].Severity != conditions.Urgent ||
!strings.Contains(found[0].Said, "not judged") {
t.Fatalf("a named agent account with no verdict: %+v", found)
}
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
if w.Headline == "" || w.Needs == "" || w.Resolved == "" {
t.Errorf("the condition has no plain words: %+v", w)
}
healthy := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: link.StateHealthy, Root: link.RootNever, Account: "agent"}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{healthy}}); err != nil {
t.Fatal(err)
}
if found, err = probeAgentAccounts(ctx, d); err != nil || len(onlyMachine(found, "anchor")) != 0 {
t.Fatalf("a judged agent account still fails: %+v %v", found, err)
}
if named, confined, why, err := agentConfined(ctx, inv, "anchor", time.Now()); err != nil || !named || !confined {
t.Fatalf("agentConfined on anchor: %v %v %q %v", named, confined, why, err)
}
if named, _, why, err := agentConfined(ctx, inv, "laptop", time.Now()); err != nil || named ||
!strings.Contains(why, "operator account") {
t.Fatalf("agentConfined on a machine naming none: %v %q %v", named, why, err)
}
}
func TestTheAgentRootWordsArePlain(t *testing.T) {
w := plainWordings[kindAgentCanBecomeRoot](conditions.Observation{Kind: kindAgentCanBecomeRoot, Machine: "anchor"})
@@ -83,6 +134,28 @@ func onlyMachine(obs []conditions.Observation, machine string) []conditions.Obse
// The snapshot a merge check composes from carries the agent account as a pseudonym (novox/hq ADR 0266),
// so a change is judged against machines that name one, and the name never leaves.
func TestTheFactsCarryTheAgentAccountAsAPseudonym(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
if err := open.inventory.SetAccount(ctx, "anchor", "keeper", ""); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetAgentAccount(ctx, "anchor", "warden", ""); err != nil {
t.Fatal(err)
}
f, err := gatherFacts(ctx, open, "2.11.17")
if err != nil {
t.Fatal(err)
}
body, _ := f.Encode()
if strings.Contains(string(body), "warden") {
t.Error("the agent account's name is in the snapshot")
}
m, ok := f.Machine(snapshot.Pseudonym("machine", "anchor"))
if !ok || m.AgentAccount != snapshot.Pseudonym("account", "warden") || m.Account == m.AgentAccount {
t.Fatalf("the anchor's agent account reads as %q (operator %q)", m.AgentAccount, m.Account)
}
}
// No verb runs a `node` command that sets something: through the generic `command` verb, `node account`,
// `node agent-account` and every other `node` subcommand but list and show are refused, naming the terminal.
@@ -132,3 +205,75 @@ func TestTheNodeVerbOnlyShows(t *testing.T) {
// controller first saw it waiting — never from the engine's own "since", which a restart resets, so an engine
// restarted in a loop does not keep it quiet. The account is still not confined, and `node show` still says not
// judged; a search that failed, or a way to root found, is urgent at once.
func TestASearchStillRunningAfterARestartIsNotUrgent(t *testing.T) {
if searchQuietFor != rootsearch.Quiet {
t.Fatalf("the quiet is %s and the node-engine's %s: they are one value", searchQuietFor, rootsearch.Quiet)
}
// A daily search that finishes within the quiet never leaves the account unjudged between two of them.
if rootsearch.FreshFor < rootsearch.Every+rootsearch.Quiet {
t.Fatalf("a complete search judges for %s, less than a day's search (%s) and the quiet (%s)",
rootsearch.FreshFor, rootsearch.Every, rootsearch.Quiet)
}
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.NodeByName(ctx, "anchor"); err != nil {
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
if err := inv.SetAgentAccount(ctx, "anchor", "agent", ""); err != nil {
t.Fatal(err)
}
d := &doctor{open: open}
say := func(state, reason string) []conditions.Observation {
t.Helper()
// The engine's since is always now: it was just restarted.
v := inventory.ResourceHealth{Module: "claude-code", Resource: "claude-code.agent-account",
Kind: link.KindAccount, Target: "agent", State: state, Reason: reason, Root: link.RootNever,
Account: "agent", Since: time.Now()}
if _, err := inv.RecordHealth(ctx, inventory.NodeHealth{Node: "anchor", Contract: link.RootContract,
SaidAt: time.Now(), HeardAt: time.Now(), Resources: []inventory.ResourceHealth{v}}); err != nil {
t.Fatal(err)
}
found, err := probeAgentAccounts(ctx, d)
if err != nil {
t.Fatal(err)
}
return onlyMachine(found, "anchor")
}
running := link.ReasonRootPending + ": the search for setuid programs, started at 19:21, has not finished yet"
healthy := func() {
t.Helper()
if found := say(link.StateHealthy, ""); len(found) != 0 {
t.Fatalf("a healthy verdict raised: %+v", found)
}
}
if found := say(link.StateUnknown, running); len(found) != 0 {
t.Fatalf("a search first seen now was raised: %+v", found)
}
if _, confined, why, _ := agentConfined(ctx, inv, "anchor", time.Now()); confined || !strings.Contains(why, "not judged") {
t.Fatalf("an account whose search runs was read as confined: %q", why)
}
// The engine restarted again and again, each statement's own since fresh: the controller's clock runs on.
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err)
}
healthy() // a complete verdict forgets when the waiting began …
if _, err := inv.RootSearchPending(ctx, "anchor", time.Now().Add(-searchQuietFor-time.Minute)); err != nil {
t.Fatal(err) // … and this restart loop began past the bound
}
if found := say(link.StateUnknown, running); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search pending past the bound, by the controller's clock, was not urgent: %+v", found)
}
healthy()
incomplete := rootsearch.ReasonIncomplete + ": the search for setuid programs did not finish (last tried at " +
"19:23: timeout); it is tried again later"
if found := say(link.StateUnknown, incomplete); len(found) != 1 || found[0].Severity != conditions.Urgent {
t.Fatalf("a search that did not finish was not urgent: %+v", found)
}
if found := say(link.StateUnhealthy, link.ReasonRoot+": in the group docker; "+running); len(found) != 1 ||
found[0].Severity != conditions.Urgent {
t.Fatalf("a way to root found while the search runs was not urgent: %+v", found)
}
}
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
+1 -47
View File
@@ -33,66 +33,20 @@ const packageManagerKey = "the package manager ("
func watchApplyFailing(f *signalFacts) []conditions.Observation {
heard := map[string]bool{}
for _, m := range f.machines {
if !m.asleep() && !m.lastHeard.IsZero() && !m.silent(f.now) {
if !m.asleep() && !m.lastHeard.IsZero() && f.now.Sub(m.lastHeard) <= heartbeatBound(m.every) {
heard[m.name] = true
}
}
var out []conditions.Observation
said := map[string]bool{}
for _, d := range f.doing {
if !d.Wrong() || d.Since == nil || f.now.Sub(*d.Since) <= applyEvery || !heard[d.Node] {
continue
}
out = append(out, applyFailing(d, f.now))
said[d.Node] = true
}
// **A module's files the provisioner failed to place are a failed apply too** (novox/hq ADR 0362 §3): its client's
// newest word, judged by the same bound — the client says a failure once, and fills again on every change —
// on a machine whose node-engine applied what it was sent.
failed := map[string][]inventory.ProvReport{}
for _, r := range f.provFailed {
if r.Status != inventory.ProvFailed || f.now.Sub(r.At) <= applyEvery || !heard[r.Node] || said[r.Node] {
continue
}
failed[r.Node] = append(failed[r.Node], r)
}
for _, node := range sortedNames(failed) {
out = append(out, provisioningFailing(node, failed[node], f.now))
}
return out
}
// provisioningFailing is one machine's S21 condition for the modules whose files its provisioner client failed to
// place.
func provisioningFailing(m string, failed []inventory.ProvReport, now time.Time) conditions.Observation {
sort.Slice(failed, func(a, b int) bool { return failed[a].Module < failed[b].Module })
var names, words []string
since := failed[0].At
for _, r := range failed {
names = append(names, r.Module)
words = append(words, r.Module+": "+oneLine(orNoReason(r.Reason)))
if r.At.Before(since) {
since = r.At
}
}
o := conditions.Observation{Scope: conditions.ScopeMachine, ID: m, Kind: kindApplyFailing, Machine: m,
Severity: conditions.Warning, Said: strings.Join(words, "; "),
Resolved: m + " applies what it is sent again"}
o.Summary = fmt.Sprintf("%s's provisioner has failed to place the files of %s since %s (bound %s)", m,
strings.Join(names, ", "), since.UTC().Format("2006-01-02 15:04 MST"), ago(applyEvery))
o.Headline = m + " is not applying its update"
o.Explanation = fmt.Sprintf("The provisioner on %s has not been able to write the files of %s for %s. The "+
"details say why.", m, strings.Join(names, ", "), humanDuration(now.Sub(since)))
return o
}
func orNoReason(s string) string {
if s == "" {
return "no reason given"
}
return s
}
// applyFailing is one machine's S21 condition.
func applyFailing(d inventory.Doing, now time.Time) conditions.Observation {
m := d.Node
@@ -1,134 +0,0 @@
//go:build integration
package main
import (
"context"
"fmt"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// Issue 1334, the look: when the bus did not answer an ask of any module of a tier, no build is in flight and no event follows,
// so the look itself comes back (it fails, and the looker retries it), asks the modules again, and past the bound fails the plan.
func TestALookComesBackForATierTheBusRefusedWhole(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
for _, name := range []string{"a", "b"} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: name, Version: "1"},
inventory.Source{Repository: "novox/" + name, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
p := inventory.Plan{ID: "plan-bus", Repository: "novox/a", Branch: "main", Commit: "c1", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"a", "b"}}, Modules: map[string]*inventory.PlanModule{"a": {}, "b": {}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
bus := false
was := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
if !bus {
return "", fmt.Errorf("%w: nats: no responders available for request", inventory.ErrNoChain)
}
return "build-" + source.Repository, nil
}
t.Cleanup(func() { askABuild = was })
look := func() bool {
t.Helper()
release, err := inv.HoldPlans(ctx, true)
if err != nil {
t.Fatal(err)
}
defer release()
return advanceHeld(ctx, open)
}
read := func() inventory.Plan {
t.Helper()
got, err := inv.PlanByID(ctx, "plan-bus")
if err != nil {
t.Fatal(err)
}
return got
}
// Every module refused: the look says it failed, which has the looker come back; the plan is building and says why.
if !look() {
t.Fatal("a look that left the whole tier unasked for the bus did not fail, so nothing would come back to it")
}
if got := read(); got.State != inventory.PlanBuilding || !strings.Contains(got.Note, "bus did not answer") ||
got.Modules["a"].State != "" || got.Modules["b"].State != "" {
t.Fatalf("the plan after a refused tier: %s %q %+v %+v", got.State, got.Note, got.Modules["a"], got.Modules["b"])
}
// Again, still refused: the same fact is not saved again, and the look still comes back.
before := read().Revision
if !look() || read().Revision != before {
t.Fatalf("a second refused look failed to come back or saved the plan again (revision %d, was %d)", read().Revision, before)
}
// The bus answers at the next look: both are asked, and nothing waits any more.
bus = true
if look() {
t.Fatal("a look that asked the whole tier failed")
}
if got := read(); got.Modules["a"].State != "asked" || got.Modules["b"].State != "asked" || got.Modules["a"].BusWaitFrom != nil {
t.Fatalf("the tier was not asked once the bus answered: %+v %+v", got.Modules["a"], got.Modules["b"])
}
}
func TestALookFailsTheTierTheBusRefusedPastTheBound(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "a", Version: "1"},
inventory.Source{Repository: "novox/a", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
long := time.Now().UTC().Add(-busWaitBound - time.Minute)
p := inventory.Plan{ID: "plan-bus", Repository: "novox/a", Branch: "main", Commit: "c1", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}}, Modules: map[string]*inventory.PlanModule{"a": {BusWaitFrom: &long}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", fmt.Errorf("%w: nats: no responders available for request", inventory.ErrNoChain)
}
t.Cleanup(func() { askABuild = was })
release, err := inv.HoldPlans(ctx, true)
if err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
release()
got, err := inv.PlanByID(ctx, "plan-bus")
if err != nil {
t.Fatal(err)
}
if got.State != inventory.PlanFailed || !strings.Contains(got.Note, "the bus did not answer an ask for 10m0s (since ") {
t.Fatalf("a bus silent past its bound did not fail the plan in words: %s %q", got.State, got.Note)
}
// Rebuilt by hand while the bus still reloads: said as waiting, not as a failure and not with an empty reason.
joined, said, err := joinAPlan(ctx, open, "a")
if err != nil || !joined || !strings.Contains(said, "not asked yet: the bus did not answer") {
t.Fatalf("a rebuild left for the bus is said as %v %q %v", joined, said, err)
}
// And retried by hand: the module is named as waiting, not as asked again.
got, _ = inv.PlanByID(ctx, "plan-bus")
if got.State == inventory.PlanFailed {
got.Modules["a"].State, got.Modules["a"].Why = "failed", "x"
if err := inv.SavePlan(ctx, &got); err != nil {
t.Fatal(err)
}
said, err := retryPlan(ctx, open, "plan-bus")
if err != nil || !strings.Contains(said, "not asked yet: the bus did not answer") {
t.Fatalf("a retry left for the bus is said as %q %v", said, err)
}
}
}
-139
View File
@@ -1,139 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// Issue 1334: a bus that does not answer for a moment (it reloads its user list, and JetStream's API has no responder for a few
// seconds) is not a failed build. The module is left unasked and the plan building, and the next look asks it again; only a
// module's own fault, or a bus silent for ten minutes, fails the plan.
func aTierOfThree() (*inventory.Plan, map[string]inventory.Entry) {
p := &inventory.Plan{ID: "plan-1", State: inventory.PlanBuilding, Tiers: [][]string{{"a", "b", "c"}},
Modules: map[string]*inventory.PlanModule{}}
byName := map[string]inventory.Entry{}
for _, m := range []string{"a", "b", "c"} {
byName[m] = inventory.Entry{Source: inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main"}}
}
return p, byName
}
func TestABusThatDoesNotAnswerForAMomentDoesNotFailTheTier(t *testing.T) {
p, byName := aTierOfThree()
bus := false
was := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
if source.Repository == "novox/b" && !bus {
return "", fmt.Errorf("%w: the bucket mesh-controller_builds is not on the bus: nats: no responders available for request",
inventory.ErrNoChain)
}
return "build-" + source.Repository, nil
}
t.Cleanup(func() { askABuild = was })
for _, m := range []string{"a", "b", "c"} {
askModule(t.Context(), p, m, byName)
}
if p.State != inventory.PlanBuilding {
t.Fatalf("a bus that did not answer for a moment failed the plan: %s, %q", p.State, p.Note)
}
if p.Modules["a"].State != "asked" || p.Modules["c"].State != "asked" {
t.Fatalf("the modules the bus did answer for were not asked: %+v %+v", p.Modules["a"], p.Modules["c"])
}
if s := p.Modules["b"]; s.State != "" {
t.Fatalf("the module the bus did not answer for is %q, not left unasked", s.State)
}
if got := unaskedOf(*p); len(got) != 1 || got[0] != "b" {
t.Fatalf("the next look would ask %v, not b", got)
}
if !strings.Contains(p.Note, "bus did not answer") {
t.Fatalf("the plan says nothing of the wait: %q", p.Note)
}
// The next look, the bus answering: b is asked and the plan goes on.
bus = true
for _, m := range unaskedOf(*p) {
askModule(t.Context(), p, m, byName)
}
if p.State != inventory.PlanBuilding || p.Modules["b"].State != "asked" || p.Modules["b"].Build == "" {
t.Fatalf("b was not asked once the bus answered: %s %+v", p.State, p.Modules["b"])
}
}
func TestABusSilentForTenMinutesFailsTheTierAsBefore(t *testing.T) {
p, byName := aTierOfThree()
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", fmt.Errorf("%w: nats: no responders available for request", inventory.ErrNoChain)
}
t.Cleanup(func() { askABuild = was })
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanBuilding {
t.Fatalf("the first refusal failed the plan: %q", p.Note)
}
long := time.Now().Add(-busWaitBound - time.Minute)
p.Modules["a"].BusWaitFrom = &long
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanFailed || p.Modules["a"].State != "failed" {
t.Fatalf("a bus silent past its bound did not fail the plan: %s %+v", p.State, p.Modules["a"])
}
// Said in plain words: the bus, the wait, and since when.
want := "a could not be asked for: the bus did not answer an ask for 10m0s (since " + long.UTC().Format(time.RFC3339) +
"): this kind of record lives on the bus"
if !strings.HasPrefix(p.Note, want) || p.Modules["a"].Why != strings.TrimPrefix(want, "a could not be asked for: ")+p.Note[len(want):] {
t.Fatalf("the plan says %q, not words starting %q", p.Note, want)
}
// The wait has ended: a Retry made while the bus still reloads waits again, and does not fail at its first refusal.
if p.Modules["a"].BusWaitFrom != nil {
t.Fatalf("a failed module still holds the start of its wait: %v", p.Modules["a"].BusWaitFrom)
}
p.State = inventory.PlanBuilding
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanBuilding || p.Modules["a"].State != "" || p.Modules["a"].BusWaitFrom == nil {
t.Fatalf("a retry made while the bus is silent did not wait afresh: %s %+v", p.State, p.Modules["a"])
}
}
func TestAPlanWithAModuleWaitingForTheBusIsOneTheLookComesBackTo(t *testing.T) {
p, _ := aTierOfThree()
if busWaiting(*p) {
t.Fatal("a plan whose modules are not asked yet waits for the bus")
}
now := time.Now()
p.Modules["b"] = &inventory.PlanModule{BusWaitFrom: &now}
p.Modules["a"] = &inventory.PlanModule{State: "asked"}
if got := busWaitingIn(*p); len(got) != 1 || got[0] != "b" {
t.Fatalf("the modules waiting for the bus are %v, not b", got)
}
p.State = inventory.PlanFailed
if busWaiting(*p) {
t.Fatal("a plan that is not open is waited for")
}
}
func TestAModulesOwnFaultStillFailsThePlan(t *testing.T) {
p, byName := aTierOfThree()
delete(byName, "a")
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanFailed || !strings.Contains(p.Note, "no longer in the catalogue") {
t.Fatalf("a module gone from the catalogue did not fail the plan: %s %q", p.State, p.Note)
}
p, byName = aTierOfThree()
was := askABuild
askABuild = func(context.Context, buildSource, string, string) (string, error) {
return "", errors.New("git.example/novox/a is no repository the catalogue builds a module from")
}
t.Cleanup(func() { askABuild = was })
askModule(t.Context(), p, "a", byName)
if p.State != inventory.PlanFailed || p.Modules["a"].State != "failed" {
t.Fatalf("a module's own refusal did not fail the plan: %s %+v", p.State, p.Modules["a"])
}
}
-87
View File
@@ -1,87 +0,0 @@
package main
import (
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/inventory"
)
// busWaitBound is how long a module whose ask the bus did not answer is asked again at every look before the plan fails
// (issue 1334).
const busWaitBound = 10 * time.Minute
// busDidNotAnswer says an ask failed because the bus's JetStream had no responder at that moment, as it has for a few seconds
// when the bus reloads its user list (novox/hq issue 353), and not because of the module: the same ask is made again.
func busDidNotAnswer(err error) bool {
if err == nil {
return false
}
if errors.Is(err, nats.ErrNoResponders) || errors.Is(err, nats.ErrJetStreamNotEnabled) ||
errors.Is(err, jetstream.ErrJetStreamNotEnabled) || errors.Is(err, inventory.ErrNoChain) {
return true
}
// An error that crossed a boundary as text only.
text := err.Error()
return strings.Contains(text, "no responders available") || strings.Contains(text, "jetstream not enabled")
}
// waitForTheBus leaves a module whose ask the bus did not answer unasked, the plan building, and says so in the plan's note: the
// next look asks the tier's unasked modules again. It says false, so the caller fails the plan as for any refusal, once the bus
// has been silent for busWaitBound.
func waitForTheBus(p *inventory.Plan, state *inventory.PlanModule, name string, err error, now time.Time) bool {
if state.BusWaitFrom == nil {
state.BusWaitFrom = &now
}
if now.Sub(*state.BusWaitFrom) > busWaitBound {
return false
}
state.State, state.Why = "", ""
p.Note = fmt.Sprintf("tier %d: %s not asked: the bus did not answer (%v); asked again at the next look, failing after %s",
p.Tier, name, firstLine(err.Error()), busWaitBound)
return true
}
// busWaiting says an open plan has a module of its current tier that the bus did not answer an ask of: the look must come back
// for it, as nothing else will when no build of the tier is in flight (issue 1334).
func busWaiting(p inventory.Plan) bool {
return len(busWaitingIn(p)) > 0
}
// busWaitingIn are the modules of the plan's current tier left unasked because the bus did not answer, sorted.
func busWaitingIn(p inventory.Plan) []string {
if !p.Open() || p.Tier >= len(p.Tiers) {
return nil
}
var out []string
for _, m := range p.Tiers[p.Tier] {
if s := p.Modules[m]; s != nil && s.State == "" && s.BusWaitFrom != nil {
out = append(out, m)
}
}
sort.Strings(out)
return out
}
// busWaitWords is how a verb says modules are waiting for the bus: not asked yet, and asked again by the plan at its next look.
func busWaitWords(p inventory.Plan, modules []string) string {
return fmt.Sprintf("%s not asked yet: the bus did not answer; %s asks again at its next look", strings.Join(modules, ", "), p.ID)
}
// slicesWithout is names without those in drop, in order.
func slicesWithout(names, drop []string) []string {
var out []string
for _, n := range names {
if !slices.Contains(drop, n) {
out = append(out, n)
}
}
return out
}
+4 -63
View File
@@ -208,11 +208,8 @@ type asker struct {
// raise keeps the asker's own condition (sourceAsker): which conditions needing the operator could not be
// asked, and why. Nil raises nothing (a test that does not look).
raise func(ctx context.Context, obs []conditions.Observation) error
// tell tells the operator one plain notice that needs no answer (operator-channel.tell, novox/hq ADR 0380):
// the refusal of an answer to a plan's ask (ADR 0449 §3). Nil tells nothing (a test that does not look).
tell func(ctx context.Context, headline string, lines []string) error
now func() time.Time
logf func(string, ...any)
now func() time.Time
logf func(string, ...any)
saidNoRouter bool
saidNoGrant bool
@@ -409,9 +406,7 @@ func (a *asker) reconcile(ctx context.Context) error {
for _, r := range all {
k := partKey(r.Condition, r.Part)
if r.State == string(asks.OutcomeChosen) && now.Sub(r.Ended) < askAgainAfterAnswer {
if prior, has := answered[k]; !has || r.Ended.After(prior.Ended) {
answered[k] = r
}
answered[k] = r
}
if r.State == string(asks.OutcomeRefused) {
if prior, has := refused[k]; !has || r.Opened.After(prior.Opened) {
@@ -502,9 +497,7 @@ func (a *asker) reconcile(ctx context.Context) error {
continue // refused lately, and nothing it was refused for has changed: asked again after the wait
}
}
// **A condition raised after the answered ask ended is a new one** (novox/hq ADR 0449 §2): a retried
// plan that fails again is asked again, not held back for the hour its earlier answer holds.
if r, done := answered[key]; done && sameAsked(r.Actions, p.actions) && !c.Raised.After(r.Ended) {
if r, done := answered[key]; done && sameAsked(r.Actions, p.actions) {
if _, held := byCondition[key]; !held {
continue
}
@@ -643,8 +636,6 @@ func doesWords(act conditions.Action) string {
return "the delivery starts"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["stop"] != "":
return "the delivery is stopped"
case act.Verb == broker.ControllerSeat+".plans" && act.Arguments["retry"] != "":
return "the failed builds are asked again"
case strings.HasSuffix(act.Verb, ".restart"):
return "its service is restarted on " + act.Machine
}
@@ -893,7 +884,6 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
ended := a.now()
if acted != nil {
outcome = "failed: " + acted.Error()
a.tellRefused(ctx, *r, act, acted)
}
r.Ended, r.Acted = ended, outcome
if _, err := a.store.Change(ctx, r.ID, func(x *asked) bool {
@@ -928,55 +918,6 @@ func (a *asker) Decided(ctx context.Context, body []byte) error {
return nil
}
// tellRefused tells the operator what became of the answer they gave to a plan's ask, when it was not done (novox/hq
// ADR 0449 §3): the controller's `plans` verb refuses a Retry, Start or Stop when the plan moved on between the ask and
// the answer, and the refusal was only recorded, so the operator who pressed the answer heard nothing. **A refusal is
// told as one, and nothing else is**: a verb that said no changed nothing, but a call that got no answer (a timeout, a
// refused grant) may have run, and is told as not known. One plain notice with the ask's headline, beside the hand act
// the caller still records. An answer to any other ask is not told here. A notice that cannot be told is said in the
// log and changes nothing.
func (a *asker) tellRefused(ctx context.Context, r asked, act conditions.Action, failure error) {
if a.tell == nil || act.Verb != broker.ControllerSeat+".plans" {
return
}
var lines []string
var refused verbRefused
if errors.As(failure, &refused) {
lines = []string{
fmt.Sprintf("Your answer, %s, could not be done: %s.", act.Label, strings.TrimRight(refusalWords(refused.words, act), ".")),
"Nothing was changed.",
}
} else {
lines = []string{
fmt.Sprintf("Your answer, %s, was given, and it is not known whether it was done.", act.Label),
"Look at the delivery before you answer again.",
}
}
if err := a.tell(ctx, r.Ask.Headline, lines); err != nil {
a.logf("the operator could NOT be told that %s was refused (ask %s): %v", act.Label, r.ID, err)
}
}
// refusalWords is a verb's refusal as the operator reads it: the plans named as "the plan" (plainRefusal), and, where
// the words still carry an identifier the operator's channel withholds, one sentence of the answer's own.
func refusalWords(words string, act conditions.Action) string {
return plainRefusalOr(words, refusalFallback(act))
}
// refusalFallback is what an answer's refusal says when its own words cannot be said: one sentence for each answer
// to a plan's ask, never another answer's.
func refusalFallback(act conditions.Action) string {
switch {
case act.Arguments["retry"] != "":
return "a retry would not run now; the details say why"
case act.Arguments["go"] != "":
return "the delivery could not be started now; the details say why"
case act.Arguments["stop"] != "":
return "the delivery could not be stopped now; the details say why"
}
return "the controller refused it; the details say why"
}
// handActWarrant is the verb an act the operator chose on a warrant is recorded under: a person's decision,
// never a repair (handActVerbs).
const handActWarrant = "warrant"
-2
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
@@ -1,141 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// 1315/AC3: the operator's Retry on a failed plan, acted on through the real callAction and the controller's own
// `plans` verb, sets the plan building again at its tier and ends its plan-failed condition (novox/hq ADR 0449 §1).
func TestAWarrantForRetryRetriesAFailedPlanInTheStoreAndEndsItsCondition(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
asked := asksRecorded(t)
planMovedHeard(t) // the bus the verb says its moves on, and the hand-act log
twoTiers(t, open)
before := time.Now().UTC().Add(-time.Hour)
ended := before.Add(time.Minute)
failed := inventory.Plan{ID: "plan-1315-retry", Repository: "novox/a", Branch: "main", Commit: "c0ffee",
Created: before, State: inventory.PlanFailed, Tier: 0, Tiers: [][]string{{"a"}, {"b"}},
Note: "a failed to build in tier 0",
Delivery: &inventory.PlanDelivery{},
Modules: map[string]*inventory.PlanModule{"a": {State: "failed", AskedAt: &before, Build: "build-1",
Why: link.KilledByHand}},
Times: &inventory.PlanTimes{Cut: &before, Started: &before, Ended: &ended, Class: inventory.ClassLeaf}}
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
conn, err := nats.Connect(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
defer conn.Close()
// The controller's seat answers `plans` as it does: by running the verb.
sub, err := conn.Subscribe(link.SeatToolSubject("mesh-controller", "plans"), func(m *nats.Msg) {
var args map[string]string
_ = json.Unmarshal(m.Data, &args)
answer := link.Answer{Result: json.RawMessage(`"ok"`)}
if len(args) != 1 || args["retry"] == "" {
answer = link.Answer{Error: "the retry was asked with more than the plan"}
} else if err := plansCommand(context.Background(), []string{"retry", args["retry"]}); err != nil {
answer = link.Answer{Error: err.Error()}
}
body, _ := json.Marshal(answer)
_ = m.Respond(body)
})
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
// What S23 raises for it, read from the store as the watchdog reads it.
raised := func(context.Context) ([]conditions.Condition, error) {
facts, err := gatherFailedPlans(ctx, open.inventory, time.Now())
if err != nil {
return nil, err
}
var out []conditions.Condition
for _, o := range watchPlansFailed(&signalFacts{now: time.Now(), failed: facts}) {
out = append(out, conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions})
}
return out, nil
}
now := time.Now()
store := memAskedStore{}
var sent []published
var acts []link.HandAct
a := &asker{open: raised, store: store,
silence: func(context.Context, string, time.Duration, string, string) error { return nil },
publish: func(_ context.Context, subject string, body []byte, id string) error {
sent = append(sent, published{subject, id, body})
return nil
},
call: callAction(conn),
record: func(_ context.Context, act link.HandAct) error { acts = append(acts, act); return nil },
now: func() time.Time { return now },
logf: t.Logf}
cs, _ := raised(ctx)
if len(cs) != 1 || len(cs[0].Actions) != 1 {
t.Fatalf("the failed plan raised %+v", cs)
}
if err := a.reconcile(ctx); err != nil {
t.Fatal(err)
}
var w asks.Warrant
for _, r := range store {
for _, o := range r.Ask.Options {
if o.Label == "Retry" {
w = asks.Warrant{Ask: r.ID, Asker: "mesh-controller", About: r.Condition, Outcome: asks.OutcomeChosen,
Option: o.ID, Label: o.Label, Level: o.Level, Channel: "telegram", Proofs: []string{"P1"}, At: now,
AskDigest: r.Ask.Digest(),
By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "42", Verified: "user id verified"}}
}
}
}
if w.Ask == "" {
t.Fatalf("no ask offers Retry: %+v", store)
}
body, _ := json.Marshal(w)
if err := a.Decided(ctx, body); err != nil {
t.Fatal(err)
}
if err := a.Decided(ctx, body); err != nil { // heard again: nothing more
t.Fatal(err)
}
p, err := open.inventory.PlanByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
m := p.Modules["a"]
if p.State != inventory.PlanBuilding || p.Tier != 0 || m.State != "asked" || m.Build == "" || m.Build == "build-1" {
t.Fatalf("after the warrant the plan is %s at tier %d and a is %+v", p.State, p.Tier, m)
}
if len(*asked) != 1 {
t.Fatalf("the build was asked %d times: %v", len(*asked), *asked)
}
if got := store[w.Ask]; got.Acted != "done" {
t.Fatalf("the ask ended %q", got.Acted)
}
if len(acts) != 1 || acts[0].Verb != handActWarrant || !strings.Contains(strings.Join(acts[0].Args, " "), "retry="+failed.ID) {
t.Fatalf("the warrant was recorded as %+v", acts)
}
if cs, _ = raised(ctx); len(cs) != 0 {
t.Fatalf("the plan-failed condition did not end: %+v", cs)
}
}
-225
View File
@@ -77,15 +77,6 @@ type askerRig struct {
silenced []string
acts []link.HandAct
now time.Time
// refuses, when set, is what every verb the asker calls answers; told is every notice the operator was told.
refuses error
told []told
}
// told is one plain notice the asker told the operator.
type told struct {
headline string
lines []string
}
func newAskerRig(t *testing.T) *askerRig {
@@ -111,10 +102,6 @@ func newAskerRig(t *testing.T) *askerRig {
call: func(_ context.Context, a conditions.Action, args map[string]string) error {
raw, _ := json.Marshal(args)
r.called = append(r.called, a.Verb+"@"+a.Machine+" "+string(raw))
return r.refuses
},
tell: func(_ context.Context, headline string, lines []string) error {
r.told = append(r.told, told{headline, lines})
return nil
},
record: func(_ context.Context, act link.HandAct) error { r.acts = append(r.acts, act); return nil },
@@ -1026,215 +1013,3 @@ func TestTheServingAskerJudgesTheGrantForReal(t *testing.T) {
t.Errorf("held %v (%q, %v) with nothing to read", held, why, err)
}
}
// failedPlanCondition is the condition S23 raises for a failed delivery plan, as the keeper holds it.
func failedPlanCondition(plan string) conditions.Condition {
p := aFailedPlan(plan, nil)
f := calm(time.Date(2026, 10, 19, 18, 30, 0, 0, time.UTC))
f.failed = []failedPlanFacts{failedFactsFor(p)}
o := watchPlansFailed(f)[0]
return conditions.Condition{Key: o.Key(), Kind: o.Kind, Severity: o.Severity, Headline: o.Headline,
Explanation: conditions.Verdict(o.Needs, o.Explanation), Needs: o.Needs, Actions: o.Actions}
}
// 1315/AC1: a failed plan's Retry is asked at the approve level, in the words "the failed builds are asked again".
func TestAFailedPlansRetryIsAskedAtApproveInItsOwnWords(t *testing.T) {
r := newAskerRig(t)
c := failedPlanCondition("plan-1792433168676870614")
r.open = []conditions.Condition{c}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
sent := r.asksSent(t)
if len(sent) != 1 || len(sent[0].Options) != 1 {
t.Fatalf("asked %+v", sent)
}
o := sent[0].Options[0]
if o.Label != "Retry" || o.Level != asks.Approve || o.Does != "the failed builds are asked again" {
t.Errorf("the option is %+v", o)
}
if err := sent[0].Check(r.now); err != nil {
t.Errorf("the ask is refused as made: %v", err)
}
if strings.Contains(sent[0].Explanation, "mesh MCP server") {
t.Errorf("the ask still says where else to answer: %q", sent[0].Explanation)
}
}
// 1315/AC2: the warrant calls plans with retry=<plan> and nothing else, once, and is recorded as the hand act;
// the same warrant heard again does nothing.
func TestAWarrantForRetryCallsPlansRetryOnceAndIsRecorded(t *testing.T) {
plan := "plan-1792433168676870614"
r := newAskerRig(t)
c := failedPlanCondition(plan)
r.open = []conditions.Condition{c}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, c.Key, "Retry")
answerWith(t, r, w)
answerWith(t, r, w)
if want := `mesh-controller.plans@ {"retry":"` + plan + `"}`; len(r.called) != 1 || r.called[0] != want {
t.Fatalf("called %v, want exactly %s", r.called, want)
}
if len(r.acts) != 1 {
t.Fatalf("hand-acts %+v", r.acts)
}
act := r.acts[0]
if act.Verb != handActWarrant || strings.Join(act.Args, " ") != "mesh-controller.plans retry="+plan ||
act.Ask != w.Ask || act.Condition != c.Key || act.Outcome != "done" || act.Cause != conditions.CauseOperatorAnswer {
t.Errorf("the hand-act is %+v", act)
}
if got := r.store[w.Ask]; got.Acted != "done" {
t.Errorf("kept %+v", got)
}
}
// 1316/AC1 and AC2: a Retry or a Stop on a plan that moved on is refused by `plans`; the operator who pressed it is
// told once, with the ask's headline and the refusal's words, and the hand act records the failure as before.
func TestARefusedAnswerToAPlansAskIsToldOnceAndStillRecorded(t *testing.T) {
plan := "plan-1792433168676870614"
waiting := conditions.Condition{Key: "plan." + plan + ".waiting", Severity: conditions.Urgent,
Headline: "openrazer delivery waiting to start", Needs: "start it, or stop it.",
Explanation: "Needs you: start it, or stop it.", Actions: waitingActions(plan, conditions.Urgent)}
for _, tc := range []struct {
name string
c conditions.Condition
label string
}{
{"retry", failedPlanCondition(plan), "Retry"},
{"stop", waiting, "Stop"},
} {
t.Run(tc.name, func(t *testing.T) {
r := newAskerRig(t)
r.refuses = verbRefused{verb: "mesh-controller.plans", words: plan + " is done; there is nothing to retry"}
r.open = []conditions.Condition{tc.c}
_ = r.a.reconcile(context.Background())
w := r.warrantFor(t, tc.c.Key, tc.label)
answerWith(t, r, w)
answerWith(t, r, w) // heard again: told once
if len(r.told) != 1 {
t.Fatalf("told %+v", r.told)
}
got := r.told[0]
if got.headline != tc.c.Headline {
t.Errorf("the headline is %q, not the ask's %q", got.headline, tc.c.Headline)
}
said := strings.Join(got.lines, " ")
if !strings.Contains(said, "the plan is done; there is nothing to retry") || !strings.Contains(said, tc.label) {
t.Errorf("the lines do not carry the answer and the refusal's words: %q", said)
}
if why, ok := conditions.Plain(said); !ok {
t.Errorf("the notice is not plain, so the channel would withhold it: %s: %q", why, said)
}
if len(r.acts) != 1 || !strings.HasPrefix(r.acts[0].Outcome, "failed: ") {
t.Errorf("the failure is not recorded as before: %+v", r.acts)
}
if kept := r.store[w.Ask]; !strings.HasPrefix(kept.Acted, "failed: ") {
t.Errorf("kept %+v", kept)
}
})
}
}
// 1316/AC3: an answer that is done tells nothing; and a refusal of an answer that is no plan's is not told here.
func TestADoneAnswerTellsNothingAndOnlyAPlansRefusalIsTold(t *testing.T) {
r := newAskerRig(t)
c := failedPlanCondition("plan-1792433168676870614")
r.open = []conditions.Condition{c}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, c.Key, "Retry"))
if len(r.told) != 0 {
t.Fatalf("a retry that was done told %+v", r.told)
}
r = newAskerRig(t)
r.refuses = verbRefused{verb: "node-service-manager.restart", words: "unit not found"}
module := conditions.Condition{Key: "module.openrazer.g14.unhealthy", Severity: conditions.Warning,
Headline: "openrazer not working on g14", Needs: "restart its service openrazer-daemon on g14.",
Explanation: "Needs you: restart it.", Actions: []conditions.Action{{Label: "Restart",
Verb: "node-service-manager.restart", Machine: "g14", Level: conditions.LevelApprove,
Arguments: map[string]string{"unit": "openrazer-daemon.service", "scope": "user"}}}}
r.open = []conditions.Condition{module}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, module.Key, "Restart"))
if len(r.told) != 0 || len(r.acts) != 1 {
t.Fatalf("a refused restart told %+v, acts %+v", r.told, r.acts)
}
}
// 1316, review round 1: a call that got no answer may have run, so it is told as not known, never as "nothing was
// changed"; and the sentence a refusal falls back to fits the answer given.
func TestAnAnswerThatGotNoAnswerIsToldAsNotKnownNotAsRefused(t *testing.T) {
plan := "plan-1792433168676870614"
r := newAskerRig(t)
r.refuses = errors.New("mesh-controller did not answer plans within 30s")
c := failedPlanCondition(plan)
r.open = []conditions.Condition{c}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, c.Key, "Retry"))
if len(r.told) != 1 {
t.Fatalf("told %+v", r.told)
}
said := strings.Join(r.told[0].lines, " ")
if !strings.Contains(said, "not known whether") || strings.Contains(said, "Nothing was changed") {
t.Errorf("a call that may have run is told as %q", said)
}
if why, ok := conditions.Plain(said); !ok {
t.Errorf("not plain: %s: %q", why, said)
}
}
func TestEachAnswersRefusalFallsBackToItsOwnSentence(t *testing.T) {
plan := "plan-1792433168676870614"
for _, tc := range []struct {
act conditions.Action
want string
}{
{retryAction(plan), "a retry would not run now"},
{conditions.Action{Label: "Start", Verb: "mesh-controller.plans", Arguments: map[string]string{"go": plan}}, "could not be started now"},
{conditions.Action{Label: "Stop", Verb: "mesh-controller.plans", Arguments: map[string]string{"stop": plan}}, "could not be stopped now"},
} {
// Words the operator's channel withholds: a hash and a dotted name.
got := refusalWords("build 48bda475dunst at mesh.seat.x refused", tc.act)
if !strings.Contains(got, tc.want) {
t.Errorf("%s: %q, want %q", tc.act.Label, got, tc.want)
}
if _, ok := conditions.Plain(got); !ok {
t.Errorf("%s: the fallback is not plain: %q", tc.act.Label, got)
}
}
}
// 1314/AC6, ADR 0449 §2: a retried plan that fails again within the hour of the answer is asked again, because the
// condition that comes back is a new one, raised after the answered ask ended; the same condition still open is not.
func TestARetriedPlanThatFailsAgainWithinTheHourIsAskedAgain(t *testing.T) {
r := newAskerRig(t)
c := failedPlanCondition("plan-1792433168676870614")
c.Raised = r.now.Add(-time.Minute)
r.open = []conditions.Condition{c}
_ = r.a.reconcile(context.Background())
answerWith(t, r, r.warrantFor(t, c.Key, "Retry"))
// Still open right after the answer: not asked again at once.
r.now = r.now.Add(time.Minute)
_ = r.a.reconcile(context.Background())
if n := len(r.asksSent(t)); n != 1 {
t.Fatalf("asked again while the same condition stayed open: %d", n)
}
// The retry ran, the plan left failed and the condition ended.
r.now = r.now.Add(time.Minute)
r.open = nil
_ = r.a.reconcile(context.Background())
// It fails again, ten minutes after the answer: a condition raised after the answered ask ended.
r.now = r.now.Add(8 * time.Minute)
again := failedPlanCondition("plan-1792433168676870614")
again.Raised = r.now
r.open = []conditions.Condition{again}
if err := r.a.reconcile(context.Background()); err != nil {
t.Fatal(err)
}
if n := len(r.asksSent(t)); n != 2 {
t.Fatalf("the second failure was asked %d times in all, want 2", n)
}
if len(r.called) != 1 {
t.Fatalf("something retried without an answer: %v", r.called)
}
}
+3 -26
View File
@@ -19,8 +19,8 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -135,12 +135,6 @@ func (b busAsked) All(ctx context.Context) ([]asked, error) {
return out, nil
}
// verbRefused is a verb's own refusal of an act: the seat answered, and said no. Any other error of callAction (a bus
// that did not answer, a grant that is refused) is not one: the verb may have run.
type verbRefused struct{ verb, words string }
func (e verbRefused) Error() string { return e.verb + " refused: " + e.words }
// callAction performs an action's verb as the controller, through the grant that names it.
func callAction(conn *nats.Conn) func(ctx context.Context, a conditions.Action, args map[string]string) error {
return func(ctx context.Context, a conditions.Action, args map[string]string) error {
@@ -174,23 +168,7 @@ func callAction(conn *nats.Conn) func(ctx context.Context, a conditions.Action,
return err
}
if answer.Error != "" {
return verbRefused{verb: a.Verb, words: answer.Error}
}
return nil
}
}
// tellThrough tells the operator one plain notice through the operator channel's `tell`, over the controller's own
// connection and grant (novox/hq ADR 0380); a holder's refusal is an error naming it.
func tellThrough(conn *nats.Conn) func(ctx context.Context, headline string, lines []string) error {
return func(ctx context.Context, headline string, lines []string) error {
answer, err := link.AskMeshSeatTool(ctx, conn, "operator-channel", "tell",
map[string]any{"headline": headline, "lines": lines}, tellWithin)
if err != nil {
return err
}
if answer.Error != "" {
return fmt.Errorf("operator-channel.tell refused: %s", answer.Error)
return fmt.Errorf("%s refused: %s", a.Verb, answer.Error)
}
return nil
}
@@ -223,7 +201,7 @@ func routerRecordOf(conn *nats.Conn, inv *inventory.Inventory) func(ctx context.
}
// asksRecords is the bucket the asks seat's declarer keeps its record of asks in.
func asksRecords(ctx context.Context, inv catalogueReader) (string, error) {
func asksRecords(ctx context.Context, inv *inventory.Inventory) (string, error) {
declared, err := inv.Catalogue(ctx)
if err != nil {
return "", err
@@ -598,7 +576,6 @@ func servingAsker(open *stores, conn *nats.Conn, js jetstream.JetStream, keeper
_, err := link.RecordHandAct(ctx, conn, act)
return err
},
tell: tellThrough(conn),
routerRecord: routerRecordOf(conn, open.inventory),
routerHere: routerHereIn(open.inventory),
grantHeld: grantHeldIn(open, conn),
@@ -1,186 +0,0 @@
//go:build integration
package main
import (
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
func TestAModuleAssignedAndNotPushedIsAwaitingAPushNotUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop") // the machine was pushed before; then the module is assigned
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
// The read the rest of the mesh asks still refuses it, with the composer's typed error: nothing
// can be compared about a secret that does not exist (status), and nothing here string-matches.
_, _, err := composeQuestion(ctx, open, "laptop", compose.Reading, false)
var notMade *catalogue.NotMadeError
if !errors.As(err, &notMade) || notMade.Module != "keeper" || notMade.Name != "repository" {
t.Fatalf("a read composition did not say which secret is not made, typed: %v", err)
}
// Foreseen, it composes, names what the push makes, and made nothing.
foreseen, _, err := composeQuestion(ctx, open, "laptop", compose.Foreseeing, false)
if err != nil || len(foreseen.foreseen) != 1 || foreseen.foreseen[0] != "keeper/repository" {
t.Fatalf("foreseen: %v %v", foreseen.foreseen, err)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "keeper", "repository"); err != nil || held {
t.Fatalf("asking ahead of the push made the secret (held %v, %v)", held, err)
}
// Within the bound: nothing at all — no urgent, no warning, nobody notified.
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a machine waiting for a push raised %+v", got)
}
// Past the bound: a warning, not urgent, saying what the push will make.
before := awaitingPushBound
awaitingPushBound = -time.Minute
t.Cleanup(func() { awaitingPushBound = before })
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.awaiting-push" || got[0].Severity != conditions.Warning ||
!strings.Contains(got[0].Summary, "keeper/repository") || !strings.Contains(got[0].Summary, "push laptop") {
t.Fatalf("a machine left un-pushed past the bound: %+v", got)
}
// Pushed: the secret is made and D1 says nothing.
pushedBy(t, open, "laptop")
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a pushed machine still raised %+v", got)
}
}
func TestASecretThePushCannotMakeIsStillUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aTalker())
if _, err := assign(ctx, open, "laptop", "talker"); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "module issue talker --node laptop") {
t.Fatalf("a push that will be refused was not said urgently: %+v", got)
}
// And a machine whose composition fails for anything else, with a secret waiting beside it, is
// uncomposable for that — the stand-in hides nothing.
register(t, open, aKeeper())
if _, err := assign(ctx, open, "anchor", "keeper"); err != nil {
t.Fatal(err)
}
one, two := rivals()
register(t, open, one)
register(t, open, two)
_, _ = assign(ctx, open, "anchor", "rival-one")
_, _ = assign(ctx, open, "anchor", "rival-two")
keys := map[string]conditions.Severity{}
for _, o := range d1(t, open) {
keys[o.Key()] = o.Severity
}
if keys["machine.anchor.uncomposable"] != conditions.Urgent {
t.Fatalf("a real failure beside a waiting secret: %v", keys)
}
}
func TestAGivenSecretUnderAnOldKeyIsNotForeseen(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "keeper", "repository", "given"); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || !strings.Contains(got[0].Summary, "issue it again") {
t.Fatalf("a given secret under an old key: %+v", got)
}
}
func TestAMachineAwaitsAPushSinceItsOldestAssignmentSinceTheLastSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
sent := time.Now()
since, err := open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.After(sent) {
t.Fatalf("nothing assigned since the send: waiting since %v (%v), the send was before %v", since, err, sent)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
since, err = open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.Before(sent.Add(-time.Second)) {
t.Fatalf("assigned after the send: waiting since %v (%v), not from the assignment", since, err)
}
}
func TestALastSendIsReadFromTheSendAndTheReport(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
// What a machine was sent is read from the commits its assignments name (issue 1109): a module is carried by a send only
// when it was built from a commit, and a hand send on a machine with nothing assigned leaves no key to mark unknown.
if err := open.inventory.RegisterModule(ctx, aKeeper(), inventory.Source{BuiltFrom: "c-keeper"}); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSent(ctx, record.ID, "carried", map[string]string{overlay.Name: "c-overlay"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
sends, err := readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("a holder assigned and not pushed is expected to answer")
}
if !sends["laptop"].settled(overlay.Name, time.Now().Add(time.Hour)) {
t.Fatal("a module the machine was sent long ago is not expected to answer")
}
pushedBy(t, open, "laptop")
sends, err = readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("pushed a moment ago and not reported, a holder is already expected")
}
if !sends["laptop"].settled("keeper", time.Now().Add(reportGrace+time.Minute)) {
t.Fatal("pushed past the grace, a holder is not expected")
}
if _, ok := sends["laptop"].carried["keeper"]; !ok {
t.Fatalf("the send's builds do not carry keeper: %v", sends["laptop"].carried)
}
pushedAndApplied(t, open, "laptop")
if sends, err = readDeliveries(ctx, open.inventory); err != nil || !sends["laptop"].settled("keeper", time.Now()) {
t.Fatalf("pushed and reported applied, a holder is not expected to answer (%v)", err)
}
}
+169 -16
View File
@@ -1,12 +1,15 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/overlay"
)
// Between `assign` and `push` a module's own secrets are not made yet: the push makes them. D1 composed
@@ -29,7 +32,7 @@ func aKeeper() catalogue.Manifest {
func pushedBy(t *testing.T, open *stores, node string) string {
t.Helper()
ctx := t.Context()
plan, settings, err := resolveNode(ctx, open, node)
plan, settings, err := planFor(ctx, open, node)
if err != nil {
t.Fatal(err)
}
@@ -46,20 +49,7 @@ func pushedBy(t *testing.T, open *stores, node string) string {
t.Fatal(err)
}
digest := digestOf(body)
// A push records the build of each module it carries (plan.go, Allocating); a module is carried by a send only when it was
// built from a commit, for what a machine was sent is read from the commits its assignments name (issue 1109).
builds := declared.Builds
if builds == nil {
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
t.Fatal(err)
}
builds = map[string]string{}
for _, m := range plan.Modules {
builds[m.Module] = current[m.Module].Commit
}
}
if err := open.inventory.RecordSent(ctx, record.ID, digest, builds); err != nil {
if err := open.inventory.RecordSent(ctx, record.ID, digest, declared.Builds); err != nil {
t.Fatal(err)
}
return digest
@@ -91,14 +81,140 @@ func d1(t *testing.T, open *stores) []conditions.Observation {
// **THE WINDOW, REPRODUCED**: assigned and not pushed, a module whose own secret the push makes is
// waiting, not uncomposable; past the bound it is a warning naming what the push makes; pushed, nothing.
func TestAModuleAssignedAndNotPushedIsAwaitingAPushNotUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop") // the machine was pushed before; then the module is assigned
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
// The read the rest of the mesh asks still refuses it, with the composer's typed error: nothing
// can be compared about a secret that does not exist (status), and nothing here string-matches.
plan, settings, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
gens, err := generators(ctx, open)
if err != nil {
t.Fatal(err)
}
_, err = declarationWith(ctx, open, "laptop", plan, settings, gens, Reading)
var notMade *catalogue.NotMadeError
if !errors.As(err, &notMade) || notMade.Module != "keeper" || notMade.Name != "repository" {
t.Fatalf("a read composition did not say which secret is not made, typed: %v", err)
}
// Foreseen, it composes, names what the push makes, and made nothing.
foreseen, err := declarationWith(ctx, open, "laptop", plan, settings, gens, Foreseeing)
if err != nil || len(foreseen.foreseen) != 1 || foreseen.foreseen[0] != "keeper/repository" {
t.Fatalf("foreseen: %v %v", foreseen.foreseen, err)
}
if _, held, err := open.inventory.ModuleSecretIfIssued(ctx, "laptop", "keeper", "repository"); err != nil || held {
t.Fatalf("asking ahead of the push made the secret (held %v, %v)", held, err)
}
// Within the bound: nothing at all — no urgent, no warning, nobody notified.
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a machine waiting for a push raised %+v", got)
}
// Past the bound: a warning, not urgent, saying what the push will make.
before := awaitingPushBound
awaitingPushBound = -time.Minute
t.Cleanup(func() { awaitingPushBound = before })
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.awaiting-push" || got[0].Severity != conditions.Warning ||
!strings.Contains(got[0].Summary, "keeper/repository") || !strings.Contains(got[0].Summary, "push laptop") {
t.Fatalf("a machine left un-pushed past the bound: %+v", got)
}
// Pushed: the secret is made and D1 says nothing.
pushedBy(t, open, "laptop")
if got := d1(t, open); len(got) != 0 {
t.Fatalf("a pushed machine still raised %+v", got)
}
}
// **A REAL FAILURE IS STILL URGENT**: a secret the push would be refused on is not one it will make.
// A bus credential nobody issued (issue 203) fails the push, so D1 says it — urgent, in the push's words.
func TestASecretThePushCannotMakeIsStillUncomposable(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aTalker())
if _, err := assign(ctx, open, "laptop", "talker"); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || got[0].Severity != conditions.Urgent ||
!strings.Contains(got[0].Summary, "module issue talker --node laptop") {
t.Fatalf("a push that will be refused was not said urgently: %+v", got)
}
// And a machine whose composition fails for anything else, with a secret waiting beside it, is
// uncomposable for that — the stand-in hides nothing.
register(t, open, aKeeper())
if _, err := assign(ctx, open, "anchor", "keeper"); err != nil {
t.Fatal(err)
}
one, two := rivals()
register(t, open, one)
register(t, open, two)
_, _ = assign(ctx, open, "anchor", "rival-one")
_, _ = assign(ctx, open, "anchor", "rival-two")
keys := map[string]conditions.Severity{}
for _, o := range d1(t, open) {
keys[o.Key()] = o.Severity
}
if keys["machine.anchor.uncomposable"] != conditions.Urgent {
t.Fatalf("a real failure beside a waiting secret: %v", keys)
}
}
// A given secret sealed to a key the machine no longer has is not the mesh's to make again: the push is
// refused on it, so D1 is too.
func TestAGivenSecretUnderAnOldKeyIsNotForeseen(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
if err := open.inventory.AcceptSecretForModule(ctx, "laptop", "keeper", "repository", "given"); err != nil {
t.Fatal(err)
}
record, err := open.inventory.NodeByName(ctx, "laptop")
if err != nil {
t.Fatal(err)
}
if err := open.inventory.RecordSealingKey(ctx, record.ID, aPublicKey(t)); err != nil {
t.Fatal(err)
}
got := d1(t, open)
if len(got) != 1 || got[0].Key() != "machine.laptop.uncomposable" || !strings.Contains(got[0].Summary, "issue it again") {
t.Fatalf("a given secret under an old key: %+v", got)
}
}
// The bound is read from when the machine began waiting: the oldest assignment since its last send.
func TestAMachineAwaitsAPushSinceItsOldestAssignmentSinceTheLastSend(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
sent := time.Now()
since, err := open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.After(sent) {
t.Fatalf("nothing assigned since the send: waiting since %v (%v), the send was before %v", since, err, sent)
}
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
since, err = open.inventory.AwaitingSince(ctx, "laptop")
if err != nil || since.Before(sent.Add(-time.Second)) {
t.Fatalf("assigned after the send: waiting since %v (%v), not from the assignment", since, err)
}
}
// **D3 AND D13 WAIT FOR THE SEND**: a holder is expected to answer on a machine once the machine was sent
// it and had time to report — never between assign and push.
@@ -126,3 +242,40 @@ func TestAHolderIsExpectedOnlyOnceSentAndReported(t *testing.T) {
}
// And through the stores: assigned, not in the last send; pushed and reported, carried and settled.
func TestALastSendIsReadFromTheSendAndTheReport(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aKeeper())
pushedBy(t, open, "laptop")
if _, err := assign(ctx, open, "laptop", "keeper"); err != nil {
t.Fatal(err)
}
sends, err := readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("a holder assigned and not pushed is expected to answer")
}
if !sends["laptop"].settled(overlay.Name, time.Now().Add(time.Hour)) {
t.Fatal("a module the machine was sent long ago is not expected to answer")
}
pushedBy(t, open, "laptop")
sends, err = readDeliveries(ctx, open.inventory)
if err != nil {
t.Fatal(err)
}
if sends["laptop"].settled("keeper", time.Now()) {
t.Fatal("pushed a moment ago and not reported, a holder is already expected")
}
if !sends["laptop"].settled("keeper", time.Now().Add(reportGrace+time.Minute)) {
t.Fatal("pushed past the grace, a holder is not expected")
}
if _, ok := sends["laptop"].carried["keeper"]; !ok {
t.Fatalf("the send's builds do not carry keeper: %v", sends["laptop"].carried)
}
pushedAndApplied(t, open, "laptop")
if sends, err = readDeliveries(ctx, open.inventory); err != nil || !sends["laptop"].settled("keeper", time.Now()) {
t.Fatalf("pushed and reported applied, a holder is not expected to answer (%v)", err)
}
}
-100
View File
@@ -1,100 +0,0 @@
package main
import (
"context"
"fmt"
"os"
"slices"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
)
// A machine out of touch when a tier is sent is not sent to and not waited for (novox/hq ADR 0435 §1, issue 1161): the
// delivery goes on with the machines heard, says the others were away, and the away ones get the builds by the send
// they are given when heard again (ADR 0353, ADR 0360 §5). Away is the silent check's one verdict, machineFacts.silent
// over the watchman's presence (ADR 0433 §3), never a second rule; a sleeping machine is away once presence says so.
// awayOf is the machines out of touch at a moment: those the silent check calls silent (a sleeping machine
// is away once presence says so). A machine
// never heard from has not joined and is not away.
func awayOf(machines []machineFacts, now time.Time) map[string]bool {
out := map[string]bool{}
for _, m := range machines {
if m.silent(now) {
out[m.name] = true
}
}
return out
}
// judgeAway is the walk's reading of which machines are away as a tier is sent: every node with its last-heard time and
// the watchman's verdict on its presence key. A bus that cannot be asked leaves none away, which is what a walk did
// before: a machine is not passed on a reading that was not made.
var judgeAway = func(ctx context.Context, open *stores, now time.Time) (map[string]bool, error) {
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return nil, err
}
address, err := broker.BusAddress()
if err != nil {
return nil, err
}
js, err := broker.Dial(address)
if err != nil {
return nil, err
}
defer js.Close()
machines := make([]machineFacts, len(nodes))
for i, n := range nodes {
machines[i] = machineFacts{name: n.Name, lastHeard: n.LastSeen}
}
if err := withPresence(ctx, js.Conn(), machines, now); err != nil {
return nil, err
}
return awayOf(machines, now), nil
}
// rememberAway keeps on the plan the machines of running that are away: sorted, each once.
func rememberAway(p *inventory.Plan, running []string, away map[string]bool) {
for _, n := range running {
if away[n] && !slices.Contains(p.Away, n) {
p.Away = append(p.Away, n)
}
}
sort.Strings(p.Away)
}
// runningHeard is the machines of running that the walk waits on for a module: those a send of the walk reached, and
// those it did not pass as away.
func runningHeard(p inventory.Plan, running []string, s *inventory.PlanModule) []string {
return slices.DeleteFunc(slices.Clone(running), func(n string) bool {
if !slices.Contains(p.Away, n) {
return false
}
_, sent := s.Rest[n]
return !sent
})
}
// awaySaying is what a plan says of the machines it passed as away, empty when it passed none.
func awaySaying(p inventory.Plan) string {
var said []string
for _, n := range p.Away {
said = append(said, n+" was away: sent when it is heard again")
}
return strings.Join(said, "; ")
}
// judgedAway reads which machines are away for a tier about to be sent, and says when it cannot.
func judgedAway(ctx context.Context, open *stores, now time.Time) map[string]bool {
away, err := judgeAway(ctx, open, now)
if err != nil {
fmt.Fprintf(os.Stderr, "which machines are away cannot be read, so none is passed over: %v\n", err)
return nil // empty-on-error: none is passed over on a reading that was not made; the error is said just above
}
return away
}
@@ -1,228 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"strings"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// 1161/AC5, with the store and a real bus and no ticker: a plan of two tiers whose first tier runs on a machine away when it
// is sent finishes that tier without waiting for it and without its bound, sends the second tier to the machine heard
// alone, and says the machine was away (issue 1161). The machine away keeps its want: it was sent nothing of the plan's
// builds, so the send it gets when it is heard again carries them (AC3).
func TestAPlanGoesOnWithoutAMachineAwayWhenItsTierIsSent(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
// laptop is away as every tier is sent; anchor is heard.
wasAway := judgeAway
judgeAway = func(context.Context, *stores, time.Time) (map[string]bool, error) {
return map[string]bool{"laptop": true}, nil
}
t.Cleanup(func() { judgeAway = wasAway })
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
legacy, err := js.Conn().JetStream()
if err != nil {
t.Fatal(err)
}
checkEvents = link.OverNATS{Conn: js.Conn(), JS: legacy}
t.Cleanup(func() { checkEvents = nil })
sentHeard := make(chan *nats.Msg, 32)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeySent), sentHeard)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
wasSaved := inventory.PlanSaved
inventory.PlanSaved = servingPlanSaved(ctx, checkEvents, open)
t.Cleanup(func() { inventory.PlanSaved = wasSaved })
// The build machine and the machines, as far as the plan meets them: an ask is counted, a send is recorded and kept.
var mu sync.Mutex
var asks []string
var sends [][]string
wasAsk := askABuild
askABuild = func(_ context.Context, source buildSource, _, _ string) (string, error) {
mu.Lock()
defer mu.Unlock()
id := link.NewBuildID(time.Now().Add(time.Duration(len(asks)) * time.Millisecond))
asks = append(asks, source.Repository+"#"+id)
return id, nil
}
t.Cleanup(func() { askABuild = wasAsk })
digests := 0
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
mu.Lock()
defer mu.Unlock()
sends = append(sends, append([]string(nil), names...))
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
builds := map[string]string{}
for _, m := range []string{"a", "b"} {
builds[m] = current[m].Commit
}
for _, node := range names {
digests++
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), fmt.Sprintf("d-%s-%d", node, digests), builds); err != nil {
return nil, err
}
}
return names, nil
}
t.Cleanup(func() { sendRollout = wasSend })
register := func(m, commit string, asked time.Time) {
t.Helper()
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: commit},
inventory.Source{Repository: "novox/" + m, Seat: "git", Path: "modules/" + m, BuiltFrom: commit, Head: commit,
Asked: asked}); err != nil {
t.Fatal(err)
}
}
record := func(id, m, commit string, at time.Time) {
t.Helper()
b := inventory.Build{ID: id, Module: m, Commit: commit, Repository: "novox/" + m, Path: "modules/" + m, Asked: at, At: at}
b.Manifest, _ = json.Marshal(catalogue.Manifest{Module: m, Version: commit})
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
old := time.Now().Add(-2 * time.Hour)
for _, m := range []string{"a", "b"} {
record("build-"+m+"-1", m, "c1", old)
register(m, "c1", old)
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.Assign(ctx, n, m); err != nil {
t.Fatal(err)
}
}
}
for _, n := range []string{"anchor", "laptop"} {
if err := inv.RecordSent(ctx, nodeID(t, open, n), "d-"+n+"-c1", map[string]string{"a": "c1", "b": "c1"}); err != nil {
t.Fatal(err)
}
}
record("build-a-2", "a", "c2", time.Now().Add(-time.Minute))
register("a", "c2", time.Now().Add(-time.Minute))
// A machine's report on what it was sent, heard as the controller hears it.
listener := nudgingListener{Enrolment: link.Enrolment{Inventory: inv}}
seq := int64(0)
reports := func(node string) {
t.Helper()
last, found, err := inv.SentTo(ctx, node)
if err != nil || !found {
t.Fatalf("%s was sent nothing: %v", node, err)
}
seq++
body, _ := json.Marshal(map[string]any{"node": node, "applied": []string{"x.server"}, "declared": last.Digest,
"report_sequence": seq})
var report link.Report
if err := json.Unmarshal(body, &report); err != nil {
t.Fatal(err)
}
if _, err := listener.Heard(ctx, report); err != nil {
t.Fatal(err)
}
}
plan := func() inventory.Plan {
t.Helper()
p, err := inv.PlanByID(ctx, "plan-1160")
if err != nil {
t.Fatal(err)
}
return p
}
built := time.Now().UTC()
p := inventory.Plan{ID: "plan-1160", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c2", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"a"}, {"b"}},
Delivery: &inventory.PlanDelivery{Order: []inventory.Edge{{From: "b", To: "a", Kind: edgeChangeOrder}}},
Modules: map[string]*inventory.PlanModule{
"a": {State: "built", BuiltAt: &built, Commit: "c2", Build: "build-a-2"},
"b": {},
}}
first, stopFirst := context.WithCancel(ctx)
defer stopFirst()
waitFirst := startRolloutEvents(first, open, js)
// Ends with the test before the cleanups restore what the look reads, which run after it (issue 1338).
t.Cleanup(func() { stopFirst(); waitFirst() })
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
eventually(t, "tier 0 sent to anchor alone", func() bool {
mu.Lock()
defer mu.Unlock()
return len(sends) == 1 && strings.Join(sends[0], ",") == "anchor"
})
reports("anchor")
// The note of the move to tier 1 is read in the step that sees it: later steps of the tier say their own.
var moved string
eventually(t, "tier 1 reached on anchor's report alone", func() bool {
if q := plan(); q.Tier == 1 {
moved = q.Note
return true
}
return false
})
if !strings.Contains(moved, "laptop was away: sent when it is heard again") || strings.Contains(moved, "anchor") {
t.Errorf("1161/AC2: the plan says %q; want laptop named away and not anchor", moved)
}
// The ask for tier 1's build is the next step, and records its build id on the module.
eventually(t, "tier 1 asked", func() bool { m := plan().Modules["b"]; return m != nil && m.Build != "" })
b := plan().Modules["b"]
askedAt, _ := link.BuildAskedAt(b.Build)
record(b.Build, "b", "c2", askedAt)
register("b", "c2", askedAt)
planBuilt(ctx, open, "b", "c2", "", askedAt, b.Build)
eventually(t, "tier 1 sent to anchor alone", func() bool {
mu.Lock()
defer mu.Unlock()
return len(sends) == 2 && strings.Join(sends[1], ",") == "anchor"
})
// 1161/AC3: laptop was sent nothing of the plan's builds, so it still wants them for the send it gets when heard again.
was, known, err := inv.SentBuilds(ctx, "laptop")
if err != nil || !known {
t.Fatalf("1161/AC3: laptop's last send: %v, known %v", err, known)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
t.Fatal(err)
}
if was["a"] == current["a"].Commit || was["a"] != "c1" {
t.Errorf("1161/AC3: laptop was last sent a at %q and a is now %q; want it still at c1, owed c2", was["a"], current["a"].Commit)
}
if got := plan().Away; strings.Join(got, ",") != "laptop" {
t.Errorf("1161/AC2: the plan keeps %v away; want laptop", got)
}
}
-67
View File
@@ -1,67 +0,0 @@
package main
import (
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A machine out of touch when a tier is sent is not sent to and not waited for (novox/hq ADR 0435 §1, issue 1161): the
// walk judges it with the silent check's one verdict, machineFacts.silent, and a sleeping machine is away once presence says so.
// 1161/AC1: away is the silent check's verdict; AC4: a machine heard is not away.
func TestAMachineIsAwayAsTheSilentCheckJudgesIt(t *testing.T) {
now := time.Now().UTC()
machines := []machineFacts{
{name: "heard", lastHeard: now.Add(-time.Minute)},
{name: "quiet", lastHeard: now.Add(-silentBound - time.Minute)},
{name: "stuck", lastHeard: now, presence: presenceSilent},
{name: "back", lastHeard: now.Add(-time.Hour), presence: presenceHeard},
{name: "new"},
}
got := awayOf(machines, now)
for node, want := range map[string]bool{"heard": false, "quiet": true, "stuck": true, "back": false,
"new": false} {
if got[node] != want {
t.Errorf("1161/AC1: %s away is %v; want %v (away: %v)", node, got[node], want, got)
}
}
}
// 1161/AC1, AC2: the plan keeps the machines of the tier it passed as away, sorted and once.
func TestAPlanKeepsTheMachinesItPassedAsAway(t *testing.T) {
p := &inventory.Plan{}
away := map[string]bool{"g14": true, "ace": true}
rememberAway(p, []string{"anchor", "g14"}, away)
rememberAway(p, []string{"g14", "ace"}, away)
if !slices.Equal(p.Away, []string{"ace", "g14"}) {
t.Fatalf("1161/AC2: the plan keeps %v away; want ace, g14", p.Away)
}
}
// 1161/AC2: the plan says it, where its other passed machines are said.
func TestAPlanSaysTheMachinesItPassedAsAway(t *testing.T) {
say := awaySaying(inventory.Plan{Away: []string{"g14"}})
if !strings.Contains(say, "g14 was away: sent when it is heard again") {
t.Fatalf("1161/AC2: the plan says %q", say)
}
if got := awaySaying(inventory.Plan{}); got != "" {
t.Fatalf("1161/AC2: a plan that passed no machine says %q", got)
}
}
// 1161/AC1: the gate does not wait for a machine passed as away, unless a send reached it; AC4: a machine heard is waited for.
func TestAGateDoesNotWaitForAMachinePassedAsAway(t *testing.T) {
p := inventory.Plan{Away: []string{"g14"}}
none := &inventory.PlanModule{}
if got := runningHeard(p, []string{"anchor", "g14"}, none); !slices.Equal(got, []string{"anchor"}) {
t.Fatalf("1161/AC1: the gate waits on %v; want anchor alone", got)
}
sent := &inventory.PlanModule{Rest: map[string]inventory.SentDeclaration{"g14": {Digest: "d"}}}
if got := runningHeard(p, []string{"anchor", "g14"}, sent); !slices.Equal(got, []string{"anchor", "g14"}) {
t.Fatalf("1161/AC4: a machine a send reached is waited for: %v", got)
}
}
@@ -1,308 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"os"
"strings"
"sync"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// aBackupSeatOnTheBus is one machine's node-backup holder as a test makes it: it counts the `now` asks it hears and
// answers `backed-up` with the last night it is told (novox/hq ADR 0404).
type aBackupSeatOnTheBus struct {
mu sync.Mutex
nows []string
night map[string]any // module → lastNight
}
func (f *aBackupSeatOnTheBus) serve(t *testing.T, conn *nats.Conn, node string) {
t.Helper()
reply := func(m *nats.Msg, result any) {
body, _ := json.Marshal(map[string]any{"result": result, "error": "", "node": node})
_ = m.Respond(body)
}
now, err := conn.Subscribe(link.NodeSeatToolSubject(catalogue.BackupSeat, "now", node), func(m *nats.Msg) {
var a struct {
Module string `json:"module"`
}
_ = json.Unmarshal(m.Data, &a)
f.mu.Lock()
f.nows = append(f.nows, a.Module)
f.mu.Unlock()
reply(m, map[string]any{"started": a.Module})
})
if err != nil {
t.Fatal(err)
}
read, err := conn.Subscribe(link.NodeSeatToolSubject(catalogue.BackupSeat, "backed-up", node), func(m *nats.Msg) {
f.mu.Lock()
defer f.mu.Unlock()
var out []map[string]any
for module, n := range f.night {
out = append(out, map[string]any{"module": module, "lastNight": n})
}
reply(m, out)
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = now.Unsubscribe(); _ = read.Unsubscribe() })
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
}
func (f *aBackupSeatOnTheBus) answers(module string, ok bool, words string) {
f.mu.Lock()
defer f.mu.Unlock()
f.night = map[string]any{module: map[string]any{"ok": ok, "at": time.Now().UTC().Add(time.Second), "error": words}}
}
func (f *aBackupSeatOnTheBus) asks() int {
f.mu.Lock()
defer f.mu.Unlock()
return len(f.nows)
}
// Acceptance test 3 of design 47, on the controller's side: a delivery plan that moves a module keeping data waits
// for its backups on both machines before it asks a build; a backup that fails holds it, with the machine, the
// module and the backup's words, and a condition says the same; the verb the delivery's release calls asks the
// backups again, and the plan goes on. One bus and one store for the scenarios.
func TestADeliveryThatMovesABackedUpModuleWaitsForItsBackupAndAFailedOneHoldsItAndIsSaid(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
asked := asksRecorded(t)
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "keeper", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BackupSeat, Scope: catalogue.ScopeNode, Serves: []string{"backed-up", "now", "restore"}}}})
notes, err := catalogue.ParseManifest([]byte(`{"module":"notes","version":"1","data":{"own":[` +
`{"id":"files","path":"${dir:files}","class":"valuable","why":"people's notes"}]},` +
`"resources":[{"id":"files","type":"directory"}]}`))
if err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, notes, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/notes", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, node := range []string{"anchor", "laptop"} {
if _, err := assign(ctx, open, node, "keeper"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, node, "notes"); err != nil {
t.Fatal(err)
}
}
seats := map[string]*aBackupSeatOnTheBus{"anchor": {}, "laptop": {}}
for node, s := range seats {
s.serve(t, js.Conn(), node)
}
// Short waits: the bound and the look are variables, so nothing here waits for the minutes of a real backup.
wasWithin, wasEvery := backupsWithin, backupsPollEvery
backupsWithin, backupsPollEvery = time.Minute, 10*time.Millisecond
t.Cleanup(func() { backupsWithin, backupsPollEvery = wasWithin, wasEvery })
newPlan := func(id string) {
t.Helper()
p := inventory.Plan{ID: id, Repository: "novox/mesh-catalog", Branch: "main", Commit: "c1", Created: time.Now().UTC(),
State: inventory.PlanBuilding, Tiers: [][]string{{"notes"}}, Modules: map[string]*inventory.PlanModule{"notes": {}}}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
}
read := func(id string) inventory.Plan {
t.Helper()
p, err := inv.PlanByID(ctx, id)
if err != nil {
t.Fatal(err)
}
return p
}
// 1. Both machines are asked; while one has not answered, no build is asked.
newPlan("plan-one")
advancePlans(ctx, open)
if seats["anchor"].asks() != 1 || seats["laptop"].asks() != 1 || len(*asked) != 0 {
t.Fatalf("asks anchor=%d laptop=%d builds=%v", seats["anchor"].asks(), seats["laptop"].asks(), *asked)
}
seats["anchor"].answers("notes", true, "")
advancePlans(ctx, open)
if len(*asked) != 0 || read("plan-one").Backups.State != inventory.BackupPending {
t.Fatalf("a tier was asked while a backup was pending: %v %+v", *asked, read("plan-one").Backups)
}
seats["laptop"].answers("notes", true, "")
advancePlans(ctx, open)
if len(*asked) != 1 || read("plan-one").Backups.State != inventory.BackupDone {
t.Fatalf("the tier was not asked once every backup was done: %v %+v", *asked, read("plan-one").Backups)
}
finish := read("plan-one")
finish.State = inventory.PlanDone
if err := inv.SavePlan(ctx, &finish); err != nil {
t.Fatal(err)
}
// 2. A backup that fails holds the plan: not failed, nothing asked, the words kept, a condition raised.
seats["anchor"].night, seats["laptop"].night = nil, nil
newPlan("plan-two")
advancePlans(ctx, open)
seats["anchor"].answers("notes", true, "")
seats["laptop"].answers("notes", false, "no space left on device")
advancePlans(ctx, open)
held := read("plan-two")
if held.State != inventory.PlanBuilding || held.Backups.State != inventory.BackupFailed || len(*asked) != 1 {
t.Fatalf("the plan was not held: %s %+v builds=%v", held.State, held.Backups, *asked)
}
var failed *inventory.PlanBackup
for i := range held.Backups.Asked {
if held.Backups.Asked[i].State == inventory.BackupFailed {
failed = &held.Backups.Asked[i]
}
}
if failed == nil || failed.Machine != "laptop" || failed.Module != "notes" || failed.Words != "no space left on device" ||
strings.Contains(held.Note, "tried again") {
t.Fatalf("the failed backup: %+v note=%q", failed, held.Note)
}
advancePlans(ctx, open) // not retried on the next look
if again := read("plan-two"); again.Backups.State != inventory.BackupFailed || len(*asked) != 1 ||
seats["laptop"].asks() != 2 {
t.Fatalf("a held plan was asked again by itself: %+v builds=%v asks=%d", again.Backups, *asked, seats["laptop"].asks())
}
said, err := probeBackupHeld(ctx, &doctor{open: open})
if err != nil || len(said) != 1 || said[0].Kind != kindBackupHeld || !strings.Contains(said[0].Said, "no space left on device") {
t.Fatalf("the condition: %+v %v", said, err)
}
// 3. The verb the release calls: the failed backup is asked again, and only it, and the plan goes on.
seats["laptop"].answers("notes", false, "no space left on device")
if err := deliveryCommand(ctx, []string{"backups-again", "plan-two", "--why", "a person released it", "--by",
"mesh-delivery for jochen"}); err != nil {
t.Fatal(err)
}
if err := deliveryCommand(ctx, []string{"backups-again", "plan-two", "--why", "again"}); err == nil {
t.Fatal("a plan with no failed backup was asked again")
}
seats["laptop"].night = nil
advancePlans(ctx, open)
if seats["laptop"].asks() != 3 || seats["anchor"].asks() != 2 {
t.Fatalf("asked again: laptop=%d anchor=%d", seats["laptop"].asks(), seats["anchor"].asks())
}
seats["laptop"].answers("notes", true, "")
advancePlans(ctx, open)
if got := read("plan-two"); got.Backups.State != inventory.BackupDone || len(*asked) != 2 {
t.Fatalf("the plan did not go on: %+v builds=%v", got.Backups, *asked)
}
if said, _ := probeBackupHeld(ctx, &doctor{open: open}); len(said) != 0 {
t.Fatalf("the condition stays: %+v", said)
}
}
// Issue 1086: a plan cut from a merge asked its first tier at the cut, so a module keeping data was sent without the
// backup the plan promised. The cut now takes the backups first (askFirstTierAtCut, batches.go), and the walk's own
// step asks the tier once they are done.
func TestAPlanCutFromAMergeOfABackedUpModuleAsksItsBackupAndNotItsTierUntilItIsDone(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
everyHeadWaits(t)
asked := asksRecorded(t)
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
before := handActConn
handActConn = js.Conn()
t.Cleanup(func() { handActConn = before })
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
register(t, open, catalogue.Manifest{Module: "keeper", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.BackupSeat, Scope: catalogue.ScopeNode, Serves: []string{"backed-up", "now", "restore"}}}})
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
notes, err := catalogue.ParseManifest([]byte(`{"module":"notes","version":"1","data":{"own":[` +
`{"id":"files","path":"${dir:files}","class":"valuable","why":"people's notes"}]},` +
`"resources":[{"id":"files","type":"directory"}]}`))
if err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, notes, inventory.Source{Repository: "novox/notes", Seat: "git", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []string{"keeper", "notes"} {
if _, err := assign(ctx, open, "anchor", m); err != nil {
t.Fatal(err)
}
}
seat := &aBackupSeatOnTheBus{}
seat.serve(t, js.Conn(), "anchor")
wasWithin, wasEvery := backupsWithin, backupsPollEvery
backupsWithin, backupsPollEvery = time.Minute, 10*time.Millisecond
t.Cleanup(func() { backupsWithin, backupsPollEvery = wasWithin, wasEvery })
// 1. A merge of notes is heard and its window closes: the batch is cut through cutBatch.
hear(t, open, repoMerge("notes", "c-notes", t0), t0.Add(time.Second))
cutAt(t, open, t0.Add(time.Second+90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walk(s)", len(ws))
}
// 2. Right after the cut no tier is asked, the record has anchor/notes pending, the seat heard one `now`.
cut, err := inv.PlanByID(ctx, ws[0].ID)
if err != nil {
t.Fatal(err)
}
if len(*asked) != 0 || askedInTier(cut) {
t.Fatalf("the first tier was asked at the cut: %v %+v", *asked, cut.Modules)
}
if cut.Backups == nil || cut.Backups.State != inventory.BackupPending || len(cut.Backups.Asked) != 1 ||
cut.Backups.Asked[0].Machine != "anchor" || cut.Backups.Asked[0].Module != "notes" {
t.Fatalf("the backups of the cut: %+v", cut.Backups)
}
if seat.asks() != 1 {
t.Fatalf("the seat heard %d `now`", seat.asks())
}
// 3. The backup answers done: the walk's own step asks the first tier, once, and asks no second backup.
seat.answers("notes", true, "")
advancePlans(ctx, open)
advancePlans(ctx, open)
after, err := inv.PlanByID(ctx, ws[0].ID)
if err != nil {
t.Fatal(err)
}
if len(*asked) != 1 || after.Backups.State != inventory.BackupDone || seat.asks() != 1 {
t.Fatalf("builds=%v backups=%+v asks=%d", *asked, after.Backups, seat.asks())
}
}
+171 -415
View File
@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"os"
"regexp"
"slices"
"sort"
"strconv"
@@ -26,10 +27,9 @@ import (
// commit in it, with at most one commit per repository, the latest of its branch, which contains the earlier
// ones. The walk names every merge it answers.
//
// - One walk at a time beside a related one (ADR 0407 decision 4, issue 973). Merges that come while a walk
// runs join the next batch; a closed batch that shares nothing with an open walk is cut beside it. A walk
// that started is never superseded; one still waiting for its delivery's word is folded into the walk of a
// batch related to it, which answers every merge it answered — the only supersession left.
// - One walk at a time. Merges that come while a walk runs join the next batch. A walk that started is never
// superseded; one still waiting for its delivery's word is folded into the next batch's walk, which
// answers every merge it answered — the only supersession left.
// - A merge heard after a later merge of its branch was put in a walk is answered by that walk: a batch
// still assembling, an open walk that builds everything it moves, or a walk done that did, which then
// names it at once. Otherwise it joins the next batch.
@@ -39,9 +39,8 @@ import (
//
// **A merge waits only for a batch it shares something with** (novox/hq ADR 0304 decision 1, ADR 0306 decisions
// 2 and 3, issue 525): it joins a batch it is related to, and one that is related to none and whose delivery
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Such a batch is cut
// at once, beside any open walk it shares nothing with, and queued behind the first one it is related to
// (cutUnrelated).
// group has no member still to merge is a batch whose window is closed at once (batchToJoin). Walks stay one at
// a time: such a batch is cut when no walk is open, and queued behind the one that is otherwise.
//
// A merge on the controller's own path (a module whose walk waits for nobody's word) never shares a batch with
// one that waits for mesh-delivery's: each kind has a batch of its own, an own-path batch is cut first and folds
@@ -62,17 +61,16 @@ var (
// mergeWindowFileVar names the file the controller's window settings are written to.
const mergeWindowFileVar = "MESH_MERGE_WINDOW_FILE"
// batchCutEvery is how often the controller looks whether a batch's window closed.
const batchCutEvery = 5 * time.Second
// batchLateAfter is how long past its maximum a batch may still be assembling, with no walk open, before
// the controller is said to have failed to cut it (ADR 0276 decision 9).
const batchLateAfter = time.Minute
// mergeGrace is how long before a batch's first merge a merge may have been made and still be told apart from one
// made before the controller began keeping merges (owedLate).
const mergeGrace = 10 * time.Minute
// edgeChangeOrder is the order the rules of a plan draw among its modules (novox/hq ADR 0419 clauses 3 and 4, delivery_edges.go)
// made an edge of its tiers: the module after depends on the module before, and is gated on it.
const edgeChangeOrder = "change-order"
// edgeGroupOrder is a delivery group's order made an edge of a walk's tiers (ADR 0276 decision 5): the
// modules of the member after depend on the modules of the member before, and are gated on them.
const edgeGroupOrder = "group-order"
// mergeWindowOf is the window and its maximum, from the controller's settings; a value unreadable is its
// default, and said.
@@ -124,56 +122,11 @@ func durationSetting(v any) (time.Duration, bool) {
return 0, false
}
// mergeFacts is what hearMerge judges a merge by, kept apart so the catch-up (ADR 0457) judges a merge the way hearing it will.
type mergeFacts struct {
merged time.Time
touches []inventory.Entry
later []inventory.BatchedMerge
owed bool
from, packaging []inventory.Entry
already int
addsModuleToBuilds bool // a first merge into a repository the mesh builds nothing from: firstBuilds acts on it
}
// factsOfMerge reads what hearing a merge turns on: the modules it touches whatever the history, the later merges of its
// branch, whether it is owed late, and the modules it moves.
func factsOfMerge(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, now time.Time,
entries []inventory.Entry, read map[string][]inventory.ReadRepository) (mergeFacts, error) {
repository := m.Owner + "/" + m.Repo
f := mergeFacts{}
var err error
f.merged, err = time.Parse(time.RFC3339Nano, m.MergedAt)
if err != nil {
f.merged = now
}
// **What it touches, history or not** (review of this change): a merge heard after a later merge of its
// branch was cut reads as history for every module that cut marked seen, and was dropped from the record
// here, never named by the walk that carried it.
raw := m
raw.MergedAt = ""
f.touches = wouldMove(raw, entries, read)
if len(f.touches) > 0 {
if f.later, err = inv.LaterMergesOf(ctx, repository, m.Base, f.merged.UTC()); err != nil {
return f, err
}
}
if f.owed, err = owedLate(ctx, inv, m, f.merged, f.touches); err != nil {
return f, err
}
f.from, f.packaging, f.already = mergeCandidates(m, entries, read)
f.addsModuleToBuilds = f.already == 0 && len(addedDirs(m, entries, read)) > 0
return f, nil
}
// plans says hearing the merge puts it in a batch or a walk.
func (f mergeFacts) plans() bool {
return len(f.from) > 0 || len(f.packaging) > 0 || len(f.later) > 0 || f.owed
}
// hearMerge puts a merge the forge announced into the batch that answers it, at the moment it was heard,
// and cuts what is due.
func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.Time) error {
// One merge acted on at a time, whoever hands it over (novox/hq issue 266).
// One merge acted on at a time, whoever hands it over: the bus, or the catch-up that reads back
// what the bus did not hand over (novox/hq issue 266).
actingOnMerges.Lock()
defer actingOnMerges.Unlock()
@@ -194,13 +147,27 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
if err != nil {
return notNow(err)
}
facts, err := factsOfMerge(ctx, inv, m, now, entries, read)
merged, err := time.Parse(time.RFC3339Nano, m.MergedAt)
if err != nil {
merged = now
}
// **What it touches, history or not** (review of this change): a merge heard after a later merge of its
// branch was cut reads as history for every module that cut marked seen, and was dropped from the record
// here, never named by the walk that carried it.
raw := m
raw.MergedAt = ""
touches := wouldMove(raw, entries, read)
var later []inventory.BatchedMerge
if len(touches) > 0 {
if later, err = inv.LaterMergesOf(ctx, repository, m.Base, merged.UTC()); err != nil {
return notNow(err)
}
}
owed, err := owedLate(ctx, inv, m, merged, touches)
if err != nil {
return notNow(err)
}
merged, touches := facts.merged, facts.touches
if !facts.plans() {
already := facts.already
if from, packaging, already := mergeCandidates(m, entries, read); len(from) == 0 && len(packaging) == 0 && len(later) == 0 && !owed {
// "Already built from it" and "nothing reads it" are different facts, and reading the first
// as the second sends somebody looking for a broken trigger when the mesh is up to date.
if already > 0 {
@@ -208,17 +175,12 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
repository, m.Base, m.Commit, already)
return nil
}
if already == 0 {
if done, err := f.firstBuilds(ctx, m, entries, read, now); err != nil || done {
return err
}
}
fmt.Print(nothingReadsIt(repository, m))
fmt.Printf("%s merged into %s (%.8s); nothing the mesh holds reads it\n", repository, m.Base, m.Commit)
return nil
}
// **Its delivery group's word, before the plans are held** (ADR 0306 decision 2): a slow answer holds up no walk.
own := ownPath(touches)
waits, holding, err := groupWaitHolding(ctx, inv, m, deliverySeatHeld(entries))
waits, err := groupWait(ctx, inv, m, deliverySeatHeld(entries), own)
if err != nil {
return notNow(err)
}
@@ -256,10 +218,10 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
}
if p.Batch() {
if err := keepBatch(ctx, inv, &p, now, ""); err != nil {
return unsaved(err)
return notNow(err)
}
} else if err := nameMerges(ctx, inv, &p); err != nil {
return unsaved(err)
return notNow(err)
}
fmt.Printf("%s merged into %s (%.8s) after a later merge of its branch was put in %s (%s): that %s "+
"answers it\n", repository, m.Base, m.Commit, p.ID, p.State, walkOrBatch(p))
@@ -274,55 +236,17 @@ func (f following) hearMerge(ctx context.Context, m link.SourceMoved, now time.T
if added, err := inv.AddMerge(ctx, heard); err != nil || !added {
return notNow(err)
}
holdFor(&batch, m.Head, waits, holding, own, now)
if err := keepBatch(ctx, inv, &batch, now, ""); err != nil {
return unsaved(err)
return notNow(err)
}
fmt.Printf("%s merged into %s (%.8s); %s, it joins %s — %s\n", repository, m.Base, m.Commit, why, batch.ID,
batchWords(batch, now))
return cutBatchesHeld(ctx, f.open, now)
}
// holdFor keeps in a batch that a merge of a group joined whether that group still has a member to merge (ADR 0418
// clause 3): held while it has, the window running again from the moment the last hold is cleared. A group with a
// merge of the controller's own path makes its batch an own-path one: such a plan waits for nobody's word, and is
// started by the controller when the group's batch closes (clause 6).
func holdFor(batch *inventory.Plan, head, why string, holding, own bool, now time.Time) {
if batch.Delivery == nil {
batch.Delivery = &inventory.PlanDelivery{}
}
if batch.Delivery.Batch == nil {
batch.Delivery.Batch = &inventory.PlanBatch{Own: own}
}
w := batch.Delivery.Batch
w.Own = w.Own || own
if head == "" {
return
}
if holding {
if w.Holds == nil {
w.Holds = map[string]string{}
}
w.Holds[head] = why
return
}
if _, held := w.Holds[head]; held {
holds := map[string]string{}
for h, y := range w.Holds {
if h != head {
holds[h] = y
}
}
w.Holds = holds
if len(holds) == 0 {
w.Holds, w.HeldAt = nil, now
}
}
}
// owedLate says a merge that reads as history is still the record's to answer (review of this change): a cut
// marks every module it moved seen at that moment, so a merge of that branch
// handed over late — made before the cut, heard after it — reads as history and was dropped. One that touches
// marks every module it moved seen at that moment, so a merge of that branch the bus lost and the catch-up
// hands over later — made before the cut, heard after it — reads as history and was dropped. One that touches
// a module the mesh holds, of a branch whose merges the controller keeps, made since it began keeping them,
// is kept; one made before it began is old news, as before.
func owedLate(ctx context.Context, inv *inventory.Inventory, m link.SourceMoved, merged time.Time,
@@ -399,7 +323,7 @@ func buildsEvery(p inventory.Plan, modules []inventory.Entry) bool {
// ownPath says a merge touches a module on the controller's own path, whose walk waits for nobody's word.
func ownPath(touches []inventory.Entry) bool {
for _, e := range touches {
if onTheOwnPath(e.Manifest.Module, e.Manifest) {
if _, own := onTheControllersPath[e.Manifest.Module]; own {
return true
}
}
@@ -416,23 +340,14 @@ func ownPath(touches []inventory.Entry) bool {
// names why): a new batch, which waits for the window, beside the ones it shares nothing with; its words say
// why.
// - none, and no member of its group is still to merge: a new batch whose window is closed at once. Cut at
// once means only that: with a walk open that it is related to the batch is queued behind it as any closed
// batch is (ADR 0276 decision 4, ADR 0407 decision 4); beside one it shares nothing with it is cut.
// once means only that: walks stay one at a time (ADR 0276 decision 4), so with a walk open the batch is
// queued behind it as any closed batch is.
func batchToJoin(ctx context.Context, inv *inventory.Inventory, heard inventory.BatchedMerge, entries []inventory.Entry,
read map[string][]inventory.ReadRepository, now time.Time, own bool, waits string) (inventory.Plan, string, error) {
edges, err := inv.Dependencies(ctx)
if err != nil {
return inventory.Plan{}, "", err
}
// **A group is one batch, whatever kinds its members are** (novox/hq ADR 0418 clause 6): a merge joins the batch
// of the other kind that holds a merge from its head.
if group, ok, err := batchOfGroup(ctx, inv, announcedOf(heard).Head, heard.Repository); err != nil {
return inventory.Plan{}, "", err
} else if ok && group.OwnPath() != own && group.Delivery != nil && group.Delivery.Batch != nil {
group.Delivery.Batch.CutAtOnce = false
return group, "its group " + announcedOf(heard).Head + " has a merge in " + group.ID + ", a batch of the other kind: " +
"a group is one batch", nil
}
joins, whys, _, err := relatedBatches(ctx, inv, []inventory.BatchedMerge{heard}, own, "", entries, read, edges)
if err != nil {
return inventory.Plan{}, "", err
@@ -491,9 +406,7 @@ func relatedBatches(ctx context.Context, inv *inventory.Inventory, merges []inve
}
carry := newFootprint()
if !own {
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch folds those it is related to
// (cutUnrelated). Every one is counted here, which only makes a merge join a batch more readily: it waits
// for the window, never walks beside what it shares something with (issue 973).
// An own-path batch folds no waiting walk (cutBatchesHeld); a catalogue batch may fold every one.
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, nil, nil, err
@@ -631,11 +544,6 @@ func newBatch(ctx context.Context, inv *inventory.Inventory, first inventory.Bat
// keepBatch writes a batch as its merges and the clock say it now: the commits it holds, the merges they
// answer, its window, and assembling or queued behind a walk. Saved only when that changed.
func keepBatch(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind string) error {
return keepBatchBecause(ctx, inv, b, now, behind, "")
}
// keepBatchBecause is keepBatch for a batch queued behind a walk it is related to, which its words say (issue 973).
func keepBatchBecause(ctx context.Context, inv *inventory.Inventory, b *inventory.Plan, now time.Time, behind, why string) error {
merges, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return err
@@ -656,42 +564,29 @@ func keepBatchBecause(ctx context.Context, inv *inventory.Inventory, b *inventor
if b.Delivery == nil {
b.Delivery = &inventory.PlanDelivery{}
}
anchor := b.Created
if was := b.Delivery.Batch; was != nil && was.HeldAt.After(anchor) {
anchor = was.HeldAt // the maximum counts from the last hold's end (ADR 0418 clause 3)
}
closes, latest := windowOf(merges, anchor, window, atMost)
closes, latest := windowOf(merges, b.Created, window, atMost)
b.Delivery.Merges = named
var closed, cutAtOnce bool
var withheld []string
var waits string
var holds map[string]string
var heldAt time.Time
if was := b.Delivery.Batch; was != nil {
holds, heldAt = was.Holds, was.HeldAt
closed, withheld = was.Closed, was.Withheld // a stop's, which no look takes back (novox/hq ADR 0299)
cutAtOnce = was.CutAtOnce // the merge's, until another joins it (ADR 0304 decision 1)
waits = was.Waits // its group's, as said when it was made (ADR 0306 decision 2)
}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, BehindWhy: why, Own: b.OwnPath(),
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits, Holds: holds, HeldAt: heldAt}
b.Delivery.Batch = &inventory.PlanBatch{ClosesAt: closes, AtMost: latest, Behind: behind, Own: b.OwnPath(),
Closed: closed, Withheld: withheld, CutAtOnce: cutAtOnce, Waits: waits}
b.State = inventory.PlanAssembling
if behind != "" && windowClosed(*b, now) {
b.State = inventory.PlanQueued
} else {
b.Delivery.Batch.Behind, b.Delivery.Batch.BehindWhy = "", ""
b.Delivery.Batch.Behind = ""
}
b.Note = batchWords(*b, now)
if unNoted(*b) == before {
return nil
}
if err := inv.SavePlan(ctx, b); err != nil {
return err
}
// The window's clock follows the record, only when the record moved: a merge made the quiet key again, a
// batch that opened made its most (ADR 0407 decision 1).
armWindow(ctx, *b, now)
return nil
return inv.SavePlan(ctx, b)
}
// windowOf is when a batch's window closes, unless another merge comes, and when at the latest: the last
@@ -710,20 +605,13 @@ func windowOf(merges []inventory.BatchedMerge, created time.Time, window, atMost
}
// windowClosed says a batch's window is closed: closed by a stop, closed at once for a merge that shares
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached. **A batch holding a group
// with a member still to merge is closed by none of the last three** (novox/hq ADR 0418 clause 3).
// nothing (ADR 0304 decision 1), no merge for the window's length, or its maximum reached.
func windowClosed(b inventory.Plan, now time.Time) bool {
if b.Delivery == nil || b.Delivery.Batch == nil {
return true
}
w := b.Delivery.Batch
if w.Closed {
return true
}
if len(w.Holds) > 0 {
return false // a group's last member is still to merge: neither the window nor its maximum closes the batch
}
return w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
return w.Closed || w.CutAtOnce || !now.Before(w.ClosesAt) || !now.Before(w.AtMost)
}
// carriedOf is a set of merges as a walk carries them: the latest merge of each repository's branch, and
@@ -779,9 +667,6 @@ func carriedOf(merges []inventory.BatchedMerge) ([]inventory.PlanCommit, []inven
type keptMerge struct {
link.SourceMoved
Moves []string `json:"moves,omitempty"`
// FirstBuild says the merge is a first merge into a repository the mesh builds nothing from, kept to ask the first
// build of its modules once mesh-delivery's word on it is kept (novox/hq ADR 0432, issue 1146).
FirstBuild bool `json:"first_build,omitempty"`
}
// announcedOf is a kept merge as the forge announced it, and what it moved; empty where the record says none.
@@ -803,7 +688,7 @@ func spelledOf(m inventory.BatchedMerge) string {
func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge {
k := announcedOf(m)
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves,
Head: k.HeadSHA, Merged: m.Merged, Heard: m.Heard}
Merged: m.Merged, Heard: m.Heard}
}
// laterMerge says a was merged after b: by the forge's merge time, then by when each was heard.
@@ -854,18 +739,33 @@ func nameMerges(ctx context.Context, inv *inventory.Inventory, p *inventory.Plan
return inv.SavePlan(ctx, p)
}
// batchCutter cuts what is due on a timer, until the context ends: a window closes with no merge to say so.
func batchCutter(ctx context.Context, open *stores) {
tick := time.NewTicker(batchCutEvery)
defer tick.Stop()
for {
select {
case <-ctx.Done():
return
case <-tick.C:
}
release, err := open.inventory.HoldPlans(ctx, false)
if err != nil {
continue // whoever holds the plans moves them, and cuts at the end
}
if err := cutBatchesHeld(ctx, open, time.Now().UTC()); err != nil {
fmt.Printf("batches: %v\n", err)
}
release()
}
}
// cutBatchesHeld takes the step that is due, for a caller holding the plans: after a failed walk, the search
// for the merge that brought the failure; with no walk open, the next merge of that search walked alone; else
// each closed batch cut into a walk with the waiting walks it is related to folded in, unless it is related to a
// walk open, behind which it is said queued (cutUnrelated). While the search runs everything waits.
// for the merge that brought the failure; with no walk open, the next merge of that search walked alone, or a
// closed batch cut into one walk with the waiting walks folded in; with a walk open, a closed batch said
// queued behind it.
func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
inv := open.inventory
// A merge found in two records is the half-done move of a crash: put right before anything is decided from it.
if repaired, err := inv.RepairMerges(ctx); err != nil {
return err
} else if repaired > 0 {
fmt.Printf("%d merge(s) found in two plans, the half of a move a restart interrupted, are kept by the plan they were moved to\n", repaired)
}
if err := searchAfterFailures(ctx, inv); err != nil {
return err
}
@@ -880,7 +780,8 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
if err != nil {
return err
}
var started, waiting []inventory.Plan
var started *inventory.Plan
var waiting []inventory.Plan
for i := range plans {
p := plans[i]
if p.Release != nil {
@@ -888,19 +789,14 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
}
if p.Waiting() {
waiting = append(waiting, p)
} else {
started = append(started, p)
} else if started == nil {
started = &plans[i]
}
}
batches, err := inv.Batches(ctx)
if err != nil {
return err
}
// **A hold is looked at again at every cut** (ADR 0418 clause 3): a group whose last member merged or closed
// since is released, and its window runs again from now.
if batches, err = releaseHolds(ctx, inv, batches, now); err != nil {
return err
}
keepAll := func(behind string) error {
for i := range batches {
if err := keepBatch(ctx, inv, &batches[i], now, behind); err != nil {
@@ -909,155 +805,46 @@ func cutBatchesHeld(ctx context.Context, open *stores, now time.Time) error {
}
return nil
}
if started != nil {
// **One walk at a time** (ADR 0276 decision 4): every batch waits behind it, whatever its window says.
return keepAll(started.ID)
}
if len(alone) > 0 {
// A failed walk's search walks its merges alone, one at a time: everything waits for it (kept as it was,
// issue 973).
if len(started) > 0 {
return keepAll(started[0].ID)
}
if len(waiting) > 0 {
// The waiting walk is the one open; the search walks after it, and the batches wait behind both.
return keepAll(waiting[0].ID)
}
return walkAlone(ctx, open, alone[0], now)
}
return cutUnrelated(ctx, open, batches, started, waiting, now)
}
// releaseHolds asks mesh-delivery again about each group a batch holds, and clears the holds of those with no member
// left to merge. A group mesh-delivery cannot be asked about stays held: the safe error is to wait. It answers the
// batches as kept.
func releaseHolds(ctx context.Context, inv *inventory.Inventory, batches []inventory.Plan, now time.Time) ([]inventory.Plan, error) {
var entries []inventory.Entry
for i := range batches {
b := &batches[i]
if b.Delivery == nil || b.Delivery.Batch == nil || len(b.Delivery.Batch.Holds) == 0 {
continue
}
if entries == nil {
var err error
if entries, err = inv.Catalogued(ctx); err != nil {
return nil, err
}
}
changed := false
for head := range b.Delivery.Batch.Holds {
open, isGroup, _, err := openMembersOf(ctx, inv, deliverySeatHeld(entries), head, "")
if err != nil || (isGroup && len(open) > 0) {
continue
}
holdFor(b, head, "", false, false, now)
changed = true
fmt.Printf("%s: group %s has no member left to merge; its window runs again from now\n", b.ID, head)
}
if changed {
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return nil, err
}
}
if err := keepAll(""); err != nil {
return err
}
return batches, nil
}
// cutUnrelated cuts the closed batches that share nothing with an open walk, each into a walk beside the open
// ones, and says the others queued behind the first walk they are related to (novox/hq ADR 0407 decision 4,
// issue 973). A batch cut in this look counts as open for the batches after it.
//
// - An own-path batch goes first and folds no waiting walk; a catalogue batch folds the waiting walks it is
// related to, and those it shares nothing with keep waiting.
// - **The controller's own path is related to everything** (decided for issue 973): a walk or batch that
// replaces the controller, the node-engine, the tool runner, the bus or mesh-delivery never runs beside
// another, and nothing runs beside it.
// - A batch whose window is still open is assembling; one related to an open walk is queued behind it, and
// ADR 0276 decision 4 stands for it: nothing supersedes a walk that started.
func cutUnrelated(ctx context.Context, open *stores, batches, started, waiting []inventory.Plan, now time.Time) error {
inv := open.inventory
// The oldest batch whose window closed is cut, an own-path one before a catalogue one (it goes first, and
// the catalogue batch queues behind it rather than being cut and overtaken). Several of a kind may be open
// since a merge that shares nothing is cut at once (ADR 0304 decision 1): walks stay one at a time, so one
// is cut per look, and once its walk started the others are queued behind it.
sort.SliceStable(batches, func(i, j int) bool { return batches[i].OwnPath() && !batches[j].OwnPath() })
var entries []inventory.Entry
var read map[string][]inventory.ReadRepository
var edges []inventory.Edge
judged := false
onPath, err := ownPathNow(ctx, inv)
if err != nil {
return err
}
openWalks, err := openWalksOf(ctx, inv, started, onPath)
if err != nil {
return err
}
for i := range batches {
b := &batches[i]
if !windowClosed(*b, now) {
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return err
}
continue
}
if !judged {
var err error
if entries, err = inv.Catalogued(ctx); err != nil {
if b.OwnPath() {
// A walk on the controller's own path folds no walk waiting for its word (decided during the build,
// 2026-10-10): that walk keeps waiting beside it, and starts once this one ended (advanceOnce).
waiting = nil
}
if err := cutBatch(ctx, open, b, waiting, now); err != nil {
return err
}
// The batches left wait behind the walk just cut when it started; the next look does the same.
if b.Open() && !b.Waiting() {
if batches, err = inv.Batches(ctx); err != nil {
return err
}
if read, err = readForPlanning(ctx, inv); err != nil {
return err
}
if edges, err = inv.Dependencies(ctx); err != nil {
return err
}
judged = true
}
held, err := inv.MergesOf(ctx, b.ID)
if err != nil {
return err
}
fp := footprintAsCut(held, entries, read, edges)
var fold, keep []inventory.Plan
if !b.OwnPath() {
for _, w := range waiting {
ww, err := openWalkOf(ctx, inv, w, onPath)
if err != nil {
return err
}
if ok, _ := related(fp, ww.fp, edges); ok {
fold = append(fold, w)
fp = unionFootprint(fp, ww.fp)
} else {
keep = append(keep, w)
}
}
}
if behind, why := relatedOpen(fp, b.OwnPath(), openWalks, edges); behind != "" {
if err := keepBatchBecause(ctx, inv, b, now, behind, why); err != nil {
return err
}
continue
}
if err := keepBatch(ctx, inv, b, now, ""); err != nil {
return err
}
if err := cutBatch(ctx, open, b, fold, now); err != nil {
return err
}
waiting = keep
if c := windowClockOf(); c != nil {
c.Forget(ctx, b.ID) // a batch cut has no window left to close
}
cut, err := inv.PlanByID(ctx, b.ID)
if err != nil {
return err
}
if !cut.Open() {
continue
}
w, err := openWalkOf(ctx, inv, cut, onPath)
if err != nil {
return err
}
if cut.Waiting() {
waiting = append(waiting, cut)
} else {
openWalks = append(openWalks, w)
return keepAll(b.ID)
}
return nil
}
return nil
}
@@ -1189,13 +976,9 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
}
}
sort.Strings(also)
ordered, err := changeOrderEdges(ctx, merges, members, entries, moved, edges)
if err != nil {
return err
}
ordered := groupOrderEdges(members, entries, read, edges)
var withheld []string
plan := planOfMoves(moved, append(append([]inventory.Edge{}, edges...), ordered...))
placedBy(&plan, ordered)
// **Withheld, a stopped merge's modules** (novox/hq ADR 0299): a module is built at its branch, which holds the
// merge a stop left out of the walk these merges were walked again from — built here only as a dependent, it
// would deliver that merge. One a merge of this batch moves is built: that merge carries the stopped change,
@@ -1217,16 +1000,10 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
plan.Delivery.Merges = named
plan.Delivery.Alone = batch.Delivery != nil && batch.Delivery.Alone
plan.Delivery.Withheld = withheld
plan.Delivery.Order = ordered
// **Left where it is, a module whose program did not change** (novox/hq issue 721): its check found it the same as
// the build the mesh holds.
for n, why := range leaveSameAsRunning(ctx, inv, &plan, moved, append(append([]inventory.Edge{}, edges...), ordered...)) {
unchanged[n] = why
}
plan.Delivery.Unchanged = unchangedLeft(unchanged, plan.Modules)
// **Its moments and its class** (novox/hq ADR 0282 decision 6): measured, never acted on.
plan.Times = walkTimesAtCut(*batch, plan, entries, now)
if len(moved) == 0 || len(plan.Modules) == 0 {
if len(moved) == 0 {
plan.State = inventory.PlanDone
plan.Tiers = [][]string{}
plan.Note = "moved nothing the mesh holds: nothing to walk"
@@ -1249,14 +1026,7 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
}
var tiers []string
for i, t := range plan.Tiers {
var names []string
for _, n := range t {
if m := plan.Modules[n]; m != nil && m.Placed != "" {
n += " " + m.Placed
}
names = append(names, n)
}
tiers = append(tiers, fmt.Sprintf("%d: %s", i, strings.Join(names, ", ")))
tiers = append(tiers, fmt.Sprintf("%d: %s", i, strings.Join(t, ", ")))
}
var answers []string
for _, n := range named {
@@ -1274,9 +1044,7 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil
}
// Written and its first tier asked as one act on the plans (novox/hq issue 213), under the caller's hold.
if err := askFirstTierAtCut(&plan,
func(p *inventory.Plan) (bool, error) { return backupsFirst(ctx, inv, p) },
func() error { return askTier(ctx, inv, &plan, plan.Tiers[plan.Tier]) }); err != nil {
if err := askTier(ctx, inv, &plan); err != nil {
return err
}
if err := inv.SavePlan(ctx, &plan); err != nil {
@@ -1286,40 +1054,6 @@ func planBatch(ctx context.Context, open *stores, batch *inventory.Plan, carry [
return nil
}
// askFirstTierAtCut asks a freshly cut plan's first tier, after the backups of the data it moves (novox/hq ADR 0404,
// issue 1086): the cut asks the tier itself, so advanceOnce, which asks the backups before a first tier, finds it
// asked and never takes them. A plan whose backups wait, failed or could not be worked out asks nothing; it is saved
// unasked and the walk's own step goes on from its record.
func askFirstTierAtCut(p *inventory.Plan, backups func(*inventory.Plan) (bool, error), ask func() error) error {
proceed, err := backups(p)
if err != nil {
p.Note = backupsNote + "not asked, the backups of the plan could not be worked out: " + err.Error() +
"; nothing is asked or sent, and the plan tries again"
return nil
}
if !proceed {
return nil
}
return ask()
}
// placedBy writes on each module of a plan the edges of its order that put it after another module of the plan, with the
// rule that drew each (novox/hq ADR 0419 clause 5).
func placedBy(plan *inventory.Plan, ordered []inventory.Edge) {
for _, e := range ordered {
m, in := plan.Modules[e.From]
if _, before := plan.Modules[e.To]; !in || !before {
continue
}
said := "after " + e.To + " (" + e.Why + ")"
if m.Placed == "" {
m.Placed = said
} else if !strings.Contains(m.Placed, said) {
m.Placed += "; " + said
}
}
}
// unchangedSaid names the modules a walk says matched what runs, for its note.
func unchangedSaid(unchanged map[string]string) string {
names := make([]string, 0, len(unchanged))
@@ -1427,9 +1161,8 @@ func classOfModules(modules []string, entries []inventory.Entry) string {
}
}
}
onPath := ownPathOf(entries)
for _, m := range modules {
if _, own := onPath[m]; own || resolvers[m] {
if _, own := onTheControllersPath[m]; own || resolvers[m] {
return inventory.ClassCore
}
}
@@ -1509,46 +1242,73 @@ type batchMember struct {
moved []string
}
// changeOrderEdges is the order of a batch's moved modules as edges of its plan's tiers (novox/hq ADR 0419): drawn
// per module from the module graph and from what the merges say of the manifests they touch (delivery_edges.go),
// whatever repository the merges are in and whether or not they form a delivery group. No `after:` line is read.
// A cycle is said, never lost: the plan puts the modules left in its last tier, and the composed check fails the
// group before it merges.
func changeOrderEdges(ctx context.Context, merges []inventory.BatchedMerge, members []batchMember,
entries []inventory.Entry, moved []string, edges []inventory.Edge) ([]inventory.Edge, error) {
// What each repository's branch held before the batch: the first parent of its earliest merge in it.
earliest := map[string]inventory.BatchedMerge{}
for _, k := range merges {
key := strings.ToLower(k.Repository) + "\x00" + k.Branch
if e, ok := earliest[key]; !ok || laterMerge(e, k) {
earliest[key] = k
}
}
var sources []changeSource
// afterLines are the `after: <repository>` lines of a pull request's description, as mesh-delivery reads them.
var afterLines = regexp.MustCompile(`(?im)^\s*after:\s*([A-Za-z0-9_.\-/]+)\s*$`)
// groupOrderEdges is the order of every delivery group among a batch's merges as edges of its walk's tiers
// (ADR 0276 decision 5, ADR 0249 unchanged): merges of two repositories or more from one head branch name are
// a group; each pair its order rules give puts the modules of the member after in a later tier than the
// member before's. A pair in a contradiction orders nothing: its group was refused before it merged.
func groupOrderEdges(members []batchMember, entries []inventory.Entry, read map[string][]inventory.ReadRepository,
edges []inventory.Edge) []inventory.Edge {
byHead := map[string][]batchMember{}
for _, m := range members {
s := changeSource{Owner: m.merge.Owner, Repo: m.merge.Repo, New: m.merge.Commit,
Paths: m.merge.Paths, Truncated: m.merge.PathsTruncated, Removed: m.merge.Removed}
if e, ok := earliest[strings.ToLower(m.merge.Owner+"/"+m.merge.Repo)+"\x00"+m.merge.Base]; ok {
s.Since = e.Commit
if m.merge.Head != "" {
byHead[m.merge.Head] = append(byHead[m.merge.Head], m)
}
if s.Truncated {
s.Paths, s.Truncated = truncatedPaths(entries, s.Owner, s.Repo, moved), false
}
var out []inventory.Edge
heads := make([]string, 0, len(byHead))
for h := range byHead {
heads = append(heads, h)
}
sort.Strings(heads)
for _, h := range heads {
group := byHead[h]
repos := map[string]bool{}
for _, m := range group {
repos[strings.ToLower(m.merge.Owner+"/"+m.merge.Repo)] = true
}
if len(repos) < 2 {
continue
}
var om []orderMember
moved := map[string][]string{}
for _, m := range group {
repository := m.merge.Owner + "/" + m.merge.Repo
id := repository + "@" + m.merge.Commit
var after []string
for _, l := range afterLines.FindAllStringSubmatch(m.merge.Body, -1) {
after = append(after, strings.TrimSuffix(l[1], ".git"))
}
om = append(om, orderMember{ID: id, Repository: repository, Base: m.merge.Base, Head: m.merge.Commit,
Number: m.merge.Number, Paths: m.merge.Paths, PathsTruncated: m.merge.PathsTruncated,
Removed: m.merge.Removed, ModuleDirs: m.merge.ModuleDirs, ModuleDirsSaid: m.merge.ModuleDirsSaid,
CloneURL: m.merge.CloneURL, After: after})
moved[id] = m.moved
}
order := orderOf(om, reachOfMembers(om, entries, read, edges), edges)
contradicted := map[string]bool{}
for _, c := range order.Contradictions {
for _, id := range c.Members {
contradicted[id] = true
}
}
for _, p := range order.Pairs {
if contradicted[p.Before] && contradicted[p.After] {
continue
}
for _, before := range moved[p.Before] {
for _, after := range moved[p.After] {
if before != after {
out = append(out, inventory.Edge{From: after, To: before, Kind: edgeGroupOrder})
}
}
}
fmt.Printf(" group %s: %s before %s (%s) inside the walk\n", h, p.Before, p.After, p.Why)
}
sources = append(sources, s)
}
changes, err := changesOf(ctx, sources)
if err != nil {
return nil, fmt.Errorf("what the merges say of their manifests cannot be read, so the order of their modules "+
"cannot be drawn: %w", err)
}
order := orderEdges(moved, changes, edges)
for _, e := range order.Edges {
fmt.Printf(" order: %s\n", edgeSaid(e))
}
if said := order.CycleSaid(); said != "" {
fmt.Printf(" %s\n", said)
}
return order.Edges, nil
return out
}
// searchAfterFailures starts the search a failed walk leaves (ADR 0276 decision 3): every merge it answered
@@ -1631,11 +1391,7 @@ func batchWords(b inventory.Plan, now time.Time) string {
grouped += " (on the controller's own path)"
}
if b.State == inventory.PlanQueued {
because := ""
if w.BehindWhy != "" {
because = " (" + w.BehindWhy + ")"
}
return fmt.Sprintf("queued behind %s%s; grouped: %s; plan not yet calculated", w.Behind, because, grouped)
return fmt.Sprintf("queued behind %s; grouped: %s; plan not yet calculated", w.Behind, grouped)
}
if w.CutAtOnce {
return fmt.Sprintf("cut at once, sharing nothing with a batch assembling; grouped: %s; plan not yet calculated",
@@ -1,698 +0,0 @@
//go:build integration
package main
import (
"bytes"
"io"
"os"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func windowed(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []struct{ module, repository, path string }{
{"app", "novox/mesh-catalog", "modules/app"},
{"notes", "novox/mesh-catalog", "modules/notes"},
{"one", "novox/one", ""},
{"two", "novox/two", ""},
{"three", "novox/three", ""},
} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1"},
inventory.Source{Repository: m.repository, Seat: "git", Path: m.path, Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
}
return open
}
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
if err := (following{open: open}).hearMerge(t.Context(), m, now); err != nil {
t.Fatal(err)
}
}
func cutAt(t *testing.T, open *stores, now time.Time) {
t.Helper()
if err := cutBatchesHeld(t.Context(), open, now); err != nil {
t.Fatal(err)
}
}
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, claude, t0.Add(time.Second))
hear(t, open, dunst, t0.Add(18*time.Second))
if ws, bs := walks(t, open); len(ws) != 0 || len(bs) != 1 {
t.Fatalf("within the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
cutAt(t, open, t0.Add(18*time.Second+89*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window closed: %+v", ws)
}
cutAt(t, open, t0.Add(18*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if w.Commit != dunst.Commit || len(w.Commits) != 1 || w.Commits[0].Commit != dunst.Commit {
t.Fatalf("the walk is at %s %+v, not the later commit", w.Commit, w.Commits)
}
for _, m := range []string{"app", "notes"} {
if _, in := w.Modules[m]; !in {
t.Fatalf("the walk does not build %s, which one of its merges moved: %v", m, w.Modules)
}
}
want := []inventory.PlanMerge{{Repository: "novox/mesh-catalog", Commit: claude.Commit, Carried: dunst.Commit},
{Repository: "novox/mesh-catalog", Commit: dunst.Commit}}
if w.Delivery == nil || !slices.EqualFunc(w.Delivery.Merges, want, sameMerge) {
t.Fatalf("the walk answers %+v, want %+v", w.Delivery, want)
}
if len(*asked) != 2 || (*asked)[0][2] != "main" || (*asked)[1][2] != "main" {
t.Fatalf("the walk did not ask its modules at the branch, which holds the commit it carries: %v", *asked)
}
// Heard again, as the bus may hand it over twice: never a second merge, nor a second walk.
hear(t, open, claude, t0.Add(5*time.Minute))
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 {
t.Fatalf("a merge heard twice made %d walk(s) and %d batch(es)", len(ws), len(bs))
}
}
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if len(w.Commits) != 3 || len(w.Delivery.Merges) != 3 {
t.Fatalf("the walk carries %+v and answers %+v", w.Commits, w.Delivery.Merges)
}
for _, r := range []string{"one", "two", "three"} {
if w.CommitOf("novox/"+r) != "c-"+r {
t.Fatalf("the walk carries %s at %q", r, w.CommitOf("novox/"+r))
}
if _, in := w.Modules[r]; !in {
t.Fatalf("the walk does not build %s: %v", r, w.Modules)
}
}
// Each module's ask is recorded at its own repository's commit.
for _, r := range []string{"one", "two", "three"} {
q, found, err := open.inventory.BuildRequestByID(t.Context(), w.Modules[r].Build)
if err != nil || !found || q.Commit != "c-"+r {
t.Fatalf("%s's build was not recorded at its own commit: %+v %v %v", r, q, found, err)
}
}
}
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
}
cutAt(t, open, t0.Add(170*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || len(ws[0].Commits) != 2 {
t.Fatalf("the window did not close 90 seconds after its last merge: %+v", ws)
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
}
break
}
}
ws, _ := walks(t, busy)
if len(ws) != 1 {
t.Fatalf("ten minutes of merges a minute apart were never cut: %d walk(s)", len(ws))
}
}
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
claude.Number, claude.Title = 174, "claude-code: an agent proposes a section"
dunst := catalogueMerge("48bda475dunst", "notes", now.Add(-2*time.Second))
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
for _, want := range []string{"assembling: ", " s left (at the latest ", "grouped: novox/mesh-catalog@48bda475 " +
"(answers 553b7191), novox/one@a6bc0931; plan not yet calculated"} {
if !strings.Contains(first, want) {
t.Fatalf("plans' first line %q does not say %q", first, want)
}
}
// Under it, one line per repository: the pull request, what it answers, what it moves.
if len(lines) < 3 || strings.TrimSpace(lines[1]) != "mesh-catalog #175 dunst: the font the operator chose (answers "+
"#174 claude-code: an agent proposes a section) · app, notes" ||
strings.TrimSpace(lines[2]) != "one a6bc0931 · one" {
t.Fatalf("the grouped list reads %q", lines[1:4])
}
if len(said) == 0 || said[len(said)-1].State != inventory.PlanAssembling || said[len(said)-1].Delivery.Batch == nil ||
len(said[len(said)-1].Delivery.Merges) != 3 {
t.Fatalf("the batch was not said as assembling with its merges: %+v", said)
}
}
func captured(t *testing.T, run func() error) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
runErr := run()
os.Stdout = was
_ = w.Close()
var b bytes.Buffer
_, _ = io.Copy(&b, r)
if runErr != nil {
t.Fatal(runErr)
}
return b.String()
}
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first batch was not cut: %+v", ws)
}
first := ws[0]
hear(t, open, repoMerge("one", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(5*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
t.Fatalf("the merge during the walk: %d walk(s), batches %+v", len(ws), bs)
}
if got, _ := open.inventory.PlanByID(t.Context(), first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the next merge", got.State)
}
if !strings.HasPrefix(batchWords(bs[0], t0.Add(5*time.Minute)), "queued behind "+first.ID) {
t.Fatalf("a queued batch reads %q", batchWords(bs[0], t0.Add(5*time.Minute)))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(6*time.Minute))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut when the walk ended: %+v %+v", ws, bs)
}
}
func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
waiting := ws[0]
hear(t, open, repoMerge("one", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
ws, bs := walks(t, open)
if len(bs) != 0 || len(ws) != 2 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
folded, err := open.inventory.PlanByID(ctx, waiting.ID)
if err != nil {
t.Fatal(err)
}
newer := ws[0]
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
}
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c2" {
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
}
if _, in := newer.Modules["one"]; !in {
t.Fatalf("what the folded walk was to build is not built: %v", newer.Modules)
}
}
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, later, t0.Add(18*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
// Moves only notes, which the done walk built from the branch after it: answered there, at once.
hear(t, open, catalogueMerge("553b7191early", "notes", t0), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if got.State != inventory.PlanDone || len(got.Delivery.Merges) != 2 ||
!sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: "553b7191early",
Carried: later.Commit}) {
t.Fatalf("the late merge is not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// Moves app, which that walk never built: the next batch walks it.
hear(t, open, catalogueMerge("1111aaaaapp", "app", t0.Add(time.Second)), t0.Add(16*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "1111aaaaapp" {
t.Fatalf("a late merge moving what the walk never built did not join the next batch: %+v", bs)
}
}
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
oldest := catalogueMerge("aaaa0001", "app", t0.Add(-3*time.Minute))
middle := catalogueMerge("bbbb0002", "app", t0.Add(-3*time.Minute+10*time.Second))
newest := catalogueMerge("cccc0003", "notes", t0.Add(-3*time.Minute+20*time.Second))
for i, m := range []link.SourceMoved{oldest, middle, newest} {
hear(t, open, m, t0.Add(time.Duration(i)*10*time.Second+time.Second))
}
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
failed := ws[0]
failed.State, failed.Note = inventory.PlanFailed, "notes failed to build in tier 0"
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, failed.ID)
if got.State != inventory.PlanFailed || len(got.Delivery.Merges) != 3 {
t.Fatalf("the failed walk's record changed: %s %+v", got.State, got.Delivery.Merges)
}
ws, _ = walks(t, open)
alone := ws[0]
if alone.ID == failed.ID || alone.Commit != middle.Commit || len(alone.Delivery.Merges) != 1 ||
!sameMerge(alone.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: middle.Commit}) {
t.Fatalf("the newest earlier merge was not walked alone on its own commit: %+v", alone)
}
if _, in := alone.Modules["app"]; !in || (*asked)[len(*asked)-1] != [3]string{"novox/mesh-catalog", "modules/app",
middle.Commit} {
t.Fatalf("the merge walked alone does not build app at its own commit: %v, asked %v", alone.Modules, *asked)
}
// Retried, the failed walk would name merges the search answers now.
if _, err := retryPlan(ctx, open, failed.ID); err == nil || !strings.Contains(err.Error(), "walked alone") {
t.Fatalf("a failed walk whose merges are searched was retried: %v", err)
}
alone.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &alone); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(4*time.Minute))
ws, bs := walks(t, open)
if ws[0].ID != alone.ID || len(bs) != 0 {
t.Fatalf("the search went on after a merge was delivered: %+v", ws[0])
}
done, _ := open.inventory.PlanByID(ctx, alone.ID)
if len(done.Delivery.Merges) != 2 || !sameMerge(done.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog",
Commit: oldest.Commit, Carried: middle.Commit}) {
t.Fatalf("the oldest merge is not answered by the walk that delivered the one after it: %+v", done.Delivery.Merges)
}
// A stopped walk starts no search: a person ended it.
hear(t, open, catalogueMerge("dddd0004", "app", t0.Add(5*time.Minute)), t0.Add(5*time.Minute))
hear(t, open, catalogueMerge("eeee0005", "notes", t0.Add(5*time.Minute+time.Second)), t0.Add(5*time.Minute+time.Second))
cutAt(t, open, t0.Add(8*time.Minute))
ws, _ = walks(t, open)
if _, err := stopWalk(ctx, open.inventory, ws[0].ID, "mesh-delivery for jochen", "not now"); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(9*time.Minute))
if again, _ := walks(t, open); again[0].ID != ws[0].ID {
t.Fatalf("a stopped walk's earlier merge was walked alone: %+v", again[0])
}
}
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1"},
inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
host := repoMerge("mesh-host", "h1", t0)
ctl := repoMerge("mesh-controller", "k1", t0.Add(time.Second))
host.Head, ctl.Head = "feat/together", "feat/together"
if !grouped {
ctl.Head = "feat/alone"
}
hear(t, open, ctl, t0.Add(2*time.Second))
hear(t, open, host, t0.Add(3*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walks", len(ws))
}
tiers := ws[0].Tiers
if grouped {
if len(tiers) != 2 || !slices.Equal(tiers[0], []string{"mesh-host"}) ||
!slices.Equal(tiers[1], []string{"mesh-controller"}) {
t.Fatalf("the group's order is not the walk's tiers: %v", tiers)
}
} else if len(tiers) != 1 {
t.Fatalf("two merges of no group were ordered: %v", tiers)
}
}
}
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
}
cutAt(t, again, t0.Add(120*time.Second))
ws, bs := walks(t, again)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller did not cut the batch at its window: %+v %+v", ws, bs)
}
}
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
hear(t, open, c2, t0)
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0.Add(-60*time.Second)), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the late merge was not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// A walk of two that built only two: a late merge of one is not its to name.
hear(t, open, repoMerge("two", "d2", t0.Add(16*time.Minute)), t0.Add(16*time.Minute))
cutAt(t, open, t0.Add(18*time.Minute))
ws, _ = walks(t, open)
w = ws[0]
delete(w.Modules, "two")
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("two", "d1", t0.Add(15*time.Minute)), t0.Add(20*time.Minute))
if got, _ := open.inventory.PlanByID(ctx, w.ID); len(got.Delivery.Merges) != 1 {
t.Fatalf("a walk that never built what the late merge moves named it: %+v", got.Delivery.Merges)
}
if _, bs := walks(t, open); len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "d1" {
t.Fatalf("the late merge did not join the next batch: %+v", bs)
}
}
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(2*time.Minute))
started, _ := walks(t, open)
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/one", Branch: "main", Commit: "d1", Created: t0,
State: inventory.PlanBuilding, Tiers: [][]string{{"one"}}, Modules: map[string]*inventory.PlanModule{"one": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if _, err := letGo(ctx, open.inventory, waiting.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatalf("the word was refused beside %s: %v", started[0].ID, err)
}
before := len(*asked)
advanceHeld(ctx, open)
got, _ := open.inventory.PlanByID(ctx, waiting.ID)
if len(*asked) != before || !strings.Contains(got.Note, "starts once "+started[0].ID) {
t.Fatalf("a walk let go beside a started one asked (%d → %d) or does not say it waits: %q", before, len(*asked), got.Note)
}
// Read as a wait, an hour on: its note on the line, never LATE, and no tier of it late for S3.
later := time.Now().Add(time.Hour)
if line := planLine(got, later); !strings.Contains(line, "starts once "+started[0].ID) || strings.Contains(line, "LATE") {
t.Fatalf("a deferred walk reads %q", line)
}
facts, _, err := gatherPlans(ctx, open.inventory, later, nil)
if err != nil {
t.Fatal(err)
}
for _, f := range facts {
if f.id == got.ID {
t.Fatalf("a deferred walk is watched as a tier running late: %+v", f)
}
}
done := started[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) != before+1 {
t.Fatalf("the walk did not start once the started one ended: asked %v", *asked)
}
}
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("aaaa0001", "app", t0), t0)
hear(t, open, repoMerge("mesh-controller", "k1", t0.Add(10*time.Second)), t0.Add(10*time.Second))
hear(t, open, catalogueMerge("bbbb0002", "notes", t0.Add(20*time.Second)), t0.Add(20*time.Second))
_, bs := walks(t, open)
var ownBatch, catalogueBatch inventory.Plan
for _, b := range bs {
if b.OwnPath() {
ownBatch = b
} else {
catalogueBatch = b
}
}
if len(bs) != 2 || ownBatch.ID == "" || catalogueBatch.ID == "" || len(catalogueBatch.Delivery.Merges) != 2 ||
len(ownBatch.Delivery.Merges) != 1 {
t.Fatalf("a controller merge and two catalogue merges in one window: %+v", bs)
}
if !strings.Contains(batchWords(ownBatch, t0.Add(30*time.Second)), "own path") {
t.Fatalf("the own-path batch does not say so: %q", batchWords(ownBatch, t0.Add(30*time.Second)))
}
// Both windows closed, the controller's batch is cut first and starts; the catalogue batch queues behind it,
// is cut when the controller's walk ended, and waits for its word with both merges.
cutAt(t, open, t0.Add(2*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the controller's batch was not cut first into a started walk: %+v", ws)
}
ownWalk := ws[0]
for _, m := range []string{"app", "notes"} {
if _, in := ownWalk.Modules[m]; in {
t.Fatalf("the controller's walk builds %s, a catalogue module: it skipped its turn", m)
}
}
if len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != ownWalk.ID || bs[0].OwnPath() {
t.Fatalf("the catalogue batch does not queue behind the controller's walk: %+v", bs)
}
ownWalk.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &ownWalk); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
ws, bs = walks(t, open)
if len(bs) != 0 || ws[0].ID != catalogueBatch.ID || !ws[0].Waiting() || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the catalogue batch was not cut into a waiting walk once the controller's ended: %+v %+v", ws, bs)
}
for _, m := range []string{"app", "notes"} {
if _, in := ws[0].Modules[m]; !in {
t.Fatalf("the catalogue walk does not build %s: %v", m, ws[0].Modules)
}
}
for _, a := range *asked {
if a[1] == "modules/app" || a[1] == "modules/notes" {
t.Fatalf("a catalogue module was asked without the word: %v", *asked)
}
}
// A catalogue walk waiting for its word when an own-path batch is cut keeps waiting beside the own-path
// walk, is not folded into it, and its word is taken meanwhile: it starts once the own-path walk ended.
catalogueWalk := ws[0]
hear(t, open, repoMerge("mesh-controller", "k2", t0.Add(4*time.Minute)), t0.Add(4*time.Minute))
cutAt(t, open, t0.Add(6*time.Minute))
ws, _ = walks(t, open)
kept, _ := open.inventory.PlanByID(ctx, catalogueWalk.ID)
if !kept.Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k2" || ws[0].Waiting() {
t.Fatalf("the waiting catalogue walk was not kept waiting beside the controller's walk: %s %+v", kept.State, ws)
}
if _, in := ws[0].Modules["app"]; in {
t.Fatalf("the controller's walk folded the waiting catalogue walk in: %v", ws[0].Modules)
}
if _, err := letGo(ctx, open.inventory, catalogueWalk.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatal(err)
}
before := len(*asked)
advanceHeld(ctx, open)
if len(*asked) != before {
t.Fatalf("the catalogue walk started beside the controller's: asked %v", (*asked)[before:])
}
own2 := ws[0]
own2.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &own2); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) < before+1 || (*asked)[before][1] != "modules/app" {
t.Fatalf("the catalogue walk did not start once the controller's ended: %v", (*asked)[before:])
}
}
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
module string
claims []catalogue.Claim
}{
{"nats", nil},
{"mesh-delivery", []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1", Claims: m.claims},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/" + m.module, Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
mixed := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "m1xed", MergedAt: t0.Format(time.RFC3339Nano),
Paths: []string{"modules/nats/module.json", "modules/app/module.json"}, ModuleDirs: []string{"modules/nats", "modules/app"},
ModuleDirsSaid: true}
hear(t, open, mixed, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() {
t.Fatalf("the mixed merge's walk waited, or was not cut: %+v", ws)
}
done := ws[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("ear1ier", "app", t0.Add(-30*time.Second)), t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, done.ID)
_, bs := walks(t, open)
if len(got.Delivery.Merges) != 1 || len(bs) != 1 || bs[0].OwnPath() || bs[0].Delivery.Merges[0].Commit != "ear1ier" {
t.Fatalf("the late catalogue merge was answered by the own-path walk (%+v) rather than the next catalogue batch (%+v)",
got.Delivery.Merges, bs)
}
}
+720 -47
View File
@@ -1,16 +1,17 @@
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"os"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -18,19 +19,42 @@ import (
// The suite's tests of a merge, but for the merge window's own (this file's), plan the merge as it is heard: a
// window of nothing closes at once, so a merge with no walk open is cut into its walk at once, as every merge
// was planned before novox/hq ADR 0276. The window's tests set it through the controller's settings.
func init() {
mergeWindowDefault = 0
// No forge in a test: no manifest is found at any commit, so a merge draws no edge of the manifests' change
// (delivery_edges.go). A test that wants some replaces these (forgeHolding).
manifestAt = func(context.Context, string, string, string, string) (catalogue.Manifest, bool, error) {
return catalogue.Manifest{}, false, nil
}
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
}
func init() { mergeWindowDefault = 0 }
// windowed is a mesh whose controller's settings give a merge window of 90 seconds and at most 10 minutes, as its
// settings file says them, and
// modules built from the catalogue (app, notes) and from three repositories of their own.
func windowed(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
settings := t.TempDir() + "/merge-window.json"
if err := os.WriteFile(settings, []byte(`{"merge-window": "90s", "merge-window-at-most": "10m"}`), 0o600); err != nil {
t.Fatal(err)
}
t.Setenv(mergeWindowFileVar, settings)
noGroups(t)
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Seat: "git", Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
for _, m := range []struct{ module, repository, path string }{
{"app", "novox/mesh-catalog", "modules/app"},
{"notes", "novox/mesh-catalog", "modules/notes"},
{"one", "novox/one", ""},
{"two", "novox/two", ""},
{"three", "novox/three", ""},
} {
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1"},
inventory.Source{Repository: m.repository, Seat: "git", Path: m.path, Ref: "main", BuiltFrom: "c0",
Head: "c0"}); err != nil {
t.Fatal(err)
}
}
return open
}
// t0 is the moment a test's clock starts: merges are dated by it, and heard after it.
var t0 = time.Now().UTC().Truncate(time.Second)
@@ -59,16 +83,15 @@ func noGroups(t *testing.T) {
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupsWaiting stubs mesh-delivery's word on groups for a test of the merge window's own timing: each head named
// is one whose group mesh-delivery cannot say, so a merge from it waits for the window (ADR 0306 decision 2) and
// nothing holds the batch open; every other head names no group. A group with a member still to merge, which holds
// its batch open however long it takes (ADR 0418 clause 3), is groupStates'.
// groupsWaiting stubs mesh-delivery's word on groups for a test: each head named is a group with a member of another
// repository whose pull request is open (novox/elsewhere, ready), so a merge from it waits for the window (ADR 0306
// decision 2); every other head names no group.
func groupsWaiting(t *testing.T, heads ...string) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, head string) ([]groupMember, bool, error) {
if slices.Contains(heads, head) {
return nil, false, errors.New("the test's mesh-delivery does not say")
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
return nil, false, nil
}
@@ -76,37 +99,12 @@ func groupsWaiting(t *testing.T, heads ...string) {
}
// everyHeadWaits stubs mesh-delivery's word on groups for a test whose merges each wait for the window as they did
// before novox/hq ADR 0304: mesh-delivery cannot say any head's group.
// before novox/hq ADR 0304: every head is a group with a member of another repository still to merge.
func everyHeadWaits(t *testing.T) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(context.Context, bool, string) ([]groupMember, bool, error) {
return nil, false, errors.New("the test's mesh-delivery does not say")
}
t.Cleanup(func() { deliveryGroupOf = was })
}
// groupStates stubs mesh-delivery's word on one group: its members and their states, which a test changes as the
// members merge.
func groupStates(t *testing.T, head string, states map[string]string) {
t.Helper()
groupStatesFrom(t, head, states, true)
}
// groupStatesFrom is groupStates for a mesh-delivery that does (composed) or does not yet say the group's gate is the
// composed check's: only the first holds the group's batch open for its last member.
func groupStatesFrom(t *testing.T, head string, states map[string]string, composed bool) {
t.Helper()
was := deliveryGroupOf
deliveryGroupOf = func(_ context.Context, _ bool, asked string) ([]groupMember, bool, error) {
if asked != head {
return nil, false, nil
}
var members []groupMember
for repository, state := range states {
members = append(members, groupMember{Repository: repository, State: state, Composed: composed})
}
return members, true, nil
return []groupMember{{Repository: "novox/elsewhere", State: "ready"}}, true, nil
}
t.Cleanup(func() { deliveryGroupOf = was })
}
@@ -126,8 +124,20 @@ func lone(m link.SourceMoved) link.SourceMoved {
}
// hear is a merge heard at a moment.
func hear(t *testing.T, open *stores, m link.SourceMoved, now time.Time) {
t.Helper()
if err := (following{open: open}).hearMerge(t.Context(), m, now); err != nil {
t.Fatal(err)
}
}
// cutAt is the cutter looking at a moment.
func cutAt(t *testing.T, open *stores, now time.Time) {
t.Helper()
if err := cutBatchesHeld(t.Context(), open, now); err != nil {
t.Fatal(err)
}
}
// walks is every plan record that is a walk, newest first, and the batches not yet cut.
func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
@@ -148,34 +158,426 @@ func walks(t *testing.T, open *stores) (walks, batches []inventory.Plan) {
// The replay of issue 362: two catalogue merges 18 seconds apart are one batch, cut once into one walk at the
// later commit, which names both merges — the earlier carried by the later.
func TestTwoMergesSecondsApartAreOneWalkAtTheLaterCommit(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
claude := catalogueMerge("553b7191claude", "app", t0)
dunst := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, claude, t0.Add(time.Second))
hear(t, open, dunst, t0.Add(18*time.Second))
if ws, bs := walks(t, open); len(ws) != 0 || len(bs) != 1 {
t.Fatalf("within the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
cutAt(t, open, t0.Add(18*time.Second+89*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatalf("cut before the window closed: %+v", ws)
}
cutAt(t, open, t0.Add(18*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("after the window: %d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if w.Commit != dunst.Commit || len(w.Commits) != 1 || w.Commits[0].Commit != dunst.Commit {
t.Fatalf("the walk is at %s %+v, not the later commit", w.Commit, w.Commits)
}
for _, m := range []string{"app", "notes"} {
if _, in := w.Modules[m]; !in {
t.Fatalf("the walk does not build %s, which one of its merges moved: %v", m, w.Modules)
}
}
want := []inventory.PlanMerge{{Repository: "novox/mesh-catalog", Commit: claude.Commit, Carried: dunst.Commit},
{Repository: "novox/mesh-catalog", Commit: dunst.Commit}}
if w.Delivery == nil || !slices.EqualFunc(w.Delivery.Merges, want, sameMerge) {
t.Fatalf("the walk answers %+v, want %+v", w.Delivery, want)
}
if len(*asked) != 2 || (*asked)[0][2] != "main" || (*asked)[1][2] != "main" {
t.Fatalf("the walk did not ask its modules at the branch, which holds the commit it carries: %v", *asked)
}
// Heard again, as the bus may hand it over twice: never a second merge, nor a second walk.
hear(t, open, claude, t0.Add(5*time.Minute))
if ws, bs := walks(t, open); len(ws) != 1 || len(bs) != 0 {
t.Fatalf("a merge heard twice made %d walk(s) and %d batch(es)", len(ws), len(bs))
}
}
// Merges of three repositories in one window are one walk, one commit for each.
func TestThreeRepositoriesInOneWindowAreOneWalk(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/three")
asksRecorded(t)
for i, r := range []string{"one", "two", "three"} {
hear(t, open, fromHead(repoMerge(r, "c-"+r, t0.Add(time.Duration(i)*20*time.Second)), "feat/three"),
t0.Add(time.Duration(i)*20*time.Second+time.Second))
}
cutAt(t, open, t0.Add(41*time.Second+90*time.Second))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 0 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
w := ws[0]
if len(w.Commits) != 3 || len(w.Delivery.Merges) != 3 {
t.Fatalf("the walk carries %+v and answers %+v", w.Commits, w.Delivery.Merges)
}
for _, r := range []string{"one", "two", "three"} {
if w.CommitOf("novox/"+r) != "c-"+r {
t.Fatalf("the walk carries %s at %q", r, w.CommitOf("novox/"+r))
}
if _, in := w.Modules[r]; !in {
t.Fatalf("the walk does not build %s: %v", r, w.Modules)
}
}
// Each module's ask is recorded at its own repository's commit.
for _, r := range []string{"one", "two", "three"} {
q, found, err := open.inventory.BuildRequestByID(t.Context(), w.Modules[r].Build)
if err != nil || !found || q.Commit != "c-"+r {
t.Fatalf("%s's build was not recorded at its own commit: %+v %v %v", r, q, found, err)
}
}
}
// Each merge restarts the window: a merge at second 80 keeps the batch open until second 170; the window
// closes at most ten minutes after its first merge however busy.
func TestAMergeRestartsTheWindowAndTheMaximumClosesIt(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(80*time.Second)), "feat/both"), t0.Add(80*time.Second))
cutAt(t, open, t0.Add(100*time.Second))
if ws, _ := walks(t, open); len(ws) != 0 {
t.Fatal("the window closed 90 seconds after its first merge, not after its last")
}
cutAt(t, open, t0.Add(170*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || len(ws[0].Commits) != 2 {
t.Fatalf("the window did not close 90 seconds after its last merge: %+v", ws)
}
busy := windowed(t)
groupsWaiting(t, "feat/both", "feat/busy")
asksWithPaths(t)
var last time.Time
for i := 0; i < 12; i++ {
last = t0.Add(time.Duration(i) * time.Minute)
repo := []string{"one", "two", "three"}[i%3]
hear(t, busy, fromHead(repoMerge(repo, "c"+string(rune('a'+i)), last), "feat/busy"), last)
if ws, _ := walks(t, busy); len(ws) > 0 {
if i != 10 {
t.Fatalf("a busy window closed at merge %d (minute %d), not at its maximum", i, i)
}
break
}
}
ws, _ := walks(t, busy)
if len(ws) != 1 {
t.Fatalf("ten minutes of merges a minute apart were never cut: %d walk(s)", len(ws))
}
}
// `plans` lists the batch being assembled first, in the operator's words: how long is left, when at the latest,
// what is grouped, and that the plan is not yet calculated. It says so on the bus as plan-moved, too.
func TestTheAssemblingBatchIsShown(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "pr/553b7191claude")
asksWithPaths(t)
var said []inventory.Plan
was := inventory.PlanSaved
inventory.PlanSaved = func(_ *inventory.Inventory, p inventory.Plan) { said = append(said, p) }
t.Cleanup(func() { inventory.PlanSaved = was })
now := time.Now().UTC()
claude := catalogueMerge("553b7191claude", "app", now.Add(-20*time.Second))
claude.Number, claude.Title = 174, "claude-code: an agent proposes a section"
dunst := catalogueMerge("48bda475dunst", "notes", now.Add(-2*time.Second))
dunst.Number, dunst.Title = 175, "dunst: the font the operator chose"
hear(t, open, claude, now.Add(-19*time.Second))
hear(t, open, dunst, now.Add(-time.Second))
hear(t, open, fromHead(repoMerge("one", "a6bc0931one", now), dunst.Head), now)
out := captured(t, func() error { return plansCommand(t.Context(), nil) })
lines := strings.Split(out, "\n")
first := lines[0]
for _, want := range []string{"assembling: ", " s left (at the latest ", "grouped: novox/mesh-catalog@48bda475 " +
"(answers 553b7191), novox/one@a6bc0931; plan not yet calculated"} {
if !strings.Contains(first, want) {
t.Fatalf("plans' first line %q does not say %q", first, want)
}
}
// Under it, one line per repository: the pull request, what it answers, what it moves.
if len(lines) < 3 || strings.TrimSpace(lines[1]) != "mesh-catalog #175 dunst: the font the operator chose (answers "+
"#174 claude-code: an agent proposes a section) · app, notes" ||
strings.TrimSpace(lines[2]) != "one a6bc0931 · one" {
t.Fatalf("the grouped list reads %q", lines[1:4])
}
if len(said) == 0 || said[len(said)-1].State != inventory.PlanAssembling || said[len(said)-1].Delivery.Batch == nil ||
len(said[len(said)-1].Delivery.Merges) != 3 {
t.Fatalf("the batch was not said as assembling with its merges: %+v", said)
}
}
// captured is what a command printed.
func captured(t *testing.T, run func() error) string {
t.Helper()
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
was := os.Stdout
os.Stdout = w
runErr := run()
os.Stdout = was
_ = w.Close()
var b bytes.Buffer
_, _ = io.Copy(&b, r)
if runErr != nil {
t.Fatal(runErr)
}
return b.String()
}
// A merge heard during a walk it is related to (the same repository, issue 973) joins the next batch, which is
// cut when the walk ends; the walk that started is never superseded (ADR 0276 decision 4, ADR 0407 decision 4).
// One walk at a time: a merge heard while a walk runs joins the next batch, which is cut when the walk ends;
// the walk that started is never superseded.
func TestAMergeDuringAWalkJoinsTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first batch was not cut: %+v", ws)
}
first := ws[0]
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(5*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != first.ID {
t.Fatalf("the merge during the walk: %d walk(s), batches %+v", len(ws), bs)
}
if got, _ := open.inventory.PlanByID(t.Context(), first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the next merge", got.State)
}
if !strings.HasPrefix(batchWords(bs[0], t0.Add(5*time.Minute)), "queued behind "+first.ID) {
t.Fatalf("a queued batch reads %q", batchWords(bs[0], t0.Add(5*time.Minute)))
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(t.Context(), &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(6*time.Minute))
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 0 || ws[0].Commit != "c2" {
t.Fatalf("the queued batch was not cut when the walk ended: %+v %+v", ws, bs)
}
}
// A walk waiting for its delivery's word is folded into the next batch's walk, which answers its merges; it
// names the walk that took it over.
func TestAWaitingWalkIsFoldedIntoTheNextBatch(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 || !ws[0].Waiting() {
t.Fatalf("the walk does not wait for its word: %+v", ws)
}
waiting := ws[0]
hear(t, open, repoMerge("two", "c2", t0.Add(2*time.Minute)), t0.Add(2*time.Minute))
cutAt(t, open, t0.Add(2*time.Minute+90*time.Second))
ws, bs := walks(t, open)
if len(bs) != 0 || len(ws) != 2 {
t.Fatalf("%d walk(s), %d batch(es)", len(ws), len(bs))
}
folded, err := open.inventory.PlanByID(ctx, waiting.ID)
if err != nil {
t.Fatal(err)
}
newer := ws[0]
if folded.State != inventory.PlanSuperseded || folded.Delivery.TakenOverBy != newer.ID {
t.Fatalf("the waiting walk is %s, taken over by %q", folded.State, folded.Delivery.TakenOverBy)
}
if len(newer.Delivery.Merges) != 2 || newer.CommitOf("novox/one") != "c1" || newer.CommitOf("novox/two") != "c2" {
t.Fatalf("the newer walk answers %+v", newer.Delivery.Merges)
}
if _, in := newer.Modules["one"]; !in {
t.Fatalf("what the folded walk was to build is not built: %v", newer.Modules)
}
}
// A merge heard after a later merge of its branch was walked is answered by that walk when it builds all it
// moves — named at once on a walk done — and joins the next batch when it does not.
func TestALateMergeIsAnsweredByTheWalkOfTheLaterOne(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
later := catalogueMerge("48bda475dunst", "notes", t0.Add(17*time.Second))
hear(t, open, later, t0.Add(18*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
// Moves only notes, which the done walk built from the branch after it: answered there, at once.
hear(t, open, catalogueMerge("553b7191early", "notes", t0), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if got.State != inventory.PlanDone || len(got.Delivery.Merges) != 2 ||
!sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: "553b7191early",
Carried: later.Commit}) {
t.Fatalf("the late merge is not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// Moves app, which that walk never built: the next batch walks it.
hear(t, open, catalogueMerge("1111aaaaapp", "app", t0.Add(time.Second)), t0.Add(16*time.Minute))
_, bs := walks(t, open)
if len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "1111aaaaapp" {
t.Fatalf("a late merge moving what the walk never built did not join the next batch: %+v", bs)
}
}
// A failed walk marks nothing delivered: its earlier merges are walked alone, on their own commit, the newest
// first; the first delivered answers the older ones, and the search stops.
func TestAFailedWalkWalksItsEarlierMergesAlone(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
// Made before the cut marks their modules seen, as merges are: walked again, they are news all the same.
oldest := catalogueMerge("aaaa0001", "app", t0.Add(-3*time.Minute))
middle := catalogueMerge("bbbb0002", "app", t0.Add(-3*time.Minute+10*time.Second))
newest := catalogueMerge("cccc0003", "notes", t0.Add(-3*time.Minute+20*time.Second))
for i, m := range []link.SourceMoved{oldest, middle, newest} {
hear(t, open, m, t0.Add(time.Duration(i)*10*time.Second+time.Second))
}
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
failed := ws[0]
failed.State, failed.Note = inventory.PlanFailed, "notes failed to build in tier 0"
if err := open.inventory.SavePlan(ctx, &failed); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, failed.ID)
if got.State != inventory.PlanFailed || len(got.Delivery.Merges) != 3 {
t.Fatalf("the failed walk's record changed: %s %+v", got.State, got.Delivery.Merges)
}
ws, _ = walks(t, open)
alone := ws[0]
if alone.ID == failed.ID || alone.Commit != middle.Commit || len(alone.Delivery.Merges) != 1 ||
!sameMerge(alone.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog", Commit: middle.Commit}) {
t.Fatalf("the newest earlier merge was not walked alone on its own commit: %+v", alone)
}
if _, in := alone.Modules["app"]; !in || (*asked)[len(*asked)-1] != [3]string{"novox/mesh-catalog", "modules/app",
middle.Commit} {
t.Fatalf("the merge walked alone does not build app at its own commit: %v, asked %v", alone.Modules, *asked)
}
// Retried, the failed walk would name merges the search answers now.
if _, err := retryPlan(ctx, open, failed.ID); err == nil || !strings.Contains(err.Error(), "walked alone") {
t.Fatalf("a failed walk whose merges are searched was retried: %v", err)
}
alone.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &alone); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(4*time.Minute))
ws, bs := walks(t, open)
if ws[0].ID != alone.ID || len(bs) != 0 {
t.Fatalf("the search went on after a merge was delivered: %+v", ws[0])
}
done, _ := open.inventory.PlanByID(ctx, alone.ID)
if len(done.Delivery.Merges) != 2 || !sameMerge(done.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/mesh-catalog",
Commit: oldest.Commit, Carried: middle.Commit}) {
t.Fatalf("the oldest merge is not answered by the walk that delivered the one after it: %+v", done.Delivery.Merges)
}
// A stopped walk starts no search: a person ended it.
hear(t, open, catalogueMerge("dddd0004", "app", t0.Add(5*time.Minute)), t0.Add(5*time.Minute))
hear(t, open, catalogueMerge("eeee0005", "notes", t0.Add(5*time.Minute+time.Second)), t0.Add(5*time.Minute+time.Second))
cutAt(t, open, t0.Add(8*time.Minute))
ws, _ = walks(t, open)
if _, err := stopWalk(ctx, open.inventory, ws[0].ID, "mesh-delivery for jochen", "not now"); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(9*time.Minute))
if again, _ := walks(t, open); again[0].ID != ws[0].ID {
t.Fatalf("a stopped walk's earlier merge was walked alone: %+v", again[0])
}
}
// A delivery group's order holds inside the walk (ADR 0249 unchanged): merges of two repositories from one head
// branch name, the node-engine's before the controller's, are tiered so; without the group, one tier.
func TestAGroupsOrderBecomesTiersInsideTheWalk(t *testing.T) {
for _, grouped := range []bool{true, false} {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []string{"mesh-host"} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m, Version: "1"},
inventory.Source{Repository: "novox/" + m, Seat: "git", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
host := repoMerge("mesh-host", "h1", t0)
ctl := repoMerge("mesh-controller", "k1", t0.Add(time.Second))
host.Head, ctl.Head = "feat/together", "feat/together"
if !grouped {
ctl.Head = "feat/alone"
}
hear(t, open, ctl, t0.Add(2*time.Second))
hear(t, open, host, t0.Add(3*time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walks", len(ws))
}
tiers := ws[0].Tiers
if grouped {
if len(tiers) != 2 || !slices.Equal(tiers[0], []string{"mesh-host"}) ||
!slices.Equal(tiers[1], []string{"mesh-controller"}) {
t.Fatalf("the group's order is not the walk's tiers: %v", tiers)
}
} else if len(tiers) != 1 {
t.Fatalf("two merges of no group were ordered: %v", tiers)
}
}
}
// A restarted controller resumes the window where it stood: the batch, its merges and their times are in the
// store, a merge handed over again is not doubled, and the batch is cut when its window closes.
func TestABatchSurvivesARestart(t *testing.T) {
open := windowed(t)
groupsWaiting(t, "feat/both")
asksWithPaths(t)
hear(t, open, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0)
hear(t, open, fromHead(repoMerge("two", "c2", t0.Add(30*time.Second)), "feat/both"), t0.Add(30*time.Second))
// A new controller: nothing of the first one's but the store.
again, err := openStores(t.Context())
if err != nil {
t.Fatal(err)
}
t.Cleanup(again.Close)
hear(t, again, fromHead(repoMerge("one", "c1", t0), "feat/both"), t0.Add(60*time.Second)) // the bus hands it over again
cutAt(t, again, t0.Add(119*time.Second))
if ws, bs := walks(t, again); len(ws) != 0 || len(bs) != 1 || len(bs[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller's batch: %d walk(s), %+v", len(ws), bs)
}
cutAt(t, again, t0.Add(120*time.Second))
ws, bs := walks(t, again)
if len(ws) != 1 || len(bs) != 0 || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the restarted controller did not cut the batch at its window: %+v %+v", ws, bs)
}
}
// The window's settings reach the controller in its settings file, read at every look; one that says no
// duration, or no file, is the default.
@@ -220,9 +622,94 @@ func TestAWaitingWalkNamesTheMergesItAnswers(t *testing.T) {
// A merge heard after a later merge of its branch was cut is named by that walk however its modules read since
// the cut, in a repository of one module (review of this change); a walk that never built what it moves names
// nothing, and the merge joins the next batch.
func TestALateMergeOfAOneModuleRepositoryIsNamed(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
c2 := repoMerge("one", "c2", t0.Add(-50*time.Second))
hear(t, open, c2, t0)
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("one", "c1", t0.Add(-60*time.Second)), t0.Add(15*time.Minute))
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the late merge was not named by the walk that carried it: %+v", got.Delivery.Merges)
}
// A related walk waits for the delivery's word too: a waiting walk let go beside a started one it is related to
// (the same repository, issue 973) takes the word and starts once that one ended, asking nothing before.
// A walk of two that built only two: a late merge of one is not its to name.
hear(t, open, repoMerge("two", "d2", t0.Add(16*time.Minute)), t0.Add(16*time.Minute))
cutAt(t, open, t0.Add(18*time.Minute))
ws, _ = walks(t, open)
w = ws[0]
delete(w.Modules, "two")
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
hear(t, open, repoMerge("two", "d1", t0.Add(15*time.Minute)), t0.Add(20*time.Minute))
if got, _ := open.inventory.PlanByID(ctx, w.ID); len(got.Delivery.Merges) != 1 {
t.Fatalf("a walk that never built what the late merge moves named it: %+v", got.Delivery.Merges)
}
if _, bs := walks(t, open); len(bs) != 1 || bs[0].Delivery.Merges[0].Commit != "d1" {
t.Fatalf("the late merge did not join the next batch: %+v", bs)
}
}
// One walk at a time holds against the delivery's word too: a waiting walk let go beside a started one takes the
// word and starts once that one ended, asking nothing before.
func TestAWalkLetGoBesideAStartedOneStartsOnceItEnded(t *testing.T) {
open := windowed(t)
asked := asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(2*time.Minute))
started, _ := walks(t, open)
waiting := inventory.Plan{ID: "plan-waiting", Repository: "novox/two", Branch: "main", Commit: "d1", Created: t0,
State: inventory.PlanBuilding, Tiers: [][]string{{"two"}}, Modules: map[string]*inventory.PlanModule{"two": {}},
Delivery: &inventory.PlanDelivery{Awaits: catalogue.DeliverySeat}}
if err := open.inventory.SavePlan(ctx, &waiting); err != nil {
t.Fatal(err)
}
if _, err := letGo(ctx, open.inventory, waiting.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatalf("the word was refused beside %s: %v", started[0].ID, err)
}
before := len(*asked)
advanceHeld(ctx, open)
got, _ := open.inventory.PlanByID(ctx, waiting.ID)
if len(*asked) != before || !strings.Contains(got.Note, "starts once "+started[0].ID) {
t.Fatalf("a walk let go beside a started one asked (%d → %d) or does not say it waits: %q", before, len(*asked), got.Note)
}
// Read as a wait, an hour on: its note on the line, never LATE, and no tier of it late for S3.
later := time.Now().Add(time.Hour)
if line := planLine(got, later); !strings.Contains(line, "starts once "+started[0].ID) || strings.Contains(line, "LATE") {
t.Fatalf("a deferred walk reads %q", line)
}
facts, _, err := gatherPlans(ctx, open.inventory, later, nil)
if err != nil {
t.Fatal(err)
}
for _, f := range facts {
if f.id == got.ID {
t.Fatalf("a deferred walk is watched as a tier running late: %+v", f)
}
}
done := started[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) != before+1 {
t.Fatalf("the walk did not start once the started one ended: asked %v", *asked)
}
}
// A file a merge of the batch removed and a later one brought back is not removed; one removed last is.
func TestARemovedFileIsTheLastMergesWord(t *testing.T) {
@@ -242,10 +729,153 @@ func TestARemovedFileIsTheLastMergesWord(t *testing.T) {
}
}
// The catch-up hands over what the bus lost after its branch's later merges were cut, and the record keeps it:
// an earlier merge is named by the walk that carried it; a merge made before a cut and heard after it, which
// no later merge carries, joins the next batch — neither reads as history and is dropped (review of this change).
func TestTheCatchUpKeepsWhatACutMadeHistory(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
base := time.Now().UTC().Add(-time.Hour).Truncate(time.Second)
hear(t, open, repoMerge("one", "c2", base.Add(10*time.Second)), base.Add(11*time.Second))
cutAt(t, open, base.Add(2*time.Minute))
ws, _ := walks(t, open)
w := ws[0]
w.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &w); err != nil {
t.Fatal(err)
}
lost := []link.AnnouncedMerge{
{SourceMoved: repoMerge("one", "c1", base), At: base.Add(time.Second)},
{SourceMoved: repoMerge("one", "c3lost", base.Add(20*time.Second)), At: base.Add(21 * time.Second)},
}
catalogued := func(ctx context.Context) ([]inventory.Entry, map[string][]inventory.ReadRepository, error) {
entries, err := open.inventory.Catalogued(ctx)
if err != nil {
return nil, nil, err
}
read, err := readForPlanning(ctx, open.inventory)
return entries, read, err
}
if err := catchUpOnMerges(ctx, time.Now(), unheardMerges{announcedList(lost), open.inventory}, catalogued,
(following{open}).SourceMoved, t.Logf); err != nil {
t.Fatal(err)
}
got, _ := open.inventory.PlanByID(ctx, w.ID)
if len(got.Delivery.Merges) != 2 || !sameMerge(got.Delivery.Merges[0], inventory.PlanMerge{Repository: "novox/one",
Commit: "c1", Carried: "c2"}) {
t.Fatalf("the earlier merge the catch-up handed over is not named by the walk that carried it: %+v",
got.Delivery.Merges)
}
if _, kept, err := open.inventory.MergeOf(ctx, "novox/one", "c3lost"); err != nil || !kept {
t.Fatalf("the merge made before the cut and heard after it was dropped: %v %v", kept, err)
}
}
// A merge on the controller's own path never shares a batch with one that waits for mesh-delivery's word
// (decided during the build, 2026-10-10): the two are batches of their own, cut one after the other; a walk on
// the own path folds no waiting catalogue walk but batches its merges again behind it; the catalogue's walk
// still waits for the word, so no catalogue delivery skips its turn.
func TestAMergeOnTheControllersPathNeverSharesABatch(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksWithPaths(t)
ctx := t.Context()
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-delivery", Version: "1",
Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/mesh-delivery", Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("aaaa0001", "app", t0), t0)
hear(t, open, repoMerge("mesh-controller", "k1", t0.Add(10*time.Second)), t0.Add(10*time.Second))
hear(t, open, catalogueMerge("bbbb0002", "notes", t0.Add(20*time.Second)), t0.Add(20*time.Second))
_, bs := walks(t, open)
var ownBatch, catalogueBatch inventory.Plan
for _, b := range bs {
if b.OwnPath() {
ownBatch = b
} else {
catalogueBatch = b
}
}
if len(bs) != 2 || ownBatch.ID == "" || catalogueBatch.ID == "" || len(catalogueBatch.Delivery.Merges) != 2 ||
len(ownBatch.Delivery.Merges) != 1 {
t.Fatalf("a controller merge and two catalogue merges in one window: %+v", bs)
}
if !strings.Contains(batchWords(ownBatch, t0.Add(30*time.Second)), "own path") {
t.Fatalf("the own-path batch does not say so: %q", batchWords(ownBatch, t0.Add(30*time.Second)))
}
// Both windows closed, the controller's batch is cut first and starts; the catalogue batch queues behind it,
// is cut when the controller's walk ended, and waits for its word with both merges.
cutAt(t, open, t0.Add(2*time.Minute))
ws, bs := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the controller's batch was not cut first into a started walk: %+v", ws)
}
ownWalk := ws[0]
for _, m := range []string{"app", "notes"} {
if _, in := ownWalk.Modules[m]; in {
t.Fatalf("the controller's walk builds %s, a catalogue module: it skipped its turn", m)
}
}
if len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != ownWalk.ID || bs[0].OwnPath() {
t.Fatalf("the catalogue batch does not queue behind the controller's walk: %+v", bs)
}
ownWalk.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &ownWalk); err != nil {
t.Fatal(err)
}
cutAt(t, open, t0.Add(3*time.Minute))
ws, bs = walks(t, open)
if len(bs) != 0 || ws[0].ID != catalogueBatch.ID || !ws[0].Waiting() || len(ws[0].Delivery.Merges) != 2 {
t.Fatalf("the catalogue batch was not cut into a waiting walk once the controller's ended: %+v %+v", ws, bs)
}
for _, m := range []string{"app", "notes"} {
if _, in := ws[0].Modules[m]; !in {
t.Fatalf("the catalogue walk does not build %s: %v", m, ws[0].Modules)
}
}
for _, a := range *asked {
if a[1] == "modules/app" || a[1] == "modules/notes" {
t.Fatalf("a catalogue module was asked without the word: %v", *asked)
}
}
// A catalogue walk waiting for its word when an own-path batch is cut keeps waiting beside the own-path
// walk, is not folded into it, and its word is taken meanwhile: it starts once the own-path walk ended.
catalogueWalk := ws[0]
hear(t, open, repoMerge("mesh-controller", "k2", t0.Add(4*time.Minute)), t0.Add(4*time.Minute))
cutAt(t, open, t0.Add(6*time.Minute))
ws, _ = walks(t, open)
kept, _ := open.inventory.PlanByID(ctx, catalogueWalk.ID)
if !kept.Waiting() || ws[0].CommitOf("novox/mesh-controller") != "k2" || ws[0].Waiting() {
t.Fatalf("the waiting catalogue walk was not kept waiting beside the controller's walk: %s %+v", kept.State, ws)
}
if _, in := ws[0].Modules["app"]; in {
t.Fatalf("the controller's walk folded the waiting catalogue walk in: %v", ws[0].Modules)
}
if _, err := letGo(ctx, open.inventory, catalogueWalk.ID, catalogue.DeliverySeat, "its turn"); err != nil {
t.Fatal(err)
}
before := len(*asked)
advanceHeld(ctx, open)
if len(*asked) != before {
t.Fatalf("the catalogue walk started beside the controller's: asked %v", (*asked)[before:])
}
own2 := ws[0]
own2.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &own2); err != nil {
t.Fatal(err)
}
advanceHeld(ctx, open)
if len(*asked) < before+1 || (*asked)[before][1] != "modules/app" {
t.Fatalf("the catalogue walk did not start once the controller's ended: %v", (*asked)[before:])
}
}
// `plans` names a walk by what it moves (asked by the operator, 2026-10-10): the repository's pull request and
// title, the modules it moves and the machines running them, then the state words; a record naming no pull
@@ -291,3 +921,46 @@ func sameMerge(a, b inventory.PlanMerge) bool {
// A catalogue merge heard after a later merge of its branch was walked without the word (a merge that touched
// the bus too, on the controller's own path) is not answered by that walk while the delivery seat has a holder:
// its delivery would skip its turn. It joins the next catalogue batch.
func TestALateCatalogueMergeIsNotAnsweredByAnOwnPathWalk(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksWithPaths(t)
ctx := t.Context()
for _, m := range []struct {
module string
claims []catalogue.Claim
}{
{"nats", nil},
{"mesh-delivery", []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}},
} {
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: m.module, Version: "1", Claims: m.claims},
inventory.Source{Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/" + m.module, Ref: "main",
BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
}
if _, err := open.inventory.Assign(ctx, "anchor", "mesh-delivery"); err != nil {
t.Fatal(err)
}
mixed := link.SourceMoved{Owner: "novox", Repo: "mesh-catalog", Base: "main", Commit: "m1xed", MergedAt: t0.Format(time.RFC3339Nano),
Paths: []string{"modules/nats/module.json", "modules/app/module.json"}, ModuleDirs: []string{"modules/nats", "modules/app"},
ModuleDirsSaid: true}
hear(t, open, mixed, t0.Add(time.Second))
cutAt(t, open, t0.Add(2*time.Minute))
ws, _ := walks(t, open)
if len(ws) != 1 || ws[0].Waiting() {
t.Fatalf("the mixed merge's walk waited, or was not cut: %+v", ws)
}
done := ws[0]
done.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &done); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("ear1ier", "app", t0.Add(-30*time.Second)), t0.Add(3*time.Minute))
got, _ := open.inventory.PlanByID(ctx, done.ID)
_, bs := walks(t, open)
if len(got.Delivery.Merges) != 1 || len(bs) != 1 || bs[0].OwnPath() || bs[0].Delivery.Merges[0].Commit != "ear1ier" {
t.Fatalf("the late catalogue merge was answered by the own-path walk (%+v) rather than the next catalogue batch (%+v)",
got.Delivery.Merges, bs)
}
}
+2 -48
View File
@@ -3,8 +3,6 @@ package main
import (
"context"
"fmt"
"sort"
"strings"
"time"
"github.com/novox/mesh-controller/internal/conditions"
@@ -19,9 +17,6 @@ import (
const (
kindBatchNotCut = "batch-not-cut"
kindBatchBehindWalk = "batch-behind-walk"
// kindGroupNotMerged is raised for a group whose last member has not merged by the merge window's maximum after
// its first (novox/hq ADR 0418 clause 3).
kindGroupNotMerged = "group-not-merged"
)
// batchFacts is one batch not yet cut, as the watchdogs read it.
@@ -33,11 +28,6 @@ type batchFacts struct {
// tiers.
behind string
walkBound time.Duration
// holds are the groups the batch is held for, by head, with why; created is when the batch was made, the first
// merge of the group heard; atMostWindow is the maximum a window may take (the controller's setting).
holds map[string]string
created time.Time
atMostWindow time.Duration
}
// gatherBatches is every batch not yet cut, with the bound of the walk it waits behind.
@@ -52,10 +42,8 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
}
var out []batchFacts
for _, b := range batches {
_, atMostWindow := mergeWindowOf()
f := batchFacts{id: b.ID, state: b.State, grouped: groupedWords(b), created: b.Created, atMostWindow: atMostWindow}
f := batchFacts{id: b.ID, state: b.State, grouped: groupedWords(b)}
if w := b.Delivery; w != nil && w.Batch != nil {
f.holds = w.Batch.Holds
f.atMost, f.closed, f.behind = w.Batch.AtMost, w.Batch.ClosesAt, w.Batch.Behind
if f.atMost.Before(f.closed) {
f.closed = f.atMost
@@ -79,8 +67,7 @@ func gatherBatches(ctx context.Context, inv *inventory.Inventory, now time.Time)
func watchBatchesNotCut(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, b := range f.batches {
if b.state != inventory.PlanAssembling || b.atMost.IsZero() || f.now.Sub(b.atMost) <= batchLateAfter ||
len(b.holds) > 0 { // a batch held for a group waits on a member, which has its own condition
if b.state != inventory.PlanAssembling || b.atMost.IsZero() || f.now.Sub(b.atMost) <= batchLateAfter {
continue
}
late := f.now.Sub(b.atMost)
@@ -118,36 +105,3 @@ func watchBatchesBehind(f *signalFacts) []conditions.Observation {
}
return out
}
// watchGroupsNotMerged is the condition of a batch held for a group whose last member has not merged by the merge
// window's maximum after the first member merged, naming the member and why (novox/hq ADR 0418 clause 3): nothing of
// the group is delivered while it waits.
func watchGroupsNotMerged(f *signalFacts) []conditions.Observation {
var out []conditions.Observation
for _, b := range f.batches {
if len(b.holds) == 0 || b.created.IsZero() || f.now.Sub(b.created) <= b.atMostWindow {
continue
}
heads := make([]string, 0, len(b.holds))
for h := range b.holds {
heads = append(heads, h)
}
sort.Strings(heads)
var why []string
for _, h := range heads {
why = append(why, b.holds[h])
}
waited := f.now.Sub(b.created)
out = append(out, conditions.Observation{Scope: conditions.ScopePlan, ID: b.id,
Kind: kindGroupNotMerged, Severity: conditions.Warning,
Summary: fmt.Sprintf("%s; its first member merged %s ago, past the merge window's maximum (%s), and nothing "+
"of the group is delivered while it waits (batch %s)", strings.Join(why, "; "), ago(waited), ago(b.atMostWindow), b.id),
Said: "a group waits for a member that has not merged",
Headline: "A change waits for the rest of its group to merge",
Explanation: fmt.Sprintf("The changes of a group are delivered together, once the last one is merged. "+
"One has not merged in %s. Nothing is lost; the merged ones are kept and delivered with it.",
humanDuration(waited)),
Resolved: "The group's last change merged"})
}
return out
}
-108
View File
@@ -1,108 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// An unrelated closed batch is cut into a walk beside an open one, and that walk asks its first tier at once
// (issue 973, novox/hq ADR 0407 decision 4): a hotfix is never held by an unrelated delivery.
func TestAnUnrelatedBatchWalksBesideAnOpenWalk(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
if ws, _ := walks(t, open); len(ws) != 1 || !ws[0].Open() {
t.Fatalf("the first walk is not open: %+v", ws)
}
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("two", "c2", heard)), heard)
advanceHeld(ctx, open)
ws, bs := walks(t, open)
if len(ws) != 2 || len(bs) != 0 {
t.Fatalf("%d walk(s) and %d batch(es); want the unrelated batch cut beside the open walk", len(ws), len(bs))
}
for _, w := range ws {
if !w.Open() || w.Waiting() || deferred(w) || !askedAny(w) {
t.Errorf("%s (%s) is not walking: open %v, deferred %v, asked %v, note %q", w.ID, w.Named(), w.Open(),
deferred(w), askedAny(w), w.Note)
}
}
}
// A closed batch related to an open walk is queued behind it with the reason, while one that shares nothing is
// cut beside it; the walk that started is never superseded (ADR 0276 decision 4, issue 973).
func TestARelatedBatchWaitsBehindTheOpenWalk(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
first := func() inventory.Plan { ws, _ := walks(t, open); return ws[len(ws)-1] }()
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("one", "c2", heard)), heard)
hear(t, open, lone(repoMerge("three", "c3", heard.Add(time.Second))), heard.Add(time.Second))
ws, bs := walks(t, open)
if len(ws) != 2 || len(bs) != 1 {
t.Fatalf("%d walk(s) and %d batch(es); want the unrelated merge walking and the related one queued", len(ws), len(bs))
}
b := bs[0]
words := batchWords(b, heard)
if b.State != inventory.PlanQueued || b.Delivery.Batch.Behind != first.ID || b.CommitOf("novox/one") != "c2" ||
!strings.Contains(words, "queued behind "+first.ID) || !strings.Contains(words, "both change novox/one") {
t.Fatalf("the related batch is %s, behind %q, reading %q", b.State, b.Delivery.Batch.Behind, words)
}
if got, _ := open.inventory.PlanByID(ctx, first.ID); !got.Open() {
t.Fatalf("the started walk was %s by the related merge", got.State)
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(2*time.Second))
if _, bs := walks(t, open); len(bs) != 0 {
t.Fatalf("the queued batch was not cut once the walk it waited for ended: %+v", bs)
}
}
// A batch on the controller's own path never walks beside another walk, and nothing walks beside it, whatever
// the two share (issue 973): it replaces the controller, so it goes alone.
func TestAnOwnPathBatchNeverWalksBeside(t *testing.T) {
open := windowed(t)
asksWithPaths(t)
ctx := t.Context()
hear(t, open, repoMerge("one", "c1", t0), t0)
cutAt(t, open, t0.Add(90*time.Second))
ws, _ := walks(t, open)
first := ws[0]
heard := t0.Add(100 * time.Second)
hear(t, open, lone(repoMerge("mesh-controller", "k1", heard)), heard)
ws, bs := walks(t, open)
if len(ws) != 1 || len(bs) != 1 || !bs[0].OwnPath() || bs[0].State != inventory.PlanQueued ||
bs[0].Delivery.Batch.Behind != first.ID || !strings.Contains(batchWords(bs[0], heard), "own path") {
t.Fatalf("an own-path batch beside an open walk: %d walk(s), batches %+v", len(ws), bs)
}
first.State = inventory.PlanDone
if err := open.inventory.SavePlan(ctx, &first); err != nil {
t.Fatal(err)
}
cutAt(t, open, heard.Add(time.Second))
ws, _ = walks(t, open)
if len(ws) != 2 || !ws[0].Open() || ws[0].CommitOf("novox/mesh-controller") != "k1" {
t.Fatalf("the own-path batch was not cut once the walk ended: %+v", ws)
}
own := ws[0]
// Unrelated to the own-path walk by every footprint, a merge still waits for it.
later := heard.Add(time.Minute)
hear(t, open, lone(repoMerge("two", "c2", later)), later)
ws, bs = walks(t, open)
if len(ws) != 2 || len(bs) != 1 || bs[0].State != inventory.PlanQueued || bs[0].Delivery.Batch.Behind != own.ID {
t.Fatalf("a merge beside the own-path walk: %d walk(s), batches %+v", len(ws), bs)
}
}
+2 -4
View File
@@ -4,7 +4,7 @@ import (
"context"
"fmt"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
@@ -73,8 +73,6 @@ func reportKept(ctx context.Context, plan catalogue.Resolution) {
// sent, on every machine — the resolver kept it there (said, urgent, until a person pins), or a pin
// moves it (said, so the data goes first), and never anything else.
func probeBindings(ctx context.Context, d *doctor) ([]conditions.Observation, error) {
// One reading of the mesh for every machine this looks at.
ctx = withInputCache(ctx)
inv := d.open.inventory
nodes, err := inv.Nodes(ctx)
if err != nil {
@@ -82,7 +80,7 @@ func probeBindings(ctx context.Context, d *doctor) ([]conditions.Observation, er
}
var out []conditions.Observation
for _, n := range nodes {
plan, _, err := resolveNode(ctx, d.open, n.Name)
plan, _, err := planFor(ctx, d.open, n.Name)
if err != nil {
if unresolvable(err) {
// D1 says it, with the binding that refused it when that is why.
@@ -1,102 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
func TestTheIncidentAConsumerStaysBesideItsDataUntilAPersonPinsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
for _, m := range storeManifests() {
register(t, open, m)
}
assignAll := func(pairs ...[2]string) {
for _, a := range pairs {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
}
// The anchor's store and resolver hold the mesh's seats, on record; the laptop runs its own of each.
assignAll([2]string{"anchor", "store"}, [2]string{"anchor", "resolver"})
for _, seat := range [][2]string{{"mesh-store", "store"}, {"mesh-dns-resolver", "resolver"}} {
if err := inv.HoldSeat(ctx, seat[0], catalogue.ScopeMesh, "anchor", seat[1]); err != nil {
t.Fatal(err)
}
}
assignAll([2]string{"laptop", "store"}, [2]string{"laptop", "resolver"}, [2]string{"laptop", "network"},
[2]string{"laptop", "board"})
plan, _, err := resolveNode(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "laptop" || n.Module != "store" || !n.KeepsData {
t.Fatalf("the consumer was bound to %s/%s (keeps data: %v); its data is beside it", n.From, n.Module, n.KeepsData)
}
if n := need(t, plan, "network", "wildcard-resolution"); n.From != "anchor" || n.KeepsData {
t.Fatalf("the resolver was bound to %s (keeps data: %v); its seat is held on anchor (issue 258)", n.From, n.KeepsData)
}
// Sent, and recorded: only the binding to data.
bindings := boundToData(plan, nil)
if len(bindings) != 1 || bindings[0].Provider != (catalogue.Chosen{Node: "laptop", Module: "store"}) {
t.Fatalf("recorded %+v", bindings)
}
if err := inv.RecordBindings(ctx, "laptop", bindings); err != nil {
t.Fatal(err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a mesh bound where it was sent: %+v, %v", found, err)
}
// The laptop's store taken away: refused, not moved to the anchor's empty one.
if err := inv.Unassign(ctx, "laptop", "store"); err != nil {
t.Fatal(err)
}
if _, _, err := resolveNode(ctx, open, "laptop"); err == nil || !unresolvable(err) ||
!strings.Contains(err.Error(), "board on laptop is bound to laptop/store") ||
!strings.Contains(err.Error(), "pin laptop postgres-database anchor store") {
t.Fatalf("the consumer's store went and it was answered elsewhere: %v", err)
}
// A person pins the anchor's: it moves, said before it is sent, and the record keeps where it was.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "anchor", "store"); err != nil {
t.Fatal(err)
}
plan, _, err = resolveNode(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "anchor" {
t.Fatalf("pinned to the anchor and bound to %s", n.From)
}
found, err := probeBindings(ctx, &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || found[0].Kind != kindBindingMoving || found[0].Severity != conditions.Warning ||
!strings.Contains(found[0].Summary, "board's postgres-database moves from laptop/store to anchor/store") {
t.Fatalf("a pinned move is not said before it is sent: %+v", found)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
all, err := inv.Bindings(ctx)
if err != nil || len(all) != 1 || all[0].MovedFrom != "laptop/store" {
t.Fatalf("%+v, %v", all, err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a move sent is still said: %+v, %v", found, err)
}
}
+91 -1
View File
@@ -4,7 +4,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
)
@@ -39,6 +39,96 @@ func need(t *testing.T, plan catalogue.Resolution, consumer, provision string) c
// The incident through the stores: the laptop runs its own store and a consumer of it, the anchor's
// store holds the mesh's seat. The consumer stays beside its data, the resolver follows its seat, the
// binding is recorded as sent, and a pin — only a pin — moves it, said before the push that carries it.
func TestTheIncidentAConsumerStaysBesideItsDataUntilAPersonPinsIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
if _, err := inv.SeedSeats(ctx, catalogue.DefaultSeats()); err != nil {
t.Fatal(err)
}
for _, m := range storeManifests() {
register(t, open, m)
}
assignAll := func(pairs ...[2]string) {
for _, a := range pairs {
if _, err := assign(ctx, open, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
}
// The anchor's store and resolver hold the mesh's seats, on record; the laptop runs its own of each.
assignAll([2]string{"anchor", "store"}, [2]string{"anchor", "resolver"})
for _, seat := range [][2]string{{"mesh-store", "store"}, {"mesh-dns-resolver", "resolver"}} {
if err := inv.HoldSeat(ctx, seat[0], catalogue.ScopeMesh, "anchor", seat[1]); err != nil {
t.Fatal(err)
}
}
assignAll([2]string{"laptop", "store"}, [2]string{"laptop", "resolver"}, [2]string{"laptop", "network"},
[2]string{"laptop", "board"})
plan, _, err := planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "laptop" || n.Module != "store" || !n.KeepsData {
t.Fatalf("the consumer was bound to %s/%s (keeps data: %v); its data is beside it", n.From, n.Module, n.KeepsData)
}
if n := need(t, plan, "network", "wildcard-resolution"); n.From != "anchor" || n.KeepsData {
t.Fatalf("the resolver was bound to %s (keeps data: %v); its seat is held on anchor (issue 258)", n.From, n.KeepsData)
}
// Sent, and recorded: only the binding to data.
bindings := boundToData(plan, nil)
if len(bindings) != 1 || bindings[0].Provider != (catalogue.Chosen{Node: "laptop", Module: "store"}) {
t.Fatalf("recorded %+v", bindings)
}
if err := inv.RecordBindings(ctx, "laptop", bindings); err != nil {
t.Fatal(err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a mesh bound where it was sent: %+v, %v", found, err)
}
// The laptop's store taken away: refused, not moved to the anchor's empty one.
if err := inv.Unassign(ctx, "laptop", "store"); err != nil {
t.Fatal(err)
}
if _, _, err := planFor(ctx, open, "laptop"); err == nil || !unresolvable(err) ||
!strings.Contains(err.Error(), "board on laptop is bound to laptop/store") ||
!strings.Contains(err.Error(), "pin laptop postgres-database anchor store") {
t.Fatalf("the consumer's store went and it was answered elsewhere: %v", err)
}
// A person pins the anchor's: it moves, said before it is sent, and the record keeps where it was.
if err := inv.PinProvision(ctx, "laptop", "postgres-database", "anchor", "store"); err != nil {
t.Fatal(err)
}
plan, _, err = planFor(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
if n := need(t, plan, "board", "postgres-database"); n.From != "anchor" {
t.Fatalf("pinned to the anchor and bound to %s", n.From)
}
found, err := probeBindings(ctx, &doctor{open: open})
if err != nil {
t.Fatal(err)
}
if len(found) != 1 || found[0].Kind != kindBindingMoving || found[0].Severity != conditions.Warning ||
!strings.Contains(found[0].Summary, "board's postgres-database moves from laptop/store to anchor/store") {
t.Fatalf("a pinned move is not said before it is sent: %+v", found)
}
if err := inv.RecordBindings(ctx, "laptop", boundToData(plan, nil)); err != nil {
t.Fatal(err)
}
all, err := inv.Bindings(ctx)
if err != nil || len(all) != 1 || all[0].MovedFrom != "laptop/store" {
t.Fatalf("%+v, %v", all, err)
}
if found, err := probeBindings(ctx, &doctor{open: open}); err != nil || len(found) != 0 {
t.Fatalf("a move sent is still said: %+v, %v", found, err)
}
}
// What one machine's resolution says against its record.
func TestBindingFindingsSayAKeptMoveAndAMoveNothingAskedFor(t *testing.T) {
+31 -137
View File
@@ -7,7 +7,6 @@ import (
"flag"
"fmt"
"github.com/novox/mesh-controller/internal/conditions"
"io"
"os"
"slices"
"strings"
@@ -15,9 +14,8 @@ import (
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -153,9 +151,6 @@ func buildFrom(result link.BuildResult) inventory.Build {
Path: result.Path,
// What it was made from (novox/hq issue 280): two builds with one are one build.
SourceFingerprint: result.SourceFingerprint,
SourceInputs: result.SourceInputs,
// And its reuse key (novox/hq ADR 0350 §1), which the chain names (ADR 0408 §1).
ReuseKey: result.ReuseKey,
}
// When it was asked, which is what orders it against another build of the same module
// (novox/hq 04-ISSUES/219) — not when it was heard.
@@ -187,10 +182,8 @@ func buildFrom(result link.BuildResult) inventory.Build {
kept.Manifest = recordedManifest(result.Manifest, announced)
// The module name comes from the manifest, which only exists when the build got that far.
if len(kept.Manifest) > 0 {
if list, err := judgedWith(result); err == nil {
if m, err := catalogue.ParseManifestWith(kept.Manifest, list); err == nil {
kept.Module = m.Module
}
if m, err := catalogue.ParseManifest(kept.Manifest); err == nil {
kept.Module = m.Module
}
}
return kept
@@ -238,56 +231,35 @@ func buildsCommand(ctx context.Context, args []string) error {
}
for _, b := range builds {
printBuild(os.Stdout, b)
what := b.Module
if what == "" {
// It failed before knowing what it was building, which is most of the interesting
// failures. The repository is what a person has to go and look at.
what = "?"
}
outcome := "built " + short(b.Commit)
if !b.Worked() {
outcome = "failed"
}
fmt.Printf("%-18s %-14s %-10s %s %s\n",
what, outcome, b.On, b.At.Local().Format("2006-01-02 15:04"), b.ID)
fmt.Printf(" %s", b.Repository)
if b.Ref != "" {
fmt.Printf(" at %s", b.Ref)
}
fmt.Println()
for _, made := range b.Made {
fmt.Printf(" %-10s %s\n", made.Kind, made.Reference)
}
if !b.Worked() {
// The builder's own first line. The whole failure is often a build log, and printing
// it here would bury every other row.
fmt.Printf(" %s\n", firstLine(b.Failed))
}
}
return nil
}
// printBuild says one build: what, its outcome, where and when, what it made and published, its reuse key, and its
// source fingerprint (novox/hq issue 1204: the one tool that reads what the builder recorded it was made from).
func printBuild(w io.Writer, b inventory.Build) {
what := b.Module
if what == "" {
// It failed before knowing what it was building, which is most of the interesting
// failures. The repository is what a person has to go and look at.
what = "?"
}
outcome := "built " + short(b.Commit)
if !b.Worked() {
outcome = "failed"
}
fmt.Fprintf(w, "%-18s %-14s %-10s %s %s\n",
what, outcome, b.On, b.At.Local().Format("2006-01-02 15:04"), b.ID)
fmt.Fprintf(w, " %s", b.Repository)
if b.Ref != "" {
fmt.Fprintf(w, " at %s", b.Ref)
}
fmt.Fprintln(w)
for _, made := range b.Made {
fmt.Fprintf(w, " %-10s %s\n", made.Kind, made.Reference)
}
// What it published and under what reuse key (novox/hq ADR 0408 §1).
if p := b.Published; p != nil {
if p.Artifact != "" {
fmt.Fprintf(w, " published %s\n", p.Artifact)
}
if p.Package != nil {
fmt.Fprintf(w, " published %s@%s\n", p.Package.Name, p.Package.Version)
}
}
if b.ReuseKey != "" {
fmt.Fprintf(w, " reuse key %s\n", b.ReuseKey)
}
if b.SourceFingerprint != "" {
fmt.Fprintf(w, " source %s\n", b.SourceFingerprint)
}
if !b.Worked() {
// The builder's own first line. The whole failure is often a build log, and printing
// it here would bury every other row.
fmt.Fprintf(w, " %s\n", firstLine(b.Failed))
}
}
// builderCommand issues a build machine its own broker credential.
//
// **A build machine is not a node**, and giving it a node's account would let it read another
@@ -484,7 +456,7 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
return "", err
}
// Through a verb, only a repository the catalogue builds from (novox/hq ADR 0266).
if err := verbMayAsk(ctx, source, repository, path); err != nil {
if err := verbMayAsk(ctx, source, repository); err != nil {
return "", err
}
@@ -511,7 +483,6 @@ func buildOneAsked(ctx context.Context, source buildSource, path, ref string, wa
Seats: seatBases(ctx),
DryRun: dryRun,
}
request.Reuse = reuseAsked(ctx, dryRun)
fmt.Printf("asked for %s", source)
if source.Seat != "" {
fmt.Printf(" (%s)", repository)
@@ -630,11 +601,6 @@ func saysWhenThePolicyActs(ctx context.Context, inv *inventory.Inventory, module
// written with the same values.
func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResult) (
catalogue.Manifest, inventory.Build, error) {
// The list of predefined keys as the store keeps it now: another controller process may have kept a newer one.
if err := loadPredefined(ctx, inv); err != nil {
return catalogue.Manifest{}, inventory.Build{}, fmt.Errorf("the list of predefined keys kept cannot be "+
"read: %w", err)
}
kept := buildFrom(result)
if err := inv.RecordBuild(ctx, kept); err != nil {
return catalogue.Manifest{}, kept, err
@@ -645,15 +611,7 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return catalogue.Manifest{}, kept, fmt.Errorf("%s could not build %s:\n%s",
result.On, result.Repository, result.Failed)
}
// Judged first against the list the build was judged against: the catalogue's at the commit built (novox/hq
// ADR 0348 §7), so a module naming a key its own commit adds is read; and again below against the list the
// mesh publishes, before it is registered.
list, err := judgedWith(result)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
result.Repository, short(result.Commit), err)
}
manifest, err := catalogue.ParseManifestWith(kept.Manifest, list)
manifest, err := catalogue.ParseManifest(kept.Manifest)
if err != nil {
return catalogue.Manifest{}, kept, fmt.Errorf("%s built %s and what came back is not a manifest: %w",
result.On, result.Repository, err)
@@ -711,12 +669,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %w", result.On,
manifest.Module, short(result.Commit), err)
}
// **Never a build in the scratch namespace** (novox/hq ADR 0239 §9, ADR 0350 §2): what a check builds is
// registered only as the walk's copy in the module's own namespace.
if scratch := inScratch(result.Made); scratch != "" {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: %s is in the scratch "+
"namespace, which only a walk's copy leaves (ADR 0350)", result.On, manifest.Module, short(result.Commit), scratch)
}
// **A build that failed its gate is never registered again** (novox/hq ADR 0236): an outcome heard
// twice, or replayed, would otherwise make the build a rollback put back what the module is again,
// and the next push would send it.
@@ -727,23 +679,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
"back: it is recorded and not registered again — a newer build is", result.On, manifest.Module,
short(result.Commit))
}
// **The newest trunk build's list of predefined keys is published** (novox/hq ADR 0348 §7): a build of the
// catalogue that may be registered keeps its list when its commit is newer than the one kept. Then the module
// is judged against the list the mesh publishes, as every manifest read again is: one naming a key that list
// does not hold, or of another type, is refused here and never enters the catalogue.
if carried, err := carriedList(result); err != nil {
return manifest, kept, err
} else if carried != nil {
if err := publishPredefined(ctx, inv, result, carried); err != nil {
fmt.Printf("%s: the list of predefined keys it carried is not published: %v\n", result.ID, err)
}
}
if manifest.NamesPredefined() {
if _, err := catalogue.ParseManifest(kept.Manifest); err != nil {
return manifest, kept, fmt.Errorf("%s built %s (%s), recorded and not registered: judged against the "+
"list of predefined keys the mesh publishes, %w", result.On, manifest.Module, short(result.Commit), err)
}
}
// **A build whose source is unchanged is never a move** (novox/hq issue 280): a rebuild made from
// what the build the mesh stands on was made from registers that build's artifacts at the new
// commit, so no machine is sent a new digest for a source nobody changed — an image is not
@@ -779,16 +714,6 @@ func takeIn(ctx context.Context, inv *inventory.Inventory, result link.BuildResu
return manifest, kept, nil
}
// inScratch is the first artifact a build made into the scratch namespace; empty when none is.
func inScratch(made []link.MadeArtifact) string {
for _, m := range made {
if path, kept := catalogue.InArtifactStore(catalogue.Recorded(m.Reference)); kept && strings.HasPrefix(path, builder.ScratchPrefix) {
return m.Reference
}
}
return ""
}
// errOffTheTrunk is a build of a commit off its repository's trunk, which is never published.
var errOffTheTrunk = errors.New("the commit is not on its repository's trunk: a commit off the trunk is checked, " +
"never published (ADR 0238) — merge it, and the merge builds it")
@@ -823,7 +748,7 @@ func buildAndShow(ctx context.Context, source buildSource, path, ref string, wai
if err != nil {
return err
}
if err := verbMayAsk(ctx, source, repository, path); err != nil {
if err := verbMayAsk(ctx, source, repository); err != nil {
return err
}
ident, err := openIdentity(ctx)
@@ -877,9 +802,6 @@ type answers struct {
sources map[string]inventory.Source
// waiting is every machine not running what the mesh would send it.
waiting []inventory.Machine
// recordedWaits is, per machine, every move of a recorded module that waits there for a person (novox/hq ADR
// 0377 §6): the machine is behind for these, and `push <node>` sends them.
recordedWaits map[string][]string
// reported is every machine's last word beside when it was last sent a declaration — the
// pair that answers "has it caught up", which waiting alone cannot (the sent digest is
// recorded at send, not at apply).
@@ -959,34 +881,6 @@ type answers struct {
//
// Routed through the artifact store as the network reaches it now (novox/hq 04-ISSUES/102): a
// base is recorded by digest and path, and a build machine needs something it can pull.
// runningBy is, keyed "<module>/<artifact>", each artifact of the build the mesh has registered for its module, as a
// builder reaches the artifact store (novox/hq issue 721): what a check compares its builds with. Empty when it cannot
// be read, said, and then nothing is said unchanged.
func runningBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
defer open.Close()
running, err := open.inventory.RegisteredHeld(ctx)
if err != nil {
fmt.Fprintf(os.Stderr, "could not read what this mesh runs, so the check compares nothing: %v\n", err)
// empty-on-error: said above; a check given nothing to compare with says no module unchanged, and all move
return nil
}
address, err := whereABuilderReachesTheStore(ctx, open.inventory)
if err != nil || address == "" {
return running
}
routed := make(map[string]string, len(running))
for key, reference := range running {
routed[key] = catalogue.Rerouted(reference, address)
}
return routed
}
func heldBy(ctx context.Context) map[string]string {
open, err := openStores(ctx)
if err != nil {
@@ -1,131 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func TestABuildHeardIsRecordedAndRegistered(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "shop", "version": "3"})
m, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, link.BuildResult{
ID: "b-1", Repository: "http://forge.internal:20000/novox/shop.git", Path: "modules/shop",
Ref: "main", On: "anchor", Commit: "abcdef0123", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/shop"},
}))
if err != nil {
t.Fatal(err)
}
if m.Module != "shop" {
t.Fatalf("registered %q", m.Module)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if _, held := shelf["shop"]; !held {
t.Fatal("the module a heard build produced is not in the catalogue")
}
src, err := open.inventory.SourceOf(ctx, "shop")
if err != nil || src.Seat != "git" || src.Repository != "novox/shop" || src.BuiltFrom != "abcdef0123" {
t.Fatalf("the source is the seat and the path, never the URL: %+v %v", src, err)
}
builds, err := open.inventory.Builds(ctx, "shop", 5)
if err != nil || len(builds) != 1 || builds[0].ID != "b-1" {
t.Fatalf("the build is not recorded once: %v %v", builds, err)
}
named, _ := json.Marshal(map[string]any{"module": "idp", "version": "1", "resources": []any{
map[string]any{"id": "server", "type": "container", "image": "x@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}}}})
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{
ID: "b-2", Repository: "/r", On: "anchor", Commit: "0123456789", Manifest: named})
if err == nil || !strings.Contains(err.Error(), "does not register it") {
t.Fatalf("a definition naming an installation was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["idp"].Module != "" {
t.Fatal("the refused module was registered anyway")
}
if builds, _ := open.inventory.Builds(ctx, "idp", 5); len(builds) != 1 {
t.Fatalf("the refused build was not recorded: %v", builds)
}
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{ID: "b-3", Repository: "/r", On: "anchor", Failed: "no compiler"})
if err == nil || !strings.Contains(err.Error(), "no compiler") {
t.Fatalf("a failure is said in the builder's words: %v", err)
}
}
func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "unifi", "version": "1"})
result := func(id, ref, commit string) link.BuildResult {
return link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/unifi", Ref: ref, On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result("b-1", "main", "1111111aaaa"))); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, result("b-2", "9c97a8a", "9c97a8a1d2c3")); err != nil {
t.Fatal(err)
}
src, err := open.inventory.SourceOf(ctx, "unifi")
if err != nil {
t.Fatal(err)
}
if src.Ref != "main" || src.BuiltFrom != "9c97a8a1d2c3" {
t.Errorf("after a build at a commit the module follows %q, built from %q; want main, 9c97a8a1d2c3", src.Ref, src.BuiltFrom)
}
// One new to the catalogue, first built at a commit, follows the default branch.
other, _ := json.Marshal(map[string]any{"module": "letta", "version": "1"})
r := result("b-3", "deadbeef", "deadbeefcafe")
r.Manifest, r.Path = other, "modules/letta"
if _, _, err := takeIn(ctx, open.inventory, r); err != nil {
t.Fatal(err)
}
if src, _ := open.inventory.SourceOf(ctx, "letta"); src.Ref != "" {
t.Errorf("a module first built at a commit follows %q, want the default branch", src.Ref)
}
}
func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
result := func(asked time.Time, image string) link.BuildResult {
manifest, _ := json.Marshal(map[string]any{"module": "postgres", "version": image})
return link.BuildResult{ID: link.NewBuildID(asked), Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result(newer, "4bcd5f73"))); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
if !errors.Is(err, inventory.ErrSuperseded) {
t.Fatalf("the older request's outcome was taken in as current: %v", err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "4bcd5f73" {
t.Errorf("postgres is %q; want the newer request's 4bcd5f73", got)
}
if builds, _ := open.inventory.Builds(ctx, "postgres", 5); len(builds) != 2 {
t.Errorf("the late build was not recorded: %v", builds)
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ package main
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
+8 -91
View File
@@ -10,7 +10,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -88,7 +88,7 @@ func newSourceForms(ctx context.Context, inv *inventory.Inventory) *sourceForms
if world == nil {
w := catalogue.World{}
if shelf, err := inv.Catalogue(ctx); err == nil {
if read, err := restOfTheMeshOf(ctx, inv, shelf, ""); err == nil {
if read, err := theRestOfTheMesh(ctx, inv, shelf, ""); err == nil {
w = read
}
}
@@ -389,31 +389,10 @@ func (f *sourceForms) onTheForge(repository, seat string) (owner, name string, o
// least one required status, no administrator merging past one.
//
// Anything else only when the build request was kept as asked at the controller's terminal, for this very
// repository and path; or, for the build of a merge, when mesh-delivery announced it on the operator's Approve of
// that head (novox/hq ADR 0420 §7 for a moved module, ADR 0432 for a new one), the forge's checks still holding. path is the module's directory as built; branch the trunk it is registered from.
// repository and path. path is the module's directory as built; branch the trunk it is registered from.
func mayRegisterFrom(ctx context.Context, inv *inventory.Inventory, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
return mayRegisterIn(ctx, inv, newSourceForms(ctx, inv), module, built, buildID, path, branch)
}
// catalogueReader is the catalogue's manifests, as the router's records are found in them.
type catalogueReader interface {
Catalogue(ctx context.Context) (map[string]catalogue.Manifest, error)
}
// registrations is what a registration's decision reads of the mesh: *inventory.Inventory, or a test's stand-in.
type registrations interface {
catalogueReader
SourceOf(ctx context.Context, module string) (inventory.Source, error)
SourceIdentity(ctx context.Context, module string) (int64, error)
Catalogued(ctx context.Context) ([]inventory.Entry, error)
BuildRequestByID(ctx context.Context, id string) (inventory.BuildRequest, bool, error)
MergeOf(ctx context.Context, repository, commit string) (inventory.BatchedMerge, bool, error)
}
// mayRegisterIn is mayRegisterFrom on what it reads of the mesh handed in.
func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, module string, built inventory.Source,
buildID, path, branch string) (int64, error) {
forms := newSourceForms(ctx, inv)
was, err := inv.SourceOf(ctx, module)
isNew := errors.Is(err, inventory.ErrNoSuchModule)
if err != nil && !isNew {
@@ -425,8 +404,7 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
}
refuse := func(why string) (int64, error) {
return 0, fmt.Errorf("%w: %s. A module is registered from such a source only by a build asked at the "+
"controller's terminal, or by the build of a merge mesh-delivery announced on the operator's Approve of that "+
"head, never through a verb: a registered module is what the next push sends, and whoever "+
"controller's terminal, never through a verb: a registered module is what the next push sends, and whoever "+
"may call a verb includes agents (novox/hq ADR 0266)", errNotItsSource, why)
}
@@ -457,23 +435,6 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
module, sourceWords(built.Repository, built.Seat))
}
}
// **Or the move the operator approved** (novox/hq ADR 0420 §7, issue 1025): a known module built from another
// repository by a merge, whose pull request mesh-delivery merged on the operator's Approve of that head.
// A verb alone, or a merge with no such warrant, is refused as before.
// **Or a new module from a repository nothing registered is built from, on the same warrant** (novox/hq ADR 0432,
// issue 1146): the build of a merge mesh-delivery announced on the operator's Approve of that head registers it.
// The forge's protection and the repository's identity are checked below as for every registration.
movedWhy := ""
if why != "" && !terminal {
accepted, err := approvedMove(ctx, inv, forms, buildID, built)
switch {
case err == nil:
movedWhy, why = accepted, ""
case !errors.Is(err, errNotAMerge):
fmt.Printf("%s: %s is built from another repository, and no approved move covers this build: %v\n",
buildID, module, err)
}
}
if why != "" && !terminal {
return refuse(why)
}
@@ -510,8 +471,7 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
}
// The same repository by the forge's id, not only its name: one deleted and made again is another.
recorded := map[string]int64{}
// A moved module is registered from the repository the operator approved, not the one it was registered from.
if !isNew && movedWhy == "" {
if !isNew {
id, err := inv.SourceIdentity(ctx, module)
if err != nil {
return 0, err
@@ -532,19 +492,12 @@ func mayRegisterIn(ctx context.Context, inv registrations, forms *sourceForms, m
owner, name, m, facts.ID, m, id))
}
}
if movedWhy != "" && isNew {
fmt.Printf("%s: %s is new, from %s/%s — registered at the operator's Approve: %s\n", buildID, module, owner, name,
movedWhy)
} else if movedWhy != "" {
fmt.Printf("%s: %s moved from %s to %s/%s — registered at the operator's Approve: %s\n", buildID, module,
sourceWords(was.Repository, was.Seat), owner, name, movedWhy)
}
return facts.ID, nil
}
// askedHereFor says the build was asked at the controller's terminal, for this repository and this path: a kept
// request marked so, whose source is the outcome's (novox/hq ADR 0266).
func askedHereFor(ctx context.Context, inv registrations, forms *sourceForms, buildID string,
func askedHereFor(ctx context.Context, inv *inventory.Inventory, forms *sourceForms, buildID string,
built inventory.Source, path string) (bool, error) {
r, found, err := inv.BuildRequestByID(ctx, buildID)
if err != nil || !found || !r.AtTerminal {
@@ -567,49 +520,13 @@ func sourceWords(repository, seat string) string {
return buildSource{Repository: repository, Seat: seat}.String()
}
// firstBuildWarrant is what the controller checked before it asks the first build of the modules a merge adds to a
// repository nothing is built from: the merge was made on the operator's Approve of its head (approvedAnnounced). It
// travels in the context of that ask alone, in this process, so no verb carries one.
type firstBuildWarrant struct {
repository string // owner/name on the git seat
paths map[string]bool
why string
}
type firstBuildWarrantKey struct{}
// withFirstBuildWarrant is ctx carrying the warrant for asking the first builds of paths in repository.
func withFirstBuildWarrant(ctx context.Context, repository string, paths []string, why string) context.Context {
w := &firstBuildWarrant{repository: repository, paths: map[string]bool{}, why: why}
for _, p := range paths {
w.paths[strings.Trim(p, "/")] = true
}
return context.WithValue(ctx, firstBuildWarrantKey{}, w)
}
func firstBuildWarrantIn(ctx context.Context) *firstBuildWarrant {
w, _ := ctx.Value(firstBuildWarrantKey{}).(*firstBuildWarrant)
return w
}
// covers says the warrant is for exactly this ask: the repository on the git seat, and a module path the merge holds.
func (w *firstBuildWarrant) covers(source buildSource, path string) bool {
return w != nil && source.Seat == gitSeat && strings.EqualFold(strings.Trim(source.Repository, "/"), w.repository) &&
w.paths[strings.Trim(path, "/")]
}
// verbMayAsk refuses, for a call through a verb, a build of a repository the catalogue builds no module from
// (novox/hq ADR 0266): the build node would run what an agent wrote, and its outcome could never be
// registered anyway. url is the repository as it is cloned. At the terminal anything may be asked.
func verbMayAsk(ctx context.Context, source buildSource, url, path string) error {
func verbMayAsk(ctx context.Context, source buildSource, url string) error {
if startedAtTheTerminal() {
return nil
}
// **Or the first build of a module a merge adds on the operator's Approve** (novox/hq ADR 0432, issue 1146): the
// controller's own ask, carrying the warrant it checked, for that repository and the module paths the merge holds.
if firstBuildWarrantIn(ctx).covers(source, path) {
return nil
}
open, err := openStores(ctx)
if err != nil {
return err
@@ -1,353 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"os"
"os/exec"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func theCatalogue(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
for _, b := range []link.BuildResult{
onTrunk("build-sudo", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "1"}),
onTrunk("build-host", "novox/mesh-host", "git", "", map[string]any{"module": "mesh-host", "version": "1"}),
} {
keptAsked(t, open.inventory, b.ID, b.Source.Repository, b.Path, true)
if _, _, err := takeIn(ctx, open.inventory, b); err != nil {
t.Fatal(err)
}
}
return open
}
func TestABuildFromAnAgentsRepositoryIsNotRegisteredAsAModuleTheMeshHolds(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
for _, c := range []struct {
name, repository, path, module string
}{
{"its own repository naming sudo", "agent/sudo", "modules/sudo", "sudo"},
{"a fork of the catalogue", "agent/mesh-catalog", "modules/sudo", "sudo"},
{"a fork of the node-engine", "agent/mesh-host", "", "mesh-host"},
} {
t.Run(c.name, func(t *testing.T) {
id := "build-" + strings.ReplaceAll(c.repository, "/", "-")
keptAsked(t, open.inventory, id, c.repository, c.path, false) // through the build verb
evil := onTrunk(id, c.repository, "git", c.path, map[string]any{"module": c.module, "version": "evil"})
_, _, err := takeIn(ctx, open.inventory, evil)
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a build of %s was taken in as %s: %v", c.repository, c.module, err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf[c.module].Version; got != "1" {
t.Fatalf("%s is now %q, from %s", c.module, got, c.repository)
}
if _, found, _ := open.inventory.BuildByID(ctx, id); !found {
t.Errorf("the refused build %s is not recorded", id)
}
})
}
}
func TestANewModuleFromARepositoryTheCatalogueDoesNotBuildFromIsNotRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-new", "agent/tools", "", false)
_, _, err := takeIn(ctx, open.inventory, onTrunk("build-new", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from an agent's repository was taken in: %v", err)
}
// And one asked of nobody here — an outcome on the bus no request was kept for — the same.
_, _, err = takeIn(ctx, open.inventory, onTrunk("build-unasked", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome nobody asked for was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["agent-tools"].Module != "" {
t.Fatal("the refused module is in the catalogue")
}
}
func TestAtTheTerminalAnotherRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-moved", "novox/sudo", "", true)
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-moved", "novox/sudo", "git", "",
map[string]any{"module": "sudo", "version": "2"})); err != nil {
t.Fatalf("a move the operator asked for at the terminal was refused: %v", err)
}
if src, _ := open.inventory.SourceOf(ctx, "sudo"); src.Repository != "novox/sudo" {
t.Fatalf("sudo is built from %q", src.Repository)
}
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-external",
Repository: "http://forge.internal:20000/someone/app.git", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-external", "someone/app", "", "",
map[string]any{"module": "app", "version": "1"})); err != nil {
t.Fatalf("a new module the operator asked for at the terminal was refused: %v", err)
}
}
func TestADeliveryFromTheRegisteredRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
rebuilt := onTrunk("build-plan", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: rebuilt.ID,
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/sudo", For: "plan"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, rebuilt); err != nil {
t.Fatalf("a plan's build of the module's own repository was refused: %v", err)
}
added := onTrunk("build-merge", "novox/mesh-catalog", "git", "modules/zram", map[string]any{"module": "zram", "version": "1"})
if _, _, err := takeIn(ctx, open.inventory, added); err != nil {
t.Fatalf("a module a merge added to the catalogue repository was refused: %v", err)
}
shelf, _ := open.inventory.Catalogue(ctx)
if shelf["sudo"].Version != "2" || shelf["zram"].Module == "" {
t.Fatalf("not registered: sudo %q, zram %q", shelf["sudo"].Version, shelf["zram"].Module)
}
}
func TestAVerbAsksNoBuildOfARepositoryTheCatalogueDoesNotBuildFrom(t *testing.T) {
theCatalogue(t)
ctx := t.Context()
t.Setenv(verbVar, "build")
t.Setenv(link.CallerVar, "node-tools.anchor, through the mesh-controller seat")
err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "http://forge.internal:20000/agent/sudo.git", "")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("a verb's build of an agent's repository was asked: %v", err)
}
if err := verbMayAsk(ctx, buildSource{Repository: "novox/mesh-catalog", Seat: "git"},
"http://forge.internal:20000/novox/mesh-catalog.git", ""); err != nil {
t.Fatalf("a verb's build of the catalogue repository was refused: %v", err)
}
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
if err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "", ""); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
func TestARollbackNeverPutsBackABuildFromAnotherRepository(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
fork := onTrunk("build-1791500000000000000", "agent/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "evil"})
fork.Commit = "c0ffee0123456789" // the commit sudo was registered at
keptAsked(t, inv, fork.ID, "agent/mesh-catalog", "modules/sudo", false)
if _, _, err := takeIn(ctx, inv, fork); !errors.Is(err, errNotItsSource) {
t.Fatalf("the fork's build was taken in: %v", err)
}
// Asked after the registered build, whenever the test runs: an id naming a fixed moment read as older
// than the registered build once the clock passed it (2026-10-10 02:40 UTC), and the test failed on main.
failed := onTrunk(fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano()), "novox/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "2"})
failed.Commit = "badbadbad0123456"
if _, _, err := takeIn(ctx, inv, failed); err != nil {
t.Fatal(err)
}
record, _, err := inv.BuildByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
previous, found, err := inv.PreviousBuild(ctx, "sudo", "c0ffee0123456789", record)
if err != nil {
t.Fatal(err)
}
if found && previous.ID == fork.ID {
t.Fatalf("a rollback would put back the fork's build %s", previous.ID)
}
if !found || previous.ID != "build-sudo" {
t.Fatalf("a rollback puts back %q (found %v), want the registered build-sudo", previous.ID, found)
}
}
func TestATrunkTheForgeDoesNotProtectRegistersNothing(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/unguarded", forgeFacts{ID: 7, Why: "the rule main lets a person push to main directly"})
t.Cleanup(func() { theForge.Delete("novox/unguarded") })
first := onTrunk("build-unguarded-1", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/unguarded", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatalf("at the terminal: %v", err)
}
again := onTrunk("build-unguarded-2", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "2"})
if _, _, err := takeIn(ctx, open.inventory, again); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "push to main directly") {
t.Fatalf("a rebuild from an unprotected trunk was taken in: %v", err)
}
theForge.Store("novox/unguarded", errors.New("nothing serves gitea.gitea_api"))
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-unguarded-3", "novox/unguarded", "git", "",
map[string]any{"module": "unguarded", "version": "3"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a forge that could not say was read as a protected trunk: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["unguarded"].Version != "1" {
t.Fatalf("unguarded is %q", shelf["unguarded"].Version)
}
}
func TestARepositoryMadeAgainUnderItsNameIsNotTheModulesSource(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/remade", forgeFacts{ID: 100, Guarded: true})
t.Cleanup(func() { theForge.Delete("novox/remade") })
first := onTrunk("build-remade-1", "novox/remade", "git", "", map[string]any{"module": "remade", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/remade", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "remade"); id != 100 {
t.Fatalf("the forge's id was not recorded: %d", id)
}
theForge.Store("novox/remade", forgeFacts{ID: 101, Guarded: true}) // deleted, and made again by an agent
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-2", "novox/remade", "git", "",
map[string]any{"module": "remade", "version": "evil"})); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "made again") {
t.Fatalf("a repository made again under the name was taken in: %v", err)
}
// And a new module from it, beside the one registered from the first, the same.
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-3", "novox/remade", "git", "modules/other",
map[string]any{"module": "other", "version": "1"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from a repository made again was taken in: %v", err)
}
}
func TestATerminalRequestCoversOnlyWhatItAsked(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-asked", Repository: "novox/app",
Seat: "git", Path: "modules/app", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
other := onTrunk("build-asked", "agent/sudo", "git", "modules/app", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, other); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another repository under a terminal request's id was taken in: %v", err)
}
elsewhere := onTrunk("build-asked", "novox/app", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, elsewhere); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another path under a terminal request's id was taken in: %v", err)
}
}
func TestTheServingControllerIsNeverTheTerminal(t *testing.T) {
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
t.Setenv(servedVar, "")
if !startedAtTheTerminal() {
t.Fatal("a process started by hand is not the terminal")
}
markServed()
if startedAtTheTerminal() {
t.Fatal("the serving controller reads as the terminal")
}
child := exec.Command(os.Args[0], "-test.run=^$")
child.Env = os.Environ()
if !slices.Contains(child.Env, servedVar+"=1") {
t.Fatal("what the serving controller starts does not carry its mark")
}
}
func TestAnUnknownIdentityKeepsTheRecordedOne(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 100); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 0); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "sudo"); id != 100 {
t.Fatalf("the recorded id became %d", id)
}
}
func TestAModuleMovesOnTheOperatorsApproveOfTheMergeThatMovedIt(t *testing.T) {
const commit = "c0ffee0123456789"
for _, c := range []struct {
what string
change func(*anApprovedMerge)
unkept bool
request string // who asked: "merge", or "build" through a verb
moved bool
}{
{what: "the operator's Approve of the merged head", request: "merge", moved: true},
{what: "no merge mesh-delivery announced", unkept: true, request: "merge"},
{what: "a warrant for other heads", request: "merge", change: func(a *anApprovedMerge) {
a.merge.Heads = map[string]string{"novox/mesh-host@aaaaaaaaaaaa": "e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5e5"}
}},
{what: "a Decline", request: "merge", change: func(a *anApprovedMerge) {
a.warrant.Option, a.warrant.Label, a.merge.Option = "decline", "Decline", "decline"
}},
{what: "a person who is not the operator", request: "merge", change: func(a *anApprovedMerge) { a.warrant.By.Who = "agent" }},
{what: "a build asked through a verb", request: "build"},
} {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
a := approvedMerge(t)
a.merge.Owner, a.merge.Repo, a.merge.Number, a.merge.HeadSHA = "novox", "mesh-host", 7, "a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1a1"
if c.change != nil {
c.change(&a)
}
wasKept, wasAsk := keptDeliveryMerge, routerAskOf
keptDeliveryMerge = func(_ context.Context, owner, repo string, number int, head string) (deliveryMerge, bool, error) {
return a.merge, !c.unkept && owner == "novox" && repo == "mesh-host" && number == 7 && head == a.merge.HeadSHA, nil
}
routerAskOf = func(_ context.Context, _ catalogueReader, _, id string) (*asks.Ask, *asks.Warrant, error) {
if id != a.merge.AskID {
t.Errorf("%s: the router was asked for %q", c.what, id)
}
return &a.ask, &a.warrant, nil
}
t.Cleanup(func() { keptDeliveryMerge, routerAskOf = wasKept, wasAsk })
event, _ := json.Marshal(map[string]any{"owner": "novox", "repo": "mesh-host", "base": "main", "number": 7,
"head_sha": a.merge.HeadSHA, "merge_commit_sha": commit})
if _, err := inv.AddMerge(ctx, inventory.BatchedMerge{Repository: "novox/mesh-host", Branch: "main", Commit: commit,
Merged: time.Now(), Heard: time.Now(), Event: event}); err != nil {
t.Fatal(err)
}
id := fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano())
if err := inv.RecordBuildRequest(ctx, inventory.BuildRequest{ID: id, Repository: "novox/mesh-host", Seat: "git",
Path: "modules/sudo", Ref: "main", Commit: commit, For: c.request}); err != nil {
t.Fatal(err)
}
moved := onTrunk(id, "novox/mesh-host", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
moved.Commit = commit
_, _, err := takeIn(ctx, inv, moved)
src, _ := inv.SourceOf(ctx, "sudo")
switch {
case c.moved && err != nil:
t.Errorf("%s: the move was refused: %v", c.what, err)
case c.moved && src.Repository != "novox/mesh-host":
t.Errorf("%s: sudo is built from %q", c.what, src.Repository)
case !c.moved && !errors.Is(err, errNotItsSource):
t.Errorf("%s: the move was not refused as ADR 0266 has it: %v", c.what, err)
case !c.moved && src.Repository != "novox/mesh-catalog":
t.Errorf("%s: a refused move changed sudo's source to %q", c.what, src.Repository)
}
}
}
+262 -6
View File
@@ -3,10 +3,16 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"hash/fnv"
"os"
"os/exec"
"slices"
"strings"
"sync"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -39,14 +45,141 @@ func keptAsked(t *testing.T, inv *inventory.Inventory, id, repository, path stri
}
// theCatalogue is a mesh whose sudo is built from the catalogue repository and whose node-engine from its own.
func theCatalogue(t *testing.T) *stores {
t.Helper()
open := aMesh(t)
ctx := t.Context()
for _, b := range []link.BuildResult{
onTrunk("build-sudo", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "1"}),
onTrunk("build-host", "novox/mesh-host", "git", "", map[string]any{"module": "mesh-host", "version": "1"}),
} {
keptAsked(t, open.inventory, b.ID, b.Source.Repository, b.Path, true)
if _, _, err := takeIn(ctx, open.inventory, b); err != nil {
t.Fatal(err)
}
}
return open
}
func TestABuildFromAnAgentsRepositoryIsNotRegisteredAsAModuleTheMeshHolds(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
for _, c := range []struct {
name, repository, path, module string
}{
{"its own repository naming sudo", "agent/sudo", "modules/sudo", "sudo"},
{"a fork of the catalogue", "agent/mesh-catalog", "modules/sudo", "sudo"},
{"a fork of the node-engine", "agent/mesh-host", "", "mesh-host"},
} {
t.Run(c.name, func(t *testing.T) {
id := "build-" + strings.ReplaceAll(c.repository, "/", "-")
keptAsked(t, open.inventory, id, c.repository, c.path, false) // through the build verb
evil := onTrunk(id, c.repository, "git", c.path, map[string]any{"module": c.module, "version": "evil"})
_, _, err := takeIn(ctx, open.inventory, evil)
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a build of %s was taken in as %s: %v", c.repository, c.module, err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf[c.module].Version; got != "1" {
t.Fatalf("%s is now %q, from %s", c.module, got, c.repository)
}
if _, found, _ := open.inventory.BuildByID(ctx, id); !found {
t.Errorf("the refused build %s is not recorded", id)
}
})
}
}
func TestANewModuleFromARepositoryTheCatalogueDoesNotBuildFromIsNotRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-new", "agent/tools", "", false)
_, _, err := takeIn(ctx, open.inventory, onTrunk("build-new", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from an agent's repository was taken in: %v", err)
}
// And one asked of nobody here — an outcome on the bus no request was kept for — the same.
_, _, err = takeIn(ctx, open.inventory, onTrunk("build-unasked", "agent/tools", "git", "",
map[string]any{"module": "agent-tools", "version": "1"}))
if !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome nobody asked for was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["agent-tools"].Module != "" {
t.Fatal("the refused module is in the catalogue")
}
}
// The operator at the terminal may still move a module, or add one from a new repository.
func TestAtTheTerminalAnotherRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
keptAsked(t, open.inventory, "build-moved", "novox/sudo", "", true)
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-moved", "novox/sudo", "git", "",
map[string]any{"module": "sudo", "version": "2"})); err != nil {
t.Fatalf("a move the operator asked for at the terminal was refused: %v", err)
}
if src, _ := open.inventory.SourceOf(ctx, "sudo"); src.Repository != "novox/sudo" {
t.Fatalf("sudo is built from %q", src.Repository)
}
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-external",
Repository: "http://forge.internal:20000/someone/app.git", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-external", "someone/app", "", "",
map[string]any{"module": "app", "version": "1"})); err != nil {
t.Fatalf("a new module the operator asked for at the terminal was refused: %v", err)
}
}
// The delivery's flow is untouched: a merge's rebuild of a module from its own repository, and a merge adding
// a module to a repository the catalogue builds from (novox/hq issue 300), are registered with no terminal.
func TestADeliveryFromTheRegisteredRepositoryIsRegistered(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
rebuilt := onTrunk("build-plan", "novox/mesh-catalog", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "2"})
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: rebuilt.ID,
Repository: "novox/mesh-catalog", Seat: "git", Path: "modules/sudo", For: "plan"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, rebuilt); err != nil {
t.Fatalf("a plan's build of the module's own repository was refused: %v", err)
}
added := onTrunk("build-merge", "novox/mesh-catalog", "git", "modules/zram", map[string]any{"module": "zram", "version": "1"})
if _, _, err := takeIn(ctx, open.inventory, added); err != nil {
t.Fatalf("a module a merge added to the catalogue repository was refused: %v", err)
}
shelf, _ := open.inventory.Catalogue(ctx)
if shelf["sudo"].Version != "2" || shelf["zram"].Module == "" {
t.Fatalf("not registered: sudo %q, zram %q", shelf["sudo"].Version, shelf["zram"].Module)
}
}
// Through a verb, a build of a repository the catalogue builds nothing from is not even asked: the build node
// would run what the agent wrote.
func TestAVerbAsksNoBuildOfARepositoryTheCatalogueDoesNotBuildFrom(t *testing.T) {
theCatalogue(t)
ctx := t.Context()
t.Setenv(verbVar, "build")
t.Setenv(link.CallerVar, "node-tools.anchor, through the mesh-controller seat")
err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, "http://forge.internal:20000/agent/sudo.git")
var policy *heldAtTheTerminal
if !errors.As(err, &policy) {
t.Fatalf("a verb's build of an agent's repository was asked: %v", err)
}
if err := verbMayAsk(ctx, buildSource{Repository: "novox/mesh-catalog", Seat: "git"},
"http://forge.internal:20000/novox/mesh-catalog.git"); err != nil {
t.Fatalf("a verb's build of the catalogue repository was refused: %v", err)
}
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
if err := verbMayAsk(ctx, buildSource{Repository: "agent/sudo", Seat: "git"}, ""); err != nil {
t.Fatalf("the terminal was refused: %v", err)
}
}
// asTheOperator keeps a build as asked at the controller's terminal, as the operator's first build of a module
// from a repository the catalogue does not yet build from is (novox/hq ADR 0266), and hands it back.
@@ -66,6 +199,40 @@ func asTheOperator(t *testing.T, inv *inventory.Inventory, b link.BuildResult) l
// A rollback puts back only a build of the module's own repository: an agent's build of the module's name,
// recorded and refused, at the very commit the machine ran before (a fork carries it), is never registered by
// the back door of a failed gate.
func TestARollbackNeverPutsBackABuildFromAnotherRepository(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
inv := open.inventory
fork := onTrunk("build-1791500000000000000", "agent/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "evil"})
fork.Commit = "c0ffee0123456789" // the commit sudo was registered at
keptAsked(t, inv, fork.ID, "agent/mesh-catalog", "modules/sudo", false)
if _, _, err := takeIn(ctx, inv, fork); !errors.Is(err, errNotItsSource) {
t.Fatalf("the fork's build was taken in: %v", err)
}
// Asked after the registered build, whenever the test runs: an id naming a fixed moment read as older
// than the registered build once the clock passed it (2026-10-10 02:40 UTC), and the test failed on main.
failed := onTrunk(fmt.Sprintf("build-%d", time.Now().Add(time.Hour).UnixNano()), "novox/mesh-catalog", "git", "modules/sudo",
map[string]any{"module": "sudo", "version": "2"})
failed.Commit = "badbadbad0123456"
if _, _, err := takeIn(ctx, inv, failed); err != nil {
t.Fatal(err)
}
record, _, err := inv.BuildByID(ctx, failed.ID)
if err != nil {
t.Fatal(err)
}
previous, found, err := inv.PreviousBuild(ctx, "sudo", "c0ffee0123456789", record)
if err != nil {
t.Fatal(err)
}
if found && previous.ID == fork.ID {
t.Fatalf("a rollback would put back the fork's build %s", previous.ID)
}
if !found || previous.ID != "build-sudo" {
t.Fatalf("a rollback puts back %q (found %v), want the registered build-sudo", previous.ID, found)
}
}
// theForge is what the forge says in a test, by owner/name: a repository not named here is protected as a
// trunk must be, with an id of its own.
@@ -89,15 +256,97 @@ func init() {
// A module's trunk the forge does not protect — direct pushes, no required status — or a forge that cannot say,
// registers nothing through a verb: the trunk rule means nothing on a branch anyone pushes to.
func TestATrunkTheForgeDoesNotProtectRegistersNothing(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/unguarded", forgeFacts{ID: 7, Why: "the rule main lets a person push to main directly"})
t.Cleanup(func() { theForge.Delete("novox/unguarded") })
first := onTrunk("build-unguarded-1", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/unguarded", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatalf("at the terminal: %v", err)
}
again := onTrunk("build-unguarded-2", "novox/unguarded", "git", "", map[string]any{"module": "unguarded", "version": "2"})
if _, _, err := takeIn(ctx, open.inventory, again); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "push to main directly") {
t.Fatalf("a rebuild from an unprotected trunk was taken in: %v", err)
}
theForge.Store("novox/unguarded", errors.New("nothing serves gitea.gitea_api"))
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-unguarded-3", "novox/unguarded", "git", "",
map[string]any{"module": "unguarded", "version": "3"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a forge that could not say was read as a protected trunk: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["unguarded"].Version != "1" {
t.Fatalf("unguarded is %q", shelf["unguarded"].Version)
}
}
// A repository deleted and made again under the module's repository's name is another repository: the forge's
// id, recorded at registration, tells them apart.
func TestARepositoryMadeAgainUnderItsNameIsNotTheModulesSource(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
theForge.Store("novox/remade", forgeFacts{ID: 100, Guarded: true})
t.Cleanup(func() { theForge.Delete("novox/remade") })
first := onTrunk("build-remade-1", "novox/remade", "git", "", map[string]any{"module": "remade", "version": "1"})
keptAsked(t, open.inventory, first.ID, "novox/remade", "", true)
if _, _, err := takeIn(ctx, open.inventory, first); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "remade"); id != 100 {
t.Fatalf("the forge's id was not recorded: %d", id)
}
theForge.Store("novox/remade", forgeFacts{ID: 101, Guarded: true}) // deleted, and made again by an agent
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-2", "novox/remade", "git", "",
map[string]any{"module": "remade", "version": "evil"})); !errors.Is(err, errNotItsSource) ||
!strings.Contains(err.Error(), "made again") {
t.Fatalf("a repository made again under the name was taken in: %v", err)
}
// And a new module from it, beside the one registered from the first, the same.
if _, _, err := takeIn(ctx, open.inventory, onTrunk("build-remade-3", "novox/remade", "git", "modules/other",
map[string]any{"module": "other", "version": "1"})); !errors.Is(err, errNotItsSource) {
t.Fatalf("a new module from a repository made again was taken in: %v", err)
}
}
// The terminal's mark is the operator's for the repository and path they asked: an outcome of another, under that
// build's id, is not theirs.
func TestATerminalRequestCoversOnlyWhatItAsked(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.RecordBuildRequest(ctx, inventory.BuildRequest{ID: "build-asked", Repository: "novox/app",
Seat: "git", Path: "modules/app", For: "build", AtTerminal: true}); err != nil {
t.Fatal(err)
}
other := onTrunk("build-asked", "agent/sudo", "git", "modules/app", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, other); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another repository under a terminal request's id was taken in: %v", err)
}
elsewhere := onTrunk("build-asked", "novox/app", "git", "modules/sudo", map[string]any{"module": "sudo", "version": "evil"})
if _, _, err := takeIn(ctx, open.inventory, elsewhere); !errors.Is(err, errNotItsSource) {
t.Fatalf("an outcome of another path under a terminal request's id was taken in: %v", err)
}
}
// The serving controller, and everything it starts, is never the terminal: a build asked in it reads as asked
// through the mesh even when no verb and no caller is named.
func TestTheServingControllerIsNeverTheTerminal(t *testing.T) {
t.Setenv(verbVar, "")
t.Setenv(link.CallerVar, "")
t.Setenv(servedVar, "")
if !startedAtTheTerminal() {
t.Fatal("a process started by hand is not the terminal")
}
markServed()
if startedAtTheTerminal() {
t.Fatal("the serving controller reads as the terminal")
}
child := exec.Command(os.Args[0], "-test.run=^$")
child.Env = os.Environ()
if !slices.Contains(child.Env, servedVar+"=1") {
t.Fatal("what the serving controller starts does not carry its mark")
}
}
// The forge applies one rule to a branch: the rule named for it, else the first glob that covers it. A stronger
// rule later in the list is not what guards the branch, and is not read as if it were (the confirmation review).
@@ -120,9 +369,16 @@ func TestTheRuleJudgedIsTheOneTheForgeApplies(t *testing.T) {
}
// An id the forge could not give keeps the id already recorded.
// **The operator's Approve of a reviewed group registers a moved module from its new repository** (novox/hq ADR 0420 §7,
// issue 1025). sudo is built from the catalogue; a merge into the repository of node-engine's module, announced by
// mesh-delivery with the ask it merged on, brings its build from there. Accepted with the operator's Approve of the
// merged head; refused with no kept merge, a warrant for other heads, a Decline, a person who is not the operator, and
// through a verb.
func TestAnUnknownIdentityKeepsTheRecordedOne(t *testing.T) {
open := theCatalogue(t)
ctx := t.Context()
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 100); err != nil {
t.Fatal(err)
}
if err := open.inventory.SetSourceIdentity(ctx, "sudo", 0); err != nil {
t.Fatal(err)
}
if id, _ := open.inventory.SourceIdentity(ctx, "sudo"); id != 100 {
t.Fatalf("the recorded id became %d", id)
}
}
+116 -43
View File
@@ -1,13 +1,12 @@
package main
import (
"bytes"
"reflect"
"encoding/json"
"errors"
"strings"
"testing"
"time"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -16,12 +15,126 @@ import (
// the module registered with its source as the seat and path when the request said so — never the
// URL. A definition naming an installation is recorded and not registered; a failure is recorded
// and said.
func TestABuildHeardIsRecordedAndRegistered(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "shop", "version": "3"})
m, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, link.BuildResult{
ID: "b-1", Repository: "http://forge.internal:20000/novox/shop.git", Path: "modules/shop",
Ref: "main", On: "anchor", Commit: "abcdef0123", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/shop"},
}))
if err != nil {
t.Fatal(err)
}
if m.Module != "shop" {
t.Fatalf("registered %q", m.Module)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if _, held := shelf["shop"]; !held {
t.Fatal("the module a heard build produced is not in the catalogue")
}
src, err := open.inventory.SourceOf(ctx, "shop")
if err != nil || src.Seat != "git" || src.Repository != "novox/shop" || src.BuiltFrom != "abcdef0123" {
t.Fatalf("the source is the seat and the path, never the URL: %+v %v", src, err)
}
builds, err := open.inventory.Builds(ctx, "shop", 5)
if err != nil || len(builds) != 1 || builds[0].ID != "b-1" {
t.Fatalf("the build is not recorded once: %v %v", builds, err)
}
named, _ := json.Marshal(map[string]any{"module": "idp", "version": "1", "resources": []any{
map[string]any{"id": "server", "type": "container", "image": "x@sha256:aa",
"env": map[string]any{"KC_HOSTNAME": "https://login.mesh-one.be"}}}})
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{
ID: "b-2", Repository: "/r", On: "anchor", Commit: "0123456789", Manifest: named})
if err == nil || !strings.Contains(err.Error(), "does not register it") {
t.Fatalf("a definition naming an installation was taken in: %v", err)
}
if shelf, _ := open.inventory.Catalogue(ctx); shelf["idp"].Module != "" {
t.Fatal("the refused module was registered anyway")
}
if builds, _ := open.inventory.Builds(ctx, "idp", 5); len(builds) != 1 {
t.Fatalf("the refused build was not recorded: %v", builds)
}
_, _, err = takeIn(ctx, open.inventory, link.BuildResult{ID: "b-3", Repository: "/r", On: "anchor", Failed: "no compiler"})
if err == nil || !strings.Contains(err.Error(), "no compiler") {
t.Fatalf("a failure is said in the builder's words: %v", err)
}
}
// novox/hq 04-ISSUES/215: a build asked at a commit is recorded as built from that commit, and the
// module keeps following the branch it followed — a new one, the default branch.
func TestABuildAtACommitKeepsTheBranchTheModuleFollows(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
manifest, _ := json.Marshal(map[string]any{"module": "unifi", "version": "1"})
result := func(id, ref, commit string) link.BuildResult {
return link.BuildResult{ID: id, Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/unifi", Ref: ref, On: "anchor", Commit: commit, Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result("b-1", "main", "1111111aaaa"))); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, open.inventory, result("b-2", "9c97a8a", "9c97a8a1d2c3")); err != nil {
t.Fatal(err)
}
src, err := open.inventory.SourceOf(ctx, "unifi")
if err != nil {
t.Fatal(err)
}
if src.Ref != "main" || src.BuiltFrom != "9c97a8a1d2c3" {
t.Errorf("after a build at a commit the module follows %q, built from %q; want main, 9c97a8a1d2c3", src.Ref, src.BuiltFrom)
}
// One new to the catalogue, first built at a commit, follows the default branch.
other, _ := json.Marshal(map[string]any{"module": "letta", "version": "1"})
r := result("b-3", "deadbeef", "deadbeefcafe")
r.Manifest, r.Path = other, "modules/letta"
if _, _, err := takeIn(ctx, open.inventory, r); err != nil {
t.Fatal(err)
}
if src, _ := open.inventory.SourceOf(ctx, "letta"); src.Ref != "" {
t.Errorf("a module first built at a commit follows %q, want the default branch", src.Ref)
}
}
// novox/hq 04-ISSUES/219: an older request heard after a newer one is recorded and not registered,
// so a push sends what the newer request built.
func TestAnOlderBuildHeardLaterDoesNotReplaceTheNewer(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
older := time.Date(2026, 10, 3, 21, 33, 45, 0, time.UTC)
newer := time.Date(2026, 10, 3, 21, 51, 57, 0, time.UTC)
result := func(asked time.Time, image string) link.BuildResult {
manifest, _ := json.Marshal(map[string]any{"module": "postgres", "version": image})
return link.BuildResult{ID: link.NewBuildID(asked), Repository: "http://forge.internal:20000/novox/mesh-catalog.git",
Path: "modules/postgres", Ref: "main", On: "anchor", Commit: "efff5415", Manifest: manifest,
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"}}
}
if _, _, err := takeIn(ctx, open.inventory, asTheOperator(t, open.inventory, result(newer, "4bcd5f73"))); err != nil {
t.Fatal(err)
}
_, _, err := takeIn(ctx, open.inventory, result(older, "0ab07fa9"))
if !errors.Is(err, inventory.ErrSuperseded) {
t.Fatalf("the older request's outcome was taken in as current: %v", err)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if got := shelf["postgres"].Version; got != "4bcd5f73" {
t.Errorf("postgres is %q; want the newer request's 4bcd5f73", got)
}
if builds, _ := open.inventory.Builds(ctx, "postgres", 5); len(builds) != 2 {
t.Errorf("the late build was not recorded: %v", builds)
}
}
func TestABuildIDSaysWhenItWasAsked(t *testing.T) {
at := time.Date(2026, 10, 3, 21, 51, 57, 392539762, time.UTC)
@@ -37,43 +150,3 @@ func TestABuildIDSaysWhenItWasAsked(t *testing.T) {
}
}
}
// novox/hq issue 1204: builds says the source fingerprint the builder recorded, the one tool that reads it.
func TestBuildsSaysTheSourceFingerprint(t *testing.T) {
at := time.Date(2026, 10, 19, 10, 40, 0, 0, time.UTC)
built := inventory.Build{ID: "build-1", Module: "mailu", Commit: "b4527d9091e972f28cb365623d9280045e7741a3", On: "ace",
Repository: "novox/mesh-catalog", At: at, ReuseKey: "r1:abc",
SourceFingerprint: "src1:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}
// 1204/AC1
var out bytes.Buffer
printBuild(&out, built)
lines := strings.Split(strings.TrimRight(out.String(), "\n"), "\n")
if last := lines[len(lines)-1]; last != " source "+built.SourceFingerprint {
t.Errorf("last line %q; want the source line after the reuse key:\n%s", last, out.String())
}
if !strings.Contains(lines[len(lines)-2], "reuse key r1:abc") {
t.Errorf("the source line does not follow the reuse key:\n%s", out.String())
}
// 1204/AC2
without := built
without.SourceFingerprint = ""
var plain bytes.Buffer
printBuild(&plain, without)
if strings.Contains(plain.String(), "source") {
t.Errorf("a build with no fingerprint says a source line:\n%s", plain.String())
}
if want := strings.Join(lines[:len(lines)-1], "\n") + "\n"; plain.String() != want {
t.Errorf("the other lines changed:\n%s\nwant\n%s", plain.String(), want)
}
}
// 1215/AC4: what the builder said of the inputs is what the controller keeps of the build.
func TestTheControllerKeepsTheInputsABuilderSaid(t *testing.T) {
in := &sdkdelivery.SourceInputs{Module: "mailu", Tree: "t", Toolchains: map[string]string{"code": "ts sha256:aa 01"}}
kept := buildFrom(link.BuildResult{ID: "build-1", Repository: "r", Commit: "c", SourceFingerprint: "src1:x", SourceInputs: in})
if !reflect.DeepEqual(kept.SourceInputs, in) {
t.Errorf("the controller keeps %+v, the builder said %+v", kept.SourceInputs, in)
}
}
+1 -1
View File
@@ -11,7 +11,7 @@ import (
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
@@ -1,92 +0,0 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **Every user's slow consumer is S9's, not the controller's alone** (novox/hq ADR 0357, issue 270). On
// 2026-10-07 the bus closed two clients on the laptop as slow consumers and nothing raised a condition:
// the controller heard only its own connection. Said by the bus module and read back, a user the bus
// closes is one condition naming the module and its machine, raised at the first look and cleared an hour
// after the last closure.
func TestAUserTheBusClosedAsASlowConsumerIsOneConditionNamingItsModuleAndMachine(t *testing.T) {
closed := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
now := closed.Add(20 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
advisories := link.BusClosures([]link.BusClosure{
{User: "laptop.claude-code", CID: 10, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Hour),
Closed: closed.Add(-10 * time.Minute)},
{User: "laptop.claude-code", CID: 44, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Minute),
Closed: closed},
{User: "anchor.desk", CID: 7, Reason: "Maximum Message Payload Exceeded", Opened: closed.Add(-time.Hour),
Closed: closed},
}, []string{link.AdvisorySlowConsumer, link.AdvisorySlowConsumer, link.AdvisoryConnectionClosed}, closed.Add(-time.Hour))
look := func() {
t.Helper()
f := &signalFacts{now: now, host: "anchor", advisories: advisories}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
}
look()
slow, found, err := conditions.ReadOne(t.Context(), store, "bus.laptop.claude-code.slow-consumer")
if err != nil || !found {
t.Fatalf("no slow-consumer condition for the user: %v %v", found, err)
}
if slow.Subject.Machine != "laptop" || slow.Observations != 2 || !slow.LastObserved.Equal(closed) {
t.Fatalf("the slow consumer is said on %q, %d closure(s), last %s: %+v", slow.Subject.Machine, slow.Observations,
slow.LastObserved, slow)
}
if !strings.Contains(slow.Summary, "claude-code on laptop") || strings.Contains(slow.Summary, "laptop.claude-code") {
t.Fatalf("the summary does not name the module in words, or carries the dotted user: %q", slow.Summary)
}
if !strings.Contains(slow.Headline, "Claude-code on laptop") {
t.Fatalf("the headline is %q", slow.Headline)
}
cut, found, err := conditions.ReadOne(t.Context(), store, "bus.anchor.desk.connection-closed")
if err != nil || !found || cut.Subject.Machine != "anchor" || !strings.Contains(cut.Summary, "maximum message payload exceeded") {
t.Fatalf("the user cut off for breaking the bus's rules: %v %v %+v", found, err, cut)
}
// An hour after the newest closure without another, both clear.
now = closed.Add(time.Hour + time.Minute)
look()
for _, key := range []string{"bus.laptop.claude-code.slow-consumer", "bus.anchor.desk.connection-closed"} {
if _, found, _ := k.Get(t.Context(), key); found {
t.Fatalf("%s is still open an hour after the last closure", key)
}
}
}
// **A read of the bus module's events that fails does not blind S9** (review of #326): the controller's own
// signals are still said, and the failed read is a condition of its own.
func TestABusModuleReadThatFailsKeepsTheControllersOwnSignals(t *testing.T) {
at := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
f := &signalFacts{now: at.Add(time.Minute), host: "anchor",
advisories: []link.Advisory{{Kind: link.AdvisorySlowConsumer, ID: "controller", Said: "the controller fell behind",
First: at, Last: at, Count: 1}},
busFaultsErr: errors.New("nats: timeout")}
keys := map[string]bool{}
for _, o := range watchAdvisories(f) {
keys[o.Key()] = true
}
if !keys["bus.controller.slow-consumer"] || !keys["bus.bus-module.bus-faults-unread"] {
t.Fatalf("observed %v: the controller's own slow consumer and the unread events, both", keys)
}
for _, row := range signalsTable {
if row.Row == "S9" && row.needs(f) != nil {
t.Fatal("the S9 row went blind for a read of the bus module's events")
}
}
}
@@ -1,70 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
func TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds(t *testing.T) {
now := time.Date(2026, 10, 8, 18, 30, 0, 0, time.UTC)
b := aBusOnAnchor()
walk := func(id, repo, note string, updated time.Time) inventory.Plan {
return inventory.Plan{ID: id, Repository: repo, Commit: "c0ffee001122", State: inventory.PlanRolling, Tier: 0,
Tiers: [][]string{{"mesh-host"}}, Modules: map[string]*inventory.PlanModule{"mesh-host": {}}, Note: note,
Updated: updated}
}
for _, c := range []struct {
name string
bus busPending
plans []inventory.Plan
first map[string]time.Time
want []string
since time.Time
}{
{"a walk refused for the bus", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-27*time.Minute))}, nil,
[]string{"plan-1"}, now.Add(-27 * time.Minute)},
{"refused earlier than its last save, as this controller saw it", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-5*time.Minute))},
map[string]time.Time{"plan-1": now.Add(-28 * time.Minute)}, []string{"plan-1"}, now.Add(-28 * time.Minute)},
{"a walk refused for another reason", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", "tier 0: the build seat is paused — tried again", now)}, nil,
nil, time.Time{}},
{"refused for an older bus build than the one held now", func() busPending { o := b; o.to = "99999999cccc"; return o }(),
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"the bus already runs the build held: its step started", func() busPending {
o := aBusOnAnchor()
o.from = map[string]string{"anchor": o.to}
return o
}(), []inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"a walk waiting for its delivery's word asks no send", b, func() []inventory.Plan {
p := walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)
p.Delivery = &inventory.PlanDelivery{Awaits: "mesh-delivery"}
return []inventory.Plan{p}
}(), nil, nil, time.Time{}},
} {
t.Run(c.name, func(t *testing.T) {
got := busWaitsOf(c.plans, c.bus, func(id string) time.Time { return c.first[id] })
var ids []string
for _, w := range got.waits {
ids = append(ids, w.plan)
}
if strings.Join(ids, ",") != strings.Join(c.want, ",") {
t.Fatalf("held for the bus: %v, want %v", ids, c.want)
}
if len(c.want) > 0 {
if !got.waits[0].since.Equal(c.since) {
t.Errorf("waiting since %s, want %s", got.waits[0].since, c.since)
}
if strings.Join(got.machines, ",") != "anchor" || got.to != b.to || got.module != "nats" {
t.Errorf("behind %+v", got)
}
}
})
}
}
+60
View File
@@ -7,6 +7,7 @@ import (
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq issue 336: a send held because it would replace the bus outside its planned step waits for a
@@ -26,6 +27,65 @@ func refusedNote(b busPending, tier int) string {
return "tier " + string(rune('0'+tier)) + ": " + errBusWaits.Error() + ": " + held + " — tried again"
}
func TestASendHeldForTheBusStepIsFoundFromTheWalksItHolds(t *testing.T) {
now := time.Date(2026, 10, 8, 18, 30, 0, 0, time.UTC)
b := aBusOnAnchor()
walk := func(id, repo, note string, updated time.Time) inventory.Plan {
return inventory.Plan{ID: id, Repository: repo, Commit: "c0ffee001122", State: inventory.PlanRolling, Tier: 0,
Tiers: [][]string{{"mesh-host"}}, Modules: map[string]*inventory.PlanModule{"mesh-host": {}}, Note: note,
Updated: updated}
}
for _, c := range []struct {
name string
bus busPending
plans []inventory.Plan
first map[string]time.Time
want []string
since time.Time
}{
{"a walk refused for the bus", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-27*time.Minute))}, nil,
[]string{"plan-1"}, now.Add(-27 * time.Minute)},
{"refused earlier than its last save, as this controller saw it", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now.Add(-5*time.Minute))},
map[string]time.Time{"plan-1": now.Add(-28 * time.Minute)}, []string{"plan-1"}, now.Add(-28 * time.Minute)},
{"a walk refused for another reason", b,
[]inventory.Plan{walk("plan-1", "novox/mesh-host", "tier 0: the build seat is paused — tried again", now)}, nil,
nil, time.Time{}},
{"refused for an older bus build than the one held now", func() busPending { o := b; o.to = "99999999cccc"; return o }(),
[]inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"the bus already runs the build held: its step started", func() busPending {
o := aBusOnAnchor()
o.from = map[string]string{"anchor": o.to}
return o
}(), []inventory.Plan{walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)}, nil, nil, time.Time{}},
{"a walk waiting for its delivery's word asks no send", b, func() []inventory.Plan {
p := walk("plan-1", "novox/mesh-host", refusedNote(b, 0), now)
p.Delivery = &inventory.PlanDelivery{Awaits: "mesh-delivery"}
return []inventory.Plan{p}
}(), nil, nil, time.Time{}},
} {
t.Run(c.name, func(t *testing.T) {
got := busWaitsOf(c.plans, c.bus, func(id string) time.Time { return c.first[id] })
var ids []string
for _, w := range got.waits {
ids = append(ids, w.plan)
}
if strings.Join(ids, ",") != strings.Join(c.want, ",") {
t.Fatalf("held for the bus: %v, want %v", ids, c.want)
}
if len(c.want) > 0 {
if !got.waits[0].since.Equal(c.since) {
t.Errorf("waiting since %s, want %s", got.waits[0].since, c.since)
}
if strings.Join(got.machines, ",") != "anchor" || got.to != b.to || got.module != "nats" {
t.Errorf("behind %+v", got)
}
}
})
}
}
// **Said at once, not as lateness, and cleared when the step starts**: the walk held only for the bus raises
// the bus's condition on the first tick, before any tier bound, naming what waits, the bus build from and to,
// since when and `bus upgrade`, for the operator; S3 says nothing of it even past its bound; and the
+5 -42
View File
@@ -49,12 +49,10 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
// composed from, so a module the mesh grants a consumer's subjects has that consumer waiting.
// Done on every raise, not only when a credential is issued: every module moved onto this bus
// by the rollout was issued on the old one, and came up with nothing to bind to (2026-09-28).
consumers, records, err := moduleConsumersAndRecords(ctx, inv)
consumers, err := moduleConsumers(ctx, inv)
if err != nil {
return nil, err
}
// And a grant of a restricted stream that composes nothing, said on every raise and send (novox/hq ADR 0376 §2).
sayWithheld(records, " ")
// And the work queues of seats that name their caller or their kind, with each holder's worker
// (novox/hq ADR 0259 §3): an ask queues until the router takes it, a channel's work until that kind
// takes it.
@@ -62,19 +60,9 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
if err != nil {
return nil, err
}
// And every module's own stream (novox/hq ADR 0363), from the catalogue, before the consumers on it.
modStreams, err := inv.DeclaredStreams(ctx)
if err != nil {
return nil, err
}
// Every one tried, and every failure named: one module's consumer the bus refuses is no reason
// the modules after it in the list hear nothing (novox/hq issue 208, where this runs on each send).
var failed []error
for _, s := range broker.ModuleStreams(modStreams) {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the module stream %s: %w", s.Name, err))
}
}
for _, s := range trafficStreams {
if err := r.EnsureStream(s); err != nil {
failed = append(failed, fmt.Errorf("the work queue %s: %w", s.Name, err))
@@ -85,30 +73,11 @@ func assertBusObjects(ctx context.Context, inv *inventory.Inventory, r broker.Ra
failed = append(failed, fmt.Errorf("the worker %s on %s: %w", c.Name, c.Stream, err))
}
}
restricted := map[string]bool{}
for _, s := range modStreams {
restricted[s.Name] = s.Restricted
}
for _, c := range consumers {
if err := broker.EnsureHeard(r, c.Consumer, restricted[c.Consumer.Stream]); err != nil {
if err := r.EnsureConsumer(c.Consumer); err != nil {
failed = append(failed, fmt.Errorf("how %s on %s hears what it consumes: %w", c.Module, c.Node, err))
}
}
// **And a consumer no longer granted loses what it reads through** (novox/hq ADR 0363 §6): every consumer
// on a module's stream the composition no longer asks for is removed, on this very send.
if p, prunes := r.(broker.Pruner); prunes && len(modStreams) > 0 {
var wanted []broker.Consumer
for _, c := range consumers {
wanted = append(wanted, c.Consumer)
}
removed, err := broker.PruneHeard(p, modStreams, wanted)
if err != nil {
failed = append(failed, err)
}
for _, gone := range removed {
fmt.Printf(" removed the consumer %s: its module is no longer granted that stream (novox/hq ADR 0363)\n", gone)
}
}
if len(failed) > 0 {
return nil, errors.Join(failed...)
}
@@ -167,21 +136,15 @@ func unassertedObservation(err error) conditions.Observation {
// moduleConsumers is every module's durable consumer, from the records the user list is composed from.
func moduleConsumers(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, error) {
consumers, _, err := moduleConsumersAndRecords(ctx, inv)
return consumers, err
}
// moduleConsumersAndRecords is moduleConsumers with the records they were composed from.
func moduleConsumersAndRecords(ctx context.Context, inv *inventory.Inventory) ([]broker.ModuleConsumer, broker.Records, error) {
records, err := inv.BusRecords(ctx)
if err != nil {
return nil, broker.Records{}, err
return nil, err
}
users, err := broker.Users(records)
if err != nil {
return nil, broker.Records{}, err
return nil, err
}
return broker.ConsumersOf(users), records, nil
return broker.ConsumersOf(users), nil
}
// seatTrafficObjects is the work queues and workers of seats that name their caller or their kind, from
+1 -3
View File
@@ -1,5 +1,3 @@
//go:build integration
package main
import (
@@ -9,8 +7,8 @@ import (
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
-103
View File
@@ -1,103 +0,0 @@
package main
import (
"fmt"
"time"
"github.com/novox/mesh-controller/internal/inventory"
)
// A first-node gate waits for a busy container (novox/hq ADR 0335 §4).
//
// When a machine reports, on what the gate sent it, that a container of a judged module was left running
// because it said busy, the judging has not started: the build is not yet what runs there. The walk says it
// waits for a busy container on that machine, counts no pass and no bound, and starts the judging — its
// Since — at the report that no longer lists the container waiting. The wait is the node-engine's, bounded
// at 15 minutes; the controller's own bound (busyGateBound past the engine's) is for an engine that stopped
// reporting while it waited, so no walk waits for ever. An engine older than the field never lists one, and
// its gate starts as before.
// busyGateSlack is how long past the engine's bound the controller still waits for the report that says the
// container replaced: a pass of the engine (its asks every 20 seconds) and the report's way here.
var busyGateSlack = 5 * time.Minute
// busyEngineBound is the node-engine's bound on a wait (novox/hq ADR 0335 §3), counted again here on this
// controller's clock from when it first read the wait.
const busyEngineBound = 15 * time.Minute
// busyGateEnds is when the gate stops waiting: the engine's own word on its bound, or this controller's count of
// it from when it first read the wait, whichever is sooner, and the slack — so neither a clock ahead on the
// machine nor a wrong word from it holds a walk longer.
func busyGateEnds(b *inventory.GateBusy) time.Time {
until := b.Seen.Add(busyEngineBound)
if !b.Until.IsZero() && b.Until.Before(until) {
until = b.Until
}
return until.Add(busyGateSlack)
}
// busyWait decides, before a judging, whether the gate waits for a busy container on a machine it judges:
// true while it waits, the gate saying so. False when nothing waits, or no longer: a wait that ended moves the
// gate's Since to its end. Pure but for the gate it is handed.
func busyWait(g *inventory.PlanGate, pairs []judged, f gateFacts, now time.Time) bool {
if g.Busy != nil && g.Busy.Ended != nil {
return false
}
for _, j := range pairs {
b, waits := busyOn(j, f)
if !waits {
continue
}
if g.Busy == nil {
g.Busy = &inventory.GateBusy{Machine: j.node, Module: j.module, Container: b.Container, Since: b.Since,
Seen: now}
}
if g.Busy.Seen.IsZero() {
g.Busy.Seen = now
}
g.Busy.Until, g.Busy.Said = b.Until, b.Said
if now.After(busyGateEnds(g.Busy)) {
// The engine said it would have replaced it by now: the judging starts, and finds the module not
// yet healthy while the machine still lists it waiting (busyStill), until the gate's own bound.
end := now
g.Busy.Ended, g.Busy.PastBound = &end, true
g.Since, g.Passes, g.LastPass, g.Healthy, g.HealthyAt, g.Applied = &end, 0, nil, nil, nil, nil
return false
}
g.Passes, g.LastPass, g.Last, g.Failing = 0, nil, g.Busy.Words(), nil
g.Read(now, false, g.Busy.Words())
return true
}
if g.Busy == nil {
return false
}
// Replaced: the judging starts at the report that said so.
end := now
if r, said := f.reports[g.Busy.Machine]; said && r.At != nil && r.At.After(g.Busy.Since) && r.At.Before(now) {
end = r.At.UTC()
}
g.Busy.Ended = &end
g.Since, g.Passes, g.LastPass, g.Last, g.Healthy, g.HealthyAt, g.Applied = &end, 0, nil, "", nil, nil, nil
return false
}
// busyOn is the container of a judged module its machine last reported, on what the gate sent it, as left
// running because it said busy.
func busyOn(j judged, f gateFacts) (inventory.BusyWait, bool) {
r, said := f.reports[j.node]
if !said || r.At == nil || !f.reportedOn(j.node, r) {
return inventory.BusyWait{}, false
}
return inventory.WaitingFor(r.Busy, j.module)
}
// busyStill is a judged module whose container its machine still lists waiting once the gate stopped waiting
// for it: not yet what was sent, so not healthy, whatever its health says of the container still running.
func busyStill(j judged, f gateFacts) (string, bool) {
b, waits := busyOn(j, f)
if !waits {
return "", false
}
return fmt.Sprintf("%s still runs its busy container %s, past its node-engine's bound of %s",
j.node, b.Container, b.Until.UTC().Format(time.RFC3339)), true
}
-148
View File
@@ -1,148 +0,0 @@
package main
import (
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A walk whose module's container waits as busy on its first machine says so, and starts that machine's
// first-machine gate only once the machine reports the container replaced (novox/hq ADR 0335 §4).
func TestAGateWaitsForABusyContainerAndStartsWhenItIsReplaced(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent,
Sent: map[string]inventory.SentDeclaration{"novox": {Digest: "d1", Sequence: 7}}}
pairs := []judged{{module: "build-agent", node: "novox"}, {module: "mesh-controller", node: "novox"}}
reported := sent.Add(20 * time.Second)
waiting := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: reported, Until: reported.Add(15 * time.Minute),
Said: "busy: novox/mesh-controller (check of #280), in hand since 00:58:00Z"}}
facts := func(at time.Time, busy []inventory.BusyWait) gateFacts {
// As judgeMoves hands it: the report held against the gate's own send.
return gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"novox": {Node: "novox",
Outcome: inventory.OutcomeApplied, At: &at, Declared: "d1", ReportedSequence: 7, Busy: busy}}}
}
now := reported.Add(time.Minute)
if !busyWait(g, pairs, facts(reported, waiting), now) {
t.Fatal("a gate whose module's container waits as busy judged it")
}
if !g.Since.Equal(sent) || g.Busy == nil || g.Busy.Machine != "novox" || g.Busy.Ended != nil {
t.Fatalf("the gate does not keep the wait: %+v %+v", g.Since, g.Busy)
}
for _, said := range []string{gateLine(g), g.Last} {
if !strings.Contains(said, "waits for a busy container on novox") {
t.Errorf("the walk does not say it waits for a busy container there: %s", said)
}
}
if !strings.Contains(g.Last, "check of #280") {
t.Errorf("the wait does not say what the container holds: %s", g.Last)
}
// Ten minutes on, still busy, within the engine's bound: still waiting, nothing judged, no bound run.
now = reported.Add(10 * time.Minute)
if !busyWait(g, pairs, facts(reported.Add(9*time.Minute), waiting), now) || g.Passes != 0 {
t.Fatalf("the gate stopped waiting within the bound: %+v", g)
}
// The machine reports it replaced: the judging starts at that report.
replacedAt := reported.Add(12 * time.Minute)
now = replacedAt.Add(5 * time.Second)
replaced := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: "replaced-when-idle", Since: reported, Until: reported.Add(15 * time.Minute)}}
if busyWait(g, pairs, facts(replacedAt, replaced), now) {
t.Fatal("the gate still waits after the container was replaced")
}
if g.Since == nil || !g.Since.Equal(replacedAt) || g.Busy.Ended == nil || !g.Busy.Ended.Equal(replacedAt) {
t.Fatalf("the gate does not start at the report that said it replaced: since %v, %+v", g.Since, g.Busy)
}
if strings.Contains(gateLine(g), "waits for a busy container") {
t.Errorf("a gate judging still says it waits: %s", gateLine(g))
}
// And a later report listing it waiting again does not hold a gate that started.
if busyWait(g, pairs, facts(now, waiting), now.Add(time.Minute)) {
t.Error("a gate that started waits again")
}
}
// An engine that stops reporting while it waits holds no walk for ever: past the engine's bound and the slack,
// the judging starts, and finds the module not yet healthy while its container is still listed waiting.
func TestAGateWaitsNoLongerThanTheEnginesBound(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &sent}
pairs := []judged{{module: "build-agent", node: "novox"}}
waiting := []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: sent, Until: sent.Add(15 * time.Minute)}}
f := gateFacts{reports: map[string]inventory.Reported{"novox": {Node: "novox", Outcome: inventory.OutcomeApplied,
At: &sent, Current: true, Busy: waiting}}}
now := sent.Add(15*time.Minute + busyGateSlack + time.Second)
if busyWait(g, pairs, f, now) {
t.Fatal("a gate waited past the engine's bound and the slack")
}
if !g.Busy.PastBound || !g.Since.Equal(now) {
t.Fatalf("the gate does not start, said past the bound: %+v %v", g.Busy, g.Since)
}
if why, still := busyStill(pairs[0], f); !still || !strings.Contains(why, "past its node-engine's bound") {
t.Fatalf("a module still running its busy container reads as what was sent: %q", why)
}
}
// A report from an engine older than the field lists nothing: the gate starts at the send, as before; and a
// report not about the gate's send says nothing of it.
func TestAGateOfAnEngineThatNeverWaitsStartsAsBefore(t *testing.T) {
sent := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"ace"}, Since: &sent,
Sent: map[string]inventory.SentDeclaration{"ace": {Digest: "d2", Sequence: 9}}}
pairs := []judged{{module: "build-agent", node: "ace"}}
at := sent.Add(time.Minute)
older := gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"ace": {Node: "ace", Outcome: inventory.OutcomeApplied,
At: &at, Declared: "d2", ReportedSequence: 9}}}
if busyWait(g, pairs, older, at) || g.Busy != nil || !g.Since.Equal(sent) {
t.Fatalf("a gate on an engine that never waits did not start at the send: %+v %v", g.Busy, g.Since)
}
elsewhere := gateFacts{sent: g.Sent, reports: map[string]inventory.Reported{"ace": {Node: "ace", Outcome: inventory.OutcomeApplied,
At: &at, Declared: "d1", ReportedSequence: 8, Busy: []inventory.BusyWait{{ID: "build-agent.server",
State: inventory.BusyWaiting, Until: at.Add(time.Hour)}}}}}
if busyWait(g, pairs, elsewhere, at) {
t.Fatal("a wait reported about an earlier declaration held the gate")
}
}
// The wait is bounded on this controller's clock: an engine whose clock runs an hour ahead, saying it waits until
// then, holds the gate no longer than 15 minutes and the slack from when the controller first read the wait.
func TestAGateWaitIsBoundedOnTheControllersClock(t *testing.T) {
seen := time.Date(2026, 10, 13, 1, 0, 0, 0, time.UTC)
g := &inventory.PlanGate{Machines: []string{"novox"}, Since: &seen}
pairs := []judged{{module: "build-agent", node: "novox"}}
ahead := seen.Add(time.Hour)
f := gateFacts{reports: map[string]inventory.Reported{"novox": {Node: "novox", Outcome: inventory.OutcomeApplied,
At: &seen, Current: true, Busy: []inventory.BusyWait{{ID: "build-agent.server", Container: "mesh-build-agent",
State: inventory.BusyWaiting, Since: ahead, Until: ahead.Add(15 * time.Minute)}}}}}
if !busyWait(g, pairs, f, seen) || !g.Busy.Seen.Equal(seen) {
t.Fatalf("the gate did not start waiting, seen now: %+v", g.Busy)
}
if !busyWait(g, pairs, f, seen.Add(15*time.Minute+busyGateSlack-time.Second)) {
t.Fatal("the gate stopped waiting within the bound on its own clock")
}
if busyWait(g, pairs, f, seen.Add(15*time.Minute+busyGateSlack+time.Second)) || !g.Busy.PastBound {
t.Fatalf("an engine's later word held the gate past the controller's bound: %+v", g.Busy)
}
}
// `plan` says, for a machine whose node-engine is older than the field, each container whose busy is not sent.
func TestAPlanSaysABusyAnOlderEngineIsNotSent(t *testing.T) {
modules := []catalogue.Manifest{{Module: "build-agent", Resources: []map[string]any{
{"id": "server", "type": "container", "busy": map[string]any{"command": "mesh-builder busy"}},
{"id": "agent-env", "type": "file"}}}}
lines := busyNotSent(modules, false)
if len(lines) != 1 || !strings.Contains(lines[0], "build-agent.server declares busy") ||
!strings.Contains(lines[0], "replaced at once") {
t.Fatalf("the plan does not say the busy not sent: %v", lines)
}
if lines := busyNotSent(modules, true); len(lines) != 0 {
t.Fatalf("a busy sent is said as not sent: %v", lines)
}
}
@@ -1,36 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"strings"
"testing"
)
func TestCallsIsServedAndSaysWhatItKeeps(t *testing.T) {
handlers, behind, err := seatToolHandlers()
if err != nil || len(behind) != 0 {
t.Fatalf("%v %v", behind, err)
}
calls, ok := handlers["calls"]
if !ok {
t.Fatal("calls is not served")
}
if _, err := calls(context.Background(), json.RawMessage(`{"node":"anchor"}`)); err == nil ||
!strings.Contains(err.Error(), `"node"`) {
t.Errorf("calls took an argument it does not declare: %v", err)
}
if _, err := calls(context.Background(), json.RawMessage(`{"call":"call-0-0"}`)); err == nil ||
!strings.Contains(err.Error(), "not across a restart") {
t.Errorf("an unknown call was not said plainly: %v", err)
}
got, err := calls(context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if _, listed := got.(map[string]any)["calls"]; !listed {
t.Errorf("calls answered %v", got)
}
}
+28
View File
@@ -1,6 +1,9 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
@@ -49,3 +52,28 @@ func TestAPushAnswersBeforeItSends(t *testing.T) {
}
// `calls` is served, takes a call's id and nothing else, and says plainly when it holds no such call.
func TestCallsIsServedAndSaysWhatItKeeps(t *testing.T) {
handlers, behind, err := seatToolHandlers()
if err != nil || len(behind) != 0 {
t.Fatalf("%v %v", behind, err)
}
calls, ok := handlers["calls"]
if !ok {
t.Fatal("calls is not served")
}
if _, err := calls(context.Background(), json.RawMessage(`{"node":"anchor"}`)); err == nil ||
!strings.Contains(err.Error(), `"node"`) {
t.Errorf("calls took an argument it does not declare: %v", err)
}
if _, err := calls(context.Background(), json.RawMessage(`{"call":"call-0-0"}`)); err == nil ||
!strings.Contains(err.Error(), "not across a restart") {
t.Errorf("an unknown call was not said plainly: %v", err)
}
got, err := calls(context.Background(), json.RawMessage(`{}`))
if err != nil {
t.Fatal(err)
}
if _, listed := got.(map[string]any)["calls"]; !listed {
t.Errorf("calls answered %v", got)
}
}
-597
View File
@@ -1,597 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"net/url"
"path"
"regexp"
"slices"
"sort"
"strconv"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **The controller catches up on merges it did not hear** (novox/hq ADR 0457, issue 1336). A merge the forge announced while the
// controller was being replaced was never heard, and nothing asked for it again: the trunk held a merge no walk answered. At its
// start, and each time it hears a merge, the controller reads the last catchUpWindow merge commits of a trunk from the forge, and
// hears each one it holds no record of as if the forge had announced it then, on the warrant the router holds for it.
// It lasts until release 6 of design 47 deletes delivery plans and batches (ADR 0429 §7, §8).
// catchUpWindow is how many of a trunk's newest merge commits are read; an older merge is a person's to decide.
const catchUpWindow = 50
// kindMergeNotHeard is the condition of a merge the controller did not hear and holds no warrant to plan.
const kindMergeNotHeard = "merge-not-heard"
// trunkMerge is a merge commit of a trunk as the forge lists it.
type trunkMerge struct {
Commit string
// Head is the merge commit's second parent: the pull request's head that was merged.
Head string
Subject string
At time.Time
}
// trunkCommit is a commit of a trunk as the forge lists it, newest first: a merge commit has a second parent.
type trunkCommit struct {
Commit, SecondParent, Subject string
At time.Time
}
// trunkCommitsPage is one page (fifty) of a trunk's commits, newest first; empty past the last. A variable so a test needs no
// forge.
var trunkCommitsPage = func(ctx context.Context, owner, repo, branch string, page int) ([]trunkCommit, error) {
var found struct {
Result []struct {
SHA string `json:"sha"`
Parents []struct {
SHA string `json:"sha"`
} `json:"parents"`
Commit struct {
Message string `json:"message"`
Committer struct {
Date string `json:"date"`
} `json:"committer"`
} `json:"commit"`
} `json:"result"`
}
apiPath := fmt.Sprintf("/repos/%s/%s/commits?sha=%s&limit=50&page=%d&stat=false&verification=false&files=false",
owner, repo, url.QueryEscape(branch), page)
missing, err := askForgeApi(ctx, apiPath, &found)
if err != nil {
return nil, err
}
if missing {
return nil, fmt.Errorf("the forge holds no branch %s of %s/%s", branch, owner, repo)
}
var out []trunkCommit
for _, c := range found.Result {
at, _ := time.Parse(time.RFC3339, c.Commit.Committer.Date)
tc := trunkCommit{Commit: c.SHA, Subject: firstLine(c.Commit.Message), At: at.UTC()}
if len(c.Parents) >= 2 {
tc.SecondParent = c.Parents[1].SHA
}
out = append(out, tc)
}
return out, nil
}
// readTrunkMerges is the last catchUpWindow merge commits of a repository's trunk, oldest first: pages are read until that many
// merges are held or the trunk ends.
func readTrunkMerges(ctx context.Context, owner, repo, branch string) ([]trunkMerge, error) {
var merges []trunkMerge
for page := 1; page <= 8 && len(merges) < catchUpWindow; page++ {
commits, err := trunkCommitsPage(ctx, owner, repo, branch, page)
if err != nil {
return nil, err
}
if len(commits) == 0 {
break
}
for _, c := range commits {
if c.SecondParent != "" {
merges = append(merges, trunkMerge{Commit: c.Commit, Head: c.SecondParent, Subject: c.Subject, At: c.At})
}
}
}
return lastMerges(merges), nil
}
// lastMerges is the newest catchUpWindow of merges, given newest first, oldest first.
func lastMerges(newestFirst []trunkMerge) []trunkMerge {
if len(newestFirst) > catchUpWindow {
newestFirst = newestFirst[:catchUpWindow]
}
out := slices.Clone(newestFirst)
slices.Reverse(out)
return out
}
var mergeSubject = regexp.MustCompile(`^Merge pull request '(.*)' \(#(\d+)\) from (\S+) into (\S+)$`)
// pullOfSubject is the pull request a merge commit's subject names: its title and number.
func pullOfSubject(subject string) (title string, number int, ok bool) {
m := mergeSubject.FindStringSubmatch(strings.TrimSpace(subject))
if m == nil {
return "", 0, false
}
n, err := strconv.Atoi(m[2])
if err != nil || n <= 0 {
return "", 0, false
}
return m[1], n, true
}
// headBranchOfSubject is the branch a merge commit's subject says was merged.
func headBranchOfSubject(subject string) string {
if m := mergeSubject.FindStringSubmatch(strings.TrimSpace(subject)); m != nil {
return m[3]
}
return ""
}
// readMergeEvent is the merge as the forge would have announced it: the commit, the pull request, the files it changed and the
// directories among them that hold a module.json. A variable so a test needs no forge.
var readMergeEvent = func(ctx context.Context, owner, repo, branch string, tm trunkMerge) (link.SourceMoved, error) {
m := link.SourceMoved{Owner: owner, Repo: repo, Base: branch, Head: headBranchOfSubject(tm.Subject), HeadSHA: tm.Head,
Commit: tm.Commit, MergedAt: tm.At.UTC().Format(time.RFC3339)}
if title, number, ok := pullOfSubject(tm.Subject); ok {
m.Number, m.Title = number, title
}
var found struct {
Result struct {
Files []struct {
Filename string `json:"filename"`
Status string `json:"status"`
} `json:"files"`
} `json:"result"`
}
missing, err := askForgeApi(ctx, "/repos/"+owner+"/"+repo+"/git/commits/"+url.PathEscape(tm.Commit), &found)
if err != nil {
return m, err
}
if missing || len(found.Result.Files) == 0 {
return m, fmt.Errorf("the forge named no files for the merge %.8s of %s/%s", tm.Commit, owner, repo)
}
for _, f := range found.Result.Files {
m.Paths = append(m.Paths, f.Filename)
if f.Status == "removed" || f.Status == "deleted" {
m.Removed = append(m.Removed, f.Filename)
}
}
// The forge lists a merge's files by the page; a list as long as a page may be a beginning.
m.PathsTruncated = len(m.Paths) >= 300
dirs := map[string]bool{}
for _, p := range m.Paths {
for d := path.Dir(p); ; d = path.Dir(d) {
dirs[d] = true
if d == "." || d == "/" {
break
}
}
}
if len(dirs) > 64 {
return m, nil // too many to look at: the old rule stands, a directory is a module when its manifest changed
}
var holding []string
for d := range dirs {
file := "module.json"
if d != "." {
file = d + "/module.json"
}
_, exists, err := manifestCached(ctx, owner, repo, file, tm.Commit)
if err != nil {
return m, err
}
if exists {
holding = append(holding, d)
}
}
sort.Strings(holding)
m.ModuleDirs, m.ModuleDirsSaid = holding, true
return m, nil
}
// mergeAskPrefix is what the id of a merge ask made by the retired module mesh-delivery begins with: the pull request's number
// and the head's first twelve digits (`merge-<number>-<head>-<try>`). The teller's ids name neither (see findMergeWarrant).
func mergeAskPrefix(number int, head string) string {
if len(head) > 12 {
head = head[:12]
}
return fmt.Sprintf("merge-%d-%s-", number, head)
}
// askersOfMerges are the askers a merge's warrant is read under: the teller, and mesh-delivery for a merge before release 3e
// (ADR 0437 §6). Release 4's controller work stops accepting mesh-delivery (ADR 0450 §8); a merge asked under it earlier is
// still found here until then.
func askersOfMerges() []string { return []string{sdkdelivery.TellerModule, theDelivery} }
// caughtUpWarrant judges a chosen merge ask of the router as the warrant of a merge: the ask is the operator's, the warrant
// answers that very ask as it was published, and it is the operator's Approve proved by the channel it came through (ADR 0437
// §6, as movedOnApprove judges an announced one). The heads the Approve was bound to are checked by the caller, where it can
// name them.
func caughtUpWarrant(asker string, ask *asks.Ask, w *asks.Warrant) (string, error) {
if err := askerAccepted(asker); err != nil {
return "", err
}
switch {
case ask == nil || w == nil:
return "", errors.New("the router holds no closed record of the ask")
case ask.Who != asks.Operator:
return "", fmt.Errorf("the ask %s is for %q to answer, not the operator", ask.ID, ask.Who)
}
option, err := w.For(asker, *ask)
if err != nil {
return "", err
}
if option.ID != optionApprove || w.Option != optionApprove {
return "", fmt.Errorf("the operator chose %q on the ask %s, not an Approve", w.Option, w.Ask)
}
if w.By == nil || w.By.Who != asks.Operator || w.By.Verified == "" {
return "", fmt.Errorf("the ask %s was not answered by the operator on a channel that proves who answered", w.Ask)
}
return w.Says() + " (ask " + w.Ask + ")", nil
}
// routerAskIDs are the ids of the asks an asker has in the router's record. A variable so a test needs no bus.
var routerAskIDs = func(ctx context.Context, inv catalogueReader, asker string) ([]string, error) {
serving := servingBus.Load()
if serving == nil {
return nil, errors.New("only the serving controller reads the router's record")
}
bucket, err := asksRecords(ctx, inv)
if err != nil {
return nil, err
}
if bucket == "" {
return nil, errors.New("no module declares the operator channel's records")
}
api, err := jetstream.New(serving.Conn())
if err != nil {
return nil, err
}
kv, err := api.KeyValue(ctx, bucket)
if err != nil {
return nil, err
}
lister, err := kv.ListKeysFiltered(ctx, asker+".>")
if err != nil {
return nil, err
}
var ids []string
for key := range lister.Keys() {
ids = append(ids, strings.TrimPrefix(key, asker+"."))
}
if err := lister.Stop(); err != nil {
return nil, err
}
return ids, nil
}
// closedRecord is a router record of an ask that ended: it never changes, so it is read once.
type closedRecord struct {
ask *asks.Ask
w *asks.Warrant
}
var closedRecords sync.Map // asker + "." + id -> closedRecord
// closedRecordOf is the router's record of an ask, kept once it is closed. A nil ask is an ask still open, ended without a
// choice, or not there.
func closedRecordOf(ctx context.Context, inv catalogueReader, asker, id string) (*asks.Ask, *asks.Warrant, error) {
if v, ok := closedRecords.Load(asker + "." + id); ok {
r := v.(closedRecord)
return r.ask, r.w, nil
}
ask, w, err := routerAskOf(ctx, inv, asker, id)
if err != nil {
return nil, nil, err
}
if ask != nil && w != nil {
closedRecords.Store(asker+"."+id, closedRecord{ask, w})
}
return ask, w, nil
}
// deliveryIDOf is the id the gate gives a delivery, and so the key of its head in the heads an Approve binds: the repository as
// owner/name, lower-cased, `@` and the first twelve characters of the head.
func deliveryIDOf(owner, repo, head string) string {
if len(head) > 12 {
head = head[:12]
}
return strings.ToLower(owner+"/"+repo) + "@" + head
}
// coversHead says an ask the operator approved is for this merge: its Approve is bound to the heads of the pull request alone
// (mergeAct over {delivery id: head}, which is how a pull request not in a group is asked), or, for a group's ask, whose other
// members' heads the router's record does not carry in full, the ask as published names this pull request at this head in its
// details ("<repo>#<number> ... at <head>:"), and the digest the warrant answers covers those words.
func coversHead(ask *asks.Ask, owner, repo string, number int, head string) (how string, ok bool) {
option, found := ask.Option(optionApprove)
if !found {
return "", false
}
heads := map[string]string{deliveryIDOf(owner, repo, head): head}
if option.Performs(mergeAct(ask.ID, optionApprove, heads)) == nil {
return "bound to the head of this pull request", true
}
short := head
if len(short) > 12 {
short = short[:12]
}
line := regexp.MustCompile(`(?m)^` + regexp.QuoteMeta(repo) + `#` + strconv.Itoa(number) + ` .* at ` + regexp.QuoteMeta(short) + `:`)
if line.MatchString(ask.Details) {
return "a group's ask naming this pull request at this head", true
}
return "", false
}
// findMergeWarrant finds the warrant of a merge, with no read the controller does not make today: the router's chosen merge
// ask for it. Under the teller (releases 3e on) the id is `merge-<group>-<heads hash>-<tail>`, which names neither the pull
// request nor the head, so the teller's closed asks are read and the one the operator approved whose heads cover the merge's
// second parent is the warrant (coversHead). Under mesh-delivery, which made the earlier merges, the id names the pull request
// and the head and is found by them. Why it holds; or found false with why not, for a person to read; or an error when the
// router could not be read, which is no answer. A variable so a test needs no bus.
var findMergeWarrant = func(ctx context.Context, inv catalogueReader, owner, repo string, number int, head string) (why string, found bool, reason string, err error) {
reason = "the router holds no chosen merge ask of " + strings.Join(askersOfMerges(), " or ") + " that covers the head " + head[:min(12, len(head))]
for _, asker := range askersOfMerges() {
ids, err := routerAskIDs(ctx, inv, asker)
if err != nil {
return "", false, "", err
}
for _, id := range ids {
named := number > 0 && strings.HasPrefix(id, mergeAskPrefix(number, head))
if asker == sdkdelivery.TellerModule && !strings.HasPrefix(id, sdkdelivery.AskKindMerge+"-") {
continue
}
if asker == theDelivery && !named {
continue
}
ask, w, err := closedRecordOf(ctx, inv, asker, id)
if err != nil {
return "", false, "", err
}
if ask == nil || w == nil {
continue
}
warrant, err := caughtUpWarrant(asker, ask, w)
if err != nil {
if named {
reason = fmt.Sprintf("%s's ask %s holds no operator's Approve: %v", asker, id, err)
}
continue
}
how, covers := coversHead(ask, owner, repo, number, head)
if !covers && !named {
continue
}
if !covers {
how = "the ask's id names this pull request and head"
}
return warrant + ", " + how, true, "", nil
}
}
return "", false, reason, nil
}
// caughtTrunk is one trunk the catalogue builds from.
type caughtTrunk struct{ Owner, Repo, Branch string }
func (t caughtTrunk) repository() string { return t.Owner + "/" + t.Repo }
// trunksOfCatalogue are the trunks of the repositories on the mesh's forge the catalogue builds a module from.
func trunksOfCatalogue(forms *sourceForms, entries []inventory.Entry) []caughtTrunk {
var out []caughtTrunk
for _, e := range entries {
if e.Provided || e.Source.Repository == "" {
continue
}
owner, name, onForge := forms.onTheForge(e.Source.Repository, e.Source.Seat)
if !onForge {
continue
}
branch := followedBranch(e.Source.Ref)
if branch == "" {
branch = "main"
}
t := caughtTrunk{owner, name, branch}
if !slices.Contains(out, t) {
out = append(out, t)
}
}
return out
}
// catchUpBeside runs a catch-up beside the hearing of the merge that asked for it, so the bus's acknowledgement of that merge
// waits for no forge. A variable so a test runs it where it stands.
var catchUpBeside = func(run func()) { go run() }
// catchingUp makes one catch-up at a time.
var catchingUp sync.Mutex
// settledMerges are the merges this process found nothing to plan for: heard, and no record kept, because nothing reads them.
// They are not read from the forge again.
var settledMerges sync.Map
// catchUp hears the merges of the catalogue's trunks (only one, when only names it) that the controller holds no record of.
// It says whether a trunk was left for later: the forge, the router or the store did not answer.
func (f following) catchUp(ctx context.Context, only *caughtTrunk) (later bool) {
catchingUp.Lock()
defer catchingUp.Unlock()
inv := f.open.inventory
entries, err := inv.Catalogued(ctx)
if err != nil {
fmt.Printf("catch-up: the catalogue cannot be read, so no merge is caught up now: %v\n", err)
return true
}
trunks := trunksOfCatalogue(newSourceForms(ctx, inv), entries)
if only != nil {
trunks = slices.DeleteFunc(trunks, func(t caughtTrunk) bool {
return !strings.EqualFold(t.repository(), only.repository()) || t.Branch != only.Branch
})
}
for _, t := range trunks {
if f.catchUpTrunk(ctx, t) {
later = true
}
}
return later
}
func (f following) catchUpTrunk(ctx context.Context, t caughtTrunk) (later bool) {
inv := f.open.inventory
repository := t.repository()
merges, err := readTrunkMerges(ctx, t.Owner, t.Repo, t.Branch)
if err != nil {
// Not nothing to catch up: left for the next start or the next heard merge (ADR 0457 clause 3).
fmt.Printf("catch-up: the forge did not answer for the last merges of %s %s, so they are not caught up now; the next start "+
"or the next merge heard there does it: %v\n", repository, t.Branch, err)
return true
}
unwarranted := map[string]bool{}
for _, tm := range merges {
id := fmt.Sprintf("%s_%.8s", repository, tm.Commit)
if _, done := settledMerges.Load(repository + "@" + tm.Commit); done {
continue
}
if _, known, err := inv.MergeOf(ctx, repository, tm.Commit); err != nil {
fmt.Printf("catch-up: the record of %s cannot be read, so %s is not caught up now: %v\n", repository, tm.Commit[:8], err)
return true
} else if known {
continue
}
m, err := readMergeEvent(ctx, t.Owner, t.Repo, t.Branch, tm)
if err != nil {
fmt.Printf("catch-up: the merge %.8s of %s cannot be read from the forge, so it is not caught up now: %v\n", tm.Commit, repository, err)
later = true
continue
}
entries, err := inv.Catalogued(ctx)
if err != nil {
return true
}
read, err := readForPlanning(ctx, inv)
if err != nil {
return true
}
facts, err := factsOfMerge(ctx, inv, m, time.Now().UTC(), entries, read)
if err != nil {
fmt.Printf("catch-up: %s %.8s cannot be judged now: %v\n", repository, tm.Commit, err)
later = true
continue
}
// The very test hearMerge makes: a merge it would plan, or take as a first build, is a gap; any other it would say
// nothing reads, and is no gap for a person to decide.
if !facts.plans() && !facts.addsModuleToBuilds {
settledMerges.Store(repository+"@"+tm.Commit, true)
continue
}
why, found, reason, err := findMergeWarrant(ctx, inv, t.Owner, t.Repo, m.Number, tm.Head)
if err != nil {
fmt.Printf("catch-up: the router cannot be read, so %s %.8s is not caught up now: %v\n", repository, tm.Commit, err)
later = true
continue
}
if !found {
unwarranted[id] = true
f.raiseMergeNotHeard(ctx, repository, t.Branch, tm, m, reason)
continue
}
now := time.Now().UTC()
said := fmt.Sprintf("caught up: merged %s, not heard, planned %s", tm.At.Format(time.RFC3339), now.Format(time.RFC3339))
fmt.Printf("catch-up: %s merged into %s (%.8s) was not heard; %s; on the warrant %s\n", repository, t.Branch, tm.Commit, said, why)
m.Title = strings.TrimSpace(m.Title + " (" + said + ")")
if err := f.hearMerge(ctx, m, now); err != nil {
fmt.Printf("catch-up: %s %.8s could not be heard now: %v\n", repository, tm.Commit, err)
later = true
continue
}
if _, known, err := inv.MergeOf(ctx, repository, tm.Commit); err == nil && !known {
settledMerges.Store(repository+"@"+tm.Commit, true) // heard, and nothing reads it
}
}
// Cleared only when every gap was judged: one left for later may still be open.
if !later {
f.clearMergesHeard(ctx, repository, unwarranted)
}
return later
}
// raiseMergeNotHeard keeps open, and observes again, the condition of a merge with no warrant to plan it. One for each
// repository and commit, however many catch-ups find it.
func (f following) raiseMergeNotHeard(ctx context.Context, repository, branch string, tm trunkMerge, m link.SourceMoved, reason string) {
fmt.Printf("catch-up: %s merged into %s (%.8s) was not heard, and is not planned: %s\n", repository, branch, tm.Commit, reason)
if conditionsFrom == nil {
return
}
o := conditions.Observation{Scope: conditions.ScopeMerge, ID: fmt.Sprintf("%s_%.8s", repository, tm.Commit), Kind: kindMergeNotHeard,
Severity: conditions.Warning, Source: "catch-up", Resolver: conditions.ResolverOperator,
Summary: fmt.Sprintf("the merge %.8s of %s into %s was not heard by the controller, and no operator's Approve of it is on "+
"record to plan it on: a person decides", tm.Commit, repository, branch),
Said: fmt.Sprintf("%s %.8s (pull request %d, head %.8s, merged %s): %s", repository, tm.Commit, m.Number, tm.Head,
tm.At.Format(time.RFC3339), reason),
Headline: fmt.Sprintf("A merge into %s was never delivered", repository),
Explanation: fmt.Sprintf("A merge into %s was made while the controller did not hear it, and no Approve of yours for it is on "+
"record that the controller can plan it on. It stays open while the merge is among the latest %d merges of that repository. "+
"The details say what was looked for.", repository, catchUpWindow),
Needs: "decide whether that merge should be delivered, and say so if it should.",
Resolved: "the merge is planned, or is no longer among the latest merges of its repository"}
if _, err := conditionsFrom.Observe(ctx, o); err != nil {
fmt.Printf("catch-up: the condition of %s %.8s could not be kept: %v\n", repository, tm.Commit, err)
}
}
// clearMergesHeard clears the open conditions of a repository's merges not heard that this catch-up did not find again: the
// gap was planned, or left the window.
func (f following) clearMergesHeard(ctx context.Context, repository string, still map[string]bool) {
if conditionsFrom == nil {
return
}
open, err := conditionsFrom.Open(ctx)
if err != nil {
fmt.Printf("catch-up: the open conditions cannot be read, so none is cleared now: %v\n", err)
return
}
for _, c := range open {
if c.Kind != kindMergeNotHeard || c.Subject.Scope != conditions.ScopeMerge ||
!strings.HasPrefix(c.Subject.ID, repository+"_") || still[c.Subject.ID] {
continue
}
if _, err := conditionsFrom.Clear(ctx, c.Key, "the merge is planned, or left the last merges of its trunk"); err != nil {
fmt.Printf("catch-up: the condition %s could not be cleared: %v\n", c.Key, err)
}
}
}
// catchUpAtStart catches up every trunk once the controller serves, and again every minute while a trunk was left for later.
func catchUpAtStart(ctx context.Context, open *stores) {
select {
case <-ctx.Done():
return
case <-time.After(30 * time.Second):
}
for try := 0; try < 20; try++ {
if !(following{open: open}).catchUp(ctx, nil) {
return
}
select {
case <-ctx.Done():
return
case <-time.After(time.Minute):
}
}
fmt.Println("catch-up: a trunk is still left for later after twenty tries; the next merge heard there does it")
}
@@ -1,291 +0,0 @@
//go:build integration
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// The checks of ADR 0457 (issue 1336): what the controller does about merges it did not hear.
// aTrunk is a forge's trunk as the catch-up reads it: its merges oldest first, the module each moves, and which of them the
// router holds an operator's Approve for.
type aTrunk struct {
merges []trunkMerge
module map[string]string
approved map[string]bool
read []string // the trunks asked of the forge, as owner/repo
events int // merges read in full from the forge
silent bool // the forge does not answer
}
func aTrunkOf(t *testing.T, repo string, modules ...string) *aTrunk {
t.Helper()
tr := &aTrunk{module: map[string]string{}, approved: map[string]bool{}}
for i, m := range modules {
c := fmt.Sprintf("%s-c%02d", repo, i+1)
tr.merges = append(tr.merges, trunkMerge{Commit: c, Head: fmt.Sprintf("%shead%02d", repo, i+1),
Subject: fmt.Sprintf("Merge pull request 'change %d' (#%d) from pr/%s into main", i+1, 100+i, c),
At: t0.Add(time.Duration(i) * time.Minute)})
tr.module[c] = m
tr.approved[c] = true
}
was, wasEvent, wasWarrant := trunkCommitsPage, readMergeEvent, findMergeWarrant
trunkCommitsPage = func(_ context.Context, owner, name, _ string, page int) ([]trunkCommit, error) {
tr.read = append(tr.read, owner+"/"+name)
if tr.silent {
return nil, errors.New("the forge did not answer")
}
if page > 1 || owner+"/"+name != "novox/"+repo {
return nil, nil
}
var newest []trunkCommit
for i := len(tr.merges) - 1; i >= 0; i-- {
m := tr.merges[i]
newest = append(newest, trunkCommit{Commit: m.Commit, SecondParent: m.Head, Subject: m.Subject, At: m.At})
}
return newest, nil
}
readMergeEvent = func(_ context.Context, owner, name, branch string, tm trunkMerge) (link.SourceMoved, error) {
tr.events++
m := catalogueMerge(tm.Commit, tr.module[tm.Commit], tm.At)
m.Owner, m.Repo, m.Base, m.HeadSHA = owner, name, branch, tm.Head
_, m.Number, _ = pullOfSubject(tm.Subject)
return m, nil
}
findMergeWarrant = func(_ context.Context, _ catalogueReader, _, _ string, number int, head string) (string, bool, string, error) {
for _, m := range tr.merges {
if strings.Contains(m.Subject, fmt.Sprintf("(#%d)", number)) && m.Head == head && tr.approved[m.Commit] {
return "the operator chose Approve (ask merge-" + fmt.Sprint(number) + ")", true, "", nil
}
}
return "", false, "the router holds no chosen merge ask for the pull request and head", nil
}
wasBeside := catchUpBeside
catchUpBeside = func(run func()) { run() }
t.Cleanup(func() {
trunkCommitsPage, readMergeEvent, findMergeWarrant, catchUpBeside = was, wasEvent, wasWarrant, wasBeside
settledMerges.Clear()
})
return tr
}
func mergeKnown(t *testing.T, open *stores, commit string) bool {
t.Helper()
_, known, err := open.inventory.MergeOf(t.Context(), "novox/mesh-catalog", commit)
if err != nil {
t.Fatal(err)
}
return known
}
// Clause 1: the trunk holds merges 1, 2 and 3 and the controller holds records of 1 and 3 only; at its start it plans 2, said
// as caught up; a second start plans nothing new.
func TestAGapBeforeANewerHeardMergeIsPlannedAtTheControllersStart(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksRecorded(t)
tr := aTrunkOf(t, "mesh-catalog", "app", "notes", "app")
hear(t, open, catalogueMerge("mesh-catalog-c01", "app", t0), t0.Add(time.Second))
hear(t, open, catalogueMerge("mesh-catalog-c03", "app", t0.Add(2*time.Minute)), t0.Add(2*time.Minute+time.Second))
if mergeKnown(t, open, "mesh-catalog-c02") {
t.Fatal("the merge to be caught up is known before the catch-up")
}
if (following{open: open}).catchUp(t.Context(), nil) {
t.Fatal("the catch-up says a trunk was left for later")
}
if !mergeKnown(t, open, "mesh-catalog-c02") {
t.Fatal("the merge the controller did not hear was not planned")
}
kept, _, _ := open.inventory.MergeOf(t.Context(), "novox/mesh-catalog", "mesh-catalog-c02")
if said := announcedOf(kept).Title; !strings.Contains(said, "caught up: merged ") || !strings.Contains(said, "not heard, planned ") {
t.Fatalf("the merge does not say it was caught up: %q", said)
}
cutAt(t, open, t0.Add(10*time.Minute))
ws, _ := walks(t, open)
if len(ws) == 0 {
t.Fatal("no walk answers the caught-up merge")
}
if _, in := ws[0].Modules["notes"]; !in {
t.Fatalf("the walk does not build notes, which the caught-up merge moved: %v", ws[0].Modules)
}
before := tr.events
if (following{open: open}).catchUp(t.Context(), nil) || tr.events != before {
t.Fatalf("a second start read %d merge(s) in full again", tr.events-before)
}
}
// Clause 1: a heard merge of a repository whose trunk holds earlier merges the controller has no record of plans them all, the
// earlier first, and the forge is read for that repository's trunk only.
func TestAHeardMergeCatchesUpTheEarlierOnesOfItsOwnTrunkOnly(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksRecorded(t)
tr := aTrunkOf(t, "mesh-catalog", "app", "notes", "app")
var order []string
inner := readMergeEvent
readMergeEvent = func(ctx context.Context, owner, name, branch string, tm trunkMerge) (link.SourceMoved, error) {
order = append(order, tm.Commit)
return inner(ctx, owner, name, branch, tm)
}
t.Cleanup(func() { readMergeEvent = inner })
tr.read = nil
heard := catalogueMerge("mesh-catalog-c03", "app", t0.Add(2*time.Minute))
if err := (following{open: open}).SourceMoved(t.Context(), heard); err != nil {
t.Fatal(err)
}
for _, c := range []string{"mesh-catalog-c01", "mesh-catalog-c02", "mesh-catalog-c03"} {
if !mergeKnown(t, open, c) {
t.Fatalf("%s is not planned: the heard merge and the earlier ones of its trunk are all wanted", c)
}
}
if len(order) != 2 || order[0] != "mesh-catalog-c01" || order[1] != "mesh-catalog-c02" {
t.Fatalf("the earlier merges were heard in the order %v, not the trunk's", order)
}
if len(tr.read) == 0 {
t.Fatal("the forge was not read for the trunk of the merge heard")
}
for _, r := range tr.read {
if r != "novox/mesh-catalog" {
t.Fatalf("the forge was read for %s, not only for the trunk of the merge heard: %v", r, tr.read)
}
}
}
// Clause 1: a trunk of fifty-one merges whose oldest is unheard plans nothing: the oldest is outside the window.
func TestAMergeOlderThanTheLastFiftyIsNotRead(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksRecorded(t)
modules := make([]string, 51)
for i := range modules {
modules[i] = "app"
}
aTrunkOf(t, "mesh-catalog", modules...)
// Fifty are known; the oldest is not.
for i := 2; i <= 51; i++ {
c := fmt.Sprintf("mesh-catalog-c%02d", i)
hear(t, open, catalogueMerge(c, "app", t0.Add(time.Duration(i)*time.Minute)), t0.Add(time.Duration(i)*time.Minute+time.Second))
}
// Paged as the forge pages: fifty a page, and a trunk of another repository is empty (a short page, or none).
pages := trunkCommitsPage
trunkCommitsPage = func(ctx context.Context, owner, name, branch string, page int) ([]trunkCommit, error) {
all, err := pages(ctx, owner, name, branch, 1)
if err != nil || page < 1 || (page-1)*50 >= len(all) {
return nil, err
}
return all[(page-1)*50 : min(page*50, len(all))], nil
}
t.Cleanup(func() { trunkCommitsPage = pages })
(following{open: open}).catchUp(t.Context(), nil)
if mergeKnown(t, open, "mesh-catalog-c01") {
t.Fatal("the oldest of fifty-one merges was read and planned, outside the window")
}
}
// Clause 2: a gap with no chosen Approve in the router is not planned and raises one condition naming its repository and
// commit; a second start leaves that one condition open, observed twice; planning the gap clears it.
func TestAGapWithNoApproveRaisesOneConditionKeptOpenAndClearedWhenPlanned(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksRecorded(t)
keeper, _ := withConditionsInMemory(t)
tr := aTrunkOf(t, "mesh-catalog", "notes")
tr.approved["mesh-catalog-c01"] = false
catch := func() { (following{open: open}).catchUp(t.Context(), nil) }
catch()
if mergeKnown(t, open, "mesh-catalog-c01") {
t.Fatal("a merge with no Approve on record was planned")
}
raised := func() []conditions.Condition {
t.Helper()
all, err := keeper.Open(t.Context())
if err != nil {
t.Fatal(err)
}
var out []conditions.Condition
for _, c := range all {
if c.Kind == kindMergeNotHeard {
out = append(out, c)
}
}
return out
}
one := raised()
if len(one) != 1 || !strings.Contains(one[0].Subject.ID, "novox/mesh-catalog") || !strings.Contains(one[0].Subject.ID, "mesh-cata") {
t.Fatalf("the gap raised %+v, not one condition naming its repository and commit", one)
}
catch()
if again := raised(); len(again) != 1 || again[0].Observations < 2 {
t.Fatalf("a second catch-up left %+v: one condition, observed twice, was wanted", again)
}
tr.approved["mesh-catalog-c01"] = true
catch()
if !mergeKnown(t, open, "mesh-catalog-c01") {
t.Fatal("the gap was not planned once its Approve was on record")
}
if left := raised(); len(left) != 0 {
t.Fatalf("planning the gap left its condition open: %+v", left)
}
}
// Clause 3: a forge that does not answer leaves the catch-up undone and says so, and the next start does it.
func TestAForgeThatDoesNotAnswerLeavesTheCatchUpForTheNextStart(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksRecorded(t)
tr := aTrunkOf(t, "mesh-catalog", "notes")
tr.silent = true
if later := (following{open: open}).catchUp(t.Context(), nil); !later {
t.Fatal("a forge that did not answer counted as nothing to catch up")
}
if mergeKnown(t, open, "mesh-catalog-c01") {
t.Fatal("a merge was planned without the forge's word")
}
tr.silent = false
if later := (following{open: open}).catchUp(t.Context(), nil); later || !mergeKnown(t, open, "mesh-catalog-c01") {
t.Fatalf("the next start did not catch up (left for later: %v)", later)
}
}
// Finding 2 of the review: a gap that adds a module is a gap, judged as hearMerge judges it. With the operator's Approve on record it
// is heard (hearMerge asks the new module in its walk); without, it raises the condition and is not lost in silence.
func TestAGapThatAddsAModuleIsHeardOnItsWarrantAndRaisedWithout(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asksRecorded(t)
keeper, _ := withConditionsInMemory(t)
tr := aTrunkOf(t, "mesh-catalog", "newmodule")
tr.approved["mesh-catalog-c01"] = false
catch := func() { (following{open: open}).catchUp(t.Context(), nil) }
catch()
if mergeKnown(t, open, "mesh-catalog-c01") {
t.Fatal("a merge adding a module was heard with no Approve on record")
}
all, err := keeper.Open(t.Context())
if err != nil {
t.Fatal(err)
}
var raised int
for _, c := range all {
if c.Kind == kindMergeNotHeard {
raised++
}
}
if raised != 1 {
t.Fatalf("a merge adding a module with no Approve raised %d condition(s), not one: it was lost in silence", raised)
}
tr.approved["mesh-catalog-c01"] = true
catch()
if !mergeKnown(t, open, "mesh-catalog-c01") {
t.Fatal("a merge adding a module was not heard once its Approve was on record")
}
}
-294
View File
@@ -1,294 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
sdkdelivery "git.novox.be/novox/mesh-sdk/go/delivery"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// The pieces of the catch-up that need no store (novox/hq ADR 0457, issue 1336).
func TestAMergeCommitsSubjectNamesItsPullRequest(t *testing.T) {
title, number, ok := pullOfSubject("Merge pull request 'Fix keycloak's sign-in (issue 1332)' (#378) from fix/1332-sign-in into main")
if !ok || number != 378 || title != "Fix keycloak's sign-in (issue 1332)" {
t.Fatalf("read %q %d %v", title, number, ok)
}
if b := headBranchOfSubject("Merge pull request 'x' (#378) from fix/1332-sign-in into main"); b != "fix/1332-sign-in" {
t.Fatalf("the head branch read as %q", b)
}
if _, _, ok := pullOfSubject("Merge branch 'main' of somewhere"); ok {
t.Fatal("a merge that names no pull request read as one")
}
}
func TestOnlyTheLastFiftyMergesOfATrunkAreRead(t *testing.T) {
pages := map[int][]trunkCommit{}
var all []trunkCommit // newest first: 51 merges, a plain commit between each pair
for i := 51; i >= 1; i-- {
all = append(all, trunkCommit{Commit: fmt.Sprintf("merge-%02d", i), SecondParent: fmt.Sprintf("head-%02d", i)},
trunkCommit{Commit: fmt.Sprintf("plain-%02d", i)})
}
for i := 0; i < len(all); i += 50 {
pages[i/50+1] = all[i:min(i+50, len(all))]
}
was := trunkCommitsPage
trunkCommitsPage = func(_ context.Context, _, _, _ string, page int) ([]trunkCommit, error) { return pages[page], nil }
t.Cleanup(func() { trunkCommitsPage = was })
got, err := readTrunkMerges(t.Context(), "novox", "mesh-catalog", "main")
if err != nil {
t.Fatal(err)
}
if len(got) != catchUpWindow || got[0].Commit != "merge-02" || got[len(got)-1].Commit != "merge-51" {
t.Fatalf("read %d merges from %s to %s: the oldest of 51 is outside the window", len(got), got[0].Commit, got[len(got)-1].Commit)
}
if got[0].Head != "head-02" {
t.Fatalf("a merge's head is its second parent, not %q", got[0].Head)
}
}
func TestAForgeThatDoesNotAnswerIsAnErrorNotAnEmptyTrunk(t *testing.T) {
was := trunkCommitsPage
trunkCommitsPage = func(context.Context, string, string, string, int) ([]trunkCommit, error) {
return nil, errors.New("the forge did not answer")
}
t.Cleanup(func() { trunkCommitsPage = was })
if got, err := readTrunkMerges(t.Context(), "novox", "mesh-catalog", "main"); err == nil {
t.Fatalf("a forge that did not answer read as %d merges: nothing to catch up", len(got))
}
}
func TestACaughtUpMergeIsAcceptedOnTheSameWarrantAsAnAnnouncedOne(t *testing.T) {
ok := approvedMerge(t)
for _, asker := range []string{sdkdelivery.TellerModule, theDelivery} {
w := ok.warrant
w.Asker = asker
if _, err := caughtUpWarrant(asker, &ok.ask, &w); err != nil {
t.Errorf("the operator's Approve under %s was refused: %v", asker, err)
}
}
if _, err := caughtUpWarrant("keycloak", &ok.ask, &ok.warrant); err == nil {
t.Error("a warrant of an asker the controller takes none from was accepted")
}
declined := ok.warrant
declined.Option, declined.Label = "decline", "Decline"
if _, err := caughtUpWarrant(theDelivery, &ok.ask, &declined); err == nil {
t.Error("a Decline was accepted as a warrant")
}
unproved := ok.warrant
by := *unproved.By
by.Verified = ""
unproved.By = &by
if _, err := caughtUpWarrant(theDelivery, &ok.ask, &unproved); err == nil {
t.Error("an answer whose channel proves nobody was accepted")
}
if _, err := caughtUpWarrant(theDelivery, nil, nil); err == nil {
t.Error("a router that holds nothing was accepted")
}
}
// aTellersAsk is the ask the delivery gate composes for a pull request, as the router holds it once the operator chose Approve
// (the gate's composeMergeAsk: its id from sdkdelivery.AskingID of the group and the heads, its Approve bound to mergeAct over the
// heads, the pull request and its head's first twelve digits in the details), with the warrant the router wrote.
func aTellersAsk(t *testing.T, group string, members ...sdkdelivery.Head) (asks.Ask, asks.Warrant) {
t.Helper()
key, err := sdkdelivery.MergeAskKey(group, members)
if err != nil {
t.Fatal(err)
}
id := sdkdelivery.AskingID(key, 1)
heads := map[string]string{}
var details []string
for _, m := range members {
owner, repo, _ := strings.Cut(m.Repository, "/")
heads[deliveryIDOf(owner, repo, m.Head)] = m.Head
details = append(details, fmt.Sprintf("%s#%d %q at %s: issues: none named; review: approved; reaches: no delivery plan yet", repo, m.Number, "title", m.Head[:12]))
}
bind := func(option string) string {
d, err := asks.ActDigest(mergeAct(id, option, heads))
if err != nil {
t.Fatal(err)
}
return d
}
at := time.Date(2026, 10, 20, 12, 2, 1, 0, time.UTC)
ask := asks.Ask{ID: id, Headline: "Merge into main", Explanation: "passed its checks.", Who: asks.Operator, Expires: at.Add(24 * time.Hour),
OnExpiry: "nothing is merged; the ask is made again", Details: strings.Join(details, "\n"),
Options: []asks.Option{
{ID: optionApprove, Label: "Approve", Does: "merges", Level: asks.Approve, Binds: bind(optionApprove)},
{ID: "decline", Label: "Decline", Does: "nothing is merged", Level: asks.Approve, Binds: bind("decline")}}}
w := asks.Warrant{Ask: id, Asker: sdkdelivery.TellerModule, Outcome: asks.OutcomeChosen, Option: optionApprove, Label: "Approve",
Level: asks.Approve, By: &asks.Person{Who: asks.Operator, Kind: "telegram", Identity: "1", Verified: "user id"},
At: at.Add(time.Minute), AskDigest: ask.Digest()}
return ask, w
}
// withRouter hands findMergeWarrant these asks as the router's records, by asker.
func withRouter(t *testing.T, records map[string]map[string]struct {
ask asks.Ask
w asks.Warrant
}) {
t.Helper()
wasIDs, wasAsk := routerAskIDs, routerAskOf
closedRecords.Range(func(k, _ any) bool { closedRecords.Delete(k); return true })
routerAskIDs = func(_ context.Context, _ catalogueReader, asker string) ([]string, error) {
var ids []string
for id := range records[asker] {
ids = append(ids, id)
}
return ids, nil
}
routerAskOf = func(_ context.Context, _ catalogueReader, asker, id string) (*asks.Ask, *asks.Warrant, error) {
r, ok := records[asker][id]
if !ok {
return nil, nil, nil
}
return &r.ask, &r.w, nil
}
t.Cleanup(func() {
routerAskIDs, routerAskOf = wasIDs, wasAsk
closedRecords.Range(func(k, _ any) bool { closedRecords.Delete(k); return true })
})
}
type aRouterRecord = struct {
ask asks.Ask
w asks.Warrant
}
// The teller's asks, as the router held them on 2026-10-20: the id is the router's own key, rebuilt here from the group and the
// head the gate composed it for (mesh-controller#405 at 8a330afd, mesh-catalog#381 at 1c702422, mesh-host#124 at 52600ec7). It names
// neither the pull request nor the head, so the warrant is found by the heads the Approve is bound to.
func TestATellersWarrantIsFoundByTheHeadItsApproveIsBoundTo(t *testing.T) {
real := []struct {
group, key, owner, repo string
number int
head string
}{
{"feat/r4b-sdk-users", "merge-feat_r4b_sdk_users-ba59b03220f8-989afe0a3bcf", "novox", "mesh-controller", 405, "8a330afd44ba852873339cf1210e90989dca46e1"},
{"fix/1351-remove-todays-delivery-module", "merge-fix_1351_remove_todays_delivery_module-64a353-d93b97b9697b", "novox", "mesh-catalog", 381, "1c7024224dd77c5b603299378883e0993b2e2fdd"},
{"feat/r4b-sdk-users", "merge-feat_r4b_sdk_users-6a59ba48da14-424e053ebc9c", "novox", "mesh-host", 124, "52600ec7bd61ee271a5cc7f4136bcf0a2a8f00fc"},
}
teller := map[string]aRouterRecord{}
for _, r := range real {
ask, w := aTellersAsk(t, r.group, sdkdelivery.Head{Repository: r.owner + "/" + r.repo, Number: r.number, Head: r.head})
if ask.ID != r.key {
t.Fatalf("the id of %s#%d is %q, and the router held it as %q: the fixture is not the router's", r.repo, r.number, ask.ID, r.key)
}
teller[ask.ID] = aRouterRecord{ask, w}
}
withRouter(t, map[string]map[string]aRouterRecord{sdkdelivery.TellerModule: teller})
for _, r := range real {
why, found, reason, err := findMergeWarrant(t.Context(), nil, r.owner, r.repo, r.number, r.head)
if err != nil || !found || !strings.Contains(why, r.key) || !strings.Contains(why, "bound to the head of this pull request") {
t.Errorf("%s#%d: the warrant was not found: %q %v %q %v", r.repo, r.number, why, found, reason, err)
}
// Another head of the same pull request is another merge: the Approve was not given for it.
if _, found, _, _ := findMergeWarrant(t.Context(), nil, r.owner, r.repo, r.number, "ffffffffffffffffffffffffffffffffffffffff"); found {
t.Errorf("%s#%d: another head found the warrant of this one", r.repo, r.number)
}
}
// One pull request's Approve is none for another repository's merge at the same head digits.
if _, found, _, _ := findMergeWarrant(t.Context(), nil, "novox", "mesh-tools", 405, real[0].head); found {
t.Error("a merge of another repository found mesh-controller#405's warrant")
}
}
// A group's ask binds several heads, and the router's record carries only twelve digits of the other members': the pull request is
// found in the ask as it was published, which the warrant's digest covers.
func TestAGroupsWarrantIsFoundByTheAsksWordsForThePullRequestAndItsHead(t *testing.T) {
a := sdkdelivery.Head{Repository: "novox/mesh-host", Number: 124, Head: "52600ec7bd61ee271a5cc7f4136bcf0a2a8f00fc"}
b := sdkdelivery.Head{Repository: "novox/mesh-controller", Number: 405, Head: "8a330afd44ba852873339cf1210e90989dca46e1"}
ask, w := aTellersAsk(t, "feat/r4b-sdk-users", a, b)
withRouter(t, map[string]map[string]aRouterRecord{sdkdelivery.TellerModule: {ask.ID: {ask, w}}})
for _, h := range []sdkdelivery.Head{a, b} {
owner, repo, _ := strings.Cut(h.Repository, "/")
why, found, reason, err := findMergeWarrant(t.Context(), nil, owner, repo, h.Number, h.Head)
if err != nil || !found || !strings.Contains(why, "a group's ask naming this pull request at this head") {
t.Errorf("%s#%d: not found in the group's ask: %q %v %q %v", repo, h.Number, why, found, reason, err)
}
}
if _, found, _, _ := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 381, a.Head); found {
t.Error("a pull request the group's ask does not name found its warrant")
}
// A Decline is no warrant.
declined := w
declined.Option, declined.Label = "decline", "Decline"
withRouter(t, map[string]map[string]aRouterRecord{sdkdelivery.TellerModule: {ask.ID: {ask, declined}}})
if _, found, _, _ := findMergeWarrant(t.Context(), nil, "novox", "mesh-host", 124, a.Head); found {
t.Error("a Decline found a warrant")
}
}
// Keycloak's merge was asked as mesh-delivery.merge-378-b9104454f9a5-1, before release 3e: the id names the pull request and
// the first twelve digits of its head, and the warrant is found by them under mesh-delivery.
func TestAWarrantAskedUnderMeshDeliveryIsFoundByThePullRequestAndItsHead(t *testing.T) {
ok := approvedMerge(t)
head := "b9104454f9a5e0c4d3b2a1f0e9d8c7b6a5948372"
askID := "merge-378-b9104454f9a5-1"
ask, w := ok.ask, ok.warrant
ask.ID, w.Ask, w.Asker = askID, askID, theDelivery
w.AskDigest = ask.Digest()
withRouter(t, map[string]map[string]aRouterRecord{theDelivery: {
"merge-377-aaaaaaaaaaaa-1": {},
askID: {ask, w},
}})
why, found, reason, err := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, head)
if err != nil || !found || !strings.Contains(why, askID) {
t.Fatalf("the warrant was not found: %q %v %q %v", why, found, reason, err)
}
// Another head names another ask, and none is found.
if _, found, _, _ := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, "ffffffffffff0000"); found {
t.Fatal("another head found a warrant")
}
// A router that cannot be read is an error, never "no warrant".
routerAskIDs = func(context.Context, catalogueReader, string) ([]string, error) { return nil, errors.New("no answer") }
if _, found, _, err := findMergeWarrant(t.Context(), nil, "novox", "mesh-catalog", 378, head); found || err == nil {
t.Fatalf("an unreadable router read as no warrant: %v %v", found, err)
}
}
// A trunk shorter than a page, or with none, is read without a fault: the pages end when the forge's do.
func TestAShortTrunkIsReadWholeAndAnEmptyOneIsNoMerges(t *testing.T) {
was := trunkCommitsPage
t.Cleanup(func() { trunkCommitsPage = was })
trunkCommitsPage = func(_ context.Context, _, _, _ string, page int) ([]trunkCommit, error) {
if page > 1 {
return nil, nil
}
return []trunkCommit{{Commit: "m2", SecondParent: "h2"}, {Commit: "p"}, {Commit: "m1", SecondParent: "h1"}}, nil
}
got, err := readTrunkMerges(t.Context(), "novox", "x", "main")
if err != nil || len(got) != 2 || got[0].Commit != "m1" || got[1].Commit != "m2" {
t.Fatalf("a short trunk read as %+v %v", got, err)
}
trunkCommitsPage = func(context.Context, string, string, string, int) ([]trunkCommit, error) { return nil, nil }
if got, err := readTrunkMerges(t.Context(), "novox", "x", "main"); err != nil || len(got) != 0 {
t.Fatalf("an empty trunk read as %+v %v", got, err)
}
}
// ADR 0253: what the operator reads of a merge not heard is plain: no hash, no identifier, nothing to do "by hand".
func TestTheConditionOfAMergeNotHeardIsInPlainWords(t *testing.T) {
keeper, _ := withConditionsInMemory(t)
tm := trunkMerge{Commit: "c0880c4d1f2e3a4b5c6d7e8f9a0b1c2d3e4f5a6b", Head: "b9104454f9a5e0c4d3b2a1f0e9d8c7b6a5948372"}
(following{}).raiseMergeNotHeard(t.Context(), "novox/mesh-catalog", "main", tm, link.SourceMoved{Number: 378},
"the router holds no chosen merge ask of mesh-delivery-teller or mesh-delivery that covers the head b9104454f9a5")
open, err := keeper.Open(t.Context())
if err != nil || len(open) != 1 {
t.Fatalf("opened %d conditions, %v", len(open), err)
}
c := open[0]
if why, ok := conditions.PlainWords(conditions.Words{Headline: c.Headline, Explanation: c.Explanation, Needs: c.Needs,
Resolved: c.Resolved}); !ok {
t.Fatalf("the condition's words are not plain: %s\n%q\n%q\n%q", why, c.Headline, c.Explanation, c.Needs)
}
}
-177
View File
@@ -1,177 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"time"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// The chain on the bus (novox/hq ADR 0408, design 47 milestone 5): builds, rollouts and assignments are keys of the
// bus's key-value store, written by the controller that holds the lease, and the controller's store keeps none of them.
// chainHolds is whether this process may write the chain now: the lease's gate, the one every act passes.
func chainHolds(ctx context.Context) error {
_, err := theLease.epoch(ctx)
return err
}
// openTheChain opens the chain's buckets on the bus: the serving controller's connection, or one dialled for a command.
func openTheChain(ctx context.Context) (*chain.Chain, error) {
js, err := aBus()
if err != nil {
return nil, err
}
// Not closed: a borrowed connection closes nothing, and a dialled one lasts as long as the command.
return chain.OnTheBus(ctx, js.Conn(), chainHolds)
}
// attachTheChain gives the inventory of the serving controller the chain on its connection.
func attachTheChain(ctx context.Context, inv *inventory.Inventory, server *link.Server) error {
bus, ok := server.Bus().(link.OverNATS)
if !ok {
return errors.New("the chain lives on the bus, and this control plane is not on it")
}
c, err := chain.OnTheBus(ctx, bus.Conn, chainHolds)
if err != nil {
return err
}
inv.UseChain(c)
inv.RequireLayout()
// And the builds laid out under their modules, once, before they are read (issue 1121) — **before this returns**, so
// nothing the start goes on to read meets a chain not laid out yet: on 2026-10-18 the layout ran beside the start, a
// read of the start was refused, and the controller exited and was started again every five seconds. The layout of
// keys laid out before skips them, so this takes seconds; one that fails is tried again beside the serving, loudly.
if err := inv.LayBuildsOut(ctx, func(format string, args ...any) { fmt.Printf("chain: "+format+"\n", args...) }); err != nil {
fmt.Printf("chain: the builds are not laid out under their modules yet, and none is read until they are: %v\n", err)
go layBuildsOutSoon(ctx, inv)
}
// And the requests the move of delivery 1 found never handed over, asked again once (askAgain), once it is serving.
go askAgainSoon(ctx, inv)
// And what delivery 1 recorded of what each machine was sent, copied onto the keys of the assignments (CopySentBuilds).
go copySentBuildsSoon(ctx, inv)
return nil
}
// askAgain asks once more for each build request the move found never handed over (novox/hq issue 981 note 7 Q5): the
// requests a merge could not ask for, which a person's hand `build` would otherwise have to repeat. **At most once**: a
// request is claimed before it is asked and settled after, so a crash between loses the ask, which is where it stood.
func askAgain(ctx context.Context, inv *inventory.Inventory) {
pending, err := inv.RequestsToAskAgain(ctx)
if err != nil {
fmt.Printf("chain: the requests never handed over cannot be read: %v\n", err)
return
}
for _, r := range pending {
if claimed, err := inv.ClaimToAskAgain(ctx, r.ID); err != nil || !claimed {
if err != nil {
fmt.Printf("chain: %s cannot be claimed to be asked again: %v\n", r.ID, err)
}
continue
}
id, err := askABuild(ctx, buildSource{Repository: r.Repository, Seat: r.Seat}, r.Path, r.Ref)
if err != nil {
if serr := inv.AskedAgainAs(ctx, r.ID, "not asked again: "+err.Error()); serr != nil {
fmt.Printf("chain: %s was not asked again, and that could not be kept: %v\n", r.ID, serr)
}
fmt.Printf("chain: %s was never handed over and cannot be asked again: %v\n", r.ID, err)
continue
}
recordBuildRequest(ctx, inv, inventory.BuildRequest{ID: id, Repository: r.Repository, Seat: r.Seat, Path: r.Path,
Ref: r.Ref, Commit: r.Commit, For: r.For})
if err := inv.AskedAgainAs(ctx, r.ID, id); err != nil {
fmt.Printf("chain: %s was asked again as %s, and that could not be kept: %v\n", r.ID, id, err)
}
fmt.Printf("chain: %s, never handed over, is asked again as %s\n", r.ID, id)
}
}
// askAgainSoon waits for the controller to be serving, which an ask needs, and then asks again.
func askAgainSoon(ctx context.Context, inv *inventory.Inventory) {
select {
case <-ctx.Done():
case <-time.After(time.Minute):
askAgain(ctx, inv)
}
}
// kindSentBuildsNotCopied is the condition of the one-time copy of node.sent_builds onto the assignments' keys that has not
// succeeded: until it has, a module an assignment pulled in may be missing from what a machine is read to run.
const kindSentBuildsNotCopied = "sent-builds-not-copied"
// copySentBuildsSoon copies what the store recorded of what each machine was sent onto the keys of the assignments, as soon
// as this controller holds the lease (the chain refuses a write before), and tries again every half minute while it fails,
// each time it fails saying so as an urgent condition until it has worked. It is idempotent: CopySentBuilds overwrites no key
// that names a commit and empties each column it has copied.
func copySentBuildsSoon(ctx context.Context, inv *inventory.Inventory) {
failed := false
for {
_, err := inv.CopySentBuilds(ctx)
o := conditions.Observation{Scope: conditions.ScopeBuild, ID: kindSentBuildsNotCopied, Token: kindSentBuildsNotCopied,
Kind: kindSentBuildsNotCopied, Severity: conditions.Urgent, Resolver: conditions.ResolverOperator,
Source: "chain", Summary: "what each machine was sent is not copied onto its assignments: "}
if err != nil {
fmt.Printf("chain: what the machines were sent is not copied onto the assignments: %v\n", err)
if keeper := conditionsFrom; keeper != nil && !failed {
o.Summary += err.Error()
if _, oerr := keeper.Observe(ctx, o); oerr == nil {
failed = true
}
}
} else {
if keeper := conditionsFrom; keeper != nil && failed {
_, _ = keeper.Clear(ctx, o.Key(), "copied")
}
return
}
select {
case <-ctx.Done():
return
case <-time.After(chainMoveRetry):
}
}
}
// chainMoveRetry is how long the copy waits before it tries again.
const chainMoveRetry = 30 * time.Second
// kindBuildsNotLaidOut is the condition of the layout of the builds under their modules that has not succeeded: until it
// has, no build is read, and a send that needs one is refused with the reason.
const kindBuildsNotLaidOut = "builds-not-laid-out"
// layBuildsOutSoon lays the builds out under their modules as soon as this controller holds the lease (the chain refuses a
// write before), and tries again every half minute while it fails, loudly: each failure is printed, and the first raises an
// urgent condition until it has worked. It is idempotent (LayBuildsOut).
func layBuildsOutSoon(ctx context.Context, inv *inventory.Inventory) {
failed := false
for {
err := inv.LayBuildsOut(ctx, func(format string, args ...any) { fmt.Printf("chain: "+format+"\n", args...) })
o := conditions.Observation{Scope: conditions.ScopeBuild, ID: kindBuildsNotLaidOut, Token: kindBuildsNotLaidOut,
Kind: kindBuildsNotLaidOut, Severity: conditions.Urgent, Resolver: conditions.ResolverOperator,
Source: "chain", Summary: "the builds are not laid out under their modules, so none is read: "}
if err != nil {
fmt.Printf("chain: the builds are not laid out under their modules, and none is read until they are: %v\n", err)
if keeper := conditionsFrom; keeper != nil && !failed {
o.Summary += err.Error()
if _, oerr := keeper.Observe(ctx, o); oerr == nil {
failed = true
}
}
} else {
if keeper := conditionsFrom; keeper != nil && failed {
_, _ = keeper.Clear(ctx, o.Key(), "laid out")
}
return
}
select {
case <-ctx.Done():
return
case <-time.After(chainMoveRetry):
}
}
}
@@ -1,245 +0,0 @@
//go:build integration
package main
import (
"context"
"fmt"
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/lease"
"github.com/novox/mesh-controller/internal/link"
)
// **Acceptance 10 of design 47: one read of an assignment names the commit, the build and the rollout that put it there,
// and one read of the rollout lists the assignments it changed** (novox/hq ADR 0408, issue 1095). It enters by the live
// trigger — a merge heard, its batch cut into a plan when the window closed, its build answered through the build seat's
// path, the plan advanced — and writes no key and no row by hand. Only the machine's end of the send is the test's: the send
// is recorded as the controller records one it made, and the gate's facts are the store's.
func TestOneReadOfAnAssignmentNamesItsCommitBuildAndRolloutAndOneReadOfTheRolloutListsItsAssignments(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksRecorded(t)
ctx := t.Context()
inv := open.inventory
if _, err := assign(ctx, open, "anchor", "notes"); err != nil {
t.Fatal(err)
}
var sent [][]string
wasSend := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
sent = append(sent, append([]string(nil), names...))
current, err := open.inventory.CurrentBuilds(ctx)
if err != nil {
return nil, err
}
for _, node := range names {
digest := fmt.Sprintf("d-%s-%d", node, len(sent))
if err := open.inventory.RecordSent(ctx, nodeID(t, open, node), digest,
map[string]string{"notes": current["notes"].Commit}); err != nil {
return nil, err
}
}
return names, nil
}
t.Cleanup(func() { sendRollout = wasSend })
wasGather := gatherGateFacts
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
f := gateFacts{now: time.Now(), reports: map[string]inventory.Reported{}, engines: map[string]string{},
rolledBack: map[string][]lease.Rollback{}, served: map[string]served{}}
reports, err := open.inventory.LastReports(ctx)
if err != nil {
return f, err
}
for _, r := range reports {
f.reports[r.Node] = r
}
return f, nil
}
t.Cleanup(func() { gatherGateFacts = wasGather })
// A merge of notes is heard, and its window closes: the batch is cut into the walk.
hear(t, open, catalogueMerge("c-notes", "notes", t0), t0.Add(time.Second))
cutAt(t, open, t0.Add(time.Second+90*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walk(s) after the cut", len(ws))
}
rollout := ws[0].ID
// The walk asks its first tier (at the cut, or at its next step); the build seat answers, and the controller takes the
// outcome in as it does.
asking, err := inv.PlanByID(ctx, rollout)
if err != nil {
t.Fatal(err)
}
for try := 0; try < 3 && (asking.Modules["notes"] == nil || asking.Modules["notes"].Build == ""); try++ {
advancePlans(ctx, open)
if asking, err = inv.PlanByID(ctx, rollout); err != nil {
t.Fatal(err)
}
}
buildID := asking.Modules["notes"].Build
if buildID == "" || len(*asked) != 1 || !strings.HasSuffix((*asked)[0], "#"+buildID) {
t.Fatalf("the walk asked %v, and notes' build is %q", *asked, buildID)
}
digest := "sha256:" + strings.Repeat("a", 64)
result := outcome(buildID, "modules/notes", "c-notes", "notes", "")
result.ReuseKey = "rk-notes"
result.Made = []link.MadeArtifact{{Name: "notes", Kind: "image", Reference: "artifact-store://notes/notes@" + digest}}
if err := (builds{inv: inv, open: open}).Built(ctx, result); err != nil {
t.Fatal(err)
}
// The walk sends the tier to its first machine.
for try := 0; try < 3 && len(sent) == 0; try++ {
advancePlans(ctx, open)
}
if !slices.Equal(slices.Concat(sent...), []string{"anchor"}) {
t.Fatalf("sent %v", sent)
}
// One read of the assignment.
on, err := inv.AssignmentsOn(ctx, "anchor")
a, ok := assignmentOf(on, "notes")
if err != nil || !ok {
t.Fatalf("the assignments of anchor: %+v %v", on, err)
}
// Beside it a module pulled in by the assignment, which no send of this walk put anywhere.
if wireguard, pulled := assignmentOf(on, "mesh-wireguard"); pulled && (wireguard.Build != "" || wireguard.Rollout != "") {
t.Fatalf("a module this rollout did not send names a build or a rollout: %+v", wireguard)
}
if a.Module != "notes" || a.Commit != "c-notes" || a.Build != buildID || a.Rollout != rollout || a.Artifact != digest ||
a.Since.IsZero() || a.Assigned.IsZero() || !a.Stands() {
t.Fatalf("the assignment names %+v, not commit c-notes, build %s, artifact %s, rollout %s", a, buildID, digest, rollout)
}
// One read of the rollout.
p, err := inv.PlanByID(ctx, rollout)
if err != nil {
t.Fatal(err)
}
wantChange := chain.Change{Node: "anchor", Module: "notes", After: chain.Ref{Build: buildID, Commit: "c-notes"}}
if !slices.Contains(p.Changed, wantChange) || !slices.Contains(p.Builds, buildID) || len(p.Steps) != 1 ||
p.Steps[0].Node != "anchor" || p.Steps[0].Sent == nil {
t.Fatalf("the rollout lists changed %+v, builds %v, steps %+v; want %+v", p.Changed, p.Builds, p.Steps, wantChange)
}
// And one read of the build names the same commit, with what it published and its reuse key.
built, found, err := inv.BuildByID(ctx, buildID)
if err != nil || !found || built.Commit != "c-notes" || built.Module != "notes" {
t.Fatalf("the build: %+v %v %v", built, found, err)
}
lately, err := inv.Builds(ctx, "notes", 1)
if err != nil || len(lately) != 1 || lately[0].ReuseKey != "rk-notes" || lately[0].Published == nil ||
lately[0].Published.Artifact != digest {
t.Fatalf("what the build published: %+v %v", lately, err)
}
// The verbs say the links.
if lines := changedLines(p); len(lines) != 2 || !strings.Contains(lines[1], "anchor/notes") ||
!strings.Contains(lines[1], buildID) {
t.Fatalf("`plans` says %v", lines)
}
if lines := runLines(on); len(lines) != 2 || !strings.Contains(lines[1], buildID) || !strings.Contains(lines[1], rollout) {
t.Fatalf("`node show` says %v", lines)
}
}
// A machine sent again what it runs says nothing new, and a push by hand with no rollout fills what is missing and blanks
// nothing (issue 1095).
func TestASendOfWhatAMachineRunsAlreadyChangesNoLinkOfItsAssignment(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: "notes", Version: "1"})
if _, err := assign(ctx, open, "anchor", "notes"); err != nil {
t.Fatal(err)
}
if err := inv.RecordBuild(ctx, inventory.Build{ID: "build-1", Module: "notes", Commit: "c1", Repository: "novox/notes"}); err != nil {
t.Fatal(err)
}
node := nodeID(t, open, "anchor")
inRollout := inventory.WithRollout(ctx, "plan-1", map[string]inventory.RolloutModule{"notes": {Commit: "c1", Build: "build-1"}})
if err := inv.RecordSent(inRollout, node, "d1", map[string]string{"notes": "c1"}); err != nil {
t.Fatal(err)
}
// Sent again by hand, with no rollout: the assignment still names the rollout that put the build there.
if err := inv.RecordSent(ctx, node, "d2", map[string]string{"notes": "c1"}); err != nil {
t.Fatal(err)
}
on, _ := inv.AssignmentsOn(ctx, "anchor")
if a, _ := assignmentOf(on, "notes"); a.Rollout != "plan-1" || a.Build != "build-1" || a.Commit != "c1" {
t.Fatalf("the assignment names %+v", on)
}
// A declaration sent by hand with no builds known makes what the machine runs unknown: a time on the key, no commit.
if err := inv.RecordSent(ctx, node, "d3", nil); err != nil {
t.Fatal(err)
}
if on, _ = inv.AssignmentsOn(ctx, "anchor"); runOf(on, "notes").Commit != "" || runOf(on, "notes").SentUnknown.IsZero() {
t.Fatalf("a send of unknown builds left %+v", on)
}
// A module sent a commit no build is kept of is named by its commit alone.
if err := inv.RecordSent(ctx, node, "d4", map[string]string{"notes": "c2"}); err != nil {
t.Fatal(err)
}
if on, _ = inv.AssignmentsOn(ctx, "anchor"); runOf(on, "notes").Commit != "c2" || runOf(on, "notes").Build != "" ||
runOf(on, "notes").Rollout != "" {
t.Fatalf("the assignment after a push of c2 by hand names %+v", on)
}
}
// assignmentOf is one module's assignment among those of a machine: assigning a module assigns what it requires beside it.
func assignmentOf(on []chain.Assignment, module string) (chain.Assignment, bool) {
for _, a := range on {
if a.Module == module {
return a, true
}
}
return chain.Assignment{}, false
}
func runOf(on []chain.Assignment, module string) chain.Assignment {
a, _ := assignmentOf(on, module)
return a
}
// **An assignment waiting for its build is one key of the assignments bucket** (ADR 0261, ADR 0408, issue 1094): entered
// through the `assign` verb for a module whose build is in flight, one read of `<node>.<module>` says state pending with the
// build it waits for, the build's outcome through the build seat's path makes the key stand with the wait carried as applied,
// and the table of pending assignments is read by nobody. No key is written by hand.
func TestAnAssignmentWaitingForItsBuildIsOneKeyOfTheAssignmentsBucket(t *testing.T) {
open := aCatalogueMesh(t)
ctx := t.Context()
inv := open.inventory
asksWithPaths(t)
mergeAdding(t, open, "modules/sensors", "3da80a4b00aa")
if _, err := assign(ctx, open, "laptop", "sensors"); err != nil {
t.Fatalf("an assignment while its build runs was refused: %v", err)
}
waiting, found, err := inv.AssignmentValue(ctx, "laptop", "sensors")
if err != nil || !found || waiting.State != chain.StatePending || waiting.Stands() || !waiting.Assigned.IsZero() ||
waiting.Pending == nil || waiting.Pending.State != chain.PendingWaiting || waiting.Pending.Build != "b-modules/sensors" ||
!strings.HasPrefix(waiting.Pending.ID, "p-") {
t.Fatalf("one read of the key says %+v (found %v, %v)", waiting, found, err)
}
if got := assignedTo(t, open, "laptop"); slices.Contains(got, "sensors") {
t.Fatalf("a module only waiting is assigned: %v", got)
}
if err := (builds{inv: inv, open: open}).Built(ctx, outcome("b-modules/sensors", "modules/sensors", "3da80a4b00aa", "sensors", "")); err != nil {
t.Fatal(err)
}
made, found, err := inv.AssignmentValue(ctx, "laptop", "sensors")
if err != nil || !found || !made.Stands() || made.State != "" || made.Assigned.IsZero() || made.Pending == nil ||
made.Pending.State != chain.PendingApplied || made.Pending.ID != waiting.Pending.ID {
t.Fatalf("one read of the key after the build says %+v (found %v, %v)", made, found, err)
}
if !slices.Contains(assignedTo(t, open, "laptop"), "sensors") {
t.Fatal("the build registered sensors and it was not assigned")
}
}
-121
View File
@@ -1,121 +0,0 @@
package main
import (
"context"
"fmt"
"slices"
"time"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/inventory"
)
// The links a rollout makes (novox/hq ADR 0408 §1, design 47 acceptance 10, issue 1095): the send of a plan's tier puts a
// build on a machine; the assignment of the module there names the build, its commit and the rollout (written by the record of
// the send, inventory.RecordSent, from the context a send is made in), and the rollout lists the assignments it changed with the
// build before and after.
// rolloutModules is what a plan sends of each module: the commit and the build of it.
func rolloutModules(p *inventory.Plan) map[string]inventory.RolloutModule {
out := make(map[string]inventory.RolloutModule, len(p.Modules))
for m, s := range p.Modules {
if s != nil && s.Commit != "" {
out[m] = inventory.RolloutModule{Commit: s.Commit, Build: s.Build}
}
}
return out
}
// sendingAs is the context a plan's send is made in: what is recorded of the send names this rollout.
func sendingAs(ctx context.Context, p *inventory.Plan) context.Context {
return inventory.WithRollout(ctx, p.ID, rolloutModules(p))
}
// ranBefore is what each module on a machine ran before a send, as the chain names it: its assignment's build and commit,
// else the commit the machine was last sent for it. Read before the send, which writes the new ones.
func ranBefore(ctx context.Context, inv *inventory.Inventory, node string) map[string]chain.Ref {
was := map[string]chain.Ref{}
if sent, known, err := inv.SentBuilds(ctx, node); err == nil && known {
for m, commit := range sent {
was[m] = chain.Ref{Commit: commit}
}
}
if ran, err := inv.AssignmentRuns(ctx, node); err == nil {
for m, r := range ran {
was[m] = r
}
}
return was
}
// noteChanges writes into a plan that a send to a machine changed the assignments of the modules it carried: the build and
// commit before and after, the builds the rollout carried, and the step taken. A change noted already for the machine and the
// module is replaced, so a send made again says it once.
func noteChanges(p *inventory.Plan, node string, modules []string, was map[string]chain.Ref, at time.Time) {
for _, m := range modules {
s := p.Modules[m]
if s == nil {
continue
}
change := chain.Change{Node: node, Module: m, Before: was[m], After: chain.Ref{Build: s.Build, Commit: s.Commit}}
if at := slices.IndexFunc(p.Changed, func(c chain.Change) bool { return c.Node == node && c.Module == m }); at >= 0 {
change.Before = p.Changed[at].Before
p.Changed[at] = change
} else {
p.Changed = append(p.Changed, change)
}
if s.Build != "" && !slices.Contains(p.Builds, s.Build) {
p.Builds = append(p.Builds, s.Build)
}
}
if !slices.ContainsFunc(p.Steps, func(s chain.Step) bool { return s.Tier == p.Tier && s.Node == node }) {
sent := at.UTC()
p.Steps = append(p.Steps, chain.Step{Tier: p.Tier, Node: node, Sent: &sent})
}
}
// refSaid is a build and the commit it was made from as a person reads them; "nothing" for a module that ran none.
func refSaid(r chain.Ref) string {
switch {
case r.Build == "" && r.Commit == "":
return "nothing"
case r.Build == "":
return "commit " + short(r.Commit)
}
return r.Build + " (" + short(r.Commit) + ")"
}
// changedLines say the assignments a rollout changed: the machine, the module, and the build before and after.
func changedLines(p inventory.Plan) []string {
if len(p.Changed) == 0 {
return nil
}
out := []string{" changed:"}
for _, c := range p.Changed {
out = append(out, fmt.Sprintf(" %s/%s %s → %s", c.Node, c.Module, refSaid(c.Before), refSaid(c.After)))
}
return out
}
// runLines say what each module assigned to a machine runs there, as its assignment names it: the build, the commit, the
// rollout that put it there and since when.
func runLines(assigned []chain.Assignment) []string {
var out []string
for _, a := range assigned {
if a.Build == "" && a.Commit == "" {
continue
}
line := fmt.Sprintf(" %-22s %s", a.Module, refSaid(chain.Ref{Build: a.Build, Commit: a.Commit}))
if a.Rollout != "" {
line += ", sent by " + a.Rollout
}
if !a.Since.IsZero() {
line += " at " + a.Since.Local().Format("2006-01-02 15:04")
}
out = append(out, line)
}
if len(out) == 0 {
return nil
}
return append([]string{" what they run, as the chain names it:"}, out...)
}
-69
View File
@@ -1,69 +0,0 @@
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/chain"
"github.com/novox/mesh-controller/internal/inventory"
)
// **A rollout lists the assignments it changed, each with the build before and after** (ADR 0408 §1, issue 1095): one
// change for a machine and a module however often the send is noted, the build before kept from the first, the builds the
// rollout carried once each, and one step for a machine and a tier.
func TestARolloutListsTheAssignmentsItChangedWithTheBuildBeforeAndAfter(t *testing.T) {
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
p := inventory.Plan{ID: "plan-1", Tier: 1, Modules: map[string]*inventory.PlanModule{
"notes": {Commit: "c2", Build: "b2"}, "wiki": {Commit: "c2w", Build: "b2w"}, "nothing": nil}}
was := map[string]chain.Ref{"notes": {Build: "b1", Commit: "c1"}, "wiki": {Commit: "c1w"}}
noteChanges(&p, "anchor", []string{"notes", "wiki", "nothing"}, was, at)
noteChanges(&p, "laptop", []string{"notes"}, nil, at)
// The send noted again, after what was sent had become what the machine runs: the first before stands.
noteChanges(&p, "anchor", []string{"notes"}, map[string]chain.Ref{"notes": {Build: "b2", Commit: "c2"}}, at.Add(time.Minute))
want := []chain.Change{
{Node: "anchor", Module: "notes", Before: chain.Ref{Build: "b1", Commit: "c1"}, After: chain.Ref{Build: "b2", Commit: "c2"}},
{Node: "anchor", Module: "wiki", Before: chain.Ref{Commit: "c1w"}, After: chain.Ref{Build: "b2w", Commit: "c2w"}},
{Node: "laptop", Module: "notes", After: chain.Ref{Build: "b2", Commit: "c2"}},
}
if len(p.Changed) != len(want) {
t.Fatalf("changed %+v", p.Changed)
}
for k := range want {
if p.Changed[k] != want[k] {
t.Errorf("change %d is %+v, want %+v", k, p.Changed[k], want[k])
}
}
if len(p.Builds) != 2 || p.Builds[0] != "b2" || p.Builds[1] != "b2w" {
t.Errorf("builds %v", p.Builds)
}
if len(p.Steps) != 2 || p.Steps[0].Node != "anchor" || p.Steps[0].Tier != 1 || p.Steps[0].Sent == nil || p.Steps[1].Node != "laptop" {
t.Errorf("steps %+v", p.Steps)
}
lines := changedLines(p)
if len(lines) != 4 || !strings.Contains(lines[1], "anchor/notes b1 (c1) → b2 (c2)") ||
!strings.Contains(lines[3], "laptop/notes nothing → b2 (c2)") {
t.Errorf("changed lines %q", lines)
}
if got := changedLines(inventory.Plan{}); got != nil {
t.Errorf("a plan that changed nothing says %q", got)
}
}
func TestWhatAMachineRunsIsSaidAsItsAssignmentNamesIt(t *testing.T) {
at := time.Date(2026, 10, 18, 3, 0, 0, 0, time.UTC)
lines := runLines([]chain.Assignment{
{Node: "anchor", Module: "notes", Build: "b2", Commit: "c2c2c2c2c2c2", Rollout: "plan-1", Since: at},
{Node: "anchor", Module: "web", Commit: "c3"},
{Node: "anchor", Module: "never-sent"},
})
if len(lines) != 3 || !strings.Contains(lines[1], "notes") || !strings.Contains(lines[1], "b2 (c2c2c2c2)") ||
!strings.Contains(lines[1], "sent by plan-1") || !strings.Contains(lines[2], "web") || !strings.Contains(lines[2], "commit c3") {
t.Fatalf("lines %q", lines)
}
if runLines([]chain.Assignment{{Node: "a", Module: "m"}}) != nil {
t.Fatal("a module never sent is said to run something")
}
}
@@ -1,85 +0,0 @@
//go:build integration
package main
import (
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/link"
)
// **Acceptance 12 of design 47: no table in the controller's store holds builds, rollouts or assignments** (novox/hq ADR
// 0408, issue 1111). A delivery is driven through the live trigger — a merge heard, its batch cut into a plan, the plan's
// build asked and answered, the assignment made — until a build, a rollout and an assignment exist; they are read from the
// chain, and then the store's own catalog is read: none of the tables of the chain is in it, nor the function that raised the generation from the assignment table; the table that stays, stays.
func TestNoTableOfTheControllersStoreHoldsBuildsRolloutsOrAssignments(t *testing.T) {
open := windowed(t)
everyHeadWaits(t)
asked := asksRecorded(t)
ctx := t.Context()
inv := open.inventory
if _, err := assign(ctx, open, "anchor", "notes"); err != nil {
t.Fatal(err)
}
hear(t, open, catalogueMerge("c-notes", "notes", t0), t0.Add(time.Second))
cutAt(t, open, t0.Add(91*time.Second))
ws, _ := walks(t, open)
if len(ws) != 1 {
t.Fatalf("%d walk(s) after the cut", len(ws))
}
rollout := ws[0].ID
asking, err := inv.PlanByID(ctx, rollout)
if err != nil {
t.Fatal(err)
}
for try := 0; try < 3 && (asking.Modules["notes"] == nil || asking.Modules["notes"].Build == ""); try++ {
advancePlans(ctx, open)
if asking, err = inv.PlanByID(ctx, rollout); err != nil {
t.Fatal(err)
}
}
buildID := asking.Modules["notes"].Build
if buildID == "" || len(*asked) != 1 {
t.Fatalf("the walk asked %v, and notes' build is %q", *asked, buildID)
}
digest := "sha256:" + strings.Repeat("b", 64)
result := outcome(buildID, "modules/notes", "c-notes", "notes", "")
result.Made = []link.MadeArtifact{{Name: "notes", Kind: "image", Reference: "artifact-store://notes/notes@" + digest}}
if err := (builds{inv: inv, open: open}).Built(ctx, result); err != nil {
t.Fatal(err)
}
// What the live path made is in the chain.
if built, found, err := inv.BuildByID(ctx, buildID); err != nil || !found || built.Commit != "c-notes" {
t.Fatalf("the build is not read from the chain: %+v %v %v", built, found, err)
}
if p, err := inv.PlanByID(ctx, rollout); err != nil || p.ID != rollout {
t.Fatalf("the rollout is not read from the chain: %+v %v", p, err)
}
if on, err := inv.AssignmentsOn(ctx, "anchor"); err != nil {
t.Fatal(err)
} else if _, ok := assignmentOf(on, "notes"); !ok {
t.Fatalf("the assignment is not read from the chain: %+v", on)
}
// And the catalog of the store holds none of them.
have := map[string]bool{}
for _, row := range inv.QueryForTest(t, `select table_name::text from information_schema.tables where table_schema = 'public'`) {
have[strings.Trim(row[0], `"`)] = true
}
for _, table := range []string{"build", "build_request", "release_plan", "batched_merge", "assignment", "pending_assignment", "chain_moved"} {
if have[table] {
t.Errorf("the table %s is in the controller's store", table)
}
}
for _, table := range []string{"build_gate", "assignment_generation"} {
if !have[table] {
t.Errorf("the table %s, which stays, is gone", table)
}
}
if rows := inv.QueryForTest(t, `select count(*) from pg_proc where proname = 'raise_assignment_generation'`); rows[0][0] != "0" {
t.Errorf("the function raise_assignment_generation is in the store: %v", rows)
}
}
+1 -6
View File
@@ -7,7 +7,7 @@ import (
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -73,11 +73,6 @@ func changePlanOf(repository, base, head string, r mergeReach, entries []invento
"(issue 280)", busUpgradeNeeded, name, busUpgradeVerb, strings.Join(e.On, ", ")))
case name == "nats" || catalogue.ProvidesBus(e.Manifest):
p.Steps = append(p.Steps, "a planned bus step: the bus is upgraded by `bus upgrade`, never by an ordinary send")
case waits && len(e.On) > 0 && (u.From == catalogue.FromModule || u.From == catalogue.FromData):
// Judged per machine by the walk, against the build each was last sent (novox/hq ADR 0377).
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s) — unless on a machine "+
"its move recreates none of its containers and moves none of its data, which the walk judges and then sends",
name, orNone(u.From)))
case waits && len(e.On) > 0:
p.Steps = append(p.Steps, fmt.Sprintf("%s waits for a person: its policy records (%s)", name,
orNone(u.From)))
+1 -1
View File
@@ -4,7 +4,7 @@ import (
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
+2 -52
View File
@@ -10,8 +10,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/catalogue"
)
// moduleCheck judges manifests where they are written, with no mesh (novox/hq ADR 0037, issue 148).
@@ -48,24 +47,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
shelf := catalogue.Shelf{}
faulted := map[string]bool{}
failed := 0
// The list of predefined keys at the root of the repository above each manifest (novox/hq ADR 0348 §7), read
// once per repository: a manifest naming a key is judged against the list its own tree holds.
lists := map[string]catalogue.PredefinedList{}
listOf := func(path string) (catalogue.PredefinedList, error) {
root := catalogue.RepositoryRoot(filepath.Dir(path))
if root == "" {
return nil, nil
}
if list, read := lists[root]; read {
return list, nil
}
list, err := catalogue.ReadPredefined(root)
if err != nil {
return nil, err
}
lists[root] = list
return list, nil
}
for _, path := range paths {
raw, err := os.ReadFile(path)
if err != nil {
@@ -73,18 +54,7 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed++
continue
}
list, err := listOf(path)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
continue
}
m, err := catalogue.ParseManifestWith(raw, list)
if err == nil {
// Each template's text read from the tree beside the manifest, and judged as inline content is
// (novox/hq ADR 0349 §2): a missing template is refused by name.
m, err = catalogue.WithTemplatesFrom(m, filepath.Dir(path))
}
m, err := catalogue.ParseManifest(raw)
if err != nil {
fmt.Fprintf(out, "%s: %v\n", path, err)
failed++
@@ -115,20 +85,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
failed += len(wrong)
faulted[m.Module] = true
}
// A module lives whole in one repository (novox/hq ADR 0420 §1, issue 1019): when the check is told the
// repository it runs for (MESH_CHECK_REPOSITORY, set by the build's check), an artifact building from
// another is refused, as the merge gate refuses it. A run by hand names no repository and judges none.
if repo := os.Getenv("MESH_CHECK_REPOSITORY"); repo != "" && m.Build != nil {
for _, a := range m.Build.Artifacts {
if a.Context != nil && a.Context.Repository != "" && !sameRepository(a.Context.Repository,
link.SourceMoved{Owner: ownerOf(repo), Repo: repoOf(repo)}) {
fmt.Fprintf(out, "%s: %s builds artifact %s from %s: a module lives whole in one repository "+
"(novox/hq ADR 0420 §1)\n", path, m.Module, a.Name, a.Context.Repository)
failed++
faulted[m.Module] = true
}
}
}
if named := catalogue.InstallationProblems(m); len(named) > 0 {
for _, p := range named {
fmt.Fprintf(out, "%s: %s\n", path, p)
@@ -259,12 +215,6 @@ func moduleCheckFor(paths []string, longestMachine int, out io.Writer) error {
checks = append(checks, fmt.Sprintf("%v by %s", r["id"], catalogue.HealthWords(h)))
}
}
// And its own runtime's, where it runs as its own account (novox/hq ADR 0366).
if own, has := catalogue.OwnRuntimeHealth(m); has {
if h, _, _ := catalogue.ReadHealth(own); h.Kind != "" {
checks = append(checks, fmt.Sprintf("%v by %s", own["id"], catalogue.HealthWords(h)))
}
}
if len(checks) > 0 {
fmt.Fprintf(out, ", ready: %s", strings.Join(checks, "; "))
}
-87
View File
@@ -1,87 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"os"
"strconv"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
)
// A pull request's check builds go when it closes (novox/hq ADR 0350 §5): the build seat keeps them in the artifact
// store's scratch namespace for the walk of its merge to take. **Let go of where the moment is heard**: the
// controller ends the walk answering a merge — after that walk took what it reuses — and the delivery's owner, which
// hears the forge say a pull request closed unmerged, asks the `forget-checks` verb; both let go through the store,
// as the controller's own collection does. The build seat's holders let go of anything a week old on their own,
// so a word lost here leaves a week's builds, never more.
// checkBuildsIn is the scratch namespace as the controller reaches the artifact store.
func checkBuildsIn(ctx context.Context, inv *inventory.Inventory) (builder.Scratch, error) {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return nil, err
}
address, err := artifactStoreAddress(ctx, inv, shelf, "")
if err != nil {
return nil, err
}
if address == "" {
return nil, errors.New("the artifact store is not on the mesh's network")
}
return builder.Registry{Address: address}, nil
}
// forgetChecks lets go of a pull request's check builds and answers what it did.
var forgetChecks = func(ctx context.Context, inv *inventory.Inventory, pull builder.PullRef) (string, error) {
scratch, err := checkBuildsIn(ctx, inv)
if err != nil {
return "", err
}
n, err := scratch.Forget(ctx, pull)
if err != nil {
return "", fmt.Errorf("%s: %d check(s)' builds let go of, and not all: %w", pull, n, err)
}
return fmt.Sprintf("%s: %d check(s)' builds let go of", pull, n), nil
}
// forgetChecksOfWalk lets go, on the save that ends a walk, of the check builds of every pull request whose merge
// it answers: the walk took what it reuses. Said, never fatal.
func forgetChecksOfWalk(ctx context.Context, inv *inventory.Inventory, p inventory.Plan) {
if p.WalkSaid != inventory.WalkEnded || p.Delivery == nil {
return
}
for _, pull := range reuseAskOf(&p).Pulls {
said, err := forgetChecks(ctx, inv, builder.PullRef{Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number})
if err != nil {
fmt.Fprintf(os.Stderr, "%s: its check builds stay until the week's sweep: %v\n", p.ID, err)
continue
}
fmt.Printf("%s: %s\n", p.ID, said)
}
}
// forgetChecksCommand is `forget-checks <owner> <repo> <number>`: the verb the delivery's owner asks when the forge
// says a pull request closed unmerged (novox/hq ADR 0350 §5).
func forgetChecksCommand(ctx context.Context, args []string) error {
if len(args) != 3 {
return errors.New("forget-checks <owner> <repo> <number>")
}
number, err := strconv.Atoi(args[2])
if err != nil || number <= 0 {
return fmt.Errorf("%q is not a pull request's number", args[2])
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
said, err := forgetChecks(ctx, open.inventory, builder.PullRef{Owner: args[0], Repo: args[1], Number: number})
if err != nil {
return err
}
fmt.Println(said)
return nil
}
@@ -1,102 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
func TestOnlyAWalksAskTakesAChecksBuildAndADryRunNever(t *testing.T) {
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}, {Repository: "http://forge.invalid/novox/web.git", Number: 3},
{Repository: "novox/shop", Number: 7}, {Repository: "novox/shop"}}}}
asked := reuseAskOf(p)
if want := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}, {Owner: "novox", Repo: "web", Number: 3}}; !reflect.DeepEqual(asked.Pulls, want) {
t.Fatalf("the walk names %+v", asked.Pulls)
}
walk := withReuse(t.Context(), asked)
if reuseAsked(walk, false) != asked {
t.Fatal("a walk's ask does not carry its leave")
}
if reuseAsked(walk, true) != nil {
t.Fatal("a dry run carries a leave to take a check's build")
}
if reuseAsked(t.Context(), false) != nil {
t.Fatal("an ask that is not a walk's carries a leave")
}
}
func TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne(t *testing.T) {
withTheGate(t) // the first-machine gate's own path (novox/hq ADR 0360 keeps it for earlier walks)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
gateEvery = 0
digest := "sha256:" + strings.Repeat("c", 64)
reused := func(id, reference string) link.BuildResult {
manifest := mustJSON(t, catalogue.Manifest{Module: "app", Version: "c3", Resources: []map[string]any{
{"id": "svc", "type": "container", "name": "app", "image": reference}}})
return link.BuildResult{ID: id, Repository: "novox/mesh-catalog", Path: "modules/app", Ref: "main", On: "anchor",
Commit: "c3", Manifest: manifest, Trunk: "main", OnTrunk: true, Branches: []string{"main"},
Made: []link.MadeArtifact{{Name: "server", Kind: "image", Reference: reference}},
Source: &link.SourceOnSeat{Seat: "git", Repository: "novox/mesh-catalog"},
Reused: "reused from the check of novox/mesh-catalog#9 at head abcdefabcdef (merged into trunk 123456123456): server " + digest,
ReusedFrom: &link.ReusedFrom{Pull: link.PullRef{Owner: "novox", Repo: "mesh-catalog", Number: 9},
Head: "abcdefabcdef", Trunk: "123456123456", Key: "r1:" + strings.Repeat("1", 64)}}
}
if _, _, err := takeIn(ctx, inv, reused(link.NewBuildID(time.Now()), "store.internal:5000/scratch/app/server@"+digest)); err == nil ||
!strings.Contains(err.Error(), "scratch") {
t.Fatalf("a build in the scratch namespace was registered: %v", err)
}
id := link.NewBuildID(time.Now().Add(time.Second))
result := reused(id, "store.internal:5000/app/server@"+digest)
if _, _, err := takeIn(ctx, inv, result); err != nil {
t.Fatal(err)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil || current["app"].Commit != "c3" {
t.Fatalf("the reused build is not what app is registered at: %+v %v", current["app"], err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
raw, _ := json.Marshal(shelf["app"].Resources)
if !strings.Contains(string(raw), "app/server@"+digest) {
t.Fatalf("app is registered as %s, not by the check's digest in its own namespace", raw)
}
// The walk asked it: the walk keeps what its build did, and its gate judges the registered build.
asked := time.Now().UTC()
p := g.plan(t)
p.Commit = "c3"
p.Modules["app"] = &inventory.PlanModule{State: "asked", AskedAt: &asked, Build: id}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
planBuiltSaying(ctx, g.open, "app", "c3", "", asked, id, result.MadeWords())
p = g.plan(t)
if m := p.Modules["app"]; m.State != "built" || !strings.HasPrefix(m.Made, "reused from the check of novox/mesh-catalog#9") {
t.Fatalf("the walk keeps %+v", m)
}
for i := 0; i < 6; i++ {
advancePlans(ctx, g.open)
}
p = g.plan(t)
gate := p.Modules["app"].Gate
if gate == nil || gate.To != "c3" || gate.Verdict != inventory.GatePassed {
t.Fatalf("the gate judged %+v, not the reused build", gate)
}
if sent, _, _ := inv.SentBuilds(ctx, "anchor"); sent["app"] != "c3" {
t.Fatalf("the first machine was sent %v, not the reused build", sent)
}
}
-193
View File
@@ -1,193 +0,0 @@
package main
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// One build per change (novox/hq ADR 0350), as the controller asks it and takes it in.
// The check of a change is asked to build what a merge of it would move, in the walk's order: a module of the
// repository checked from the merge, one standing on it from its own repository as the walk builds it, and one
// that reads the repository through a context not at all. A change that moves no module builds nothing.
func TestACheckIsAskedToBuildWhatTheMergeWouldMove(t *testing.T) {
entries := []inventory.Entry{
{Manifest: catalogue.Manifest{Module: "app"}, Source: inventory.Source{Repository: "novox/shop", Seat: "git", Path: "modules/app"}},
{Manifest: catalogue.Manifest{Module: "web"}, Source: inventory.Source{Repository: "novox/web", Seat: "git", Ref: "main"}},
{Manifest: catalogue.Manifest{Module: "docs"}, Source: inventory.Source{Repository: "novox/docs", Seat: "git"}},
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Base: "main", Commit: "abc"}
scope := checkScope{Reach: mergeReach{Packaging: entries[2:], Plan: inventory.Plan{Tiers: [][]string{{"app", "docs"}, {"web"}}}}}
clone := func(s inventory.Source) (string, error) { return "http://forge.invalid/" + s.Repository + ".git", nil }
got := checkBuildsOf(p, scope, entries, clone)
want := []link.CheckBuild{
{Module: "app", Path: "modules/app", Merged: true, Tier: 0},
{Module: "docs", Tier: 0, Skip: "its build reads novox/shop through a context, which a check cannot hand the merge"},
{Module: "web", Repository: "http://forge.invalid/novox/web.git", Ref: "main", Tier: 1},
}
if !reflect.DeepEqual(got, want) {
t.Fatalf("the check is asked to build\n%+v\nwant\n%+v", got, want)
}
if got := checkBuildsOf(p, checkScope{}, entries, clone); got != nil {
t.Fatalf("a change that moves no module asks for builds: %+v", got)
}
}
// A walk's ask carries its leave to take a check's build, naming the pull requests of the merges it answers; a dry
// run never does, and neither does an ask that is not a walk's.
// A walk ended lets go of its merges' check builds — only once it ended, after it took what it reuses.
func TestAWalkEndedLetsGoOfItsMergesCheckBuilds(t *testing.T) {
var forgot []builder.PullRef
was := forgetChecks
forgetChecks = func(_ context.Context, _ *inventory.Inventory, pull builder.PullRef) (string, error) {
forgot = append(forgot, pull)
return "", nil
}
t.Cleanup(func() { forgetChecks = was })
p := inventory.Plan{ID: "plan-1", Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
{Repository: "novox/shop", Number: 7}}}}
p.WalkSaid = inventory.WalkStarted
forgetChecksOfWalk(t.Context(), nil, p)
p.WalkSaid = ""
forgetChecksOfWalk(t.Context(), nil, p)
if len(forgot) != 0 {
t.Fatalf("a walk not ended let go of %v, which it may still take", forgot)
}
p.WalkSaid = inventory.WalkEnded
forgetChecksOfWalk(t.Context(), nil, p)
if want := []builder.PullRef{{Owner: "novox", Repo: "shop", Number: 7}}; !reflect.DeepEqual(forgot, want) {
t.Fatalf("the walk ended let go of %v", forgot)
}
}
// A build in the scratch namespace is never registered: only the walk's copy in the module's own namespace is.
func TestABuildInTheScratchNamespaceIsNeverRegistered(t *testing.T) {
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: "store.internal:5000/app/server@sha256:aa"}}); got != "" {
t.Fatalf("a module's own build reads as a scratch build: %s", got)
}
scratch := "store.internal:5000/scratch/app/server@sha256:aa"
if got := inScratch([]link.MadeArtifact{{Name: "server", Reference: scratch}}); got != scratch {
t.Fatalf("a scratch build was not found: %q", got)
}
if got := inScratch([]link.MadeArtifact{{Name: "look", Reference: "http://store.internal:5000/v2/scratch/app/look/blobs/sha256:bb"}}); got == "" {
t.Fatal("a scratch archive was not found")
}
}
// The walk says per module what it did: "reused from …" or "built: <why>" — and a build seat from before says
// neither, which is said.
func TestAWalksBuildSaysWhetherItReusedOrBuilt(t *testing.T) {
if got := (link.BuildResult{Reused: "reused from the check of novox/shop#7"}).MadeWords(); got != "reused from the check of novox/shop#7" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{BuiltWhy: "no check built it"}).MadeWords(); got != "built: no check built it" {
t.Fatalf("said %q", got)
}
if got := (link.BuildResult{}).MadeWords(); !strings.Contains(got, "predates ADR 0350") {
t.Fatalf("said %q", got)
}
}
// A walk's module the check's build stood in for is registered by the digest the check built — the copy in the
// module's namespace — and the walk keeps "reused from …" for it; its first-node gate judges that registered
// build as it judges one the walk built. A scratch reference is refused. Needs the store (MESH_TEST_POSTGRES), and
// is named in testdata/store-tests.
// A walk's leave names the builds its own pull requests' verdicts made at the head each merge merged — never an
// earlier head's — read once per walk, and none when they cannot be read: the builder takes nothing a verdict does
// not name (novox/hq ADR 0350 §2, issue 703).
func TestAWalksLeaveNamesTheBuildsItsPullRequestsVerdictsMade(t *testing.T) {
was := checkedBuildsOf
forgetWalkVerdicts()
t.Cleanup(func() { checkedBuildsOf = was; forgetWalkVerdicts() })
var asked [][]link.PullRef
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
atHead := link.CheckedBuild{Pull: pull, Check: "build-2", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:2"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{atHead, earlier}, nil
}
pulls := []link.PullRef{pull}
heads := map[link.PullRef]string{pull: "bbbbbbbbbbbb0123456789"}
for range 3 {
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, heads); !reflect.DeepEqual(got.Builds,
[]link.CheckedBuild{atHead}) {
t.Fatalf("the leave names %+v, not only the merged head's build", got.Builds)
}
}
if !reflect.DeepEqual(asked, [][]link.PullRef{pulls}) {
t.Fatalf("the verdicts were read %d times for one walk: %v", len(asked), asked)
}
// A merge whose head was not said takes nothing.
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, nil); got.Builds != nil {
t.Fatalf("a merge of no said head names %+v", got.Builds)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := reuseWithVerdicts(t.Context(), "plan-2", &link.ReuseAsk{Pulls: pulls}, heads); got.Builds != nil {
t.Fatalf("verdicts that could not be read named %+v", got.Builds)
}
}
// A check is handed only the builds earlier checks of its own pull request at its own head made: another pull
// request's build, or an earlier head's, is never kept (novox/hq issue 703). None when they cannot be read.
func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
was := checkedBuildsOf
t.Cleanup(func() { checkedBuildsOf = was })
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
mine := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-0", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
another := link.CheckedBuild{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 8}, Check: "build-9",
Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
var asked [][]link.PullRef
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{mine, earlier, another}, nil
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Commit: "bbbbbbbbbbbb0123456789"}
builds := []link.CheckBuild{{Module: "app"}}
if got := namedForTheCheck(t.Context(), p, builds); !reflect.DeepEqual(got, []link.CheckedBuild{mine}) ||
!reflect.DeepEqual(asked, [][]link.PullRef{{pull}}) {
t.Fatalf("the check is handed %+v, read for %v", got, asked)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := namedForTheCheck(t.Context(), p, builds); got != nil {
t.Fatalf("verdicts that could not be read named %+v", got)
}
}
func forgetWalkVerdicts() {
walkVerdicts.Lock()
defer walkVerdicts.Unlock()
walkVerdicts.read = map[string]verdictsRead{}
}
// A walk's merged heads are the forge's word for each merge; a merge announced without its head, or a pull request
// merged at two heads in one walk, has none, and its walk takes no check's build (novox/hq issue 703).
func TestAWalksMergedHeadsAreTheForgesWord(t *testing.T) {
merge := func(number int, head string) inventory.PlanMerge {
return inventory.PlanMerge{Repository: "novox/shop", Number: number, Head: head}
}
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
merge(7, "aaaaaaaaaaaa1"), merge(8, ""), merge(9, "bbbbbbbbbbbb"), merge(9, "cccccccccccc")}}}
heads := mergedHeadsOf(p)
shop := func(n int) link.PullRef { return link.PullRef{Owner: "novox", Repo: "shop", Number: n} }
if heads[shop(7)] != "aaaaaaaaaaaa1" || heads[shop(8)] != "" || heads[shop(9)] != "" {
t.Fatalf("merged heads %v", heads)
}
}
+1 -1
View File
@@ -11,8 +11,8 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/builder"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
-44
View File
@@ -1,44 +0,0 @@
package main
import (
"reflect"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// 1212/AC1 and AC2: the check request names the pulls bucket of each module claiming the delivery seat, in the order
// of the claimants' names.
func TestACheckNamesThePullsBucketsOfTheSeatsClaimants(t *testing.T) {
for name, c := range map[string]struct {
entries []inventory.Entry
want []string
}{
"only the gate": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), namedOnly("mesh-delivery")}, []string{"mesh-delivery-gate_pulls"}},
"only today's": {[]inventory.Entry{claimingTheSeat("mesh-delivery"), namedOnly("gitea")}, []string{"mesh-delivery_pulls"}},
"both, name order": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls", "mesh-delivery-gate_pulls"}},
"none": {[]inventory.Entry{namedOnly("gitea")}, nil},
} {
if got := pullsBucketsOf(c.entries); !reflect.DeepEqual(got, c.want) {
t.Errorf("%s: %v, want %v", name, got, c.want)
}
}
}
// 1212/AC1 and AC2: the check a request carries names those buckets (the wiring, not only the helper).
func TestTheCheckRequestCarriesThePullsBuckets(t *testing.T) {
for name, c := range map[string]struct {
entries []inventory.Entry
want []string
}{
"only the gate": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate")}, []string{"mesh-delivery-gate_pulls"}},
"only today's": {[]inventory.Entry{claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls"}},
"both, name order": {[]inventory.Entry{claimingTheSeat("mesh-delivery-gate"), claimingTheSeat("mesh-delivery")}, []string{"mesh-delivery_pulls", "mesh-delivery-gate_pulls"}},
} {
got := checkOf(link.PullUpdated{Owner: "novox", Repo: "a", Number: 7}, checkScope{}, c.entries, nil, nil, nil, nil)
if !reflect.DeepEqual(got.Pulls, c.want) || got.Owner != "novox" || got.Number != 7 {
t.Errorf("%s: %+v, want pulls %v", name, got, c.want)
}
}
}
-165
View File
@@ -1,165 +0,0 @@
package main
import (
"context"
"fmt"
"regexp"
"sort"
"strings"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382, issue 867).
//
// A mesh-host change and a mesh-controller change that need each other could not both pass their own merge checks:
// each was read beside the other's repository at the commit the mesh runs, which holds neither change. So the
// controller asks mesh-delivery's `groups` once for every pull request's check it asks (§1), and names, in the ask,
// the group's other open pull requests into a base of the same name, in a repository the build seat reads beside the
// check — mesh-lab never — each with its whole head. The build seat reads those repositories at the sibling's merge (§2).
// A pull request for which mesh-delivery answered and named no sibling is asked exactly as before (§7).
// wholeCommit is a commit as the forge says a head: forty hexadecimal characters.
var wholeCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
// besideRepositories are the repositories (lower case, owner/repo) a check reads beside its own, by the directory each
// is found under — the core's, as besideOf clones them, and mesh-sdk beside the controller; never mesh-lab, which is
// always read at its trunk.
func besideRepositories(entries []inventory.Entry) map[string]string {
read := map[string]string{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
read[nameOf(e.Source.Repository)] = dir
if dir == "mesh-controller" && e.Source.Seat != "" {
read[nameOf(siblingOf(e.Source.Repository, "mesh-sdk"))] = "mesh-sdk"
}
}
delete(read, "")
return read
}
// nameOf is a recorded repository as owner/repo in lower case, empty when it names no owner.
func nameOf(repository string) string {
owner, repo := sourceOwner(repository), sourceName(repository)
if owner == "" || repo == "" {
return ""
}
return strings.ToLower(owner + "/" + repo)
}
// siblingsFor is the pull request's siblings, as mesh-delivery's groups say them (ADR 0382 §1): the other members of
// the group its head branch names that are open, unmerged and into a base of its own base's name, in a repository the
// check reads beside it (beside: the directories the ask already names). The second answer is why they could not be
// learned, empty when they could: the group's members carry no head or base — a mesh-delivery from before the rule.
func siblingsFor(p link.PullUpdated, entries []inventory.Entry, beside map[string]link.CheckedOut,
lines []groupsLine) ([]link.CheckSibling, string) {
own := strings.ToLower(p.Owner + "/" + p.Repo)
var group *groupsLine
if p.Head == "" {
// A recheck (mesh-delivery's `recheck` through `delivery check`) carries no head branch name: the group is the one
// whose members hold this very pull request, by repository and number.
learnable := false
for i := range lines {
for _, m := range lines[i].Details {
learnable = true
if p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
group = &lines[i]
}
}
}
if group == nil {
for _, l := range lines {
if len(l.Details) == 0 && len(l.Members) > 0 && !learnable {
return nil, "mesh-delivery named the groups' members without their pull requests (a mesh-delivery from before novox/hq ADR 0382): " +
"the group of a pull request announced without its head branch cannot be found"
}
}
return nil, "" // a lone pull request
}
} else {
for i := range lines {
if lines[i].ID == p.Head {
group = &lines[i]
}
}
if group == nil {
return nil, ""
}
}
if len(group.Details) == 0 && len(group.Members) > 0 {
return nil, "mesh-delivery named the group's members without their heads and bases (a mesh-delivery from before novox/hq ADR 0382)"
}
read := besideRepositories(entries)
var siblings []link.CheckSibling
for _, m := range group.Details {
repository := strings.ToLower(m.Repository)
if repository == own {
continue
}
if !m.Open {
continue
}
if m.Head == "" || m.Base == "" {
return nil, "mesh-delivery named " + m.ID + " without its head or base (a mesh-delivery from before novox/hq ADR 0382)"
}
dir, readBeside := read[repository]
if m.Base != p.Base || !readBeside || dir == "mesh-lab" {
continue
}
if _, there := beside[dir]; !there {
continue
}
head := strings.ToLower(m.Head)
if !wholeCommit.MatchString(head) {
return nil, fmt.Sprintf("mesh-delivery named %s with a head that is not a whole commit (%q)", m.ID, m.Head)
}
siblings = append(siblings, link.CheckSibling{Repository: m.Repository, Number: m.Number, Head: head, Base: m.Base, Dir: dir})
}
sort.Slice(siblings, func(i, j int) bool { return siblings[i].Dir < siblings[j].Dir })
return siblings, ""
}
// withSiblings asks mesh-delivery's `groups` once and says in the request the pull request's siblings, or why they
// could not be learned (ADR 0382 §1): a check never waits on mesh-delivery past groupAskWithin and never fails for it,
// so a fix to mesh-delivery is still checked without it.
func withSiblings(ctx context.Context, request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry) {
lines, err := groupsAsked(ctx, deliverySeatHeld(entries))
siblingsFrom(request, p, entries, lines, err)
}
// siblingsFrom is withSiblings from mesh-delivery's answer already had.
func siblingsFrom(request *link.BuildRequest, p link.PullUpdated, entries []inventory.Entry, lines []groupsLine, err error) {
if request.Check == nil {
return
}
if err != nil {
request.Check.SiblingsUnknown = fmt.Sprintf("mesh-delivery did not say the group's pull requests (%v): "+
"the check reads no siblings", err)
return
}
request.Check.Siblings, request.Check.SiblingsUnknown = siblingsFor(p, entries, request.Check.Beside, lines)
}
// composedMember says the pull request is a member of a delivery group whose gate is the composed check's (novox/hq
// ADR 0403 clauses 1 and 7): the group has two members or more and mesh-delivery says so on its line. The group is
// found as siblingsFor finds it, by its head branch name, or, for a recheck that carries none, by the pull request's
// repository and number — not by siblings, which name only what the check reads beside it.
func composedMember(p link.PullUpdated, lines []groupsLine) bool {
own := strings.ToLower(p.Owner + "/" + p.Repo)
for _, l := range lines {
found := p.Head != "" && l.ID == p.Head
for _, m := range l.Details {
if p.Head == "" && p.Number != 0 && m.Number == p.Number && strings.ToLower(m.Repository) == own {
found = true
}
}
if found {
return l.ComposedGate && max(len(l.Details), len(l.Members)) >= 2
}
}
return false
}
-248
View File
@@ -1,248 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A delivery group's member is checked beside the other members' heads** (novox/hq ADR 0382 §1, §7, issue 867): the
// controller asks mesh-delivery's `groups` once for every check it asks, and names in the ask the group's other open
// pull requests into a base of the same name, in a repository the build seat reads beside it — mesh-lab never.
const (
headHost = "1111111111111111111111111111111111111111"
headController = "2222222222222222222222222222222222222222"
headSDK = "3333333333333333333333333333333333333333"
headLab = "4444444444444444444444444444444444444444"
headHq = "5555555555555555555555555555555555555555"
)
func theCoreOnTheGitSeat() []inventory.Entry {
onSeat := func(module, repository string) inventory.Entry {
e := fromRepo(module, repository, "")
e.Source.Seat = "git"
return e
}
nats := onSeat("nats", "novox/mesh-catalog")
nats.Source.Path = "modules/nats"
held := catalogue.Manifest{Module: "mesh-delivery", Claims: []catalogue.Claim{{Name: catalogue.DeliverySeat, Scope: catalogue.ScopeMesh}}}
return []inventory.Entry{onSeat("mesh-controller", "novox/mesh-controller"), onSeat("mesh-host", "novox/mesh-host"),
onSeat("node-tools", "novox/mesh-tools"), nats, {Manifest: held, On: []string{"anchor"}}}
}
func besideTheCore() map[string]link.CheckedOut {
return map[string]link.CheckedOut{"mesh-controller": {Repository: "c"}, "mesh-host": {Repository: "h"},
"mesh-tools": {Repository: "t"}, "mesh-catalog": {Repository: "k"}, "mesh-sdk": {Repository: "s"}, "mesh-lab": {Repository: "l"}}
}
func member(repository string, number int, head, base string, open bool) memberLine {
return memberLine{ID: repository + "@" + head[:12], Repository: repository, State: "checked", Number: number, Head: head,
Base: base, Open: open}
}
func group(id string, members ...memberLine) groupsLine {
line := groupsLine{ID: id, Details: members}
for _, m := range members {
line.Members = append(line.Members, m.ID+" ("+m.State+")")
}
return line
}
func TestASiblingIsTheGroupsOtherOpenPullRequestIntoABaseOfTheSameName(t *testing.T) {
entries := theCoreOnTheGitSeat()
pull := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
for _, c := range []struct {
what string
lines []groupsLine
want []string
unknown string
}{
{"the other member, open, same base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
[]string{"mesh-controller=novox/mesh-controller#346@" + headController + " onto main"}, ""},
{"a member into a base of another name is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "release", true))},
nil, ""},
{"a member not open (merged or closed) is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", false))},
nil, ""},
{"mesh-lab is never one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-lab", 5, headLab, "main", true))},
nil, ""},
{"a repository the check reads nothing of is not one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 9, headHq, "main", true))},
nil, ""},
{"mesh-sdk, read beside the controller, is one", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-sdk", 7, headSDK, "main", true))},
[]string{"mesh-sdk=novox/mesh-sdk#7@" + headSDK + " onto main"}, ""},
{"another group is none of its business", []groupsLine{group("feat/y",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))},
nil, ""},
{"a group of its own alone has none", []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))},
nil, ""},
{"an older mesh-delivery names no heads", []groupsLine{{ID: "feat/x",
Members: []string{"novox/mesh-host@111111111111 (checked)", "novox/mesh-controller@222222222222 (checked)"}}},
nil, "without their heads and bases"},
{"a member without its base", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "", true))},
nil, "without its head or base"},
{"a head that is no commit", []groupsLine{group("feat/x",
member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController[:12]+"zzzzzzzzzzzzzzzzzzzzzzzzzzzz", "main", true))},
nil, "not a whole commit"},
} {
got, unknown := siblingsFor(pull, entries, besideTheCore(), c.lines)
var said []string
for _, s := range got {
said = append(said, s.Dir+"="+s.Repository+"#"+itoa(s.Number)+"@"+s.Head+" onto "+s.Base)
}
if strings.Join(said, ";") != strings.Join(c.want, ";") {
t.Errorf("%s: siblings %v, wanted %v", c.what, said, c.want)
}
if (c.unknown == "") != (unknown == "") || !strings.Contains(unknown, c.unknown) {
t.Errorf("%s: could not learn %q, wanted %q", c.what, unknown, c.unknown)
}
}
// A pull request into a release branch has siblings into the release branch of the same name (ADR 0358 §1).
release := pull
release.Base = "feat/57"
lines := []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "feat/57", true),
member("novox/mesh-controller", 346, headController, "feat/57", true), member("novox/mesh-tools", 8, headLab, "main", true))}
if got, _ := siblingsFor(release, entries, besideTheCore(), lines); len(got) != 1 || got[0].Dir != "mesh-controller" {
t.Errorf("into a release branch: %+v", got)
}
// A recheck announces the pull request without its head branch: its group is the one holding it (repository, number).
pull.Head = ""
got, unknown := siblingsFor(pull, entries, besideTheCore(), lines)
if len(got) != 1 || got[0].Dir != "mesh-tools" || unknown != "" {
t.Errorf("a recheck of a group member: %+v %q", got, unknown)
}
// A pull request in no group, rechecked, is alone: no sibling, and nothing said.
pull.Number = 999
if got, unknown := siblingsFor(pull, entries, besideTheCore(), lines); got != nil || unknown != "" {
t.Errorf("a lone pull request: %+v %q", got, unknown)
}
if got, unknown := siblingsFor(pull, entries, besideTheCore(), nil); got != nil || unknown != "" {
t.Errorf("no groups at all: %+v %q", got, unknown)
}
// A mesh-delivery that names members without their pull requests cannot find the group of a recheck: it says so.
older := []groupsLine{{ID: "feat/x", Members: []string{"novox/mesh-host@111111111111 (checked)"}}}
pull.Number = 114
if got, unknown := siblingsFor(pull, entries, besideTheCore(), older); got != nil || unknown == "" {
t.Errorf("an older mesh-delivery: %+v %q", got, unknown)
}
}
func itoa(n int) string { b, _ := json.Marshal(n); return string(b) }
// Every check asks mesh-delivery's `groups` once; the ask names the sibling's repository, head and base.
func TestEveryCheckAsksMeshDeliveryOnceAndNamesItsSiblings(t *testing.T) {
entries := theCoreOnTheGitSeat()
asked := 0
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
asked++
if !held {
t.Error("the seat is held, yet it was asked as not held")
}
return []groupsLine{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true),
member("novox/mesh-controller", 346, headController, "main", true))}, nil
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if asked != 1 {
t.Fatalf("mesh-delivery was asked %d times for one check", asked)
}
want := link.CheckSibling{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", Dir: "mesh-controller"}
if got := request.Check.Siblings; len(got) != 1 || got[0] != want || request.Check.SiblingsUnknown != "" {
t.Fatalf("the ask names siblings %+v, unknown %q", got, request.Check.SiblingsUnknown)
}
body, _ := json.Marshal(request.Check)
if !strings.Contains(string(body), `"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"`+headController+
`","base":"main","dir":"mesh-controller"}]`) {
t.Fatalf("on the wire: %s", body)
}
}
// When mesh-delivery does not answer in time (or is not held, or answered without the fields), the check reads no
// sibling and its request says it could not learn them, so a fix to mesh-delivery is still checked without it.
func TestWhenMeshDeliveryDoesNotAnswerTheCheckReadsNoSiblingAndSaysSo(t *testing.T) {
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, c := range []struct {
what string
err error
held bool
}{{"it did not answer", context.DeadlineExceeded, true}, {"nothing serves it", errors.New("nothing serves mesh-delivery.groups"), true},
{"no holder on record", errors.New("no holder of mesh-delivery is on record to say it"), false}} {
entries := theCoreOnTheGitSeat()
if !c.held {
entries = entries[:4]
}
groupsAsked = func(_ context.Context, held bool) ([]groupsLine, error) {
if held != c.held {
t.Errorf("%s: held %v", c.what, held)
}
return nil, c.err
}
request := link.BuildRequest{Check: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Base: "main", Beside: besideTheCore()}}
withSiblings(t.Context(), &request, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
if len(request.Check.Siblings) != 0 || !strings.Contains(request.Check.SiblingsUnknown, c.err.Error()) ||
!strings.Contains(request.Check.SiblingsUnknown, "reads no siblings") {
t.Errorf("%s: siblings %+v, unknown %q", c.what, request.Check.Siblings, request.Check.SiblingsUnknown)
}
}
}
// **A pull request for which mesh-delivery answered and named no sibling is asked exactly as before** (ADR 0382 §7):
// byte for byte the request it was without the rule.
func TestAPullRequestWithNoSiblingIsAskedByteForByteAsBefore(t *testing.T) {
entries := theCoreOnTheGitSeat()
was := groupsAsked
t.Cleanup(func() { groupsAsked = was })
for _, answer := range [][]groupsLine{nil, {group("feat/other", member("novox/mesh-controller", 346, headController, "main", true))},
{group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/hq", 3, headHq, "main", true))}} {
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, nil }
build := func() link.BuildRequest {
return link.BuildRequest{ID: "build-1", Repository: "r", Ref: headHost, Check: &link.CheckRequest{Owner: "novox",
Repo: "mesh-host", Number: 114, Base: "main", Beside: besideTheCore(), Modules: []string{"mesh-host"}, Judge: "validator"}}
}
before, _ := json.Marshal(build())
after := build()
withSiblings(t.Context(), &after, link.PullUpdated{Owner: "novox", Repo: "mesh-host", Base: "main", Head: "feat/x"}, entries)
got, _ := json.Marshal(after)
if string(before) != string(got) {
t.Fatalf("the request changed:\nbefore %s\nafter %s", before, got)
}
}
}
// What the build seat's verdict says of its siblings reaches the forge's holder in the controller's `checked`.
func TestTheVerdictCarriesTheSiblingsAndWhereTheJudgeCameFrom(t *testing.T) {
read := []link.CheckSibling{{Repository: "novox/mesh-controller", Number: 346, Head: headController, Base: "main", BaseCommit: headHost}}
c := checkedOf(link.BuildResult{ID: "b", Ref: headHost, Checked: &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114},
Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok", Siblings: read, JudgeFrom: "sibling novox/mesh-controller#346 merge abc",
Validator: "the change's own validator"}})
if len(c.Siblings) != 1 || c.Siblings[0] != read[0] || c.JudgeFrom == "" || c.Validator == "" {
t.Fatalf("the verdict says %+v", c)
}
body, _ := json.Marshal(c)
for _, want := range []string{`"siblings":[{"repository":"novox/mesh-controller","number":346,"head":"` + headController +
`","base":"main","base_commit":"` + headHost + `"}]`, `"judge_from":"sibling novox/mesh-controller#346 merge abc"`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the verdict lacks %s:\n%s", want, body)
}
}
unknown := checkedOf(link.BuildResult{ID: "b", Check: &link.CheckOutcome{Verdict: "pass", SiblingsUnknown: "mesh-delivery did not answer"}})
if body, _ := json.Marshal(unknown); !strings.Contains(string(body), `"siblings_unknown":"mesh-delivery did not answer"`) ||
strings.Contains(string(body), `"siblings":`) {
t.Fatalf("the verdict that could not learn them says %s", body)
}
}
+1 -38
View File
@@ -8,8 +8,8 @@ import (
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/beside"
"github.com/novox/mesh-controller/internal/catalogue"
)
// The check anybody can run is the check registration runs (novox/hq issue 148, ADR 0037): a manifest
@@ -122,40 +122,3 @@ func TestASeatAnotherModuleDeclaresIsANoteOverSomeManifests(t *testing.T) {
t.Fatalf("the note was not said:\n%s", out.String())
}
}
// **A module lives whole in one repository** (novox/hq ADR 0420 §1, issue 1019): `module check`, told the
// repository it runs for, refuses an artifact whose build context names another; run by hand, it does not know.
func TestModuleCheckRefusesAnArtifactContextNamingAnotherRepository(t *testing.T) {
dir := t.TempDir()
write := func(name, context string) string {
path := filepath.Join(dir, name+".json")
raw := `{"module":"` + name + `","version":"1","build":{"artifacts":[{"name":"server","kind":"image","from":"Dockerfile"` + context + `}]}}`
if err := os.WriteFile(path, []byte(raw), 0o600); err != nil {
t.Fatal(err)
}
return path
}
other := write("other", `,"context":{"seat":"git","repository":"novox/mesh-controller","ref":"main"}`)
own := write("own", `,"context":{"seat":"git","repository":"novox/mesh-catalog","ref":"main"}`)
none := write("none", ``)
t.Setenv("MESH_CHECK_REPOSITORY", "novox/mesh-catalog")
var out bytes.Buffer
if err := moduleCheck([]string{other}, &out); err == nil {
t.Fatalf("a context naming another repository passed:\n%s", out.String())
}
if !strings.Contains(out.String(), "other builds artifact server from novox/mesh-controller: a module lives whole in one repository (novox/hq ADR 0420 §1)") {
t.Errorf("the refusal is not in the words of the gate:\n%s", out.String())
}
for _, path := range []string{own, none} {
out.Reset()
if err := moduleCheck([]string{path}, &out); err != nil {
t.Errorf("%s was refused: %v\n%s", path, err, out.String())
}
}
t.Setenv("MESH_CHECK_REPOSITORY", "")
out.Reset()
if err := moduleCheck([]string{other}, &out); err != nil {
t.Errorf("a local run, with no repository named, refused it: %v\n%s", err, out.String())
}
}
+40 -228
View File
@@ -4,8 +4,6 @@ import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"path"
"regexp"
"runtime/debug"
@@ -14,8 +12,7 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
@@ -75,7 +72,7 @@ type checkScope struct {
// Mesh says the repository is the mesh's: modules are built from it on some branch, its owner is the
// core's, or the change adds a module to it.
Mesh bool
// Judge is who judges the gate (link.JudgeSelf, or the running controller).
// Judge is who judges the gate (link.JudgeSelf, link.JudgeValidator, or the running controller).
Judge string
// From is a module built from the repository, for how the mesh clones it; nil when none is.
From *inventory.Entry
@@ -101,6 +98,10 @@ func pullScope(p link.PullUpdated, entries []inventory.Entry, read map[string][]
switch e.Manifest.Module {
case "mesh-controller":
s.Judge = link.JudgeSelf
case "mesh-host":
if s.Judge == "" {
s.Judge = link.JudgeValidator
}
}
}
for _, d := range r.Added {
@@ -139,116 +140,63 @@ func sourceOwner(repository string) string {
// PullUpdated decides a pull request's merge check, and asks for it when there is something to run.
func (f following) PullUpdated(ctx context.Context, p link.PullUpdated) error {
_, err := f.pullUpdated(ctx, p, os.Stdout)
return err
}
// pullUpdated is PullUpdated saying what it did to w. It returns the id of the check it asked of the build seat, or
// "" when it said a verdict itself and asked nothing. A command that answers JSON on standard output (`delivery
// check`) gives it standard error, so its output stays the document alone (novox/hq issue 1311).
func (f following) pullUpdated(ctx context.Context, p link.PullUpdated, w io.Writer) (string, error) {
inv := f.open.inventory
entries, err := inv.Catalogued(ctx)
if err != nil {
return "", err
return err
}
read, err := readForPlanning(ctx, inv)
if err != nil {
return "", err
return err
}
edges, err := inv.Dependencies(ctx)
if err != nil {
return "", err
return err
}
scope := pullScope(p, entries, read, edges)
// **A cycle of edges fails the check, naming every module and edge** (novox/hq ADR 0419 clause 6), a lone pull
// request's as a group's composed check is: no order exists for what it changes.
alone := orderMember{ID: p.Owner + "/" + p.Repo + "@" + twelve(p.Commit), Repository: p.Owner + "/" + p.Repo, Base: p.Base,
Head: p.Commit, Number: p.Number, Paths: p.Paths, PathsTruncated: p.PathsTruncated, Removed: p.Removed, BaseSHA: p.BaseSHA}
if failing, cyclic, err := cycleVerdict(ctx, "", []orderMember{alone},
map[string]orderReach{alone.ID: {Moved: scope.Modules}}, edges, entries); err != nil {
return "", err
} else if cyclic {
failing.Commit, failing.Trunk = p.Commit, twelve(p.BaseSHA)
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s\n", p.Owner, p.Repo, p.Number, p.Commit, failing.Summary)
sayChecked(ctx, w, failing)
return "", nil
}
// **A member of a group whose gate is the composed check's runs its repository's check alone** (novox/hq ADR 0403
// clause 1): mesh-delivery is asked once, and says so on the group's line. Told nothing, the pull request is
// checked alone, gate included (clause 6).
lines, groupsErr := groupsAsked(ctx, deliverySeatHeld(entries))
composed := groupsErr == nil && composedMember(p, lines)
// The change plan of the commit at hand (ADR 0238): what a merge of it would build and send, posted
// with the verdict whatever the verdict is.
plan := changePlanOf(p.Owner+"/"+p.Repo, p.Base, p.Commit, scope.Reach, entries, func(module string) (inventory.Upgrade, bool) {
u, err := inv.UpgradeOf(ctx, module)
return u, err == nil
})
fmt.Fprint(w, planText(plan))
fmt.Print(planText(plan))
direct := directVerdict(p, plan)
if composed {
scope.Modules, scope.Dependents, scope.New, scope.Manifests = nil, nil, nil, nil
composedGate(&direct)
}
switch {
case !scope.gated() && !scope.Mesh:
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
fmt.Printf("%s/%s#%d (%.8s): %s, and the repository is not the mesh's: said, nothing run\n",
p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched)
sayChecked(ctx, w, direct)
return "", nil
sayChecked(ctx, direct)
return nil
case !scope.gated() && p.MergeCheckSaid && !p.MergeCheck:
direct.RepoCheck = &link.CheckLayer{Verdict: "warning", Summary: noMergeCheck}
fmt.Fprintf(w, "%s/%s#%d (%.8s): %s; %s\n", p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched, noMergeCheck)
sayChecked(ctx, w, direct)
return "", nil
fmt.Printf("%s/%s#%d (%.8s): %s; %s\n", p.Owner, p.Repo, p.Number, p.Commit, noModuleTouched, noMergeCheck)
sayChecked(ctx, direct)
return nil
}
request, err := checkRequestFor(ctx, f.open, p, scope, entries)
if err != nil {
return "", err
return err
}
request.Check.Plan = &plan
request.Check.Composed = composed
// **The group's other open pull requests are read beside it** (novox/hq ADR 0382 §1).
siblingsFrom(&request, p, entries, lines, groupsErr)
seat := buildSeatHeld(ctx)
ask, err := askOverOn(seat)
if err != nil {
return "", err
return err
}
defer ask.Close()
if err := ask.Ask(ctx, request); err != nil {
return "", err
return err
}
// **A newer request replaces the one still waiting, and a newer key stops the one running** (novox/hq ADR 0406 §3, §4,
// issue 974): the check queue keeps one request a pull request, and a running check watches its key. Nothing is asked
// of the forge here (issue 726).
what := "its own merge-check.sh alone: " + noModuleTouched
if composed {
what = "its own merge-check.sh alone: the gate is its group's composed check's"
}
if scope.gated() {
what = fmt.Sprintf("the gate over %s (a merge would build %d module(s) in %d tier(s))",
strings.Join(append(append([]string{}, scope.Modules...), prefixedAll("new:", scope.New)...), ", "),
scope.Width, scope.Tiers)
}
fmt.Fprintf(w, "%s/%s#%d (%.8s): asked %s to check it before it merges — %s — as %s\n", p.Owner, p.Repo, p.Number,
fmt.Printf("%s/%s#%d (%.8s): asked %s to check it before it merges — %s — as %s\n", p.Owner, p.Repo, p.Number,
p.Commit, seat, what, request.ID)
return request.ID, nil
}
// composedWords is what a group member's own verdict says of its gate layer (novox/hq ADR 0403 clause 1).
const composedWords = "judged by its group's composed check"
// gateComposed is a gate layer's verdict when no gate was run for the member alone: never a pass. Mesh-delivery reads it
// as the composed check's to judge, and the forge's holder sets no mesh/merge-gate from it.
const gateComposed = "composed"
// composedGate says a group member's own verdict has no gate of its own: its gate layer, and the verdict that is
// the gate's, read "composed" in place of a pass.
func composedGate(c *link.Checked) {
c.Verdict, c.Summary = gateComposed, composedWords
c.Gate = &link.CheckLayer{Verdict: gateComposed, Summary: composedWords}
return nil
}
// directVerdict is the verdict of a change no build judges — it touches no module, and either the repository is not
@@ -277,7 +225,7 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
for _, e := range entries {
shelf[e.Manifest.Module] = e.Manifest
}
world, err := restOfTheMesh(ctx, open, shelf, "")
world, err := theRestOfTheMesh(ctx, open.inventory, shelf, "")
if err != nil {
return link.BuildRequest{}, err
}
@@ -300,35 +248,7 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
if err != nil {
return link.BuildRequest{}, err
}
beside, err := besideOf(entries, func(module string) string { return current[module].Commit }, clone)
if err != nil {
return link.BuildRequest{}, err
}
builds := checkBuildsOf(p, scope, entries, clone)
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
Ref: p.Commit,
Held: heldBy(ctx),
Seats: seatBases(ctx),
Source: sourceOnSeat(source),
Check: checkOf(p, scope, entries, beside, builds, namedForTheCheck(ctx, p, builds), runningBy(ctx)),
}, nil
}
// checkOf is the check a request carries: the pull request, what the change touches, and the buckets of the pull
// requests' keys of the modules claiming the delivery seat (novox/hq ADR 0437 §3, issue 1212).
func checkOf(p link.PullUpdated, scope checkScope, entries []inventory.Entry, beside map[string]link.CheckedOut,
builds []link.CheckBuild, named []link.CheckedBuild, running map[string]string) *link.CheckRequest {
return &link.CheckRequest{Pulls: pullsBucketsOf(entries), Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Trunk: p.BaseSHA, Ready: p.Ready, Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents,
New: scope.New, Manifests: scope.Manifests, Judge: scope.Judge, Builds: builds, Named: named, Running: running}
}
// besideOf is what a check reads beside its own repository, by the directory it is found under: each core repository
// by the module the mesh builds from it, at besideRefs. running is the commit the mesh runs of a module.
func besideOf(entries []inventory.Entry, running func(module string) string,
clone func(inventory.Source) (string, error)) (map[string]link.CheckedOut, error) {
// Beside it, at what the mesh runs: each core repository by the module the mesh builds from it.
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
@@ -337,12 +257,11 @@ func besideOf(entries []inventory.Entry, running func(module string) string,
}
url, err := clone(e.Source)
if err != nil {
return nil, err
return link.BuildRequest{}, err
}
refs := besideRefs(dir, running(e.Manifest.Module))
refs := besideRefs(dir, current[e.Manifest.Module].Commit)
beside[dir] = link.CheckedOut{Repository: url, Ref: refs[dir]}
if dir == "mesh-controller" {
// The controller's main stays its main: it judges only a check the running controller predates.
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
@@ -354,98 +273,17 @@ func besideOf(entries []inventory.Entry, running func(module string) string,
}
}
}
return beside, nil
}
// trunkOf is the branch a module's source follows: main when it records none, or records a commit.
func trunkOf(s inventory.Source) string {
if t := strings.TrimSpace(followedBranch(s.Ref)); t != "" {
return t
}
return "main"
}
// sourceName is the name of a recorded repository, a path on the git seat or a URL: novox/mesh-host → mesh-host.
func sourceName(repository string) string {
parts := strings.Split(strings.Trim(strings.TrimSuffix(repository, ".git"), "/"), "/")
if len(parts) < 2 {
return ""
}
return parts[len(parts)-1]
}
// namedForTheCheck are the builds earlier checks of the same pull request at the same head named, read once for the
// check (novox/hq issue 703): asked again because only the trunk moved, the check keeps what they built of a key
// still the same. Another pull request's build, or one of an earlier head, is never kept. None when they cannot be
// read, and the check builds, as before.
func namedForTheCheck(ctx context.Context, p link.PullUpdated, builds []link.CheckBuild) []link.CheckedBuild {
if len(builds) == 0 || p.Commit == "" {
return nil
}
pull := link.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number}
read, err := checkedBuildsOf(ctx, []link.PullRef{pull})
if err != nil {
fmt.Printf(" the earlier checks' builds of %s/%s#%d could not be read, so the check builds: %v\n",
p.Owner, p.Repo, p.Number, err)
// empty-on-error: said above; a check handed no named build builds every module, as before issue 703
return nil
}
var named []link.CheckedBuild
for _, b := range read {
if b.Pull == pull && link.SameHead(b.Head, p.Commit) {
named = append(named, b)
}
}
return named
}
// checkBuildsOf are the modules a merge of the change would build, in the walk's order, as the check builds them
// for the walk to take (novox/hq ADR 0350 §1): a module built from the repository checked, at the branch it merges
// into, from the merge the check makes; any other — a module standing on one — from its own repository at the
// branch it follows, as the walk builds it. A module whose build reads the repository checked through a context,
// or that the merge deletes, cannot be built as the walk would: said, and nothing after it is built. A change that
// moves no module builds nothing.
func checkBuildsOf(p link.PullUpdated, scope checkScope, entries []inventory.Entry,
clone func(inventory.Source) (string, error)) []link.CheckBuild {
if len(scope.Reach.Plan.Tiers) == 0 {
return nil
}
m := link.SourceMoved{Owner: p.Owner, Repo: p.Repo, Base: p.Base, CloneURL: p.CloneURL}
byName := map[string]inventory.Entry{}
for _, e := range entries {
byName[e.Manifest.Module] = e
}
skip := map[string]string{}
for _, e := range scope.Reach.Packaging {
skip[e.Manifest.Module] = "its build reads " + p.Owner + "/" + p.Repo + " through a context, which a check " +
"cannot hand the merge"
}
for _, e := range scope.Reach.Deleted {
skip[e.Manifest.Module] = "the merge deletes it"
}
var out []link.CheckBuild
for tier, names := range scope.Reach.Plan.Tiers {
for _, name := range names {
e, known := byName[name]
b := link.CheckBuild{Module: name, Tier: tier, Skip: skip[name]}
switch {
case !known:
b.Skip = "the catalogue does not hold it"
case b.Skip != "":
case sourceIs(e.Source, m):
b.Merged, b.Path = true, e.Source.Path
default:
url, err := clone(e.Source)
if err != nil {
b.Skip = "its repository cannot be cloned: " + err.Error()
break
}
b.Repository, b.Path, b.Ref = url, e.Source.Path, followedBranch(e.Source.Ref)
}
out = append(out, b)
}
}
return out
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
Ref: p.Commit,
Held: heldBy(ctx),
Seats: seatBases(ctx),
Source: sourceOnSeat(source),
Check: &link.CheckRequest{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents, New: scope.New,
Manifests: scope.Manifests, Judge: scope.Judge},
}, nil
}
// besideRefs is the ref each repository is cloned at beside a check, by the directory it is found under:
@@ -527,14 +365,7 @@ const maxCheckReport = 60 << 10
// checked says a merge check's verdict as the controller's `checked`. Nothing is recorded or
// registered: a check builds nothing (issue 240's rule for a dry run, kept for a check).
func checked(ctx context.Context, result link.BuildResult) {
// **A check stopped by its key says nothing** (novox/hq ADR 0406 §4, issue 974, review 2): it stopped because the head or
// the trunk commit moved on, and the newer request will judge. Said as an error it would name the live head — red
// statuses on the forge, a delivery refused — and written to the key it would stand in the newer request's way.
if result.Failed == link.SupersededByKey {
fmt.Printf("%s: stopped, its pull request's key moved on; the newer check will judge\n", result.ID)
return
}
sayChecked(ctx, os.Stdout, checkedOf(result))
sayChecked(ctx, checkedOf(result))
}
// checkedOf is what a check's outcome says: each layer, and an error — never a pass — for a check that
@@ -551,9 +382,6 @@ func checkedOf(result link.BuildResult) link.Checked {
// What was judged (novox/hq ADR 0323 §3): the head merged into the trunk commit, and the commits of the
// repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
c.Built = result.Check.Built
c.Validator = result.Check.Validator
c.Siblings, c.SiblingsUnknown, c.JudgeFrom = result.Check.Siblings, result.Check.SiblingsUnknown, result.Check.JudgeFrom
if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
@@ -571,12 +399,6 @@ func checkedOf(result link.BuildResult) link.Checked {
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
c.RepoCheck = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
}
// **A group member's own check has no gate of its own** (novox/hq ADR 0403 clause 1): what ran was the repository's
// check alone, and its gate layer is "composed", never a pass. A failure or an error stays what it was.
if result.Checked != nil && result.Checked.Composed && result.Check != nil && c.Gate != nil &&
(c.Gate.Verdict == "pass" || c.Gate.Verdict == "warning") {
composedGate(&c)
}
if result.Checked != nil && c.Gate != nil && len(c.Gate.Modules) == 0 {
c.Gate.Modules = append(append([]string{}, result.Checked.Modules...), prefixedAll("new:", result.Checked.New)...)
}
@@ -650,7 +472,7 @@ func prefixedAll(prefix string, items []string) []string {
}
// sayChecked says a merge check's verdict on the bus, where the forge's holder hears it.
func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
func sayChecked(ctx context.Context, c link.Checked) {
repo := "none"
if c.RepoCheck != nil {
repo = strings.ToUpper(c.RepoCheck.Verdict) + " — " + c.RepoCheck.Summary
@@ -659,7 +481,7 @@ func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
if c.Trunk != "" {
judged = " merged into trunk " + c.Trunk
}
fmt.Fprintf(w, "%s: %s/%s#%d at %.8s%s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
fmt.Printf("%s: %s/%s#%d at %.8s%s checked on %s: gate %s — %s; repository %s\n", c.ID, c.Owner, c.Repo, c.Number,
c.Commit, judged, orSomewhere(c.On), strings.ToUpper(c.Verdict), c.Summary, repo)
if checkEvents == nil {
return
@@ -671,16 +493,6 @@ func sayChecked(ctx context.Context, w io.Writer, c link.Checked) {
stating, stop := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Second)
defer stop()
if err := checkEvents.PublishSeatEvent(stating, link.MeshControllerSeat, link.KeyChecked, body); err != nil {
fmt.Fprintf(w, "%s: the verdict could not be said, so the pull request is not told it: %v\n", c.ID, err)
fmt.Printf("%s: the verdict could not be said, so the pull request is not told it: %v\n", c.ID, err)
}
}
// pullsBucketsOf are the buckets of the pull requests' keys a check names for the build agent: one per module
// claiming the delivery seat, in the order of their names (novox/hq ADR 0437 §3, issue 1212).
func pullsBucketsOf(entries []inventory.Entry) []string {
var out []string
for _, m := range claimantsOf(entries, catalogue.DeliverySeat) {
out = append(out, broker.BucketName(m, "pulls"))
}
return out
}
@@ -1,56 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
func TestACheckStoppedByItsKeyIsAnnouncedAsNoVerdict(t *testing.T) {
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
legacy, err := js.Conn().JetStream()
if err != nil {
t.Fatal(err)
}
was := checkEvents
checkEvents = link.OverNATS{Conn: js.Conn(), JS: legacy}
t.Cleanup(func() { checkEvents = was })
heard := make(chan *nats.Msg, 4)
sub, err := js.Conn().ChanSubscribe(link.SeatEventSubject(link.MeshControllerSeat, link.KeyChecked), heard)
if err != nil {
t.Fatal(err)
}
defer func() { _ = sub.Unsubscribe() }()
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 3}
checked(t.Context(), link.BuildResult{ID: "stopped", Ref: "abc", Checked: asked, Failed: link.SupersededByKey})
checked(t.Context(), link.BuildResult{ID: "broken", Ref: "abc", Checked: asked, Failed: "the facts snapshot cannot be read"})
select {
case m := <-heard:
var c link.Checked
_ = json.Unmarshal(m.Data, &c)
if c.ID != "broken" || c.Verdict != "error" {
t.Fatalf("the first verdict said is %+v; want only the broken check's error", c)
}
case <-time.After(5 * time.Second):
t.Fatal("nothing was said")
}
select {
case m := <-heard:
t.Fatalf("a second verdict was said: %s", m.Data)
case <-time.After(300 * time.Millisecond):
}
}
-97
View File
@@ -1,97 +0,0 @@
//go:build integration
package main
import (
"context"
"encoding/json"
"fmt"
"testing"
"github.com/nats-io/nats.go"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// Acceptance 5 of milestone 2 (novox/hq ADR 0406 §3, issue 974), live, with the store and a real bus: two merges into one
// repository within a minute, with two open pull requests, leave each pull request with exactly one queued check, by the
// stream's rule. Each merge moves the trunk, and mesh-delivery asks the controller to check both open pull requests again
// against it (`delivery check`, which is the controller's own path for a head announced); nothing moves a request, nothing
// asks the forge, and the queue is read back whole.
func TestTwoMergesWithinAMinuteLeaveEachOpenPullRequestOneQueuedCheck(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
holdTheDeliverySeat(t, open)
was := askDeliveryOwner
t.Cleanup(func() { askDeliveryOwner = was })
askDeliveryOwner = func(_ context.Context, _ *nats.Conn, verb string, _ map[string]any) (json.RawMessage, error) {
if verb != "groups" {
t.Fatalf("asked the owner %s", verb)
}
return json.RawMessage(`[]`), nil
}
// The repository the two pull requests are into, as a module the mesh builds from it.
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
heads := map[int]string{7: "7777777777777777777777777777777777777777", 8: "8888888888888888888888888888888888888888"}
check := func(number int, trunk string) {
t.Helper()
p := orderMember{ID: fmt.Sprintf("novox/mesh-controller@%.12s", heads[number]), Repository: "novox/mesh-controller",
Base: "main", Head: heads[number], Number: number, Paths: []string{"main.go"}, ModuleDirsSaid: true,
BaseSHA: trunk, Ready: number == 8}.pull()
p.MergeCheck, p.MergeCheckSaid = true, true
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
}
// Announced against the trunk as it stands, then asked again after each of two merges.
for _, trunk := range []string{"1111111111111111111111111111111111111111", "2222222222222222222222222222222222222222",
"3333333333333333333333333333333333333333"} {
check(7, trunk)
check(8, trunk)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil {
t.Fatal(err)
}
seen := map[int]string{}
for _, a := range q.Asks {
c := a.Request.Check
if c == nil {
t.Fatalf("%s is no check", a.ID)
}
if _, twice := seen[c.Number]; twice {
t.Errorf("pull request %d has two queued checks", c.Number)
}
seen[c.Number] = c.Trunk
if a.State != link.AskWaiting {
t.Errorf("%s is %s", a.ID, a.State)
}
if want := "3333333333333333333333333333333333333333"; c.Trunk != want {
t.Errorf("pull request %d is queued against %.8s; want the newest trunk %.8s", c.Number, c.Trunk, want)
}
}
if len(seen) != 2 {
t.Fatalf("queued checks for %v; want one for each of the two open pull requests", seen)
}
// By the streams' own count: the ready one on its stream, the other on its.
for stream, want := range map[string]uint64{broker.ChecksReadyStream: 1, broker.ChecksOtherStream: 1} {
info, err := js.Context().StreamInfo(stream)
if err != nil {
t.Fatal(err)
}
if info.State.Msgs != want {
t.Errorf("%s holds %d request(s); want %d", stream, info.State.Msgs, want)
}
}
}
+2 -5
View File
@@ -53,8 +53,8 @@ func TestThePullRequestIsMappedOntoTheModuleGraph(t *testing.T) {
"", "modules/newmod", "modules/newmod/module.json", true, ""},
{"the controller judges itself", pull("novox", "mesh-controller", "main", []string{"cmd/x.go"}),
"mesh-controller", "", "module.json", true, "self"},
{"the node-engine is judged by the controller the mesh runs, as any module is (issue 1391)",
pull("novox", "mesh-host", "main", []string{"validate/v.go"}), "mesh-host", "", "module.json", true, ""},
{"the node-engine is judged with its validator", pull("novox", "mesh-host", "main", []string{"validate/v.go"}),
"mesh-host", "", "module.json", true, "validator"},
{"the core's owner, no module: the mesh's, its own check alone", pull("novox", "hq", "main", []string{"README.md"}),
"", "", "", true, ""},
{"a branch nothing is built from: the mesh's repository, no module", pull("novox", "photos", "master",
@@ -288,6 +288,3 @@ func TestARecheckedHeadsDirectVerdictNamesItsTrunk(t *testing.T) {
t.Fatalf("a rechecked head's direct verdict names trunk %q", c.Trunk)
}
}
// A check stopped by its pull request's key moving on says no verdict (issue 974, review 2): an error said for it would name the
// live head, and the newer request is the one that judges. A check that failed for any other reason still says its error.
@@ -1,59 +0,0 @@
package main
import (
"go/ast"
"go/parser"
"go/token"
"path/filepath"
"sort"
"strings"
"testing"
)
// 1389: the controller's own composition is gone. No function of the package is named for it any more, and no function
// calls one by that name: a name reappearing is a second copy of what mesh-sdk's go/compose holds (novox/hq ADR 0453).
func TestTheControllersOwnCompositionIsGone(t *testing.T) {
gone := map[string]bool{"planFor": true, "declarationWith": true, "composedAndValidated": true, "generators": true,
"renderingFor": true, "theRestOfTheMesh": true, "whoResolves": true, "grantsFor": true, "namesInTheMesh": true,
"onThePrivateNetwork": true, "passSeatsOn": true, "theMeshPassesSeatsOn": true, "assignmentsOf": true,
"licencesFor": true, "seatsOn": true, "composeBusUsers": true, "adoptionFor": true, "localNetworksOf": true}
files, err := filepath.Glob("*.go")
if err != nil {
t.Fatal(err)
}
fset := token.NewFileSet()
var found []string
for _, f := range files {
if strings.HasSuffix(f, "_test.go") {
continue
}
file, err := parser.ParseFile(fset, f, nil, 0)
if err != nil {
t.Fatal(err)
}
for _, d := range file.Decls {
fn, ok := d.(*ast.FuncDecl)
if !ok {
continue
}
if gone[fn.Name.Name] {
found = append(found, f+": "+fn.Name.Name+" is declared")
}
if fn.Body == nil {
continue
}
ast.Inspect(fn.Body, func(n ast.Node) bool {
if c, ok := n.(*ast.CallExpr); ok {
if id, ok := c.Fun.(*ast.Ident); ok && gone[id.Name] {
found = append(found, f+": "+fn.Name.Name+" calls "+id.Name)
}
}
return true
})
}
}
sort.Strings(found)
for _, one := range found {
t.Errorf("%s: the controller's own composition is back; compose through mesh-sdk (compose_sdk.go)", one)
}
}
@@ -1,114 +0,0 @@
package main
import (
"encoding/base64"
"errors"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/broker"
)
// aCoreInput is an Input of one machine that takes a core module whose credential the vault has not boxed: the composition
// stands in a vault box for it, and the machine holds the controller's own seal of it.
func aCoreInput(t *testing.T) (*meshInput, string) {
t.Helper()
raw := make([]byte, 32)
raw[0] = 9
key := base64.StdEncoding.EncodeToString(raw)
client := catalogue.Manifest{Module: "mesh-provisioner-client", Version: "1",
Claims: []catalogue.Claim{{Name: broker.NodeProvisionerSeat, Scope: catalogue.ScopeNode}},
OwnSecrets: catalogue.OwnSecrets{broker.ProvisionerSecret: {Path: "/var/lib/mesh-provisioner-client/broker"}}}
return &meshInput{Input: compose.Input{
Catalogue: map[string]catalogue.Manifest{client.Module: client},
Nodes: []compose.Node{{Name: "laptop", Assigned: []string{client.Module}, SealingKey: key, Stated: true,
Contract: compose.VaultBoxContract}},
OwnSecrets: []compose.OwnSecretRow{{Node: "laptop", Module: client.Module, Name: broker.ProvisionerSecret,
Origin: "made", SealedTo: key, Sealed: "SEALED-BY-THE-CONTROLLER"}},
}}, client.Module
}
// 1385 (the lead's answer 1): where the vault is silent and no box is kept, the declaration carries the controller's own seal
// for the core credential, as ADR 0396 has it, and stands in for nothing. It failed first: the composition's declaration
// carried the stand-in box, and a send was refused.
func TestACoreCredentialTheVaultDidNotBoxIsSealedByTheController(t *testing.T) {
in, module := aCoreInput(t)
d, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
if len(d.CoreBoxesWanted) != 1 || len(stillForeseen(d)) == 0 {
t.Fatalf("the composition does not stand in for the vault's box: %v", d.CoreBoxesWanted)
}
sealed, _, err := sealedByTheController(in, "laptop", d, []string{module})
if err != nil {
t.Fatal(err)
}
if left := stillForeseen(sealed); len(left) != 0 {
t.Fatalf("the declaration still stands in for %v", left)
}
var file map[string]any
for _, r := range sealed.Resources {
if r["id"] == module+"."+catalogue.NeedID(broker.ProvisionerSecret) {
file = r
}
}
if file == nil || file["sealed"] != "SEALED-BY-THE-CONTROLLER" || file[catalogue.VaultSealedField] != nil {
t.Fatalf("the core credential is carried as %v: the controller's own seal, not the vault's box", file)
}
// And a module the vault did answer for is left as it is.
other, _, err := sealedByTheController(in, "laptop", d, []string{"another-module"})
if err != nil {
t.Fatal(err)
}
if len(stillForeseen(other)) == 0 {
t.Fatal("a module the vault answered for was sealed by the controller")
}
}
// 1383 (the lead's answer 4): the machine that runs the bus is refused where no user has a credential, with busUserList's
// words, and a machine that does not run the bus is not. It failed first: the Input stood a list in and the bus node composed.
func TestTheMachineThatRunsTheBusIsRefusedWhereNoUserHasACredential(t *testing.T) {
bus := catalogue.Manifest{Module: "nats", Version: "1", BusUsers: "/etc/nats/users.conf",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}}}
other := catalogue.Manifest{Module: "files", Version: "1"}
why := errors.New("this machine runs the bus and not one user has a credential, so the composed list would refuse every connection in the mesh")
in := &meshInput{busRefused: why, Input: compose.Input{
Catalogue: map[string]catalogue.Manifest{"nats": bus, "files": other},
Nodes: []compose.Node{{Name: "anchor", Assigned: []string{"nats"}}, {Name: "laptop", Assigned: []string{"files"}}},
}}
if err := refusedBus(in, "anchor"); err == nil || !strings.Contains(err.Error(), "not one user has a credential") {
t.Fatalf("the machine that runs the bus is not refused: %v", err)
}
if err := refusedBus(in, "laptop"); err != nil {
t.Fatalf("a machine that does not run the bus is refused: %v", err)
}
if err := refusedBus(&meshInput{Input: in.Input}, "anchor"); err != nil {
t.Fatalf("a bus with users is refused: %v", err)
}
}
// The controller's seal replaces the stand-in box whatever path the resource carries: the path is the manifest's with the
// machine's words filled in, not the manifest's own. It failed first, with the path compared (the live try: "composed twice and
// still names ... asked of the vault by the next send").
func TestTheControllersSealReplacesTheBoxWhateverPathTheFileCarries(t *testing.T) {
in, module := aCoreInput(t)
d, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
for _, r := range d.Resources {
if _, box := r[catalogue.VaultSealedField]; box {
r["path"] = "/home/someone/.var/lib/mesh-provisioner-client/broker"
}
}
sealed, _, err := sealedByTheController(in, "laptop", d, []string{module})
if err != nil {
t.Fatal(err)
}
if left := stillForeseen(sealed); len(left) != 0 {
t.Fatalf("the declaration still stands in for %v", left)
}
}
@@ -1,33 +0,0 @@
package main
import (
"context"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
)
// grantsOf is the grants a provider creates for its consumers, the consumers whose identity overflows the provision's bound
// and the credentials bound to another consumer, as mesh-sdk's rendering gives them for the machine (it replaced the
// controller's grantsFor, issue 1389).
func grantsOf(ctx context.Context, open *stores, node string) ([]catalogue.Grant, []catalogue.Overflow, []catalogue.Unbound, error) {
in, err := inputFor(ctx, open)
if err != nil {
return nil, nil, nil, err
}
rendered, err := compose.Render(in.Input, node, compose.Reading)
if err != nil {
return nil, nil, nil, err
}
return rendered.Rendering.Grants, rendered.Withheld, rendered.Unbound, nil
}
// madeAndRendered sends to the machine (so everything a send makes is made) and answers its rendering as a question reads it
// afterwards: what the controller's Allocating rendering was.
func madeAndRendered(ctx context.Context, open *stores, node string) (catalogue.Rendering, error) {
if _, err := composeSend(ctx, open, node); err != nil {
return catalogue.Rendering{}, err
}
forgetInput(ctx)
return renderingOf(ctx, open, node)
}
-436
View File
@@ -1,436 +0,0 @@
package main
import (
"context"
"os"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/inventory"
)
// The controller's adapter from its stores to mesh-sdk's composition input (novox/hq ADR 0453 clause 3, design 47 R4b,
// issues 1382 and 1383).
//
// **The adapter reads, and perform writes.** readComposeInput asks the stores once for everything the composition reads
// and returns compose.Input, which the composition never changes; it makes, seals, allocates and records nothing. A
// pair credential, a port, an own secret or a core box that the next send would make is not made here: the composition
// names it foreseen, and compose_perform.go makes it for a send, then reads the stores again and composes again. A
// question (status, facts, probes, the merge gate's reading) never performs.
//
// **The Input is built once per operation.** A verb that composes every node reads the stores once and hands the Input to
// each node's composition of it.
// composeInputBuilt is called once for each Input the adapter builds: the seam a test counts the reads of an operation by
// (issue 1382, criterion 5).
var composeInputBuilt = func() {}
// readComposeInput reads the stores once and returns what the composition needs. ctx carries what a send keeps (the
// recorded modules kept at the build their machine runs, recorded_kept.go): the same contexts every send composes under.
func readComposeInput(ctx context.Context, open *stores) (*meshInput, error) {
in, shape, err := readMeshAndNodes(ctx, open)
if err != nil {
return nil, err
}
// What each node resolves to, with no credential given (the settings are read before it): the modules whose settings are read, and the
// needs whose pair credentials are asked of the stores. Nothing is written by it.
unread, err := readSettings(ctx, open, &in)
if err != nil {
return nil, err
}
plans := map[string]compose.Resolved{}
for _, n := range in.Nodes {
if shape.refused[n.Name] != nil {
continue // refused for what it keeps: nothing of it is read further
}
resolved, err := compose.Resolve(in, n.Name)
switch {
case compose.IsUnresolvable(err):
// Their set does not compose: nothing of theirs is read further, as the controller never read a thing for a
// node that did not resolve.
continue
case err != nil:
return nil, err
}
plans[n.Name] = resolved
}
if err := unreadSettings(unread, plans); err != nil {
return nil, err
}
var busRefused error
wouldMake, err := readCredentialsAndBuilds(ctx, open, &in, plans, shape, &busRefused)
if err != nil {
return nil, err
}
composeInputBuilt()
return &meshInput{Input: in, wouldMake: wouldMake, busRefused: busRefused, refused: shape.refused}, nil
}
// meshInput is the Input an operation composes over, and the pair credentials its reading found none held for: the ones the
// next send makes, which perform makes.
type meshInput struct {
compose.Input
wouldMake []inventory.PairCredential
// busRefused is why the bus's user list cannot be composed (no user has a credential): the machine that runs the bus is
// refused with it, as the controller's own composition refused it, and no other machine is.
busRefused error
// refused is, by node, why that machine alone cannot be composed (nodeRefusal).
refused map[string]error
}
// nodeOf is the node of that name in an Input.
func nodeOf(in compose.Input, name string) (compose.Node, bool) {
for _, n := range in.Nodes {
if n.Name == name {
return n, true
}
}
return compose.Node{}, false
}
// inputShape is what the first half of the reading found that the second half needs.
type inputShape struct {
sent map[string]map[string]string // by node: module to the commit last sent
kept map[string]map[string]string // by node: module to the commit kept
current map[string]inventory.CurrentBuild
// refused is, by node, why a recorded module cannot be kept at the build the machine runs (keepRecorded's refusal, in
// its words): that machine alone is refused with it, as planFor refused it, and nothing else of the mesh is.
refused map[string]error
}
// readMeshAndNodes is the mesh-wide and per-node half of the Input (issue 1382).
func readMeshAndNodes(ctx context.Context, open *stores) (compose.Input, inputShape, error) {
inv := open.inventory
var in compose.Input
shape := inputShape{sent: map[string]map[string]string{}, kept: map[string]map[string]string{}, refused: map[string]error{}}
shelf, err := inv.Catalogue(ctx)
if err != nil {
return in, shape, err
}
in.Catalogue = shelf
if in.Holdings, err = inv.Holdings(ctx); err != nil {
return in, shape, err
}
if in.Licences, err = readLicences(ctx, open); err != nil {
return in, shape, err
}
nodes, err := inv.Nodes(ctx)
if err != nil {
return in, shape, err
}
places, err := inv.Overlays(ctx)
if err != nil {
return in, shape, err
}
placed := map[string]inventory.Overlay{}
for _, p := range places {
placed[p.Name] = p
}
if shape.current, err = inv.CurrentBuilds(ctx); err != nil {
return in, shape, err
}
// The range the mesh allocates from and what its internal names end in, read from the environment here and nowhere else.
in.OverlayCIDR = strings.TrimSpace(os.Getenv(OverlayCIDRVar))
in.InternalSuffix = overlay.Suffix()
if in.Tunnels, err = readTunnels(ctx, inv); err != nil {
return in, shape, err
}
if in.Adopted, err = readAdopted(ctx, inv); err != nil {
return in, shape, err
}
carried, err := inv.CarriedPeers(ctx)
if err != nil {
return in, shape, err
}
for _, c := range carried {
in.CarriedPeers = append(in.CarriedPeers, compose.CarriedPeer{
PublicKey: c.PublicKey, Address: c.Address, EnrolledAs: c.EnrolledAs, Named: c.Named})
}
if in.Joining, err = inv.NodesWithALiveToken(ctx); err != nil {
return in, shape, err
}
keepsSecrets := false
for _, m := range shelf {
keepsSecrets = keepsSecrets || m.Keeps != ""
}
for _, n := range nodes {
var refusal error
node, err := readNode(ctx, open, n, placed[n.Name], &refusal)
if err != nil {
return in, shape, err
}
if refusal != nil {
shape.refused[n.Name] = refusal
}
shape.sent[n.Name], shape.kept[n.Name] = node.SentBuilds, node.KeptBuilds
in.Nodes = append(in.Nodes, node)
}
if keepsSecrets {
if in.OperatorExport, err = inv.OperatorExport(ctx); err != nil {
return in, shape, err
}
}
return in, shape, nil
}
// readNode is one node of the Input: everything the composition reads about it, in the order the controller read it.
func readNode(ctx context.Context, open *stores, n inventory.Node, place inventory.Overlay,
refused *error) (compose.Node, error) {
inv := open.inventory
out := compose.Node{
Name: n.Name,
Place: compose.Place{Key: place.Key, Endpoint: place.Endpoint, Site: place.Site, Hub: place.Hub,
Address: place.Address},
Account: n.Account, AccountHome: n.AccountHome, AgentAccount: n.AgentAccount, AgentAccountHome: n.AgentAccountHome,
Adopted: n.Adopted,
}
var err error
if out.Assigned, err = inv.Assigned(ctx, n.Name); err != nil {
return out, err
}
if out.Capabilities, err = inv.ProfileOf(ctx, n.Name); err != nil {
return out, err
}
if out.PublicDomain, err = inv.PublicDomainOf(ctx, n.Name); err != nil {
return out, err
}
if out.Pins, err = inv.PinsFor(ctx, n.Name); err != nil {
return out, err
}
if out.Bound, err = inv.BindingsFor(ctx, n.Name); err != nil {
return out, err
}
given, err := inv.PortsFor(ctx, n.Name)
if err != nil {
return out, err
}
for _, a := range given {
if out.Ports == nil {
out.Ports = map[string]map[int]int{}
}
if out.Ports[a.Module] == nil {
out.Ports[a.Module] = map[int]int{}
}
out.Ports[a.Module][a.Wanted] = a.Machine
}
if out.SealingKey, err = inv.SealingKeyOf(ctx, n.Name); err != nil {
return out, err
}
if n.Adopted {
if out.Taken, err = inv.Taken(ctx, n.Name); err != nil {
return out, err
}
}
if out.OutwardLinks, err = inv.OutwardLinksOf(ctx, n.Name); err != nil {
return out, err
}
account, had, err := inv.MachineOf(ctx, n.Name)
if err != nil {
return out, err
}
if had && account.Machine != nil {
out.NetworksSaid = true
for _, a := range account.Machine.Networks {
out.Networks = append(out.Networks, catalogue.AttachedNetwork{Link: a.Link, Network: a.Network})
}
}
if out.Sequence, err = inv.Sequence(ctx, n.ID); err != nil {
return out, err
}
if out.Epoch, err = inv.SentEpoch(ctx, n.ID); err != nil {
return out, err
}
if out.Generation, err = inv.SentGeneration(ctx, n.ID); err != nil {
return out, err
}
if out.ReadsModules, err = inv.ReadsModules(ctx, n.ID); err != nil {
return out, err
}
if out.SentBuilds, out.SentKnown, err = inv.SentBuilds(ctx, n.Name); err != nil {
return out, err
}
runs, err := inv.AssignmentRuns(ctx, n.Name)
if err != nil {
return out, err
}
if len(runs) > 0 {
out.Runs = make(map[string]compose.Run, len(runs))
for module, ref := range runs {
out.Runs[module] = compose.Run{Commit: ref.Commit, Build: ref.Build}
}
}
// A recorded module kept at the build this machine runs, in the context a send composes under (recorded_kept.go): its
// manifest and the commit, applied to this node's own resolution only.
// What cannot be kept refuses this machine alone (keepRecorded's refusal, as planFor returned it).
kept, err := recordedKept(ctx, open, n.Name)
if err != nil {
*refused = err
} else if len(kept) > 0 {
out.KeptBuilds = kept
if out.Kept, err = keptManifests(ctx, open, n.Name, kept); err != nil {
*refused = err
out.KeptBuilds, out.Kept = nil, nil
}
}
stated, has, err := inv.HealthOf(ctx, n.Name)
if err != nil {
return out, err
}
out.Stated = has
if has {
out.Contract = stated.Contract
}
return out, nil
}
// keptManifests is the manifest of each module kept at the build a machine runs: keepRecorded's reads, and its refusals in its
// words, without putting any of it into a shelf (that is the node's own Kept, applied by the composition).
func keptManifests(ctx context.Context, open *stores, node string, kept map[string]string) (map[string]catalogue.Manifest, error) {
// applyKept puts each manifest into the shelf it is given and refuses a build that cannot be read; the shelf here is a
// scratch one, whose entries are the kept manifests.
scratch := map[string]catalogue.Manifest{}
if err := applyKept(ctx, open, node, kept, scratch); err != nil {
return nil, err
}
return scratch, nil
}
func readTunnels(ctx context.Context, inv *inventory.Inventory) (map[string]compose.FoundTunnel, error) {
found, err := inv.Tunnels(ctx)
if err != nil {
return nil, err
}
if len(found) == 0 {
return nil, nil
}
out := make(map[string]compose.FoundTunnel, len(found))
for name, t := range found {
out[name] = compose.FoundTunnel{Tunnel: tunnelOf(t.Tunnel), NodeAdopted: t.NodeAdopted}
}
return out, nil
}
func readAdopted(ctx context.Context, inv *inventory.Inventory) (*compose.AdoptedTunnel, error) {
tunnel, hub, adopted, err := inv.AdoptedTunnel(ctx)
if err != nil {
return nil, err
}
if hub == "" {
return nil, nil
}
return &compose.AdoptedTunnel{Tunnel: tunnelOf(tunnel), Hub: hub, Adopted: adopted}, nil
}
func tunnelOf(t inventory.Tunnel) compose.Tunnel {
out := compose.Tunnel{Interface: t.Interface, Unit: t.Unit, Config: t.Config, Port: t.Port, MTU: t.MTU,
Address: t.Address, Range: t.Range, PublicKey: t.PublicKey}
for _, p := range t.Peers {
out.Peers = append(out.Peers, compose.TunnelPeer{PublicKey: p.PublicKey, Address: p.Address})
}
return out
}
// readLicences is every licence the mesh holds, with who uses it and the key sealed to each holder: given even where empty,
// so a holder the mesh holds no key for composes as that, and not as a key that was not asked for.
func readLicences(ctx context.Context, open *stores) ([]compose.Licence, error) {
held, err := open.Licences(ctx)
if err != nil {
return nil, err
}
all, err := held.All(ctx)
if err != nil {
return nil, err
}
var out []compose.Licence
for _, one := range all {
l := compose.Licence{Name: one.Name, Serves: one.Serves, Keys: map[compose.Holder]string{}}
holders, err := held.HoldersOf(ctx, one.Name)
if err != nil {
return nil, err
}
for _, h := range holders {
holder := compose.Holder{Node: h.Node, Module: h.Module}
l.Holders = append(l.Holders, holder)
key, err := held.KeyFor(ctx, one.Name, h.Node, h.Module)
if err != nil {
return nil, err
}
l.Keys[holder] = key
}
node, module, err := held.ManagerOf(ctx, one.Name)
if err != nil {
return nil, err
}
l.Manager = compose.Holder{Node: node, Module: module}
out = append(out, l)
}
return out, nil
}
// readSettings is each node's settings layers for every module that has any, read BEFORE any node is resolved: the first pass
// resolves a node whose set takes a provider's served facts from the node that runs it, with that node's settings on them
// (servedOnNode), so a mesh whose provider says ${setting:domain} resolves only with the settings in the Input. A module's
// settings that fail to read are returned in unread, by node and module, and are an error only for a module a node resolves to
// (unreadSettings): the controller read none for a module nothing runs.
func readSettings(ctx context.Context, open *stores, in *compose.Input) (unread map[string]map[string]error, err error) {
have, err := open.inventory.ModulesWithSettings(ctx)
if err != nil {
return nil, err
}
names := make([]string, 0, len(have))
for module := range have {
if _, onShelf := in.Catalogue[module]; onShelf {
names = append(names, module)
}
}
sort.Strings(names)
for i := range in.Nodes {
n := &in.Nodes[i]
for _, module := range names {
layers, err := open.inventory.SettingsFor(ctx, n.Name, module)
if err != nil {
if unread == nil {
unread = map[string]map[string]error{}
}
if unread[n.Name] == nil {
unread[n.Name] = map[string]error{}
}
unread[n.Name][module] = err
continue
}
if len(layers) == 0 {
continue
}
if n.Settings == nil {
n.Settings = map[string][]catalogue.Layer{}
}
n.Settings[module] = layers
}
}
return unread, nil
}
// unreadSettings is the error of a module's settings that did not read, for a module the node resolves to.
func unreadSettings(unread map[string]map[string]error, plans map[string]compose.Resolved) error {
names := make([]string, 0, len(plans))
for node := range plans {
names = append(names, node)
}
sort.Strings(names)
for _, node := range names {
for _, m := range plans[node].Plan.Modules {
if err := unread[node][m.Module]; err != nil {
return err
}
}
}
return nil
}
@@ -1,285 +0,0 @@
package main
import (
"context"
"sort"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/overlay"
"github.com/novox/mesh-controller/internal/inventory"
)
// The second half of the adapter (issue 1383): the credential and build fields of the Input, read and never made.
// readCredentialsAndBuilds fills what the first half left: the credentials, the own secrets, the vault's boxes, the bus,
// each node's serving key and certificate, and the builds.
func readCredentialsAndBuilds(ctx context.Context, open *stores, in *compose.Input, plans map[string]compose.Resolved,
shape inputShape, busRefused *error) ([]inventory.PairCredential, error) {
if err := readIdentity(ctx, open, in); err != nil {
return nil, err
}
wouldMake, err := readCredentials(ctx, open, in, plans, busRefused)
if err != nil {
return nil, err
}
return wouldMake, readBuilds(ctx, open, in, plans, shape)
}
// readIdentity is the mesh's authority and each node's serving key and the certificate kept for it, for a mesh in which a
// module asks for a certificate. It establishes and issues nothing: a certificate not kept yet is composed with a stand-in, and
// issued by the send (certificateFor).
func readIdentity(ctx context.Context, open *stores, in *compose.Input) error {
needs := false
for _, m := range in.Catalogue {
needs = needs || m.Certificate != nil
}
if !needs {
return nil
}
ident, err := open.Identity(ctx)
if err != nil {
return err
}
if certificate, known, err := ident.KnownAuthority(ctx); err != nil {
return err
} else if known {
in.Authority = certificate
}
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
return err
}
ids := map[string]string{}
for _, n := range nodes {
ids[n.Name] = n.ID
}
for i := range in.Nodes {
n := &in.Nodes[i]
if n.ServingKey, err = ident.ServingKeyOf(ctx, ids[n.Name]); err != nil {
return err
}
if n.ServingKey == "" {
continue
}
if certificate, known, err := ident.KeptCertificate(ctx, overlay.InternalName(n.Name), n.ServingKey); err != nil {
return err
} else if known {
n.Certificate = certificate
}
}
return nil
}
// readCredentials is how each pair credential a node takes from elsewhere stands, the credentials made for each provider's
// consumers, the own secrets, the vault's boxes and the bus. It reads and never makes: a credential a send would make is
// left out (the composition's would-make), a credential only a person gives and nobody gave is not-given, and one the send
// would be refused is refused with the controller's words.
func readCredentials(ctx context.Context, open *stores, in *compose.Input, plans map[string]compose.Resolved,
busRefused *error) ([]inventory.PairCredential, error) {
var wouldMake []inventory.PairCredential
inv := open.inventory
shelf := in.Catalogue
nodeOf := map[string]compose.Node{}
for _, n := range in.Nodes {
nodeOf[n.Name] = n
}
names := make([]string, 0, len(plans))
for name := range plans {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
resolved := plans[name]
nodeShelf := shelfWith(shelf, nodeOf[name].Kept)
for _, need := range resolved.Plan.Needs {
if need.ByRecord {
continue
}
p := inventory.PairCredential{Name: need.Name, Consumer: name, ConsumerModule: need.For,
Provider: need.From, Local: need.Local}
if module, outside := issuedOutsideBy(nodeShelf, need); outside {
p.Kind, p.ProviderModule = inventory.PairOutside, module
} else if need.SharedOwn != "" {
p.Kind, p.Own = inventory.PairShared, need.SharedOwn
p.ProviderModule = sharedProviderModule(resolved.Plan, shelf, need)
}
state, err := inv.PairCredentialState(ctx, p)
if err != nil {
return nil, err
}
c := compose.Credential{Provision: need.Name, Consumer: name, ConsumerModule: need.For,
Provider: need.From, Local: need.Local}
switch state.State {
case inventory.PairHeld:
c.State, c.Sealed = compose.CredentialHeld, state.Sealed
case inventory.PairNotGiven:
c.State, c.Why = compose.CredentialNotGiven, state.Why
case inventory.PairRefused:
c.State, c.Why = compose.CredentialRefused, state.Why
default:
// Left out: a need with no entry is the credential the next send makes, and perform makes it.
wouldMake = append(wouldMake, p)
continue
}
in.Credentials = append(in.Credentials, c)
}
}
// The credentials made for each provider's consumers, so the grants it must create are read from them.
for _, n := range in.Nodes {
rows, err := inv.SecretsFrom(ctx, n.Name)
if err != nil {
return nil, err
}
for _, s := range rows {
in.Issued = append(in.Issued, compose.Issued{Provision: s.Name, Consumer: s.Consumer,
ConsumerModule: s.ConsumerModule, Local: s.Local, Provider: n.Name, ForProvider: s.ForProvider})
}
}
own, err := inv.OwnSecretRows(ctx)
if err != nil {
return nil, err
}
for _, r := range own {
in.OwnSecrets = append(in.OwnSecrets, compose.OwnSecretRow{Node: r.Node, Module: r.Module, Name: r.Name,
Origin: r.Origin, SealedTo: r.SealedTo, Sealed: r.Sealed})
}
for _, n := range in.Nodes {
boxes, err := inv.KeptCoreBoxes(ctx, n.Name)
if err != nil {
return nil, err
}
for module, byName := range boxes {
for name, b := range byName {
in.KeptCoreBoxes = append(in.KeptCoreBoxes, compose.CoreBox{Node: n.Name, Module: module, Name: name,
Generation: b.Generation, SealedTo: b.SealedTo, Box: b.Box})
}
}
}
sort.Slice(in.KeptCoreBoxes, func(i, j int) bool {
a, b := in.KeptCoreBoxes[i], in.KeptCoreBoxes[j]
if a.Node != b.Node {
return a.Node < b.Node
}
if a.Module != b.Module {
return a.Module < b.Module
}
return a.Name < b.Name
})
// The bus: its user list as the controller composes it, for the machine that runs it, and who the mesh has minted a
// credential for.
var bus []catalogue.Manifest
for _, m := range shelf {
if m.BusUsers != "" && m.ClaimsSeat("mesh-broker") {
bus = append(bus, m)
}
}
if len(bus) > 0 {
list, missing, err := busUserList(ctx, inv, bus)
if err != nil && !strings.Contains(err.Error(), "not one user has a credential") {
return nil, err
}
if err != nil {
// The machine that runs the bus is refused, as the controller's own composition refused it: a list that
// refuses every connection in the mesh is never composed (refusedBus); the other machines compose.
*busRefused = err
}
in.BusUsers, in.BusMissing = list, missing
}
if in.BusMinted, err = inv.BusMinted(ctx); err != nil {
return nil, err
}
if in.BusMemberships, err = inv.BusMemberships(ctx); err != nil {
return nil, err
}
return wouldMake, nil
}
// readBuilds is the build half of the Input: every repository the mesh built, each module's current build, and the build made
// from a commit of a module for the machines whose node-engine reads them.
func readBuilds(ctx context.Context, open *stores, in *compose.Input, plans map[string]compose.Resolved, shape inputShape) error {
inv := open.inventory
held, err := inv.Held(ctx)
if err != nil {
return err
}
for repository := range held {
in.Built = append(in.Built, repository)
}
sort.Strings(in.Built)
in.CurrentBuilds = make(map[string]string, len(shape.current))
for module, b := range shape.current {
in.CurrentBuilds[module] = b.Commit
}
// The build made from each commit a declaration may carry, for the machines that read a build per module.
for _, n := range in.Nodes {
if !n.ReadsModules {
continue
}
resolved, ok := plans[n.Name]
if !ok {
continue
}
for _, m := range resolved.Plan.Modules {
for _, commit := range []string{shape.current[m.Module].Commit, shape.kept[n.Name][m.Module], shape.sent[n.Name][m.Module]} {
if commit == "" {
continue
}
if _, done := in.BuildIDs[m.Module][commit]; done {
continue
}
build, err := inv.BuildOf(ctx, m.Module, commit)
if err != nil {
return err
}
if in.BuildIDs == nil {
in.BuildIDs = map[string]map[string]string{}
}
if in.BuildIDs[m.Module] == nil {
in.BuildIDs[m.Module] = map[string]string{}
}
in.BuildIDs[m.Module][commit] = build
}
}
}
return nil
}
// shelfWith is the shelf with each module a node keeps at the build it runs put in place of the current one: a copy.
func shelfWith(shelf map[string]catalogue.Manifest, kept map[string]catalogue.Manifest) map[string]catalogue.Manifest {
if len(kept) == 0 {
return shelf
}
out := make(map[string]catalogue.Manifest, len(shelf)+len(kept))
for name, m := range shelf {
out[name] = m
}
for name, m := range kept {
out[name] = m
}
return out
}
// sharedProviderModule is which module answers a shared credential on the providing node: the one in the plan, else the first
// the catalogue names (providerModuleOf, in an order that does not move between calls).
func sharedProviderModule(plan catalogue.Resolution, shelf map[string]catalogue.Manifest, n catalogue.Needed) string {
for _, m := range plan.Modules {
if _, shared := m.SharedCredentialOf(n.Name); shared {
return m.Module
}
}
names := make([]string, 0, len(shelf))
for name := range shelf {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
if _, shared := shelf[name].SharedCredentialOf(n.Name); shared {
return name
}
}
return ""
}
@@ -1,354 +0,0 @@
//go:build integration
package main
import (
"encoding/json"
"fmt"
"sort"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/inventory"
)
// tableState is every table's rows and every sequence's last value, as the adapter's reads must leave them.
func tableState(t *testing.T, inv *inventory.Inventory) map[string]string {
t.Helper()
out := map[string]string{}
for _, row := range inv.QueryForTest(t, `select table_name from information_schema.tables
where table_schema = 'public' and table_type = 'BASE TABLE' order by table_name`) {
name := strings.Trim(row[0], `"`)
count := inv.QueryForTest(t, fmt.Sprintf(`select count(*)::text from %q`, name))
out["rows:"+name] = fmt.Sprint(count)
digest := inv.QueryForTest(t, fmt.Sprintf(`select coalesce(md5(string_agg(t::text, '|' order by t::text)), '') from %q t`, name))
out["content:"+name] = fmt.Sprint(digest)
}
for _, row := range inv.QueryForTest(t, `select sequencename, coalesce(last_value::text, '') from pg_sequences
where schemaname = 'public' order by sequencename`) {
out["sequence:"+row[0]] = row[1]
}
return out
}
// 1382/AC4 and 1383/AC6: building the Input writes nothing. The store's rows, their contents and its sequences are the same
// before and after, on a mesh whose credentials are not made, whose ports are not given and which keeps a recorded build.
func TestBuildingTheInputWritesNothing(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
// A consumer whose credential nothing has made yet, on a mesh that already composed.
register(t, open, catalogue.Manifest{Module: "reports", Version: "1", Requires: []string{"s3-bucket"},
Resources: []map[string]any{{"id": "config", "type": "file", "path": "/etc/reports/config.json", "mode": "0600",
"content": "{}", "merge": "json"}}})
if _, err := inv.Assign(ctx, "laptop", "reports"); err != nil {
t.Fatal(err)
}
before := tableState(t, inv)
if _, err := readComposeInput(ctx, open); err != nil {
t.Fatal(err)
}
after := tableState(t, inv)
var moved []string
for k, v := range after {
if before[k] != v {
moved = append(moved, k)
}
}
sort.Strings(moved)
if len(moved) > 0 {
t.Fatalf("building the Input wrote: %v", moved)
}
}
// a consumer of the s3-bucket aMeshWithSecrets' provider offers, with a file the credential goes into.
func aConsumerOfTheBucket(name string) catalogue.Manifest {
return catalogue.Manifest{Module: name, Version: "1", Requires: []string{"s3-bucket"},
Secrets: map[string]string{"s3-bucket": "/etc/" + name + "/s3"},
Resources: []map[string]any{{"id": "config", "type": "file", "path": "/etc/" + name + "/config.json", "mode": "0600",
"content": "{}", "merge": "json"}}}
}
// 1383/AC1: of a provider's three consumers, two have their credentials made and one has not: Issued holds the two rows, with
// the copy sealed to the provider, the third is a credential the next send makes, and no row was inserted for it (planFor
// inserted it, plan.go: SecretFor in every mode).
func TestAPairCredentialNoRowHoldsIsAWouldMakeAndNoRowIsInserted(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
register(t, open, aConsumerOfTheBucket("reports"))
if _, err := inv.Assign(ctx, "laptop", "reports"); err != nil {
t.Fatal(err)
}
// A send makes the credentials of files and reports; a question makes none.
if _, err := composeSend(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
register(t, open, aConsumerOfTheBucket("metrics"))
if _, err := inv.Assign(ctx, "laptop", "metrics"); err != nil {
t.Fatal(err)
}
before := tableState(t, inv)
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
var rows []compose.Issued
for _, r := range in.Issued {
if r.Provider == "anchor" && r.Provision == "s3-bucket" {
rows = append(rows, r)
if r.ForProvider == "" {
t.Errorf("the row of %s carries no copy sealed to the provider", r.ConsumerModule)
}
}
}
if len(rows) != 2 {
t.Fatalf("Issued holds %d row(s) for anchor's s3-bucket, want files and reports: %+v", len(rows), rows)
}
var would []string
for _, p := range in.wouldMake {
would = append(would, p.ConsumerModule)
}
if len(would) != 1 || would[0] != "metrics" {
t.Fatalf("the credentials the next send makes are %v, want metrics", would)
}
for _, c := range in.Credentials {
if c.ConsumerModule == "metrics" {
t.Errorf("a credential nobody holds has an entry: %+v", c)
}
}
after := tableState(t, inv)
for k, v := range after {
if before[k] != v {
t.Errorf("reading the Input moved %s", k)
}
}
}
// 1383/AC2: a credential only a person gives that nobody gave is not given, and so is one given under a key the machine no
// longer has; a credential given by a person for the pair and sealed to a key one end no longer has is refused: in the words
// plan.go and secrets.go use.
func TestACredentialOnlyAPersonGivesIsNotGivenAndAGivenOneUnderAnOldKeyIsSaidSo(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, catalogue.Manifest{Module: "indexers", Version: "1", Provides: []catalogue.Offer{{
Name: "indexer-api", Scope: catalogue.ScopeMesh, Credential: &catalogue.OfferCredential{IssuedBy: catalogue.IssuedOutside}}}})
register(t, open, catalogue.Manifest{Module: "grabber", Version: "1", Requires: []string{"indexer-api"},
Secrets: map[string]string{"indexer-api": "/etc/grabber/key"}})
register(t, open, aConsumerOfTheBucket("ledger"))
register(t, open, catalogue.Manifest{Module: "objects", Version: "1", Provides: catalogue.FromAnywhere("s3-bucket")})
for _, a := range [][2]string{{"anchor", "indexers"}, {"laptop", "grabber"}, {"anchor", "objects"}, {"laptop", "ledger"}} {
if _, err := inv.Assign(ctx, a[0], a[1]); err != nil {
t.Fatalf("assign %s %s: %v", a[0], a[1], err)
}
}
stateOf := func(provision, consumerModule string) (compose.Credential, bool) {
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
for _, c := range in.Credentials {
if c.Provision == provision && c.ConsumerModule == consumerModule {
return c, true
}
}
return compose.Credential{}, false
}
c, found := stateOf("indexer-api", "grabber")
if !found || c.State != compose.CredentialNotGiven || !strings.Contains(c.Why, "nobody gave it") {
t.Fatalf("a credential nobody gave reads as %+v", c)
}
if err := inv.AcceptSecretForPair(ctx, "indexer-api", "laptop", "grabber", "anchor", "", "the-key"); err != nil {
t.Fatal(err)
}
if c, found := stateOf("indexer-api", "grabber"); !found || c.State != compose.CredentialHeld || c.Sealed == "" {
t.Fatalf("a credential a person gave reads as %+v", c)
}
// The consumer's machine generated a key of its own since.
if err := inv.RecordSealingKey(ctx, nodeID(t, open, "laptop"), aPublicKey(t)); err != nil {
t.Fatal(err)
}
if c, found := stateOf("indexer-api", "grabber"); !found || c.State != compose.CredentialNotGiven ||
!strings.Contains(c.Why, "a sealing key one end no longer has") {
t.Fatalf("a credential given under a key the machine no longer has reads as %+v", c)
}
// A credential for a pair, given by a person, under a key one end has since replaced: the mesh cannot make another.
if err := inv.RecordSealingKey(ctx, nodeID(t, open, "laptop"), aPublicKey(t)); err != nil {
t.Fatal(err)
}
if err := inv.AcceptSecretForPair(ctx, "s3-bucket", "laptop", "ledger", "anchor", "", "the-bucket-key"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSealingKey(ctx, nodeID(t, open, "laptop"), aPublicKey(t)); err != nil {
t.Fatal(err)
}
if c, found := stateOf("s3-bucket", "ledger"); !found || c.State != compose.CredentialRefused ||
!strings.Contains(c.Why, "was accepted from a person, and a sealing key at one end has changed since") {
t.Fatalf("a credential given for the pair under an old key reads as %+v", c)
}
}
// 1383/AC3: the own secrets carry their origin, the key they are sealed to and the sealed value; the vault's kept boxes, the
// licences (with their keys given, so an empty key means the mesh holds none), the users the bus has minted and the
// memberships equal what the inventory answers.
func TestTheAdaptersInputCarriesTheOwnSecretsTheBoxesTheLicencesAndTheBus(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
key, err := inv.SealingKeyOf(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
sealed, err := inv.SecretForModule(ctx, "anchor", "objects", "admin")
if err != nil {
t.Fatal(err)
}
box := json.RawMessage(`{"v":1,"eph":"e","iv":"i","tag":"t","ct":"c"}`)
if err := inv.KeepCoreBox(ctx, "anchor", inventory.CoreBox{Module: "objects", Name: "admin", Generation: 4,
SealedTo: key, Box: box}); err != nil {
t.Fatal(err)
}
if err := inv.SeedBusUser(ctx, inventory.BusUser{Username: "anchor.objects", Kind: "module", Node: "anchor",
Module: "objects"}, "a-password"); err != nil {
t.Fatal(err)
}
if err := inv.PutBusMembership(ctx, "laptop", "sealed-membership"); err != nil {
t.Fatal(err)
}
held, err := open.Licences(ctx)
if err != nil {
t.Fatal(err)
}
if err := held.Add(ctx, "claude", "static", map[string]any{"model": "x"}); err != nil {
t.Fatal(err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
var own *compose.OwnSecretRow
for i := range in.OwnSecrets {
if r := &in.OwnSecrets[i]; r.Node == "anchor" && r.Module == "objects" && r.Name == "admin" {
own = r
}
}
if own == nil || own.Origin != "made" || own.SealedTo != key || own.Sealed != sealed {
t.Errorf("the own secret reads as %+v; the inventory holds %q sealed to %q", own, sealed, key)
}
var kept *compose.CoreBox
for i := range in.KeptCoreBoxes {
if b := &in.KeptCoreBoxes[i]; b.Node == "anchor" && b.Module == "objects" && b.Name == "admin" {
kept = b
}
}
if kept == nil || kept.Generation != 4 || kept.SealedTo != key || string(kept.Box) != string(box) {
t.Errorf("the kept box reads as %+v", kept)
}
if got := strings.Join(in.BusMinted, ","); !strings.Contains(got, "anchor.objects") {
t.Errorf("BusMinted is %v", in.BusMinted)
}
if in.BusMemberships["laptop"] != "sealed-membership" {
t.Errorf("BusMemberships is %v", in.BusMemberships)
}
if len(in.Licences) != 1 || in.Licences[0].Name != "claude" || in.Licences[0].Keys == nil {
t.Errorf("Licences read as %+v", in.Licences)
}
}
// 1383/AC4: the bus's user list is what busUserList gives for the machine that runs the bus, and the Input carries none when
// no module of the shelf runs the bus.
func TestTheBusUsersAreWhatTheControllerComposesForTheMachineThatRunsTheBus(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
if in.BusUsers != "" || len(in.BusMissing) != 0 {
t.Fatalf("a mesh whose shelf runs no bus carries a bus user list: %q %v", in.BusUsers, in.BusMissing)
}
bus := catalogue.Manifest{Module: "nats", Version: "1", BusUsers: "/etc/nats/users.conf",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}}}
register(t, open, bus)
if err := open.inventory.SeedBusUser(ctx, inventory.BusUser{Username: "controller", Kind: "controller"}, "pw"); err != nil {
t.Fatal(err)
}
in, err = readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
list, missing, _ := busUserList(ctx, open.inventory, []catalogue.Manifest{bus})
if in.BusUsers != list || fmt.Sprint(in.BusMissing) != fmt.Sprint(missing) {
t.Errorf("the Input carries %q (%v); busUserList gives %q (%v)", in.BusUsers, in.BusMissing, list, missing)
}
}
// 1383/AC5: Built, CurrentBuilds and BuildIDs equal the inventory's Held, CurrentBuilds and BuildOf, and the serving key and
// the certificate are the machine's.
func TestTheAdaptersInputCarriesTheBuildsAndTheMachinesKey(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
held, err := inv.Held(ctx)
if err != nil {
t.Fatal(err)
}
var wantBuilt []string
for repository := range held {
wantBuilt = append(wantBuilt, repository)
}
sort.Strings(wantBuilt)
if fmt.Sprint(in.Built) != fmt.Sprint(wantBuilt) && !(len(in.Built) == 0 && len(wantBuilt) == 0) {
t.Errorf("Built is %v, the inventory's %v", in.Built, wantBuilt)
}
current, err := inv.CurrentBuilds(ctx)
if err != nil {
t.Fatal(err)
}
for module, b := range current {
if in.CurrentBuilds[module] != b.Commit {
t.Errorf("CurrentBuilds[%s] is %q, the inventory's %q", module, in.CurrentBuilds[module], b.Commit)
}
}
for module, byCommit := range in.BuildIDs {
for commit, build := range byCommit {
if want, _ := inv.BuildOf(ctx, module, commit); want != build {
t.Errorf("BuildIDs[%s][%s] is %q, the inventory's %q", module, commit, build, want)
}
}
}
}
// The lead's answer 4: where the mesh holds no credential for any bus user, the machine that runs the bus is refused with
// busUserList's words (the controller's composition refused it), the Input still builds, and another machine composes.
func TestTheBusMachineIsRefusedWhereNoUserHasACredentialAndTheOthersCompose(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
bus := catalogue.Manifest{Module: "nats", Version: "1", BusUsers: "/etc/nats/users.conf",
Claims: []catalogue.Claim{{Name: "mesh-broker", Scope: catalogue.ScopeMesh}}}
register(t, open, bus)
register(t, open, aWebModule())
if _, err := assign(ctx, open, "anchor", "nats"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "web"); err != nil {
t.Fatal(err)
}
if _, _, err := composeQuestion(ctx, open, "anchor", compose.Foreseeing, false); err == nil ||
!strings.Contains(err.Error(), "not one user has a credential") {
t.Fatalf("the bus machine is composed: %v", err)
}
if _, _, err := composeQuestion(ctx, open, "laptop", compose.Foreseeing, false); err != nil {
t.Fatalf("a machine that does not run the bus is refused: %v", err)
}
}
@@ -1,329 +0,0 @@
//go:build integration
package main
import (
"reflect"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/link"
)
// 1382: the adapter's Input names, for every node, what the inventory answers for it, each compared with the read named
// beside the field in go/compose/input.go (novox/hq ADR 0453 clause 3).
func TestTheAdaptersInputNamesWhatTheInventoryAnswersForEachNode(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
if err := inv.SetPublicDomain(ctx, "anchor", "example.test"); err != nil {
t.Fatal(err)
}
if _, err := inv.PortFor(ctx, "laptop", "files", 8080, false); err != nil {
t.Fatal(err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
shelf, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(in.Catalogue, shelf) {
t.Error("the Input's Catalogue is not the shelf the inventory holds")
}
nodes, err := inv.Nodes(ctx)
if err != nil {
t.Fatal(err)
}
if len(in.Nodes) != len(nodes) {
t.Fatalf("the Input names %d node(s), the inventory has %d", len(in.Nodes), len(nodes))
}
places, err := inv.Overlays(ctx)
if err != nil {
t.Fatal(err)
}
for _, record := range nodes {
n, ok := nodeOf(in.Input, record.Name)
if !ok {
t.Fatalf("%s is not in the Input", record.Name)
}
for _, p := range places {
if p.Name == record.Name {
if want := (compose.Place{Key: p.Key, Endpoint: p.Endpoint, Site: p.Site, Hub: p.Hub, Address: p.Address}); n.Place != want {
t.Errorf("%s: Place is %+v, the inventory's %+v", record.Name, n.Place, want)
}
}
}
assigned, _ := inv.Assigned(ctx, record.Name)
if !reflect.DeepEqual(n.Assigned, assigned) {
t.Errorf("%s: Assigned is %v, the inventory's %v", record.Name, n.Assigned, assigned)
}
caps, _ := inv.ProfileOf(ctx, record.Name)
if !reflect.DeepEqual(n.Capabilities, caps) {
t.Errorf("%s: Capabilities are %v, the inventory's %v", record.Name, n.Capabilities, caps)
}
domain, _ := inv.PublicDomainOf(ctx, record.Name)
if n.PublicDomain != domain {
t.Errorf("%s: PublicDomain is %q, the inventory's %q", record.Name, n.PublicDomain, domain)
}
if n.Account != record.Account || n.AccountHome != record.AccountHome || n.AgentAccount != record.AgentAccount ||
n.AgentAccountHome != record.AgentAccountHome || n.Adopted != record.Adopted {
t.Errorf("%s: the accounts or the mode differ from the inventory's: %+v", record.Name, n)
}
pins, _ := inv.PinsFor(ctx, record.Name)
if len(pins) != len(n.Pins) || (len(pins) > 0 && !reflect.DeepEqual(n.Pins, pins)) {
t.Errorf("%s: Pins are %v, the inventory's %v", record.Name, n.Pins, pins)
}
bound, _ := inv.BindingsFor(ctx, record.Name)
if len(bound) != len(n.Bound) || (len(bound) > 0 && !reflect.DeepEqual(n.Bound, bound)) {
t.Errorf("%s: Bound is %v, the inventory's %v", record.Name, n.Bound, bound)
}
key, _ := inv.SealingKeyOf(ctx, record.Name)
if n.SealingKey != key {
t.Errorf("%s: SealingKey is %q, the inventory's %q", record.Name, n.SealingKey, key)
}
ports, _ := inv.PortsFor(ctx, record.Name)
want := map[string]map[int]int{}
for _, a := range ports {
if want[a.Module] == nil {
want[a.Module] = map[int]int{}
}
want[a.Module][a.Wanted] = a.Machine
}
if len(want) != len(n.Ports) || (len(want) > 0 && !reflect.DeepEqual(n.Ports, want)) {
t.Errorf("%s: Ports are %v, the inventory's %v", record.Name, n.Ports, want)
}
// The settings of every module it resolves to, layer for layer.
resolved, err := compose.Resolve(in.Input, record.Name)
if err != nil {
continue
}
for _, m := range resolved.Plan.Modules {
layers, err := inv.SettingsFor(ctx, record.Name, m.Module)
if err != nil {
t.Fatal(err)
}
if len(layers) != len(n.Settings[m.Module]) || (len(layers) > 0 && !reflect.DeepEqual(n.Settings[m.Module], layers)) {
t.Errorf("%s: the settings of %s are %v, the inventory's %v", record.Name, m.Module, n.Settings[m.Module], layers)
}
}
}
if domain := nodeDomain(in.Input, "anchor"); domain != "example.test" {
t.Errorf("anchor's PublicDomain is %q", domain)
}
if got := nodePorts(in.Input, "laptop"); got["files"][8080] == 0 {
t.Errorf("laptop's port for files:8080 is not in the Input: %v", got)
}
}
func nodeDomain(in compose.Input, name string) string {
n, _ := nodeOf(in, name)
return n.PublicDomain
}
func nodePorts(in compose.Input, name string) map[string]map[int]int {
n, _ := nodeOf(in, name)
return n.Ports
}
// 1382/AC2: a node keeping a recorded module at the build it runs has that module in Kept and KeptBuilds, and the Input's
// Catalogue is the current shelf, not the kept manifest (R4a review round 1, finding 1).
func TestANodeKeepingARecordedBuildHasItInKeptAndTheCatalogueStaysCurrent(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
start := time.Now().Add(-time.Hour)
image := "registry.invalid:5000/postgres/server@sha256:" + strings.Repeat("a", 64)
for _, b := range []link.BuildResult{
aContainerBuild(t, "postgres", "c1111111", catalogue.PolicyRecord, start, map[string][2]string{"server": {image, ""}}),
} {
if _, _, err := takeIn(ctx, inv, asTheOperator(t, inv, b)); err != nil {
t.Fatal(err)
}
}
if _, err := inv.Assign(ctx, "anchor", "postgres"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, open, "anchor"), "d-anchor", map[string]string{"postgres": "c1111111"}); err != nil {
t.Fatal(err)
}
if _, _, err := takeIn(ctx, inv, aContainerBuild(t, "postgres", "c2222222", catalogue.PolicyRecord, start.Add(time.Minute),
map[string][2]string{"server": {image, "runtime"}})); err != nil {
t.Fatal(err)
}
// A send that is not a person's push keeps the build the machine runs.
in, err := readComposeInput(keepingRecorded(ctx), open)
if err != nil {
t.Fatal(err)
}
anchor, _ := nodeOf(in.Input, "anchor")
if anchor.KeptBuilds["postgres"] != "c1111111" {
t.Fatalf("anchor keeps %v, want postgres at c1111111", anchor.KeptBuilds)
}
kept, found := anchor.Kept["postgres"]
if !found {
t.Fatalf("the kept manifest of postgres is not in anchor's Kept: %v", anchor.Kept)
}
current, err := inv.Catalogue(ctx)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(in.Catalogue["postgres"], current["postgres"]) {
t.Error("the Input's Catalogue is not the current shelf")
}
if reflect.DeepEqual(kept, current["postgres"]) {
t.Error("the kept manifest is the current one: nothing is kept")
}
if laptop, _ := nodeOf(in.Input, "laptop"); len(laptop.Kept) != 0 || len(laptop.KeptBuilds) != 0 {
t.Errorf("laptop keeps what only anchor runs: %v", laptop.KeptBuilds)
}
// A person's push keeps nothing.
pushed, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
if a, _ := nodeOf(pushed.Input, "anchor"); len(a.Kept) != 0 || len(a.KeptBuilds) != 0 {
t.Errorf("a person's push keeps %v", a.KeptBuilds)
}
}
// 1382/AC3: the seats on record, the tunnels, the adopted tunnel, its carried peers and the nodes joining equal the
// inventory's answers; the range and the suffix are read from the environment once, here.
func TestTheAdaptersInputCarriesTheNetworkAsTheInventoryHoldsIt(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
t.Setenv(OverlayCIDRVar, "10.99.0.0/16")
t.Setenv("MESH_INTERNAL_SUFFIX", "mesh.test")
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
holdings, err := inv.Holdings(ctx)
if err != nil {
t.Fatal(err)
}
if !reflect.DeepEqual(in.Holdings, holdings) {
t.Errorf("Holdings are %v, the inventory's %v", in.Holdings, holdings)
}
tunnels, err := inv.Tunnels(ctx)
if err != nil {
t.Fatal(err)
}
if len(tunnels) != len(in.Tunnels) {
t.Errorf("Tunnels are %v, the inventory's %v", in.Tunnels, tunnels)
}
if _, hub, _, err := inv.AdoptedTunnel(ctx); err != nil {
t.Fatal(err)
} else if (hub == "") != (in.Adopted == nil) {
t.Errorf("Adopted is %v, the inventory's hub is %q", in.Adopted, hub)
}
carried, err := inv.CarriedPeers(ctx)
if err != nil {
t.Fatal(err)
}
if len(carried) != len(in.CarriedPeers) {
t.Errorf("CarriedPeers are %v, the inventory's %v", in.CarriedPeers, carried)
}
joining, err := inv.NodesWithALiveToken(ctx)
if err != nil {
t.Fatal(err)
}
if len(joining) != len(in.Joining) {
t.Errorf("Joining is %v, the inventory's %v", in.Joining, joining)
}
if in.OverlayCIDR != "10.99.0.0/16" || in.InternalSuffix != "mesh.test" {
t.Errorf("the range and the suffix are %q and %q", in.OverlayCIDR, in.InternalSuffix)
}
// Read once: what the environment says afterwards moves nothing already read.
t.Setenv(OverlayCIDRVar, "10.1.0.0/16")
if in.OverlayCIDR != "10.99.0.0/16" {
t.Error("the range moved with the environment")
}
}
// The lead's equality finding (build 1792487777757012513): a provider on one machine serves a fact that says ${setting:domain},
// the setting is set for it, and a consumer on another machine takes that fact. The first pass of the adapter resolved every
// node with no settings in the Input, so the served fact could not be settled and the whole Input failed. The settings are in
// the Input before any node is resolved, and the Input builds, with the same plan for the consumer as the controller's own
// composition. It failed first, with the finding's words.
func TestAServedFactThatSaysASettingIsResolvedWithTheSettingInTheInput(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "step-ca", Version: "1",
Provides: []catalogue.Offer{{Name: "acme-ca", Scope: catalogue.ScopeMesh}},
Serves: map[string]map[string]any{"acme-ca": {"domain": "${setting:domain}"}}})
register(t, open, catalogue.Manifest{Module: "route-proxy", Version: "1", Requires: []string{"acme-ca"}})
// The setting is there before the machines are assigned: an assignment resolves the mesh, and a served fact that says
// ${setting:domain} resolves only with it.
if err := open.inventory.KeepSettings(ctx, "", "step-ca", map[string]any{"domain": "example.test"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "step-ca"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "route-proxy"); err != nil {
t.Fatal(err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatalf("the Input cannot be built: %v", err)
}
if anchor, _ := nodeOf(in.Input, "anchor"); len(anchor.Settings["step-ca"]) == 0 {
t.Fatalf("the Input does not carry the setting of the provider: %v", anchor.Settings)
}
oldPlan, _, err := resolveNode(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
newPlan, _, err := resolveNode(withInputCache(ctx), open, "laptop")
if err != nil {
t.Fatalf("the consumer does not resolve: %v", err)
}
if !reflect.DeepEqual(oldPlan.Needs, newPlan.Needs) {
t.Errorf("the consumer's needs are %+v, the controller's own %+v", newPlan.Needs, oldPlan.Needs)
}
t.Logf("a served fact that says a setting resolves with the setting in the Input: the Input builds and the consumer's needs equal planFor's")
}
// A setting that nothing sets, in a fact a provider serves, is not the adapter's to hide: the controller's own composition
// refuses it as well (theRestOfTheMesh returns servedOnNode's error), and the adapter refuses with the same words rather than
// building an Input that composes a placeholder.
func TestAServedFactThatSaysAnUnsetSettingIsRefusedAsTheControllerRefusesIt(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, catalogue.Manifest{Module: "step-ca", Version: "1",
Provides: []catalogue.Offer{{Name: "acme-ca", Scope: catalogue.ScopeMesh}},
Serves: map[string]map[string]any{"acme-ca": {"domain": "${setting:domain}"}}})
register(t, open, catalogue.Manifest{Module: "route-proxy", Version: "1", Requires: []string{"acme-ca"}})
// Assigned with the setting there, then the setting is cleared: the mesh the controller holds when someone clears it.
if err := open.inventory.KeepSettings(ctx, "", "step-ca", map[string]any{"domain": "example.test"}); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "anchor", "step-ca"); err != nil {
t.Fatal(err)
}
if _, err := assign(ctx, open, "laptop", "route-proxy"); err != nil {
t.Fatal(err)
}
if err := open.inventory.ClearSettings(ctx, "", "step-ca"); err != nil {
t.Fatal(err)
}
_, _, oldErr := resolveNode(ctx, open, "laptop")
_, newErr := readComposeInput(ctx, open)
if (oldErr == nil) != (newErr == nil) {
t.Fatalf("the controller's composition says %v, the adapter says %v", oldErr, newErr)
}
if oldErr != nil && !strings.Contains(newErr.Error(), "nothing sets \"domain\"") {
t.Errorf("the adapter's words are %q, the controller's %q", newErr, oldErr)
}
t.Logf("an unset setting in a served fact is refused alike: the controller says %v, the adapter says %v", oldErr != nil, newErr != nil)
}
@@ -1,54 +0,0 @@
//go:build integration
package main
import (
"context"
"fmt"
"sort"
)
// made is what a send made in a store, by key and never by value: the machine ports given, the pair credentials and the
// own secrets.
type made struct {
Ports []string
Pairs []string
OwnSecrets []string
}
// madeIn is what a store holds that a send makes, by key and never by value.
func madeIn(ctx context.Context, open *stores) (made, error) {
var out made
inv := open.inventory
nodes, err := inv.Nodes(ctx)
if err != nil {
return out, err
}
for _, n := range nodes {
ports, err := inv.PortsFor(ctx, n.Name)
if err != nil {
return out, err
}
for _, p := range ports {
out.Ports = append(out.Ports, fmt.Sprintf("%s %s:%d->%d", n.Name, p.Module, p.Wanted, p.Machine))
}
rows, err := inv.SecretsFrom(ctx, n.Name)
if err != nil {
return out, err
}
for _, s := range rows {
out.Pairs = append(out.Pairs, fmt.Sprintf("%s %s on %s from %s local=%s", s.Name, s.ConsumerModule, s.Consumer, n.Name, s.Local))
}
}
own, err := inv.OwnSecretRows(ctx)
if err != nil {
return out, err
}
for _, r := range own {
out.OwnSecrets = append(out.OwnSecrets, fmt.Sprintf("%s %s/%s (%s)", r.Node, r.Module, r.Name, r.Origin))
}
sort.Strings(out.Ports)
sort.Strings(out.Pairs)
sort.Strings(out.OwnSecrets)
return out, nil
}
@@ -1,44 +0,0 @@
//go:build integration
package main
import (
"testing"
"git.novox.be/novox/mesh-sdk/go/compose"
)
// 1382/AC5: composing every node of one operation builds the Input once; without an operation's cache, once for each.
func TestComposingEveryNodeOfAnOperationBuildsTheInputOnce(t *testing.T) {
open, _ := aMeshWithSecrets(t)
built := 0
was := composeInputBuilt
composeInputBuilt = func() { built++ }
t.Cleanup(func() { composeInputBuilt = was })
ctx := withInputCache(t.Context())
nodes, err := open.inventory.Nodes(ctx)
if err != nil {
t.Fatal(err)
}
for _, n := range nodes {
if _, _, err := resolveNode(ctx, open, n.Name); err != nil && !unresolvable(err) {
t.Fatal(err)
}
if _, _, err := composeQuestion(ctx, open, n.Name, compose.Reading, false); err != nil && !unresolvable(err) {
t.Fatal(err)
}
}
if _, err := generatorsOf(ctx, open); err != nil {
t.Fatal(err)
}
if built != 1 {
t.Fatalf("an operation over %d node(s) built the Input %d time(s)", len(nodes), built)
}
built = 0
if _, _, err := resolveNode(t.Context(), open, nodes[0].Name); err != nil && !unresolvable(err) {
t.Fatal(err)
}
if built != 1 {
t.Fatalf("a composition with no cache built the Input %d time(s)", built)
}
}
-259
View File
@@ -1,259 +0,0 @@
package main
import (
"context"
"errors"
"fmt"
"io"
"os"
"slices"
"sort"
"strconv"
"strings"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/inventory"
)
// perform makes what a composition named foreseen (novox/hq ADR 0453 clause 3, design 47 R4b, issue 1385).
//
// **The composition makes nothing; a send does.** mesh-sdk's composition over an Input names what the next send would make
// (the ports it would assign, the credentials it would seal, the core boxes it would ask the vault for, the seats it would
// pass on, the certificate it would issue) and stands in for each. A send composes, performs, reads the stores again and
// composes again (send in compose_sdk.go), and refuses to send a declaration that still names anything foreseen. A question
// (status, facts, probes, the merge gate's reading) never reaches this file.
//
// What is made is exactly what the controller's own composition made on the way to a declaration it was going to send:
// - every pair credential no row holds, for every machine that resolves, since planning one node resolved them all;
// - the seats the mesh passes on, with the hand act (not on a throwaway mesh, whose hand act has nowhere to go);
// - the machine ports the node's modules were given none for;
// - the node's own secrets, by the controller's own rules (a bus credential nobody issued was refused by the composition);
// - the core credentials, asked of the vault every send, so a rotated one reaches the machine;
// - the certificate of a machine that was never certified.
//
// perform returns whether it made or recorded anything: the reason the caller reads the stores again.
func perform(ctx context.Context, open *stores, in *meshInput, node string, d compose.Declared) (bool, error) {
did := false
inv := open.inventory
// The seats passed on: recorded in the store with the controller's own hand act (ADR 0449 §4, issue 1317).
if len(d.Passes.Passes) > 0 {
var w io.Writer = os.Stderr
if lw, ok := ctx.Value(leftOutKey{}).(io.Writer); ok {
w = lw
}
for _, line := range d.Passes.Said {
fmt.Fprintln(w, line)
}
if performSeatPasses(ctx, seatPassingOn(inv, w), d.Passes.Passes) {
did = true
}
}
// Every pair credential the stores hold none for: made once and kept, and sealed to both ends before it was ever written
// down. The error is the one planFor met: the machine that needs it, and why none could be made.
for _, p := range in.wouldMake {
if err := makePair(ctx, inv, p); err != nil {
return did, fmt.Errorf("%s on %s needs %s from %s and no credential could be made for it: %w",
p.ConsumerModule, p.Consumer, p.Name, p.Provider, err)
}
did = true
}
// The ports this node's modules were given none for.
if len(d.ForeseenPorts) > 0 {
shelf, err := inv.Catalogue(ctx)
if err != nil {
return did, err
}
for _, one := range d.ForeseenPorts {
module, port, ok := strings.Cut(one, ":")
wanted, perr := strconv.Atoi(port)
if !ok || perr != nil {
return did, fmt.Errorf("the composition named a port the controller cannot read: %q", one)
}
fixed := false
for _, l := range nodeShelf(in, shelf, node)[module].Listens {
if l.Port == wanted {
fixed = l.Fixed
}
}
if _, err := inv.PortFor(ctx, node, module, wanted, fixed); err != nil {
return did, fmt.Errorf("%s needs %d reachable on %s and it could not be assigned: %w", module, wanted, node, err)
}
did = true
}
}
// The node's own secrets: a module's superuser password, an administrator, an account. The composition refused what the
// send refuses (a bus credential nobody issued, a given value under an old key).
for _, one := range d.StoodIn {
module, name, ok := strings.Cut(one, "/")
if !ok || strings.Contains(one, " ") {
continue // a grant's sealed copy, the certificate or the bus's list: made below or by nobody
}
if _, err := inv.SecretForModule(ctx, node, module, name); err != nil {
return did, err
}
did = true
}
// The certificate of a machine that was never certified: issued and kept, with the authority when there is none yet.
for _, one := range d.StoodIn {
if strings.HasPrefix(one, "the certificate and the authority of ") {
if _, _, err := certificateFor(ctx, open, node); err != nil {
return did, err
}
did = true
break
}
}
return did, nil
}
// makePair makes one pair credential by the controller's own read of it: the credential for the pair, or the provider's one
// credential sealed to this consumer too. A credential only a person gives is never made (it is not a would-make).
func makePair(ctx context.Context, inv *inventory.Inventory, p inventory.PairCredential) error {
var err error
switch p.Kind {
case inventory.PairShared:
_, err = inv.SharedSecretFor(ctx, p.Name, p.Consumer, p.ConsumerModule, p.Provider, p.ProviderModule, p.Local, p.Own)
case inventory.PairPlain:
_, err = inv.SecretFor(ctx, p.Name, p.Consumer, p.ConsumerModule, p.Provider, p.Local)
default:
return errors.New("a credential only a person gives is not made")
}
if inventory.IsNotGiven(err) {
return nil
}
return err
}
// stillForeseen says what a composed declaration still stands in for: the lists the composition names, and the stand-in itself
// anywhere in the body (a pair credential's stand-in is not in any list). A declaration that has any is never sent.
func stillForeseen(d compose.Declared) []string {
var left []string
left = append(left, d.Foreseen...)
left = append(left, d.StoodIn...)
for _, p := range d.ForeseenPorts {
left = append(left, "the port "+p)
}
for _, c := range d.CoreBoxesWanted {
left = append(left, "the vault's box for "+c.Module+"/"+c.Name)
}
for _, p := range d.Passes.Passes {
left = append(left, "the pass of "+p.Seat.Name+" to "+p.To.Module+" on "+p.To.Node)
}
if body, err := d.Body(); err == nil {
for _, marker := range []string{catalogue.ForeseenSealed, "asked of the vault by the next send", compose.ForeseenCertificate,
compose.ForeseenBusUsers} {
if strings.Contains(string(body), marker) {
left = append(left, "a stand-in: "+marker)
}
}
}
sort.Strings(left)
return slicesCompact(left)
}
func slicesCompact(in []string) []string {
var out []string
for _, s := range in {
if len(out) == 0 || out[len(out)-1] != s {
out = append(out, s)
}
}
return out
}
// nodeShelf is the shelf as one node composes its own set: the current one with each module it keeps at the build it runs put
// in place of the current.
func nodeShelf(in *meshInput, shelf map[string]catalogue.Manifest, node string) map[string]catalogue.Manifest {
n, _ := nodeOf(in.Input, node)
return shelfWith(shelf, n.Kept)
}
// prepareSend is what a send does before it composes, every send and whatever the composition would have named: the
// controller's own composition did it on the way, so a module's file that names an own secret composes (mesh-sdk's
// composition can stand in for none: a file naming a secret not made yet is refused in every mode), and the core's credentials
// are asked of the vault on every send, so a rotated one reaches the machine and a silent vault is said (core_box.go).
//
// - Each own secret of a module the node runs (not one handed to the provisioner, not one issued outside the mesh, not one a
// person gives) that no row holds is made, unless the composition would refuse it (a bus credential nobody issued): that
// refusal is the composition's to say, in its words.
// - The core's credentials are asked of the vault, the boxes newly answered are kept, and the modules the vault did not answer
// for with no box kept are returned: the controller seals those itself until the vault holds them (ADR 0396).
func prepareSend(ctx context.Context, open *stores, node string) ([]string, error) {
in, err := inputFor(ctx, open)
if err != nil {
return nil, err
}
if in.refused[node] != nil {
return nil, nil // empty-on-error: refused alone for what it keeps; the composition that follows says so, in its words
}
resolved, err := compose.Resolve(in.Input, node)
if err != nil {
return nil, nil // empty-on-error: it does not compose for its own reasons; the composition that follows says them
}
n, _ := nodeOf(in.Input, node)
hands := n.Stated && n.Contract >= compose.HandedContract
handing := catalogue.Rendering{Hands: hands, Provisioner: provisionerAccount(resolved.Plan)}
inv := open.inventory
did := false
if n.SealingKey != "" {
held := map[string]bool{}
for _, r := range in.OwnSecrets {
// Held only when sealed to the key the machine has now, or given by a person (which is not the mesh's
// to make again): a made row under an old key is re-made, as SecretForModule did.
if r.Node == node && (r.SealedTo == n.SealingKey || r.Origin == inventory.OriginAccepted) {
held[r.Module+"/"+r.Name] = true
}
}
for _, m := range resolved.Plan.Modules {
if handing.Handing(m) {
continue
}
plain := m.OwnSecrets.Plain()
names := make([]string, 0, len(plain))
for name := range plain {
names = append(names, name)
}
sort.Strings(names)
for _, name := range names {
if plain[name].IssuedBy == catalogue.IssuedOutside || held[m.Module+"/"+name] {
continue
}
switch name {
case "broker":
if !slices.Contains(in.BusMinted, node+"."+m.Module) {
continue
}
case compose.ProvisionerSecret:
if !slices.Contains(in.BusMinted, "provisioner."+node) {
continue
}
}
if _, err := inv.SecretForModule(ctx, node, m.Module, name); err == nil {
did = true
}
}
}
}
var byController []string
if n.Stated && n.Contract >= compose.VaultBoxContract {
boxes, err := coreBoxesFor(ctx, inv, node, resolved.Plan.Modules, func(m catalogue.Manifest) bool {
return handing.Handing(m)
}, true)
if err != nil {
return nil, err
}
did = did || len(boxes.keep) > 0
byController = boxes.byController
}
if did {
// What was made is in the stores now: the composition reads them afresh.
forgetInput(ctx)
}
return byController, nil
}
@@ -1,235 +0,0 @@
//go:build integration
package main
import (
"errors"
"fmt"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"github.com/novox/mesh-controller/internal/link"
)
func aWebModule() catalogue.Manifest {
return catalogue.Manifest{Module: "web", Version: "1",
Listens: []catalogue.Listening{{Port: 8080, From: catalogue.FromEverywhere}},
OwnSecrets: catalogue.OwnSecrets{"admin": {Path: "/etc/web/admin"}},
Resources: []map[string]any{
{"id": "server", "type": "container", "name": "web", "ports": []any{"8080"},
"image": "registry.example/web@sha256:" + strings.Repeat("a", 64)},
}}
}
// 1385/AC1: a port the composition names foreseen is allocated with the inventory's PortFor for that module and port, a second
// composition names it no more, and the number is the one the controller's own Allocating composition gives in the same store.
func TestPerformAllocatesAForeseenPortAndTheNumberIsTheControllers(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
register(t, open, aWebModule())
if _, err := assign(ctx, open, "laptop", "web"); err != nil {
t.Fatal(err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
d, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
if len(d.ForeseenPorts) != 1 || d.ForeseenPorts[0] != "web:8080" {
t.Fatalf("the composition names the ports %v foreseen, want web:8080", d.ForeseenPorts)
}
did, err := perform(ctx, open, in, "laptop", d)
if err != nil || !did {
t.Fatalf("perform did %v: %v", did, err)
}
held, err := open.inventory.PortsFor(ctx, "laptop")
if err != nil || len(held) != 1 || held[0].Module != "web" || held[0].Wanted != 8080 {
t.Fatalf("the port perform allocated reads %+v: %v", held, err)
}
in, err = readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
if again, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing); err != nil || len(again.ForeseenPorts) != 0 {
t.Fatalf("a second composition names %v foreseen: %v", again.ForeseenPorts, err)
}
// The number is the one the controller allocates in the same store.
plan, settings, err := resolveNode(ctx, open, "laptop")
if err != nil {
t.Fatal(err)
}
old, err := declarationFor(ctx, open, "laptop", plan, settings)
if err != nil {
t.Fatal(err)
}
after, _, err := compose.ComposeIn(in.Input, "laptop", compose.Foreseeing)
if err != nil {
t.Fatal(err)
}
if got, want := portsOf(after.Resources), portsOf(old.Resources); got != want {
t.Fatalf("the declaration publishes %s; the controller's allocation publishes %s", got, want)
}
}
func portsOf(resources []map[string]any) string {
var out []string
for _, r := range resources {
if r["type"] == "container" {
out = append(out, fmt.Sprint(r["name"], r["ports"]))
}
}
return strings.Join(out, ";")
}
// 1385/AC2: an own secret named foreseen is made with SecretForModule, and a pair credential the Input names would-make is
// made as planFor chose; both exist after the send's composition and a second composition names neither.
func TestPerformMakesTheOwnSecretsAndThePairCredentialsTheCompositionNamed(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
inv := open.inventory
register(t, open, aWebModule())
register(t, open, aConsumerOfTheBucket("reports"))
for _, m := range []string{"web", "reports"} {
if _, err := assign(ctx, open, "laptop", m); err != nil {
t.Fatal(err)
}
}
before, err := madeIn(ctx, open)
if err != nil {
t.Fatal(err)
}
declared, err := composeSend(ctx, open, "laptop")
if err != nil {
t.Fatalf("the send is refused: %v", err)
}
if len(declared.foreseen) != 0 {
t.Fatalf("the declaration still names %v foreseen", declared.foreseen)
}
after, err := madeIn(ctx, open)
if err != nil {
t.Fatal(err)
}
madeNow := func(have, was []string) []string {
seen := map[string]bool{}
for _, s := range was {
seen[s] = true
}
var out []string
for _, s := range have {
if !seen[s] {
out = append(out, s)
}
}
return out
}
if got := madeNow(after.OwnSecrets, before.OwnSecrets); len(got) != 1 || !strings.Contains(got[0], "web/admin") {
t.Errorf("the own secrets made are %v, want laptop web/admin", got)
}
if got := madeNow(after.Pairs, before.Pairs); len(got) != 1 || !strings.Contains(got[0], "reports") {
t.Errorf("the pair credentials made are %v, want the one of reports", got)
}
// And the credential is the one a second composition reads as held.
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
if len(in.wouldMake) != 0 {
t.Errorf("after the send the next send would still make %v", in.wouldMake)
}
if _, err := inv.SecretsFrom(ctx, "anchor"); err != nil {
t.Fatal(err)
}
}
// 1385/AC3 (the lead's answer 1): with the vault unreachable and no box of the credential kept, the send carries the
// controller's own seal for it (ADR 0396), says so as a condition, and keeps no box.
func TestAVaultThatIsSilentSealsTheSendByTheControllerWhereNoBoxIsKept(t *testing.T) {
open, vault := aCoreMesh(t, link.VaultBoxContract)
vault.silent = errors.New("no responders")
declared, err := composeSend(t.Context(), open, "laptop")
if err != nil {
t.Fatalf("the send is refused: %v", err)
}
f := clientsBroker(declared.Resources)
if f == nil || f["sealed"] == nil || f[catalogue.VaultSealedField] != nil {
t.Fatalf("the core credential is composed as %v: the controller's own seal, until the vault holds it", f)
}
if len(declared.foreseen) != 0 {
t.Fatalf("the declaration still names %v foreseen", declared.foreseen)
}
if c, found := vaultSilence(t, "laptop"); !found || !strings.Contains(c.Summary, "sealed by the controller until the vault holds it") {
t.Fatalf("no condition says why: %+v", c)
}
kept, err := open.inventory.KeptCoreBoxes(t.Context(), "laptop")
if err != nil {
t.Fatal(err)
}
if len(kept["mesh-provisioner-client"]) != 0 {
t.Fatalf("a box was kept for a credential the vault never boxed: %v", kept)
}
}
// 1385/AC6: perform is idempotent: a second send's composition over the stores it left makes and records nothing.
func TestAFinishedSendMakesNothingTheSecondTime(t *testing.T) {
open, _ := aMeshWithSecrets(t)
ctx := t.Context()
register(t, open, aWebModule())
if _, err := assign(ctx, open, "laptop", "web"); err != nil {
t.Fatal(err)
}
if _, err := composeSend(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
before := tableState(t, open.inventory)
if _, err := composeSend(ctx, open, "laptop"); err != nil {
t.Fatal(err)
}
for k, v := range tableState(t, open.inventory) {
if before[k] != v {
t.Errorf("a second send of the same machine moved %s", k)
}
}
}
// Review round 1, finding 1: a machine that rejoined with a new sealing key keeps its made rows sealed to the old key. A
// send re-makes them for the new key, as SecretForModule did on main, and the file naming the secret is not refused.
func TestASendReMakesAnOwnSecretSealedToAKeyTheMachineNoLongerHas(t *testing.T) {
open := aMesh(t)
ctx := t.Context()
inv := open.inventory
register(t, open, aBot("bot", false))
if _, err := assign(ctx, open, "laptop", "bot"); err != nil {
t.Fatal(err)
}
if _, err := composeSend(ctx, open, "laptop"); err != nil {
t.Fatalf("the first send is refused: %v", err)
}
if err := inv.RecordSealingKey(ctx, nodeID(t, open, "laptop"), aPublicKey(t)); err != nil {
t.Fatal(err)
}
if _, err := composeSend(ctx, open, "laptop"); err != nil {
t.Fatalf("a machine that rejoined cannot be sent to: %v", err)
}
in, err := readComposeInput(ctx, open)
if err != nil {
t.Fatal(err)
}
n, _ := nodeOf(in.Input, "laptop")
found := false
for _, r := range in.OwnSecrets {
if r.Node == "laptop" && r.Module == "bot" && r.Name == "token" {
found = true
if r.SealedTo != n.SealingKey {
t.Fatalf("the own secret is still sealed to a key the machine no longer has")
}
}
}
if !found {
t.Fatal("the own secret has no row")
}
}
-500
View File
@@ -1,500 +0,0 @@
package main
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"os"
"slices"
"sort"
"strings"
"sync"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
"git.novox.be/novox/mesh-sdk/go/validate"
"github.com/novox/mesh-controller/internal/inventory"
)
// The controller's composition on mesh-sdk's (novox/hq ADR 0453 clause 3, design 47 R4b, issues 1386 to 1388): every
// caller that planned or composed a node reads the Input once for its operation (compose_input.go), asks the SDK to
// compose over it, and, when it sends, performs what the composition names foreseen (compose_perform.go) and composes again.
// inputCacheKey carries an operation's Input in its context, so the planning of every node of one verb reads the stores once.
type inputCacheKey struct{}
type inputCache struct {
mu sync.Mutex
by map[string]*meshInput
}
// withInputCache is ctx for an operation: the Input read for it is kept until a send changes what it read. A context that
// carries none reads the stores for each composition, which is correct and slow.
func withInputCache(ctx context.Context) context.Context {
if _, ok := ctx.Value(inputCacheKey{}).(*inputCache); ok {
return ctx
}
return context.WithValue(ctx, inputCacheKey{}, &inputCache{by: map[string]*meshInput{}})
}
// freshInput is ctx for one composition whose Input is read afresh and kept only until the composition makes something: a
// send that is composed after another one made and recorded things (a push's machines, one after the other).
func freshInput(ctx context.Context) context.Context {
return context.WithValue(ctx, inputCacheKey{}, &inputCache{by: map[string]*meshInput{}})
}
// keptSignature is what the Input read depends on in ctx: which recorded modules each machine keeps at the build it runs
// (recorded_kept.go), so an Input read for a person's push is never taken for one read for a gated send.
func keptSignature(ctx context.Context) string {
skip, on := keptExcept(ctx)
var names []string
for m := range skip {
names = append(names, m)
}
sort.Strings(names)
every, _ := ctx.Value(keepEveryBuildKey{}).(string)
settings, _ := ctx.Value(keepForSettingsKey{}).(string)
return fmt.Sprintf("kept=%v;except=%v;every=%s;settings=%s", on, names, every, settings)
}
// inputFor is the Input for ctx's operation.
func inputFor(ctx context.Context, open *stores) (*meshInput, error) {
cache, ok := ctx.Value(inputCacheKey{}).(*inputCache)
if !ok {
return readComposeInput(ctx, open)
}
sig := keptSignature(ctx)
cache.mu.Lock()
defer cache.mu.Unlock()
if in := cache.by[sig]; in != nil {
return in, nil
}
in, err := readComposeInput(ctx, open)
if err != nil {
return nil, err
}
cache.by[sig] = in
return in, nil
}
// forgetInput drops what the operation read, because a send made or recorded something it read.
func forgetInput(ctx context.Context) {
if cache, ok := ctx.Value(inputCacheKey{}).(*inputCache); ok {
cache.mu.Lock()
cache.by = map[string]*meshInput{}
cache.mu.Unlock()
}
}
// resolveNode works out everything a node should run, from what was assigned to it (planFor on the SDK): over the Input,
// writing nothing. A failure to compose the node's own modules is marked notResolvable, as planFor marked it; every other
// failure is returned as it is.
func resolveNode(ctx context.Context, open *stores, node string) (catalogue.Resolution, catalogue.SettingsBy, error) {
in, err := inputFor(ctx, open)
if err != nil {
return catalogue.Resolution{}, nil, err
}
if err := in.refused[node]; err != nil {
return catalogue.Resolution{}, nil, err
}
resolved, err := compose.Resolve(in.Input, node)
if err != nil {
if compose.IsUnresolvable(err) {
return catalogue.Resolution{}, nil, notResolvable{err}
}
return catalogue.Resolution{}, nil, err
}
logUnheld(node, resolved.Plan.Unheld)
return resolved.Plan, resolved.Settings, nil
}
// generatorsOf is what every node's declaration is composed with, over the whole mesh (generators on the SDK).
func generatorsOf(ctx context.Context, open *stores) (map[string]catalogue.Generator, error) {
in, err := inputFor(ctx, open)
if err != nil {
return nil, err
}
return compose.Generators(in.Input)
}
// renderingOf is what a machine is rendered with as a question reads it (nothing made), with the private network: what the
// filter's rules, the roster and the like are derived from.
func renderingOf(ctx context.Context, open *stores, node string) (catalogue.Rendering, error) {
in, err := inputFor(ctx, open)
if err != nil {
return catalogue.Rendering{}, err
}
rendered, err := compose.Render(in.Input, node, compose.Reading)
if err != nil {
return catalogue.Rendering{}, markUnresolvable(err)
}
with := rendered.Rendering
if with.Generators, err = compose.Generators(in.Input); err != nil {
return catalogue.Rendering{}, err
}
return with, nil
}
// whoResolvesOn is the machines whose resolution answers a requirement, and why the others did not (whoResolves over the
// Input): by what a module provides, not by its name. A machine whose set does not compose is in refused, said; one the mesh
// could not be asked about is an error, never a refusal (novox/hq 04-ISSUES/152).
func whoResolvesOn(ctx context.Context, open *stores, requirement string) (map[string]bool, map[string]string, error) {
in, err := inputFor(ctx, open)
if err != nil {
return nil, nil, err
}
on := map[string]bool{}
refused := map[string]string{}
for _, n := range in.sortedNodes() {
if err := in.refused[n.Name]; err != nil {
return nil, nil, fmt.Errorf("whether %s answers %q cannot be read: %w", n.Name, requirement, err)
}
resolved, err := compose.Resolve(in.Input, n.Name)
switch {
case compose.IsUnresolvable(err):
refused[n.Name] = err.Error()
continue
case err != nil:
return nil, nil, fmt.Errorf("whether %s answers %q cannot be read: %w", n.Name, requirement, err)
}
for _, m := range resolved.Plan.Modules {
for _, offered := range m.Offers() {
if offered == requirement {
on[n.Name] = true
}
}
}
}
return on, refused, nil
}
// sortedNodes is the Input's nodes by name, the order the controller reads them in.
func (in *meshInput) sortedNodes() []compose.Node {
out := append([]compose.Node(nil), in.Nodes...)
sort.Slice(out, func(i, j int) bool { return out[i].Name < out[j].Name })
return out
}
// composeQuestion composes a node's declaration as a question asks it: nothing is made, and what a send would make is a
// stand-in (Foreseeing) or left out (Reading). It also answers what the node-engine's validator refuses of it. The lines the
// composition found are said where the controller says its own (standard error, or the writer an act gave).
//
// withOrder says the declaration carries the order it was last sent under (composedAndValidated's, which a comparison with the
// last send needs); without it the order is none, as declarationWith left it for the sender to stamp.
func composeQuestion(ctx context.Context, open *stores, node string, mode compose.Mode, withOrder bool) (sendable, []string, error) {
in, err := inputFor(ctx, open)
if err != nil {
return sendable{}, nil, err
}
if err := in.refused[node]; err != nil {
return sendable{}, nil, err
}
d, problems, err := compose.ComposeIn(in.Input, node, mode)
if err != nil {
return sendable{}, nil, markUnresolvable(err)
}
if err := refusedBus(in, node); err != nil {
return sendable{}, nil, err
}
sayFound(ctx, d)
out := sendableOf(d)
if !withOrder {
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
}
return out, problems, nil
}
// refusedBus is why the node cannot be composed because it runs the bus and the bus's user list cannot be: no user has a
// credential, so the list would refuse every connection in the mesh (busUserList). The controller's own composition refused that
// machine with those words; no other machine is refused for it. nil when nothing refuses the node.
func refusedBus(in *meshInput, node string) error {
if in.busRefused == nil {
return nil
}
resolved, err := compose.Resolve(in.Input, node)
if err != nil {
return nil // it does not compose for its own reasons, which are said
}
for _, m := range resolved.Plan.Modules {
if m.BusUsers != "" && m.ClaimsSeat("mesh-broker") {
return in.busRefused
}
}
return nil
}
// markUnresolvable is an error of a node whose own set does not compose, as planFor marked it.
func markUnresolvable(err error) error {
if compose.IsUnresolvable(err) {
return notResolvable{err}
}
return err
}
// composeSend composes a node's declaration to send it, as the Allocating composition did: composed over what is read,
// then what it named foreseen is made (perform), the stores are read again and it is composed again. A declaration that
// still names anything foreseen is refused, said, and never sent.
func composeSend(ctx context.Context, open *stores, node string) (sendable, error) {
declared, _, err := composeSendValidated(ctx, open, node)
return declared, err
}
// composeSendValidated is composeSend and what the node-engine's validator refuses of the declaration it composed, for the
// merge gate's two passes (composedAndValidated in its Allocating mode).
func composeSendValidated(ctx context.Context, open *stores, node string) (sendable, []string, error) {
// The core modules whose credentials the controller sealed itself because the vault was silent and no box was kept
// (ADR 0396): the composition stands in a vault box for each, and a send carries the controller's own seal instead.
byController, err := prepareSend(ctx, open, node)
if err != nil {
return sendable{}, nil, err
}
for attempt := 0; ; attempt++ {
in, err := inputFor(ctx, open)
if err != nil {
return sendable{}, nil, err
}
if err := in.refused[node]; err != nil {
return sendable{}, nil, err
}
d, problems, err := compose.ComposeIn(in.Input, node, compose.Foreseeing)
if err != nil {
return sendable{}, nil, markUnresolvable(err)
}
if err := refusedBus(in, node); err != nil {
return sendable{}, nil, err
}
if len(byController) > 0 {
if d, problems, err = sealedByTheController(in, node, d, byController); err != nil {
return sendable{}, nil, err
}
}
left := stillForeseen(d)
resolved, rerr := compose.Resolve(in.Input, node)
if rerr != nil {
return sendable{}, nil, markUnresolvable(rerr)
}
if len(resolved.Foreseen) > 0 {
left = append(left, resolved.Foreseen...)
}
if len(left) == 0 && len(in.wouldMake) == 0 {
sayFound(ctx, d)
out := sendableOf(d)
// The sender stamps the order it allots (order.stamp); a composition for a send carries none of its own.
out.Sequence, out.Epoch, out.Generation = 0, 0, 0
here, _ := nodeOf(in.Input, node)
out.Builds = buildsOf(in.Input, here, resolved.Plan, d.LeftOutWhy)
out.Bindings = boundToData(resolved.Plan, d.LeftOutWhy)
if len(d.Filled) > 0 {
if out.plans, err = handedPlans(in, node, resolved); err != nil {
return sendable{}, nil, err
}
}
return out, problems, nil
}
if attempt > 0 {
return sendable{}, nil, fmt.Errorf("%s was composed twice and still names what only a send makes (%s): nothing is sent "+
"that stands in for a value", node, strings.Join(left, "; "))
}
if _, err := perform(ctx, open, in, node, d); err != nil {
return sendable{}, nil, err
}
// What was made is in the stores now: the next read is a fresh one.
forgetInput(ctx)
}
}
// restOfTheMesh is what every other machine holds and offers (theRestOfTheMesh over the Input), for a question: the lines it
// finds are said where the controller says its own, and a seat it would pass on is passed by the next send (perform), not by
// the question. shelf, when given, replaces the Input's catalogue: a caller composing a change's shelf.
func restOfTheMesh(ctx context.Context, open *stores, shelf map[string]catalogue.Manifest, exclude string) (catalogue.World, error) {
in, err := inputFor(ctx, open)
if err != nil {
return catalogue.World{}, err
}
input := in.Input
if shelf != nil {
input.Catalogue = shelf
}
world, passes, left, err := compose.TheRestOfTheMesh(input, exclude)
if err != nil {
return catalogue.World{}, err
}
var w io.Writer = os.Stderr
if lw, ok := ctx.Value(leftOutKey{}).(io.Writer); ok {
w = lw
}
for _, l := range left {
fmt.Fprintln(w, l.String())
}
for _, line := range passes.Said {
fmt.Fprintln(w, line)
}
return world, nil
}
// restOfTheMeshOf is restOfTheMesh for a caller that holds the inventory alone: the other stores the Input reads are opened for
// the call and let go of again.
func restOfTheMeshOf(ctx context.Context, inv *inventory.Inventory, shelf map[string]catalogue.Manifest, exclude string) (catalogue.World, error) {
open := &stores{inventory: inv}
defer func() {
if open.licences != nil {
open.licences.Close()
}
if open.identity != nil {
open.identity.Close()
}
}()
return restOfTheMesh(ctx, open, shelf, exclude)
}
// sayFound says the lines a composition found while composing, once each: a machine left out of the rest of the mesh, a seat
// that two claimants could take, the bus's users left out for want of a credential.
func sayFound(ctx context.Context, d compose.Declared) {
if len(d.Said) == 0 {
return
}
var w io.Writer = os.Stderr
if lw, ok := ctx.Value(leftOutKey{}).(io.Writer); ok {
w = lw
}
for _, line := range d.Said {
fmt.Fprintln(w, line)
}
}
// sendableOf is a composed declaration as the controller's sends know it.
func sendableOf(d compose.Declared) sendable {
out := sendable{Resources: d.Resources, Sequence: d.Sequence, Epoch: d.Epoch, Generation: d.Generation,
ReadsModules: d.ReadsModules, moduleOf: d.ModuleOf, buildIDs: d.BuildIDs, modules: d.Modules,
localNetwork: d.LocalNetwork, Received: d.Received, Mesh: d.Mesh, BusUsers: d.BusUsers,
LeftOut: d.LeftOut, leftOutWhy: d.LeftOutWhy, withheld: d.Withheld, unbound: d.Unbound, foreseen: d.Foreseen,
unplaced: d.Unplaced, handedOnly: d.HandedOnly, Filled: d.Filled, Provisioner: d.Provisioner,
requiredBy: d.RequiredBy}
if d.Adoption != nil {
out.Adoption = &adoptionEnvelope{Taken: d.Adoption.Taken, Untaken: d.Adoption.Untaken}
}
return out
}
// buildsOf is the build of each module a declaration for the node carries, as a send records it (novox/hq issue 259): the
// module's current build for each module in it, the build it was last sent for one left out of it, and a module kept at the
// build the machine runs carried at that one (ADR 0245). Never nil.
func buildsOf(in compose.Input, node compose.Node, plan catalogue.Resolution, leftOut map[string]string) map[string]string {
before := node.SentBuilds
if !node.SentKnown {
before = nil
}
out := map[string]string{}
for _, m := range plan.Modules {
if _, left := leftOut[m.Module]; left {
if was, kept := before[m.Module]; kept {
out[m.Module] = was
}
continue
}
out[m.Module] = in.CurrentBuilds[m.Module]
}
for m, was := range node.KeptBuilds {
if _, carried := out[m]; carried {
out[m] = was
}
}
return out
}
// handedPlans is the plan of each module handed to the provisioner, which the assignments mirror keeps with the send (design
// 57 P6): the composition's declaration does not carry it, so it is composed from the same rendering again.
func handedPlans(in *meshInput, node string, resolved compose.Resolved) (map[string]catalogue.Plan, error) {
rendered, err := compose.Render(in.Input, node, compose.Foreseeing)
if err != nil {
return nil, err
}
gens, err := compose.Generators(in.Input)
if err != nil {
return nil, err
}
with := rendered.Rendering
with.Generators = gens
composed, err := resolved.Plan.Compose(with)
if err != nil {
return nil, err
}
return composed.Plans, nil
}
// sealedByTheController is the declaration with the controller's own seal in place of the vault's box for each core module the
// vault did not answer for and no box was kept for (core_box.go, ADR 0396): the file carries "sealed", as the controller's
// composition sent it, until the vault holds the credential. What the composition stood in for the box is replaced and no longer
// wanted; the validator is run again over the body.
func sealedByTheController(in *meshInput, node string, d compose.Declared, modules []string) (compose.Declared, []string, error) {
rendered, err := compose.Render(in.Input, node, compose.Foreseeing)
if err != nil {
return d, nil, err
}
stood, _ := compactBox(catalogue.ForeseenVaultBox)
var wanted []compose.CoreBoxWanted
var foreseen []string
for _, w := range d.CoreBoxesWanted {
if !slices.Contains(modules, w.Module) {
wanted = append(wanted, w)
continue
}
sealed := rendered.Rendering.Needed[w.Module][w.Name]
if sealed == "" {
return d, nil, fmt.Errorf("%s/%s on %s has no seal of the controller's to carry until the vault holds it", w.Module, w.Name, node)
}
for _, r := range d.Resources {
box, isBox := r[catalogue.VaultSealedField]
// By the resource's id alone: its path is the manifest's with the machine's words filled in, and so is not the
// manifest's own.
if !isBox || r["id"] != w.Module+"."+catalogue.NeedID(w.Name) {
continue
}
if had, err := compactBox(box); err != nil || had != stood {
continue
}
delete(r, catalogue.VaultSealedField)
r["sealed"] = sealed
}
}
// A stand-in box that is left is said by its resource, never sent and never guessed at.
for _, r := range d.Resources {
if box, has := r[catalogue.VaultSealedField]; has {
if had, err := compactBox(box); err == nil && had == stood {
if id, _ := r["id"].(string); slices.ContainsFunc(modules, func(m string) bool { return strings.HasPrefix(id, m+".") }) {
return d, nil, fmt.Errorf("%s on %s still carries a stand-in for the vault's box, and the controller has "+
"no seal of its own for it", id, node)
}
}
}
}
d.CoreBoxesWanted = wanted
for _, f := range d.Foreseen {
module, _, _ := strings.Cut(f, "/")
if !slices.Contains(modules, module) {
foreseen = append(foreseen, f)
}
}
d.Foreseen = foreseen
body, err := d.Body()
if err != nil {
return d, nil, err
}
return d, validate.Declaration(body), nil
}
// compactBox is a value as the compact JSON it marshals to, so a box is compared by what it says and not by the type that
// holds it.
func compactBox(v any) (string, error) {
raw, err := json.Marshal(v)
if err != nil {
return "", err
}
var out bytes.Buffer
if err := json.Compact(&out, raw); err != nil {
return "", err
}
return out.String(), nil
}
-111
View File
@@ -1,111 +0,0 @@
package main
import (
"context"
"reflect"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
)
// A declaration as the SDK composes it is the one the controller sends, byte for byte, whatever it carries: the body is the
// envelope's only marshaller on either side (design 47 R4b, issue 1386).
func TestASendableIsTheComposedDeclarationByteForByte(t *testing.T) {
d := compose.Declared{
Resources: []map[string]any{{"id": "files.config", "type": "file"}, {"id": "mesh.hosts", "type": "file"}},
Sequence: 7, Epoch: 3, Generation: 9, ReadsModules: true,
ModuleOf: map[string]string{"files.config": "files"}, BuildIDs: map[string]string{"files": "b-1"},
Adoption: &compose.Adoption{Taken: []string{"files"}, Untaken: map[string][]string{"nats": {"nats.server"}}},
Filled: []string{"/etc/files/filled.json"}, Provisioner: "mesh-provisioner",
LeftOut: []string{"cache"}, LeftOutWhy: map[string]string{"cache": "a setting cannot compose"},
}
want, err := d.Body()
if err != nil {
t.Fatal(err)
}
got, err := sendableOf(d).Body()
if err != nil {
t.Fatal(err)
}
if string(got) != string(want) {
t.Fatalf("the controller would send\n%s\nand the composition made\n%s", got, want)
}
if empty, err := sendableOf(compose.Declared{}).Body(); err != nil || !strings.Contains(string(empty), `"owns_nothing":true`) {
t.Fatalf("an empty declaration is not deliberate: %s %v", empty, err)
}
}
// 1386: what a send records as carried: the current build of each module it carries, the build a module left out of it was last
// sent (when that is known), and a module kept at the build the machine runs at that one. (The controller's carriedOn recorded
// the same; mesh-sdk's node_test TestAModuleLeftOutKeepsTheBuildItWasLastSent holds the left-out case.)
func TestWhatASendRecordsAsCarriedIsWhatTheControllerRecorded(t *testing.T) {
plan := catalogue.Resolution{Modules: []catalogue.Manifest{{Module: "postgres"}, {Module: "files"}, {Module: "cache"}}}
leftOut := map[string]string{"cache": "a setting cannot compose"}
for name, c := range map[string]struct {
sent map[string]string
known bool
kept map[string]string
want map[string]string
}{
"nothing sent": {want: map[string]string{"postgres": "p2", "files": "f2"}},
"sent and kept": {sent: map[string]string{"postgres": "p1", "cache": "c1", "files": "f1"}, known: true,
kept: map[string]string{"postgres": "p1"}, want: map[string]string{"postgres": "p1", "files": "f2", "cache": "c1"}},
"sent but not known": {sent: map[string]string{"cache": "c1"}, want: map[string]string{"postgres": "p2", "files": "f2"}},
"kept a module not in": {known: true, kept: map[string]string{"mail": "m1"}, want: map[string]string{"postgres": "p2", "files": "f2"}},
} {
in := compose.Input{CurrentBuilds: map[string]string{"postgres": "p2", "files": "f2", "cache": "c2"}}
node := compose.Node{SentBuilds: c.sent, SentKnown: c.known, KeptBuilds: c.kept}
if got := buildsOf(in, node, plan, leftOut); !reflect.DeepEqual(got, c.want) {
t.Errorf("%s: a send records %v, want %v", name, got, c.want)
}
}
}
// 1385: a declaration that stands in for a value anywhere is still foreseen, in a list the composition names or in the body
// alone (a pair credential's stand-in is in no list); one that stands in for nothing is not. It failed first, before the body
// was scanned.
func TestADeclarationThatStandsInForAValueIsStillForeseen(t *testing.T) {
clean := compose.Declared{Resources: []map[string]any{{"id": "files.config", "type": "file", "content": "ok"}}}
if left := stillForeseen(clean); len(left) != 0 {
t.Fatalf("a declaration that stands in for nothing is foreseen: %v", left)
}
for name, d := range map[string]compose.Declared{
"an own secret": {StoodIn: []string{"files/admin"}},
"a port": {ForeseenPorts: []string{"files:80"}},
"a core box": {CoreBoxesWanted: []compose.CoreBoxWanted{{Node: "anchor", Module: "mesh-vault", Name: "root"}}},
"a pass of a seat": {Passes: compose.Passes{Passes: []catalogue.SeatPass{{Seat: catalogue.Seat{Name: "mesh-store"}}}}},
"a pair credential": {Resources: []map[string]any{{"id": "files.needs-db", "sealed": catalogue.ForeseenSealed}}},
"the certificate": {Resources: []map[string]any{{"id": "proxy.cert", "content": compose.ForeseenCertificate}}},
"the bus's list": {Resources: []map[string]any{{"id": "nats.users", "content": compose.ForeseenBusUsers}}},
"the vault's box": {Resources: []map[string]any{{"id": "x", "box": catalogue.ForeseenVaultBox}}},
} {
if left := stillForeseen(d); len(left) == 0 {
t.Errorf("%s: a declaration that stands in for it is not foreseen", name)
}
}
}
// 1386: an Input read for a person's push is never the one read for a send that keeps recorded builds, and the cache is
// emptied by what a send made.
func TestAnInputIsCachedPerWhatItKeepsAndForgottenAfterASend(t *testing.T) {
ctx := withInputCache(context.Background())
plain := keptSignature(ctx)
kept := keptSignature(keepingRecorded(ctx))
bus := keptSignature(keepingRecorded(ctx, "nats"))
every := keptSignature(keepingEveryBuild(ctx, "anchor"))
for name, sig := range map[string]string{"a push": plain, "a kept send": kept, "the bus step": bus, "the grants step": every} {
for other, o := range map[string]string{"a push": plain, "a kept send": kept, "the bus step": bus, "the grants step": every} {
if name != other && sig == o {
t.Errorf("%s and %s read the same Input", name, other)
}
}
}
cache := ctx.Value(inputCacheKey{}).(*inputCache)
cache.by[plain] = &meshInput{}
forgetInput(ctx)
if len(cache.by) != 0 {
t.Fatal("what a send made left the Input it read in the cache")
}
}
@@ -1,68 +0,0 @@
//go:build integration
package main
import (
"context"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A member of a group whose gate is the composed check's is asked of the build seat with no module: its repository's
// check alone (novox/hq ADR 0403 clause 1). With no word from mesh-delivery, or one that does not say the gate is
// composed, it is checked alone with its gate (clause 6).
func askedForMember(t *testing.T, answer []groupsLine, answerErr error) link.CheckRequest {
t.Helper()
open := aMesh(t)
ctx := t.Context()
js, _ := aBuildQueue(t)
servingBus.Store(js)
t.Cleanup(func() { servingBus.Store(nil) })
if err := open.inventory.RegisterModule(ctx, catalogue.Manifest{Module: "mesh-controller", Version: "1"},
inventory.Source{Repository: "novox/mesh-controller", Ref: "main", BuiltFrom: "c0", Head: "c0"}); err != nil {
t.Fatal(err)
}
was := groupsAsked
groupsAsked = func(context.Context, bool) ([]groupsLine, error) { return answer, answerErr }
t.Cleanup(func() { groupsAsked = was })
p := link.PullUpdated{Owner: "novox", Repo: "mesh-controller", Number: 346, Base: "main", Head: "feat/x", Commit: headController,
Paths: []string{"main.go"}, ModuleDirsSaid: true, MergeCheck: true, MergeCheckSaid: true}
if err := (following{open: open, oneShot: true}).PullUpdated(ctx, p); err != nil {
t.Fatal(err)
}
q, err := link.ReadQueue(ctx, js, link.TheBuildMachine)
if err != nil || len(q.Asks) != 1 {
t.Fatalf("asks %d: %v", len(q.Asks), err)
}
return *q.Asks[0].Request.Check
}
func TestAComposedMembersCheckIsItsRepositoryChecksAlone(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
line.ComposedGate = true
c := askedForMember(t, []groupsLine{line}, nil)
if len(c.Modules) != 0 || len(c.New) != 0 || len(c.Dependents) != 0 || !c.Composed {
t.Fatalf("the member's check is asked as %+v; want no module and composed", c)
}
}
func TestAMemberOfAGroupOfAnOlderMeshDeliveryKeepsItsOwnGate(t *testing.T) {
line := group("feat/x", member("novox/mesh-controller", 346, headController, "main", true), member("novox/mesh-catalog", 350, headHost, "main", true))
c := askedForMember(t, []groupsLine{line}, nil)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed {
t.Fatalf("an older mesh-delivery's member is asked as %+v; want its own gate", c)
}
}
func TestAMemberWhoseGroupCannotBeLearnedIsCheckedAloneWithItsGate(t *testing.T) {
c := askedForMember(t, nil, context.DeadlineExceeded)
if !strings.Contains(strings.Join(c.Modules, ","), "mesh-controller") || c.Composed ||
!strings.Contains(c.SiblingsUnknown, "reads no siblings") {
t.Fatalf("the member is asked as %+v; want its gate and why its siblings are unknown", c)
}
}
@@ -1,136 +0,0 @@
package main
import (
"context"
"encoding/json"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// A delivery group's gate is the composed check's, and its members' own checks run their repository's check alone
// (novox/hq ADR 0403 clauses 1 and 7). Pure: no store, no bus.
// A member of a group whose line says composed_gate is a composed member; one whose group says nothing of it (an older
// mesh-delivery), one of a group of a single pull request and a lone pull request are not.
func TestAMemberOfAComposedGroupIsKnownByTheGroupsLine(t *testing.T) {
two := func(composed bool) groupsLine {
l := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true), member("novox/mesh-controller", 346, headController, "main", true))
l.ComposedGate = composed
return l
}
p := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Head: "feat/x", Commit: headHost}
recheck := link.PullUpdated{Owner: "novox", Repo: "mesh-host", Number: 114, Base: "main", Commit: headHost}
for name, c := range map[string]struct {
p link.PullUpdated
lines []groupsLine
want bool
}{
"composed": {p, []groupsLine{two(true)}, true},
"a recheck, by its number": {recheck, []groupsLine{two(true)}, true},
"an older mesh-delivery": {p, []groupsLine{two(false)}, false},
"another group": {p, []groupsLine{group("feat/y", member("novox/mesh-host", 1, headHost, "main", true))}, false},
"no group": {p, nil, false},
} {
if got := composedMember(c.p, c.lines); got != c.want {
t.Errorf("%s: composed %v, want %v", name, got, c.want)
}
}
alone := group("feat/x", member("novox/mesh-host", 114, headHost, "main", true))
alone.ComposedGate = true
if composedMember(p, []groupsLine{alone}) {
t.Error("a group of a single pull request is composed")
}
}
// A composed member's verdict never says its gate passed: the gate layer, and the verdict that is the gate's, read
// "composed"; a failure or an error stays what it was.
func TestAComposedMembersVerdictHasNoPassingGate(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114, Composed: true}
pass := link.BuildResult{ID: "b", Ref: headHost, Checked: asked, Check: &link.CheckOutcome{Verdict: "pass", Summary: "no module touched",
Gate: &link.CheckLayer{Verdict: "pass", Summary: "no module touched"}, RepoCheck: &link.CheckLayer{Verdict: "pass", Summary: "ok"}}}
c := checkedOf(pass)
if c.Gate == nil || c.Gate.Verdict != "composed" || c.Verdict != "composed" || c.RepoCheck == nil || c.RepoCheck.Verdict != "pass" {
t.Fatalf("the member's verdict says %+v gate %+v", c, c.Gate)
}
asked.Composed = false
if c := checkedOf(pass); c.Gate.Verdict != "pass" || c.Verdict != "pass" {
t.Fatalf("a member of no composed group has its own gate: %+v", c.Gate)
}
asked.Composed = true
failing := pass
failing.Check = &link.CheckOutcome{Verdict: "fail", Summary: "conflict", Gate: &link.CheckLayer{Verdict: "fail", Summary: "conflict"},
RepoCheck: &link.CheckLayer{Verdict: "fail", Summary: "conflict"}}
if c := checkedOf(failing); c.Gate.Verdict != "fail" || c.Verdict != "fail" {
t.Fatalf("a failure was read as composed: %+v", c.Gate)
}
broken := link.BuildResult{ID: "b", Ref: headHost, Checked: asked, Failed: "the facts snapshot cannot be read"}
if c := checkedOf(broken); c.Verdict != "error" || c.Gate.Verdict != "error" {
t.Fatalf("an error was read as composed: %+v", c)
}
}
// The composed verdict on the bus carries every member, each with its trunk commit, for the forge's holder to set each
// member's mesh/merge-gate from it.
func TestTheComposedVerdictCarriesEveryMember(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-host", Number: 114, Group: "feat/x",
Members: []link.GroupMember{{Owner: "novox", Repo: "mesh-controller", Number: 346, Ref: headController}}}
c := checkedOf(link.BuildResult{ID: "g", Ref: headHost, Checked: asked, Check: &link.CheckOutcome{Verdict: "pass", Summary: "ok",
Trunk: "aaaaaaaaaaaa", Members: []link.CheckedMember{{Owner: "novox", Repo: "mesh-controller", Commit: headController[:12], Trunk: "bbbbbbbbbbbb"}}}})
body, _ := json.Marshal(c)
for _, want := range []string{`"group":"feat/x"`, `"members":[{"owner":"novox","repo":"mesh-host","number":114,"commit":"` + headHost + `","trunk":"aaaaaaaaaaaa"}`,
`{"owner":"novox","repo":"mesh-controller","number":346,"commit":"` + headController + `","trunk":"bbbbbbbbbbbb"}`} {
if !strings.Contains(string(body), want) {
t.Fatalf("the composed verdict lacks %s:\n%s", want, body)
}
}
}
// Members whose manifests order their modules in a cycle fail the composed check, naming every module and edge, and no
// ask is made of the build seat.
func TestAGroupInACycleFailsItsComposedCheckWithoutABuild(t *testing.T) {
members := []orderMember{
{ID: "novox/x@" + headHost[:12], Repository: "novox/x", Base: "main", Head: headHost, Number: 7, Paths: []string{"module.json"}},
{ID: "novox/y@" + headController[:12], Repository: "novox/y", Base: "main", Head: headController, Number: 9, Paths: []string{"module.json"}},
}
reach := map[string]orderReach{members[0].ID: {Moved: []string{"mx"}}, members[1].ID: {Moved: []string{"my"}}}
was, wasParent := manifestAt, parentOf
t.Cleanup(func() { manifestAt, parentOf = was, wasParent })
parentOf = func(context.Context, string, string, string) (string, error) { return "", nil }
manifestAt = func(_ context.Context, _, repo, _, ref string) (catalogue.Manifest, bool, error) {
if ref == "main" {
return catalogue.Manifest{}, false, nil // what the pull requests merge into holds no such manifest
}
if repo == "x" {
return catalogue.Manifest{Module: "mx", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "a"}}}},
Invokes: []string{"seat:s.b"}}, true, nil
}
return catalogue.Manifest{Module: "my", DefinesSeats: []catalogue.SeatDeclaration{{Name: "s", Serves: []catalogue.Verb{{Name: "b"}}}},
Invokes: []string{"seat:s.a"}}, true, nil
}
c, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, []inventory.Edge(nil), nil)
if err != nil || !cyclic {
t.Fatalf("no cycle found: %v", err)
}
if c.Verdict != "fail" || c.Group != "feat/cycle" || len(c.Members) != 2 || c.Gate == nil || c.Gate.Verdict != "fail" {
t.Fatalf("the verdict is %+v", c)
}
for _, w := range []string{"mx", "my", "s.a", "s.b"} {
if !strings.Contains(c.Summary, w) {
t.Errorf("the cycle does not name %s: %s", w, c.Summary)
}
}
// Members with no edge between them are no cycle.
manifestsReadMu.Lock()
clear(manifestsRead)
manifestsReadMu.Unlock()
manifestAt = func(context.Context, string, string, string, string) (catalogue.Manifest, bool, error) {
return catalogue.Manifest{}, false, nil
}
if _, cyclic, err := cycleVerdict(context.Background(), "feat/cycle", members, reach, nil, nil); err != nil || cyclic {
t.Fatalf("a cycle where there is none: %v %v", cyclic, err)
}
}
@@ -1,5 +1,3 @@
//go:build integration
package main
import (

Some files were not shown because too many files have changed in this diff Show More