Compare commits

..
Author SHA1 Message Date
mesh-adminandClaude Opus 5.5 7a691edd3a Find a bare or aliased mirror reader, and say what the reader test misses
mesh/delivery rejected: the gate failed or could not run, or the repository's own check failed
mesh/merge-gate out of date: judged against trunk dc36536e199e, the trunk is now 60d390adeddc; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk dc36536e199e, the trunk is now 60d390adeddc; judged again on a recheck or when it is merged
Review 257 of mesh-controller#315 (hq issue 652): the test that only the audit
and a mirror's writer read a mirror (ADR 0289 §19, ADR 0356 §2) matched only
names qualified as broker. or recordmirror., so a reader inside package broker,
or one importing broker under another name, passed. It now matches by import
path and bare names in the defining packages, proves it on planted readers,
and names what it cannot see.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:40:33 +00:00
mesh-adminandClaude Opus 5.5 35c7662f89 Check the mirrors as ADR 0356 says, and prove a removal asks for the lease
mesh/merge-gate head 35c7662f895f merged into trunk dc36536e199e — pass: the change touches no module of the mesh's graph
mesh/repo-check head 35c7662f895f merged into trunk dc36536e199e — fail: its merge-check.sh failed: --- FAIL: TestTheInstallersFirstUserListIsWhatTheContro…
mesh/delivery superseded: a newer head of the same pull request
Reviews 251 of mesh-controller#315 and ADR 0356 v4 (hq issue 652): a removal
without the lease removes nothing; a module no longer registered loses its
declared-config key with its event; a listed mirror is never reported as
undeclared state; only the audit and a mirror's own writer name a mirror's
bucket; and the mirrors' list and the writers table agree. The store test's
line moves where it merges with the integration branch's own new lines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:26:29 +00:00
mesh-adminandClaude Opus 5.5 69367aa786 Correct a mirror written behind the writer's back, and write none without the lease
mesh/merge-gate out of date: judged against trunk 617881c3093d, the trunk is now 71362c196dbc; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk 617881c3093d, the trunk is now 71362c196dbc; judged again on a recheck or when it is merged
mesh/delivery superseded: a newer head of the same pull request
Review of mesh-controller#315 (hq issue 652): the writer trusted the bucket as
it read it once, so a former lease holder's write stayed for good; it wrote
without the lease, asking only after a refused compare-and-set; and the
declared config carried a manifest's declarations whole, so a field the
manifest gains would reach the bus unreviewed (ADR 0289 §9, §10, §12). Every
pass now reads the bucket and writes back the store's value where it differs,
asks for the lease before the pass and before every write, and a declaration
is mirrored as its listed fields (kind, default, why), with a test that fails
on a declaration field the list does not name.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 01:01:00 +00:00
mesh-adminandClaude Opus 5.5 8563b90918 Capture the genesis template from mesh-host's branch of this package
mesh/delivery superseded: a newer head of the same pull request
mesh/merge-gate out of date: judged against trunk bfc79c64d989, the trunk is now 617881c3093d; judged again on a recheck or when it is merged
mesh/repo-check out of date: judged against trunk bfc79c64d989, the trunk is now 617881c3093d; judged again on a recheck or when it is merged
The template at mesh-host c8920622 carries the controller's grant to write the
two record mirrors and say their events; the genesis template test holds the
controller's composition to it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:34:24 +00:00
mesh-adminandClaude Opus 5.5 3218fc3c1b Hold the composed grant and the captured genesis template to the mirrors
The controller's composition gains the two mirror buckets and their events, so
the golden composition moves, and the genesis template is captured again from
mesh-host's branch of this package, whose installer user list carries them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:33:05 +00:00
mesh-adminandClaude Opus 5.5 ea5cc6fc35 Prove a record mirror against a real bus and the store
An assignment and a registration each change exactly one key and raise exactly
one event; the same records written again change nothing (hq issue 652, the
package's done-when). Listed as a store test, so the merge check runs it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:32:29 +00:00
mesh-adminandClaude Opus 5.5 176a3d4fdf Write a record mirror only where its store differs, one event per write
The writer behind the assignments and declared-config mirrors (hq issue 652,
ADR 0289, design 54 §2-3): filled from the bucket on its first pass, it
compares each value byte for byte, writes by compare-and-set, purges a key
whose record is gone with a marker that lives a day, and says each write once
after it succeeded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:31:17 +00:00
mesh-adminandClaude Opus 5.5 8f8f7dff23 Keep the assignments and each module's declared config on the bus
mesh-config resolves a module's config values per node from which module runs
where and what each registered module declares, and both lived only in the
controller's store (hq issue 652, design 57 F3). The controller now keeps two
record mirrors, mesh-controller_assignments (<node>.<module>) and
mesh-controller_declared-config (<module>), built from the store by the lease
holder every ten seconds and written only where a value differs, by
compare-and-set, with one assignment-changed or declared-config-changed per
write and a purge said as removed (ADR 0289, design 54 §2-3). A module reads
them through `reads` as mesh-controller.<mirror>; the writers table keeps every
other writer out. Neither holds a secret: an assignment names its node and
module, a declaration is what the public manifest declares.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 00:00:25 +00:00
mesh-admin bfc79c64d9 Open the integration branch of release 57 (config and provisioning)
So the packages of hq design 57 meet here, reviewed one by one, and reach main as one delivery group.
2026-10-13 23:39:22 +00:00
74 changed files with 1800 additions and 5570 deletions
+6
View File
@@ -0,0 +1,6 @@
# Release 57: config and provisioning
This branch carries the work of hq design 57 (ADR 0348, ADR 0349) in this repository. Each package
merges here through its own reviewed pull request; the branch reaches main as one pull request, in one
delivery group with the branch of the same name in mesh-controller, mesh-host, mesh-sdk, mesh-catalog
and hq, after a rehearsal in the lab. This file is deleted by the last commit before that pull request.
+14 -57
View File
@@ -313,36 +313,25 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
registry = r.Address
}
spec := checkSpecOf(request)
// The judge cache the build agent declares as its own data, never in the workspace: only judge builds and
// mesh-lab's replays mount it, never a check's containers, so no pull request's script can write what a judge
// is built with or kept as, or what the replays run (novox/hq ADR 0358 §6, issue 709).
spec.JudgeCache = judgeCache()
// What the merge would move, built by the check for the walk to take (novox/hq ADR 0350 §1): into the
// scratch namespace. **Never with the package registry's credential**: this is code nobody has approved
// yet, and a recipe could read the credential it is handed (issue 462). A module that resolves packages
// at build time has no reuse key anyway, so nothing is lost by refusing it here.
if isRegistry && len(spec.Builds) > 0 {
spec.BuildWith = &builder.CheckBuilding{Publish: publisher, Scratch: registryHere, Held: request.Held,
Seats: request.Seats, Named: namedOf(request.Check.Named)}
Seats: request.Seats}
}
var v builder.CheckVerdict
v, err = builder.Check(building, builder.Command, spec, workspace, registry, forgeFrom(), say)
if err == nil {
result.Check = &link.CheckOutcome{Verdict: v.Verdict, Summary: v.Summary, Report: v.Report,
Took: v.Took.Round(time.Second).String(), Gate: layerOf(v.Gate), RepoCheck: layerOf(v.Repo),
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built),
Release: v.Release, JudgedBy: v.JudgedBy, BesideAt: v.BesideAt, Validator: v.Validator}
Head: v.Head, Trunk: v.Trunk, Beside: v.Beside, Built: checkBuiltOf(v.Built)}
// What each of a group's heads was judged as (novox/hq ADR 0323 §6).
for _, m := range v.Members {
result.Check.Members = append(result.Check.Members, link.CheckedMember{Owner: m.Owner, Repo: m.Repo,
Commit: m.Head, Trunk: m.Trunk})
}
// A branch tried before its pull request reads its outcome from this log, whole (novox/hq issue 686).
if request.Check.Try {
if line, err := json.Marshal(triedOf(v)); err == nil {
say(link.StepTried, string(line))
}
}
}
} else if err == nil {
// The package-registry credential is a build input, so it is resolved before the clone: a
@@ -357,7 +346,15 @@ func answer(ctx context.Context, publisher builder.Publisher, on, workspace stri
reuse.Pulls = append(reuse.Pulls, builder.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number})
}
// Only what a verdict names is taken (ADR 0350 §2).
reuse.Named = namedOf(request.Reuse.Builds)
for _, b := range request.Reuse.Builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
reuse.Named = append(reuse.Named, named)
}
built, err = builder.BuildReusing(building, builder.Labelled(builder.Command, request.ID), publisher, reuse,
request.Repository, request.Path, request.Ref, workspace, request.Held, npmrc,
forgeFrom(), say, request.Seats)
@@ -460,10 +457,9 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
spec := builder.CheckSpec{ID: request.ID, Repository: request.Repository, Ref: request.Ref,
Owner: c.Owner, Repo: c.Repo, Number: c.Number, Paths: c.Paths, Beside: map[string]builder.Beside{},
Modules: c.Modules, New: c.New, Manifests: c.Manifests, Judge: c.Judge, Base: c.Base,
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held),
Tests: c.Tests, StoreOnly: c.StoreOnly, Release: c.Release}
Toolchain: builder.ToolchainOf(request.Held), Toolchains: builder.ToolchainsOf(request.Held)}
for dir, b := range c.Beside {
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref, Else: b.Else}
spec.Beside[dir] = builder.Beside{Repository: b.Repository, Ref: b.Ref}
}
for _, b := range c.Builds {
spec.Builds = append(spec.Builds, builder.CheckBuildSpec{Module: b.Module, Repository: b.Repository, Path: b.Path,
@@ -476,50 +472,11 @@ func checkSpecOf(request link.BuildRequest) builder.CheckSpec {
return spec
}
// judgeCache is the judge cache the build agent's manifest declares (MESH_JUDGE_CACHE, novox/hq ADR 0358 §6): a
// directory mounted at the same path on the host, so a judge build's mounts name what the agent sees. Empty when the
// agent declares none, and every judge is then built cold under its check's root.
func judgeCache() string { return os.Getenv("MESH_JUDGE_CACHE") }
// triedOf is a try's outcome as its log line says it (novox/hq issue 686): the repository's own check's verdict,
// or the check's where that did not run — a head that does not merge — and the lines naming what failed.
func triedOf(v builder.CheckVerdict) link.Tried {
t := link.Tried{Verdict: v.Verdict, Summary: v.Summary, Took: v.Took.Round(time.Second).String(),
Head: v.Head, Trunk: v.Trunk}
if v.Repo != nil && (v.Gate == nil || v.Gate.Verdict == "pass") {
t.Verdict, t.Summary = v.Repo.Verdict, v.Repo.Summary
}
if v.Repo != nil && v.Repo.Failed != "" {
t.Failed = strings.Split(v.Repo.Failed, "\n")
}
if t.Verdict == "" {
t.Verdict = "error"
}
return t
}
// namedOf are the builds the seat's verdicts name, as the builder believes a scratch record by them (novox/hq ADR
// 0350 §2, issue 703): each module's key with every artifact's digest.
func namedOf(builds []link.CheckedBuild) []builder.NamedBuild {
var out []builder.NamedBuild
for _, b := range builds {
named := builder.NamedBuild{Pull: builder.PullRef{Owner: b.Pull.Owner, Repo: b.Pull.Repo,
Number: b.Pull.Number}, Check: b.Check, Module: b.Built.Module, Key: b.Built.Key,
Digests: map[string]string{}}
for _, m := range b.Built.Made {
named.Digests[m.Name] = builder.DigestOf(m.Reference)
}
out = append(out, named)
}
return out
}
// checkBuiltOf is a check's builds for its merge as the outcome carries them (novox/hq ADR 0350 §1).
func checkBuiltOf(built []builder.CheckBuilt) []link.CheckBuilt {
var out []link.CheckBuilt
for _, b := range built {
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String(),
Kept: b.Kept}
c := link.CheckBuilt{Module: b.Module, Key: b.Key, NotBuilt: b.NotBuilt, Took: b.Took.Round(time.Second).String()}
for _, m := range b.Made {
c.Made = append(c.Made, link.MadeArtifact{Name: m.Name, Kind: m.Kind, Reference: m.Reference})
}
-27
View File
@@ -1,27 +0,0 @@
package main
import (
"testing"
"github.com/novox/mesh-controller/internal/link"
)
// **A pull request into a release branch reaches the check as one** (novox/hq ADR 0358): the build seat hands the
// check the release branch and, beside it, the trunk each repository is read at when it has no branch of that name;
// and it hands the check the judge cache the build agent declares (MESH_JUDGE_CACHE), never a path of its own.
func TestTheBuildSeatHandsTheCheckItsReleaseBranch(t *testing.T) {
spec := checkSpecOf(link.BuildRequest{ID: "b", Repository: "r", Ref: "abc", Check: &link.CheckRequest{
Owner: "novox", Repo: "mesh-catalog", Base: "feat/57", Release: "feat/57",
Beside: map[string]link.CheckedOut{"mesh-host": {Repository: "h", Ref: "feat/57", Else: "main"}}}})
if spec.Release != "feat/57" || spec.Beside["mesh-host"].Ref != "feat/57" || spec.Beside["mesh-host"].Else != "main" {
t.Fatalf("the check was handed release %q and mesh-host %+v", spec.Release, spec.Beside["mesh-host"])
}
t.Setenv("MESH_JUDGE_CACHE", "/var/lib/mesh/build-agent/judge-cache")
if got := judgeCache(); got != "/var/lib/mesh/build-agent/judge-cache" {
t.Fatalf("the judge cache the manifest declares is read as %q", got)
}
t.Setenv("MESH_JUDGE_CACHE", "")
if got := judgeCache(); got != "" {
t.Fatalf("with no judge cache declared, %q is used", got)
}
}
-31
View File
@@ -1,31 +0,0 @@
package main
import (
"testing"
"time"
"github.com/novox/mesh-controller/internal/builder"
)
// A try's outcome, as the build seat says it in the build's log (novox/hq issue 686): the repository's own
// check's verdict, how long it took, and each line naming what failed.
func TestATrysOutcomeNamesWhatFailed(t *testing.T) {
tried := triedOf(builder.CheckVerdict{Verdict: "pass", Summary: "the change touches no module", Took: 303 * time.Second,
Head: "c0ffee", Trunk: "ca3f43c", Gate: &builder.Layer{Verdict: "pass"},
Repo: &builder.Layer{Verdict: "fail", Summary: "its merge-check.sh failed: TestX",
Failed: "--- FAIL: TestX (0.01s)\n x_test.go:12: got 1, want 2"}})
if tried.Verdict != "fail" || tried.Took != "5m3s" || len(tried.Failed) != 2 || tried.Trunk != "ca3f43c" ||
tried.Summary != "its merge-check.sh failed: TestX" {
t.Fatalf("a failed try: %+v", tried)
}
// A head that does not merge fails both layers: the check's own word.
conflict := triedOf(builder.CheckVerdict{Verdict: "fail", Summary: "does not merge", Gate: &builder.Layer{Verdict: "fail"},
Repo: &builder.Layer{Verdict: "fail", Summary: "does not merge"}})
if conflict.Verdict != "fail" || conflict.Summary != "does not merge" {
t.Fatalf("a try that does not merge: %+v", conflict)
}
// No verdict is an error, never a pass.
if none := triedOf(builder.CheckVerdict{}); none.Verdict != "error" {
t.Fatalf("a try with no verdict: %+v", none)
}
}
+1 -1
View File
@@ -688,7 +688,7 @@ func spelledOf(m inventory.BatchedMerge) string {
func namedOf(m inventory.BatchedMerge, repository string) inventory.PlanMerge {
k := announcedOf(m)
return inventory.PlanMerge{Repository: repository, Commit: m.Commit, Number: k.Number, Title: k.Title, Moves: k.Moves,
Head: k.HeadSHA, Merged: m.Merged, Heard: m.Heard}
Merged: m.Merged, Heard: m.Heard}
}
// laterMerge says a was merged after b: by the forge's merge time, then by when each was heard.
@@ -1,92 +0,0 @@
package main
import (
"errors"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/conditions"
"github.com/novox/mesh-controller/internal/link"
)
// **Every user's slow consumer is S9's, not the controller's alone** (novox/hq ADR 0357, issue 270). On
// 2026-10-07 the bus closed two clients on the laptop as slow consumers and nothing raised a condition:
// the controller heard only its own connection. Said by the bus module and read back, a user the bus
// closes is one condition naming the module and its machine, raised at the first look and cleared an hour
// after the last closure.
func TestAUserTheBusClosedAsASlowConsumerIsOneConditionNamingItsModuleAndMachine(t *testing.T) {
closed := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
now := closed.Add(20 * time.Second)
store := conditions.NewInMemory()
k := conditions.NewKeeper(t.Context(), conditions.Options{Store: store, History: store,
Teller: &conditions.Told{}, Now: func() time.Time { return now }})
t.Cleanup(func() { k.Close(t.Context()) })
advisories := link.BusClosures([]link.BusClosure{
{User: "laptop.claude-code", CID: 10, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Hour),
Closed: closed.Add(-10 * time.Minute)},
{User: "laptop.claude-code", CID: 44, Reason: "Slow Consumer (Write Deadline)", Opened: closed.Add(-time.Minute),
Closed: closed},
{User: "anchor.desk", CID: 7, Reason: "Maximum Message Payload Exceeded", Opened: closed.Add(-time.Hour),
Closed: closed},
}, []string{link.AdvisorySlowConsumer, link.AdvisorySlowConsumer, link.AdvisoryConnectionClosed}, closed.Add(-time.Hour))
look := func() {
t.Helper()
f := &signalFacts{now: now, host: "anchor", advisories: advisories}
if err := k.Reconcile(t.Context(), "S9", watchAdvisories(f)); err != nil {
t.Fatal(err)
}
}
look()
slow, found, err := conditions.ReadOne(t.Context(), store, "bus.laptop.claude-code.slow-consumer")
if err != nil || !found {
t.Fatalf("no slow-consumer condition for the user: %v %v", found, err)
}
if slow.Subject.Machine != "laptop" || slow.Observations != 2 || !slow.LastObserved.Equal(closed) {
t.Fatalf("the slow consumer is said on %q, %d closure(s), last %s: %+v", slow.Subject.Machine, slow.Observations,
slow.LastObserved, slow)
}
if !strings.Contains(slow.Summary, "claude-code on laptop") || strings.Contains(slow.Summary, "laptop.claude-code") {
t.Fatalf("the summary does not name the module in words, or carries the dotted user: %q", slow.Summary)
}
if !strings.Contains(slow.Headline, "Claude-code on laptop") {
t.Fatalf("the headline is %q", slow.Headline)
}
cut, found, err := conditions.ReadOne(t.Context(), store, "bus.anchor.desk.connection-closed")
if err != nil || !found || cut.Subject.Machine != "anchor" || !strings.Contains(cut.Summary, "maximum message payload exceeded") {
t.Fatalf("the user cut off for breaking the bus's rules: %v %v %+v", found, err, cut)
}
// An hour after the newest closure without another, both clear.
now = closed.Add(time.Hour + time.Minute)
look()
for _, key := range []string{"bus.laptop.claude-code.slow-consumer", "bus.anchor.desk.connection-closed"} {
if _, found, _ := k.Get(t.Context(), key); found {
t.Fatalf("%s is still open an hour after the last closure", key)
}
}
}
// **A read of the bus module's events that fails does not blind S9** (review of #326): the controller's own
// signals are still said, and the failed read is a condition of its own.
func TestABusModuleReadThatFailsKeepsTheControllersOwnSignals(t *testing.T) {
at := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
f := &signalFacts{now: at.Add(time.Minute), host: "anchor",
advisories: []link.Advisory{{Kind: link.AdvisorySlowConsumer, ID: "controller", Said: "the controller fell behind",
First: at, Last: at, Count: 1}},
busFaultsErr: errors.New("nats: timeout")}
keys := map[string]bool{}
for _, o := range watchAdvisories(f) {
keys[o.Key()] = true
}
if !keys["bus.controller.slow-consumer"] || !keys["bus.bus-module.bus-faults-unread"] {
t.Fatalf("observed %v: the controller's own slow consumer and the unread events, both", keys)
}
for _, row := range signalsTable {
if row.Row == "S9" && row.needs(f) != nil {
t.Fatal("the S9 row went blind for a read of the bus module's events")
}
}
}
+13 -76
View File
@@ -190,90 +190,27 @@ func TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne(t *testing.
}
}
// A walk's leave names the builds its own pull requests' verdicts made at the head each merge merged — never an
// earlier head's — read once per walk, and none when they cannot be read: the builder takes nothing a verdict does
// not name (novox/hq ADR 0350 §2, issue 703).
// A walk's leave names the builds its own pull requests' verdicts made, and none when they cannot be read: the
// builder takes nothing a verdict does not name (novox/hq ADR 0350 §2).
func TestAWalksLeaveNamesTheBuildsItsPullRequestsVerdictsMade(t *testing.T) {
was := checkedBuildsOf
forgetWalkVerdicts()
t.Cleanup(func() { checkedBuildsOf = was; forgetWalkVerdicts() })
var asked [][]link.PullRef
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
atHead := link.CheckedBuild{Pull: pull, Check: "build-2", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:2"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
t.Cleanup(func() { checkedBuildsOf = was })
var asked []link.PullRef
named := []link.CheckedBuild{{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 7}, Check: "build-1",
Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}}
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{atHead, earlier}, nil
asked = pulls
return named, nil
}
pulls := []link.PullRef{pull}
heads := map[link.PullRef]string{pull: "bbbbbbbbbbbb0123456789"}
for range 3 {
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, heads); !reflect.DeepEqual(got.Builds,
[]link.CheckedBuild{atHead}) {
t.Fatalf("the leave names %+v, not only the merged head's build", got.Builds)
}
}
if !reflect.DeepEqual(asked, [][]link.PullRef{pulls}) {
t.Fatalf("the verdicts were read %d times for one walk: %v", len(asked), asked)
}
// A merge whose head was not said takes nothing.
if got := reuseWithVerdicts(t.Context(), "plan-1", &link.ReuseAsk{Pulls: pulls}, nil); got.Builds != nil {
t.Fatalf("a merge of no said head names %+v", got.Builds)
pulls := []link.PullRef{{Owner: "novox", Repo: "shop", Number: 7}}
if got := reuseWithVerdicts(t.Context(), &link.ReuseAsk{Pulls: pulls}); !reflect.DeepEqual(got.Builds, named) ||
!reflect.DeepEqual(asked, pulls) {
t.Fatalf("the leave names %+v, read for %+v", got.Builds, asked)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := reuseWithVerdicts(t.Context(), "plan-2", &link.ReuseAsk{Pulls: pulls}, heads); got.Builds != nil {
if got := reuseWithVerdicts(t.Context(), &link.ReuseAsk{Pulls: pulls}); got.Builds != nil {
t.Fatalf("verdicts that could not be read named %+v", got.Builds)
}
}
// A check is handed only the builds earlier checks of its own pull request at its own head made: another pull
// request's build, or an earlier head's, is never kept (novox/hq issue 703). None when they cannot be read.
func TestACheckIsHandedOnlyItsOwnHeadsBuilds(t *testing.T) {
was := checkedBuildsOf
t.Cleanup(func() { checkedBuildsOf = was })
pull := link.PullRef{Owner: "novox", Repo: "shop", Number: 7}
mine := link.CheckedBuild{Pull: pull, Check: "build-1", Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
earlier := link.CheckedBuild{Pull: pull, Check: "build-0", Head: "aaaaaaaaaaaa", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
another := link.CheckedBuild{Pull: link.PullRef{Owner: "novox", Repo: "shop", Number: 8}, Check: "build-9",
Head: "bbbbbbbbbbbb", Built: link.CheckBuilt{Module: "app", Key: "r1:1"}}
var asked [][]link.PullRef
checkedBuildsOf = func(_ context.Context, pulls []link.PullRef) ([]link.CheckedBuild, error) {
asked = append(asked, pulls)
return []link.CheckedBuild{mine, earlier, another}, nil
}
p := link.PullUpdated{Owner: "novox", Repo: "shop", Number: 7, Commit: "bbbbbbbbbbbb0123456789"}
builds := []link.CheckBuild{{Module: "app"}}
if got := namedForTheCheck(t.Context(), p, builds); !reflect.DeepEqual(got, []link.CheckedBuild{mine}) ||
!reflect.DeepEqual(asked, [][]link.PullRef{{pull}}) {
t.Fatalf("the check is handed %+v, read for %v", got, asked)
}
checkedBuildsOf = func(context.Context, []link.PullRef) ([]link.CheckedBuild, error) {
return nil, errors.New("no bus")
}
if got := namedForTheCheck(t.Context(), p, builds); got != nil {
t.Fatalf("verdicts that could not be read named %+v", got)
}
}
func forgetWalkVerdicts() {
walkVerdicts.Lock()
defer walkVerdicts.Unlock()
walkVerdicts.read = map[string]verdictsRead{}
}
// A walk's merged heads are the forge's word for each merge; a merge announced without its head, or a pull request
// merged at two heads in one walk, has none, and its walk takes no check's build (novox/hq issue 703).
func TestAWalksMergedHeadsAreTheForgesWord(t *testing.T) {
merge := func(number int, head string) inventory.PlanMerge {
return inventory.PlanMerge{Repository: "novox/shop", Number: number, Head: head}
}
p := &inventory.Plan{Delivery: &inventory.PlanDelivery{Merges: []inventory.PlanMerge{
merge(7, "aaaaaaaaaaaa1"), merge(8, ""), merge(9, "bbbbbbbbbbbb"), merge(9, "cccccccccccc")}}}
heads := mergedHeadsOf(p)
shop := func(n int) link.PullRef { return link.PullRef{Owner: "novox", Repo: "shop", Number: n} }
if heads[shop(7)] != "aaaaaaaaaaaa1" || heads[shop(8)] != "" || heads[shop(9)] != "" {
t.Fatalf("merged heads %v", heads)
}
}
+25 -191
View File
@@ -3,7 +3,6 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"path"
"regexp"
@@ -249,16 +248,31 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
if err != nil {
return link.BuildRequest{}, err
}
// **A pull request into a release branch is judged by that branch** (novox/hq ADR 0358): asked of the forge once.
release, err := releaseBranchOf(ctx, p, scope, entries)
if err != nil {
return link.BuildRequest{}, err
// Beside it, at what the mesh runs: each core repository by the module the mesh builds from it.
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
url, err := clone(e.Source)
if err != nil {
return link.BuildRequest{}, err
}
refs := besideRefs(dir, current[e.Manifest.Module].Commit)
beside[dir] = link.CheckedOut{Repository: url, Ref: refs[dir]}
if dir == "mesh-controller" {
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
if url, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
sibling)}); err == nil {
beside[sibling] = link.CheckedOut{Repository: url, Ref: refs[sibling]}
}
}
}
}
}
beside, err := besideOf(entries, func(module string) string { return current[module].Commit }, clone, release)
if err != nil {
return link.BuildRequest{}, err
}
builds := checkBuildsOf(p, scope, entries, clone)
return link.BuildRequest{
ID: link.NewBuildID(time.Now()),
Repository: repository,
@@ -268,188 +282,10 @@ func checkRequestFor(ctx context.Context, open *stores, p link.PullUpdated, scop
Source: sourceOnSeat(source),
Check: &link.CheckRequest{Owner: p.Owner, Repo: p.Repo, Number: p.Number, Base: p.Base,
Paths: p.Paths, Beside: beside, Modules: scope.Modules, Dependents: scope.Dependents, New: scope.New,
Manifests: scope.Manifests, Judge: scope.Judge, Builds: builds, Named: namedForTheCheck(ctx, p, builds),
Release: release},
Manifests: scope.Manifests, Judge: scope.Judge, Builds: checkBuildsOf(p, scope, entries, clone)},
}, nil
}
// besideOf is what a check reads beside its own repository, by the directory it is found under: each core repository
// by the module the mesh builds from it, at besideRefs — or, for a pull request into a release branch (novox/hq ADR
// 0358 §3), at its branch of that name where it has one and at its trunk otherwise, the controller's main kept for
// its judge, and mesh-lab at its trunk whatever the base: its replays run with the container runtime's socket. running
// is the commit the mesh runs of a module.
func besideOf(entries []inventory.Entry, running func(module string) string,
clone func(inventory.Source) (string, error), release string) (map[string]link.CheckedOut, error) {
beside := map[string]link.CheckedOut{}
for _, e := range entries {
dir, core := coreModules[e.Manifest.Module]
if !core || e.Provided || e.Source.Repository == "" {
continue
}
url, err := clone(e.Source)
if err != nil {
return nil, err
}
refs := besideRefs(dir, running(e.Manifest.Module))
at := func(dir, trunk string) link.CheckedOut {
if release == "" {
return link.CheckedOut{Repository: url, Ref: refs[dir]}
}
if trunk == "" {
trunk = "main"
}
return link.CheckedOut{Repository: url, Ref: release, Else: trunk}
}
beside[dir] = at(dir, followedBranch(e.Source.Ref))
if dir == "mesh-controller" {
// The controller's main stays its main: it judges only a check the running controller predates.
beside["mesh-controller-main"] = link.CheckedOut{Repository: url, Ref: refs["mesh-controller-main"]}
if release != "" {
// mesh-controller has the release branch: that is what makes it one (ADR 0358 §1).
beside[dir] = link.CheckedOut{Repository: url, Ref: release}
}
if e.Source.Seat != "" {
for _, sibling := range []string{"mesh-lab", "mesh-sdk"} {
if url, err := clone(inventory.Source{Seat: e.Source.Seat, Repository: siblingOf(e.Source.Repository,
sibling)}); err == nil {
b := link.CheckedOut{Repository: url, Ref: refs[sibling]}
// **mesh-lab is read at its trunk, always** (ADR 0358): its replays run with the container
// runtime's socket, so they are only ever reviewed code — never a branch anyone may create.
if release != "" && sibling != "mesh-lab" {
b = link.CheckedOut{Repository: url, Ref: release, Else: "main"}
}
beside[sibling] = b
}
}
}
}
}
return beside, nil
}
// releaseBranchOf is the release branch a pull request merges into (novox/hq ADR 0358 §1): its base, when that is not
// its repository's trunk and mesh-controller has a branch of that name; empty otherwise, and the pull request is
// checked as before it. **A base that is a trunk asks nothing**: the branch a module built from the repository follows
// (main when it records none), or mesh-controller's own trunk. Any other base is asked of the forge once, and an
// answer it cannot give is an error, never read as "no release branch".
func releaseBranchOf(ctx context.Context, p link.PullUpdated, scope checkScope, entries []inventory.Entry) (string, error) {
base := p.Base
if base == "" {
return "", nil
}
if scope.From != nil && base == trunkOf(scope.From.Source) {
return "", nil
}
var controller *inventory.Entry
for i, e := range entries {
if e.Manifest.Module == "mesh-controller" && !e.Provided && e.Source.Repository != "" {
controller = &entries[i]
break
}
}
if controller == nil || base == trunkOf(controller.Source) {
return "", nil
}
if !tryRef.MatchString(base) || strings.Contains(base, "..") {
return "", nil // no branch the forge could hold: checked as before
}
owner, repo := sourceOwner(controller.Source.Repository), sourceName(controller.Source.Repository)
if owner == "" || repo == "" {
return "", nil
}
has, err := forgeHasBranch(ctx, owner, repo, base)
if err != nil {
return "", fmt.Errorf("whether %s/%s#%d merges into a release branch cannot be told — the forge did not say "+
"whether %s/%s has a branch %s (novox/hq ADR 0358): %w", p.Owner, p.Repo, p.Number, owner, repo, base, err)
}
if !has {
return "", nil
}
return base, nil
}
// trunkOf is the branch a module's source follows: main when it records none, or records a commit.
func trunkOf(s inventory.Source) string {
if t := strings.TrimSpace(followedBranch(s.Ref)); t != "" {
return t
}
return "main"
}
// sourceName is the name of a recorded repository, a path on the git seat or a URL: novox/mesh-host → mesh-host.
func sourceName(repository string) string {
parts := strings.Split(strings.Trim(strings.TrimSuffix(repository, ".git"), "/"), "/")
if len(parts) < 2 {
return ""
}
return parts[len(parts)-1]
}
// forgeHasBranch asks the forge, through its module's tools on the bus, whether a repository has a branch. A
// variable so a test needs no forge.
var forgeHasBranch = func(ctx context.Context, owner, repo, branch string) (bool, error) {
js, err := aBus()
if err != nil {
return false, err
}
defer js.Close()
raw, _ := json.Marshal(map[string]any{"path": "/repos/" + owner + "/" + repo + "/branches/" + branch})
answer, err := link.Ask(ctx, link.OverNATS{Conn: js.Conn()}, "gitea", "gitea_api", raw, 30*time.Second)
if err != nil {
return false, err
}
return branchAnswered(answer, branch)
}
// branchAnswered reads the forge's answer about a branch: a 404 is "no such branch"; an answer naming the branch is
// "it has it"; anything else — another error, an unreadable answer, one naming no branch or another — is an error,
// never read as "no release branch".
func branchAnswered(answer link.Answer, branch string) (bool, error) {
if answer.Error != "" {
if strings.Contains(answer.Error, " 404 ") {
return false, nil
}
return false, errors.New(answer.Error)
}
var found struct {
Result struct {
Name string `json:"name"`
} `json:"result"`
}
if err := json.Unmarshal(answer.Result, &found); err != nil {
return false, fmt.Errorf("the forge's answer about branch %s cannot be read: %w", branch, err)
}
if found.Result.Name != branch {
return false, fmt.Errorf("the forge answered about branch %s naming %q", branch, found.Result.Name)
}
return true, nil
}
// namedForTheCheck are the builds earlier checks of the same pull request at the same head named, read once for the
// check (novox/hq issue 703): asked again because only the trunk moved, the check keeps what they built of a key
// still the same. Another pull request's build, or one of an earlier head, is never kept. None when they cannot be
// read, and the check builds, as before.
func namedForTheCheck(ctx context.Context, p link.PullUpdated, builds []link.CheckBuild) []link.CheckedBuild {
if len(builds) == 0 || p.Commit == "" {
return nil
}
pull := link.PullRef{Owner: p.Owner, Repo: p.Repo, Number: p.Number}
read, err := checkedBuildsOf(ctx, []link.PullRef{pull})
if err != nil {
fmt.Printf(" the earlier checks' builds of %s/%s#%d could not be read, so the check builds: %v\n",
p.Owner, p.Repo, p.Number, err)
// empty-on-error: said above; a check handed no named build builds every module, as before issue 703
return nil
}
var named []link.CheckedBuild
for _, b := range read {
if b.Pull == pull && link.SameHead(b.Head, p.Commit) {
named = append(named, b)
}
}
return named
}
// checkBuildsOf are the modules a merge of the change would build, in the walk's order, as the check builds them
// for the walk to take (novox/hq ADR 0350 §1): a module built from the repository checked, at the branch it merges
// into, from the merge the check makes; any other — a module standing on one — from its own repository at the
@@ -596,8 +432,6 @@ func checkedOf(result link.BuildResult) link.Checked {
// repositories read beside it.
c.Trunk, c.Beside, c.Took = result.Check.Trunk, result.Check.Beside, result.Check.Took
c.Built = result.Check.Built
c.Release, c.JudgedBy, c.BesideAt = result.Check.Release, result.Check.JudgedBy, result.Check.BesideAt
c.Validator = result.Check.Validator
if c.Gate == nil {
// A build seat from before the layers: its verdict is the gate's.
c.Gate = &link.CheckLayer{Verdict: c.Verdict, Summary: c.Summary}
-12
View File
@@ -1443,12 +1443,6 @@ func probeGates(ctx context.Context, d *doctor) ([]conditions.Observation, error
}
// A release held after a failed one, while builds still wait for a gate (ADR 0236).
out = append(out, backlogObservation()...)
// And a rest that went silent (novox/hq ADR 0353 §4).
silent, err := silentObservation(ctx, d.open.inventory)
if err != nil {
return nil, err
}
out = append(out, silent...)
return sortedFound(dedupeObservations(out)), nil
}
@@ -1538,12 +1532,6 @@ func gateLine(g *inventory.PlanGate) string {
case inventory.GateFailed:
what = "FAILED"
}
switch {
case g.Verdict == "" && g.Pending:
what = "kept, not yet sent" // novox/hq ADR 0353 §8
case g.Verdict == "" && g.Silent:
what = "silent on its send, left" // ADR 0353 §4
}
if g.Verdict == "" && g.Waiting() {
what = "waits for a busy container on " + g.Busy.Machine
}
+1 -1
View File
@@ -151,7 +151,7 @@ func heldMachines(ctx context.Context, open *stores, names []string) (map[string
for _, mv := range f.moves(node, modules, same, known, false) {
if planStillToSend(plans, mv.Module, node) == "" {
why = append(why, fmt.Sprintf("%s would move from %s to %s, which has passed no gate yet — a release "+
"plan sends it, one machine first and then the rest at once, judged", mv.Module, buildName(mv.From), buildName(mv.To)))
"plan sends it, one machine at a time, judged", mv.Module, buildName(mv.From), buildName(mv.To)))
}
}
if len(why) > 0 {
-87
View File
@@ -1,87 +0,0 @@
package main
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// A tier whose module is no move — made from the source every machine running it runs, so nothing is
// sent — is not waited on for a report after its marking (novox/hq issue 648). Its machines answered the
// send that last carried that build before the walk got there, and nothing will make them report again:
// the walk asks the next tier in the same step, rather than "waiting for app on anchor, laptop" for ever.
func TestANoMoveTierWaitsForNoNewReport(t *testing.T) {
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
inv := g.open.inventory
// app is the build machine, so a module of the next tier is built by it: the tier gates the next on app
// running on its machines.
for _, m := range []catalogue.Manifest{
{Module: "app", Version: "c2", Claims: []catalogue.Claim{{Name: "node-build-agent", Scope: catalogue.ScopeNode}}},
{Module: "web", Version: "w1"},
} {
if err := inv.RegisterModule(ctx, m, inventory.Source{Repository: "novox/mesh-catalog", Seat: "git",
Path: "modules/" + m.Module, BuiltFrom: m.Version, Head: m.Version, Asked: time.Now().Add(-time.Minute)}); err != nil {
t.Fatal(err)
}
}
// What anchor and laptop were last sent (c1) was made from the source the plan's build of c2 is.
for _, b := range []inventory.Build{
{ID: "build-1s", Module: "app", Commit: "c1", SourceFingerprint: "src1:app",
Asked: time.Now().Add(-30 * time.Second), At: time.Now().Add(-30 * time.Second)},
{ID: "build-2s", Module: "app", Commit: "c2", SourceFingerprint: "src1:app", Asked: time.Now(), At: time.Now()},
} {
b.Manifest, _ = json.Marshal(catalogue.Manifest{Module: "app", Version: "1"})
if err := inv.RecordBuild(ctx, b); err != nil {
t.Fatal(err)
}
}
// Each machine answered that send, applied, before the walk's tier was built.
for _, n := range []string{"anchor", "laptop"} {
if _, err := inv.RecordDoing(ctx, nodeID(t, g.open, n), inventory.Doing{Node: n,
Outcome: inventory.OutcomeApplied, Declared: "d-" + n + "-c1", Applied: 1,
At: time.Now().Add(-10 * time.Second)}); err != nil {
t.Fatal(err)
}
}
built := time.Now().UTC()
p := g.plan(t)
p.Tiers = [][]string{{"app"}, {"web"}}
p.Modules["app"].Build, p.Modules["app"].BuiltAt = "build-2s", &built
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, g.open)
if len(g.sent) != 0 {
t.Fatalf("a no-move build was sent: %v", g.sent)
}
p = g.plan(t)
if s := p.Modules["app"]; s == nil || !strings.HasPrefix(s.Why, "no move") {
t.Fatalf("the build was not marked no move: %+v", s)
}
if p.Tier < 1 || strings.Contains(p.Note, "waiting for app") {
t.Fatalf("the walk waits on a no-move tier for a report nothing will cause: tier %d, %s %q", p.Tier, p.State, p.Note)
}
// A machine that has not answered its last send is still waited on: no move is not "never sent".
p.Tier, p.State, p.Note = 0, inventory.PlanRolling, ""
if err := inv.RecordSent(ctx, nodeID(t, g.open, "laptop"), "d-laptop-later", map[string]string{"app": "c1"}); err != nil {
t.Fatal(err)
}
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, g.open)
p = g.plan(t)
if p.Tier != 0 || !strings.Contains(p.Note, "waiting for app on laptop") {
t.Fatalf("a machine that has not answered its last send was not waited on: tier %d, %q", p.Tier, p.Note)
}
}
-9
View File
@@ -106,8 +106,6 @@ func run() error {
return killCommand(ctx, args[1:])
case "forget-checks":
return forgetChecksCommand(ctx, args[1:])
case "try":
return tryCommand(ctx, args[1:])
case "pause", "resume":
return pauseCommand(ctx, args[0], args[1:])
case "collection":
@@ -383,13 +381,6 @@ func (b builds) Built(ctx context.Context, result link.BuildResult) error {
// **A merge check builds nothing** (novox/hq to-be 45 §9): its verdict is said, and nothing of it is
// recorded or registered.
if result.Check != nil || result.Checked != nil {
if isTry(result) {
// A branch tried before its pull request (novox/hq issue 686): its asker reads the outcome from the
// build's log. Never said as `checked`, so the forge's holder sets no status.
fmt.Printf("%s: a try of %s at %.8s, read with `try --id`: said to nobody, nothing taken in\n", result.ID,
result.Repository, result.Ref)
return nil
}
checked(ctx, result)
return nil
}
-28
View File
@@ -456,9 +456,6 @@ var plainWordings = map[string]func(conditions.Observation) words{
"release-held": worded(func(o conditions.Observation) words {
return releaseHeldWords(nil, o.Also)
}),
"release-silent": worded(func(o conditions.Observation) words {
return releaseSilentWords(o.Also)
}),
"facts-stale": worded(func(o conditions.Observation) words {
return words{Headline: "Merge checks use outdated facts",
Explanation: "The facts a merge check judges a change against are out of date; they are gathered again " +
@@ -620,18 +617,6 @@ var plainWordings = map[string]func(conditions.Observation) words{
Explanation: "The bus reports a listener too slow to keep up, so messages to it are late.",
Resolved: "The listener keeps up again"}
}),
kindBusFaultsUnread: worded(func(o conditions.Observation) words {
return words{Headline: "Slow clients of the bus cannot be checked",
Explanation: "The controller could not read what the bus said about its clients' connections, so a " +
"client the bus cut off would go unnoticed. It tries again at every check.",
Resolved: "Resolved: the bus's clients are checked again"}
}),
"connection-closed": worded(func(o conditions.Observation) words {
return words{Headline: "The bus cut a client off",
Explanation: "The bus closed a client's connection because it broke the bus's rules, so what it was " +
"sending was lost.",
Resolved: "Resolved: the client keeps to the bus's rules"}
}),
kindBusReconnects: worded(func(o conditions.Observation) words {
return words{Headline: "A client keeps losing the bus",
Explanation: "One of the mesh's clients lost its connection to the bus again and again in the last hour. " +
@@ -856,19 +841,6 @@ func releaseHeldWords(modules, machines []string) words {
Resolved: "Resolved: the held updates are released"}
}
// releaseSilentWords are the plain words of machines that said nothing on an update they were sent (ADR 0353 §4).
func releaseSilentWords(machines []string) words {
where := "Some machines"
if len(machines) > 0 {
where = conditions.Capital(namesWords(machines, 4))
}
return words{Headline: "Machines did not answer an update",
Needs: "check that they are awake and on the mesh; the next walk sends them what waits.",
Explanation: where + " were sent module updates and said nothing about them within fifteen minutes, so " +
"they were left as they were. What they run is not known to be the update.",
Resolved: "Resolved: the next walk reached them"}
}
// reloginNeeds is what an account waiting for its groups needs (ADR 0252).
func reloginNeeds(node string) string {
return fmt.Sprintf("log out of %s completely and log in again, or restart it.", node)
+3
View File
@@ -264,6 +264,9 @@ func serve(ctx context.Context) (err error) {
// The facts snapshot a merge check is fed (novox/hq to-be 45 §9): kept current while this
// controller holds the lease, read by the build seat from the artifact store.
go exportingFacts(ctx, open, bus.Conn.ConnectedServerVersion)
// The record mirrors (novox/hq ADR 0289, design 57 F3): the assignments and each registered module's
// declared config on the bus, written from the store only when a record changed, by the lease holder.
go mirroringRecords(ctx, inv, bus.Conn, bus)
// Every call carries the lease's epoch, and its record is written only under the lease (novox/hq
// to-be 45 §6).
link.Calls.UnderLease(func() (uint64, error) { return theLease.epoch(ctx) })
+105
View File
@@ -0,0 +1,105 @@
package main
import (
"context"
"fmt"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/recordmirror"
)
// The controller's record mirrors on the bus (novox/hq ADR 0289, design 54, design 57 F3): the assignments,
// and each registered module's declared config, which mesh-config resolves a module's config values from.
//
// **Built from the store, on a tick of their own, by the lease holder alone.** A record changes through a
// verb that runs as a command of its own, a reconcile, a merge's registration or a pending assignment made,
// so no one call site sees every change; what every change has in common is that it is committed to the
// store. So the serving controller reads the store every mirrorEvery and writes only what differs from what
// the bucket holds (ADR 0289 §1): a quiet mesh writes nothing, and a change rolled back is never read. A
// mirror is therefore at most mirrorEvery behind its store.
// mirrorEvery is how often the mirrors are compared with the store: two small reads of the store.
const mirrorEvery = 10 * time.Second
// mirroringRecords keeps the mirrors current for as long as this controller serves: ctx ends when it stops.
func mirroringRecords(ctx context.Context, inv *inventory.Inventory, conn *nats.Conn, announce recordmirror.Announcer) {
api, err := jetstream.New(conn)
if err != nil {
fmt.Printf("the record mirrors are not kept: %v\n", err)
return
}
holds := func() bool {
epoch, err := theLease.epoch(ctx)
return err == nil && epoch > 0
}
epoch := func() uint64 {
e, _ := theLease.epoch(ctx)
return e
}
sources := map[string]func(context.Context) (map[string][]byte, error){
recordmirror.Assignments.Name: inv.AssignmentsMirror,
recordmirror.DeclaredConfig.Name: inv.DeclaredConfigMirror,
}
writers := map[string]*recordmirror.Writer{}
failing := map[string]string{}
tick := time.NewTicker(mirrorEvery)
defer tick.Stop()
for {
if holds() {
for _, m := range recordmirror.Mirrors() {
why := mirrorOnce(ctx, api, m, writers, sources[m.Name], announce, epoch, holds)
if why != failing[m.Name] {
if why != "" {
fmt.Printf("the %s mirror cannot be kept: %s\n", m.Name, why)
} else {
fmt.Printf("the %s mirror is kept again\n", m.Name)
}
failing[m.Name] = why
}
}
}
select {
case <-ctx.Done():
return
case <-tick.C:
}
}
}
// mirrorOnce brings one mirror to what the store holds, and answers why it could not; empty when it could.
func mirrorOnce(ctx context.Context, api jetstream.JetStream, m recordmirror.Mirror, writers map[string]*recordmirror.Writer,
source func(context.Context) (map[string][]byte, error), announce recordmirror.Announcer, epoch func() uint64,
holds func() bool) string {
reading, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
w := writers[m.Name]
if w == nil {
kv, err := api.KeyValue(reading, m.Bucket)
if err != nil {
return fmt.Sprintf("the bucket %s cannot be read: %v", m.Bucket, err)
}
w = &recordmirror.Writer{Mirror: m, KV: kv, Announce: announce, Epoch: epoch, Holds: holds}
writers[m.Name] = w
}
want, err := source(reading)
if err != nil {
return fmt.Sprintf("the store cannot be read: %v", err)
}
lost := w.LostEvents
r, err := w.Sync(reading, want)
if r.Written+r.Removed > 0 {
fmt.Printf("the %s mirror moved: %d written, %d removed\n", m.Name, r.Written, r.Removed)
}
if w.LostEvents > lost {
fmt.Printf("the %s mirror lost %d event(s): the bucket holds the change, its notice was refused twice\n",
m.Name, w.LostEvents-lost)
}
if err != nil {
return err.Error()
}
return ""
}
+41 -536
View File
@@ -39,8 +39,8 @@ import (
// from the same manifest is no move.
//
// **The release plan** is what walks the waiting moves through: whenever moves wait for a gate that no
// started plan is walking, the mesh opens one — every such machine, the first sent and judged, then the rest
// at once, each judged by its own gate (novox/hq ADR 0353, in place of one at a time). A release plan that fails stops the next one opening on its own:
// started plan is walking, the mesh opens one — every such machine, one at a time, the control node last,
// each sent and judged before the next. A release plan that fails stops the next one opening on its own:
// a person releases it again (`upgrade release-backlog --why`), after the condition says why.
// sendScope is what a send may carry that no gate has seen.
@@ -81,15 +81,6 @@ type moveFacts struct {
srcs map[string]map[string]string
passed map[string]map[string]bool
plans []inventory.Plan
// self is the release walking these moves, whose own gates are no other walk's (novox/hq ADR 0353).
self string
}
type walkingKey struct{}
// withWalking says a release plan is the one sending: its own gates never hold its own sends back.
func withWalking(ctx context.Context, id string) context.Context {
return context.WithValue(ctx, walkingKey{}, id)
}
func readMoveFacts(ctx context.Context, inv *inventory.Inventory) (moveFacts, error) {
@@ -108,7 +99,6 @@ func readMoveFacts(ctx context.Context, inv *inventory.Inventory) (moveFacts, er
return f, err
}
f.plans, err = inv.OpenPlans(ctx)
f.self, _ = ctx.Value(walkingKey{}).(string)
return f, err
}
@@ -203,24 +193,17 @@ func (f moveFacts) moves(node string, modules []string, sent map[string]string,
// feet. A release keeps no module records: its open gate's carried moves are its walk.
func (f moveFacts) walkedBy(module, node, to string) string {
for _, p := range f.plans {
if !p.Open() || (p.Release != nil && p.ID == f.self) {
if !p.Open() {
continue
}
if p.Release != nil {
// The first machine's gate, and every gate of the rest still judging (novox/hq ADR 0353).
gates := []*inventory.PlanGate{p.Release.Gate}
for _, g := range p.Release.Gates {
gates = append(gates, g)
g := p.Release.Gate
if g == nil || g.Verdict != "" {
continue
}
for _, g := range gates {
// A send kept and not yet made walks nothing yet (ADR 0353 §8).
if g == nil || g.Verdict != "" || g.Silent || g.Pending {
continue
}
for _, c := range g.Carried {
if c.Module == module && !(slices.Contains(g.Machines, node) && sameCommit(c.To, to)) {
return p.ID
}
for _, c := range g.Carried {
if c.Module == module && !(slices.Contains(g.Machines, node) && sameCommit(c.To, to)) {
return p.ID
}
}
continue
@@ -320,8 +303,8 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
}
}
if len(refused) > 0 {
return nil, fmt.Errorf("%w: %s — a walk sends them, one machine first and then the rest at once, each judged (novox/hq ADR "+
"0236, ADR 0353); `upgrade backlog` lists them", errUngated, strings.Join(refused, "; "))
return nil, fmt.Errorf("%w: %s — a walk sends them, one machine at a time, each judged (novox/hq ADR "+
"0236); `upgrade backlog` lists them", errUngated, strings.Join(refused, "; "))
}
return kept, nil
}
@@ -337,35 +320,19 @@ func ungatedIn(ctx context.Context, open *stores, names []string, addedHolder st
// And the grants step first, when the send changes what the bus's user list must say (novox/hq issue 490):
// answered for the gate to keep, nil when there was none.
func gatedSend(ctx context.Context, open *stores, node string, owns []inventory.CarriedMove) ([]inventory.CarriedMove, []string, *inventory.GrantsStep, error) {
moves, err := gatedMoves(ctx, open, node, owns)
if err != nil || len(moves) == 0 {
return nil, nil, nil, err
}
sayRecreations(ctx, open, moves)
sent, grants, err := sendJudged(ctx, open, node)
if err != nil {
return nil, nil, nil, err
}
return moves, sent, grants, nil
}
// gatedMoves is what a gated send to one machine would carry, with the build each moves to, made of nothing:
// gatedSend's reading, which a walk also keeps before it sends (novox/hq ADR 0353 §8). Refused the same way
// while a plan that has started walks one of those builds elsewhere.
func gatedMoves(ctx context.Context, open *stores, node string, owns []inventory.CarriedMove) ([]inventory.CarriedMove, error) {
inv := open.inventory
f, err := readMoveFacts(ctx, inv)
if err != nil {
return nil, err
return nil, nil, nil, err
}
moves, err := machineMoves(ctx, open, f, node, true)
if err != nil {
return nil, err
return nil, nil, nil, err
}
// A move the send keeps is not made, and is no move for the gate to judge (novox/hq issue 201): a core module
// kept over an older record judged as moving back to it would read the gate as superseded by its own send.
if moves, err = withoutKept(ctx, open, node, moves); err != nil {
return nil, err
return nil, nil, nil, err
}
own := func(module string) bool {
return slices.ContainsFunc(owns, func(o inventory.CarriedMove) bool { return o.Module == module })
@@ -375,7 +342,7 @@ func gatedMoves(ctx context.Context, open *stores, node string, owns []inventory
continue
}
if id := f.walkedBy(mv.Module, node, mv.To); id != "" {
return nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
return nil, nil, nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
mv.Module, short(mv.To), node, id)
}
}
@@ -383,7 +350,7 @@ func gatedMoves(ctx context.Context, open *stores, node string, owns []inventory
// plan, or a release, on this machine is not crossed with a newer build; this send waits for its gate.
for _, o := range owns {
if id := f.walkedBy(o.Module, node, o.To); id != "" {
return nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
return nil, nil, nil, fmt.Errorf("%w: %s's build %s waits on %s, which %s is walking", errWalkedElsewhere,
o.Module, short(o.To), node, id)
}
}
@@ -396,26 +363,30 @@ func gatedMoves(ctx context.Context, open *stores, node string, owns []inventory
moves[i].Build = o.Build
}
}
if len(moves) == 0 {
return nil, nil
if len(moves) == 0 && len(owns) == 0 {
return nil, nil, nil, nil
}
for i := range moves {
if moves[i].Build == "" {
if moves[i].Build, err = inv.BuildOf(ctx, moves[i].Module, moves[i].To); err != nil {
return nil, err
return nil, nil, nil, err
}
}
}
sort.Slice(moves, func(i, j int) bool { return moves[i].Module < moves[j].Module })
return moves, nil
sayRecreations(ctx, open, moves)
sent, grants, err := sendJudged(ctx, open, node)
if err != nil {
return nil, nil, nil, err
}
return moves, sent, grants, nil
}
// passCarried keeps a pass as the verdict of every build the gate judged beside its own module.
func passCarried(ctx context.Context, open *stores, p *inventory.Plan, g *inventory.PlanGate, except string) {
seen := map[string]bool{}
for _, c := range g.Carried {
// A build put back with another machine's failure is not this gate's to pass (novox/hq ADR 0353 §3).
if c.Module == except || c.Build == "" || seen[c.Build] || slices.Contains(g.Returned, c.Module) {
if c.Module == except || c.Build == "" || seen[c.Build] {
continue
}
seen[c.Build] = true
@@ -724,7 +695,7 @@ func releaseBacklog(ctx context.Context, open *stores, by string) (*inventory.Pl
p := inventory.Plan{ID: fmt.Sprintf("release-%d", now.UnixNano()), Repository: releaseRepository,
Created: now, State: inventory.PlanRolling, Tiers: [][]string{names}, Modules: map[string]*inventory.PlanModule{},
Release: &inventory.PlanRelease{Order: order, By: by},
Note: fmt.Sprintf("%d build(s) wait for a gate on %s; one machine first, then the rest at once, each judged", len(names),
Note: fmt.Sprintf("%d build(s) wait for a gate on %s; one machine at a time, each judged", len(names),
strings.Join(order, ", "))}
if err := inv.SavePlan(ctx, &p); err != nil {
return nil, err
@@ -733,23 +704,14 @@ func releaseBacklog(ctx context.Context, open *stores, by string) (*inventory.Pl
return &p, nil
}
// advanceRelease takes one step of a release plan: the first machine sent everything waiting there under a
// gate and judged; once it passed, every other machine sent at once, each under its own gate (novox/hq ADR
// 0353 §1), and judged; the plan done when every machine is.
//
// **A send is kept before it is made** (ADR 0353 §8): the step that chooses a machine keeps its gate pending with
// what it will carry, and the next step makes the send. A controller replaced between the send and keeping it
// finds the gate pending, reads the machine's last send, and neither sends again nor leaves the machine unjudged.
// advanceRelease takes one step of a release plan: the next machine sent everything waiting there under a
// gate, or the machine being judged judged once more; the plan done when every machine is.
func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool, error) {
r := p.Release
if r.Settings != nil {
return advanceSettingsWalk(ctx, open, p)
}
ctx = withWalking(ctx, p.ID)
now := time.Now().UTC()
if r.Rest {
return advanceRest(ctx, open, p, now)
}
if r.Gate == nil {
heard, err := releaseHeard(ctx, open)
if err != nil {
@@ -762,7 +724,7 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
r.Next++
continue
}
moves, err := gatedMoves(ctx, open, node, nil)
moves, sent, grants, err := gatedSend(ctx, open, node, nil)
if errors.Is(err, errWalkedElsewhere) {
note := fmt.Sprintf("waiting before %s: %v", node, err)
changed := p.Note != note
@@ -770,15 +732,23 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
return changed, nil
}
if err != nil {
return false, fmt.Errorf("reading what waits on %s: %w", node, err)
return false, fmt.Errorf("sending %s what waits there: %w", node, err)
}
if len(moves) == 0 {
r.Done = append(r.Done, node)
r.Next++
continue
}
r.Gate = &inventory.PlanGate{Machines: []string{node}, Carried: moves, Pending: true}
p.Note = fmt.Sprintf("sending %s %d build(s) that wait for a gate, first", node, len(moves))
r.Gate = &inventory.PlanGate{Machines: sent, Since: &now, Carried: moves, Sent: sentNow(ctx, open.inventory, sent),
Grants: grants}
p.Note = fmt.Sprintf("sent %s %d build(s) that waited for a gate; judging them there", node, len(moves))
if said := grantsSaid(grants); said != "" {
p.Note += "; " + said
}
if said := recreationsSaid(moves); said != "" {
p.Note += "; " + said
}
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
}
p.State, p.Tier = inventory.PlanDone, len(p.Tiers)
@@ -789,19 +759,6 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
return true, nil
}
g := r.Gate
if g.Pending {
node := firstOf(g.Machines)
out, err := sendKept(ctx, open, p, node, g, now)
if err != nil || out == keptWaiting {
return out == keptWaiting, err
}
if out == keptNothing {
r.Gate = nil
r.Done = append(r.Done, node)
r.Next++
}
return true, nil
}
verdict, err := judgeMoves(ctx, open, g, nil, now)
if err != nil {
return false, err
@@ -820,11 +777,6 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
r.Done = append(r.Done, firstOf(g.Machines))
r.Next++
r.Gate = nil
// **Then the rest at once** (novox/hq ADR 0353 §1): every other machine heard from, each judged by its own
// gate, in place of one machine at a time (ADR 0236 §22).
r.Rest = true
p.Note = fmt.Sprintf("passed on %s; the rest are sent at once, each judged", firstOf(g.Machines))
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
case inventory.GateSuperseded:
// Another send moved a judged module on the judged machine (novox/hq issue 352): no verdict on what
@@ -845,453 +797,6 @@ func advanceRelease(ctx context.Context, open *stores, p *inventory.Plan) (bool,
return true, nil
}
// What sendKept did with a kept send.
const (
keptSent = iota // sent now
keptAlready // the machine was already sent it: judged from that send, nothing sent
keptNothing // nothing waits there any more: no gate
keptWaiting // another walk judges a build it would carry: it waits
)
// sendKept makes a send a gate kept before it was made (novox/hq ADR 0353 §8) — unless the machine's last send
// already carries what the gate is for, made before a controller was replaced and before it was kept: then
// nothing is sent, and the gate judges from that send (§6).
func sendKept(ctx context.Context, open *stores, p *inventory.Plan, node string, g *inventory.PlanGate,
now time.Time) (int, error) {
inv := open.inventory
moves, sent, grants, err := gatedSend(ctx, open, node, nil)
if errors.Is(err, errWalkedElsewhere) {
p.Note = fmt.Sprintf("waiting before %s: %v", node, err)
return keptWaiting, nil
}
if err != nil {
return keptSent, fmt.Errorf("sending %s what waits there: %w", node, err)
}
if len(moves) > 0 {
g.Machines, g.Carried, g.Since, g.Grants, g.Pending = sent, moves, &now, grants, false
g.Sent = sentNow(ctx, inv, sent)
p.Note = fmt.Sprintf("sent %s %d build(s) that waited for a gate; judging them there", node, len(moves))
if said := grantsSaid(grants); said != "" {
p.Note += "; " + said
}
if said := recreationsSaid(moves); said != "" {
p.Note += "; " + said
}
fmt.Printf("%s: %s\n", p.ID, p.Note)
return keptSent, nil
}
if !alreadySent(ctx, inv, node, g.Carried) {
return keptNothing, nil
}
since := now
if s, found, err := inv.SentTo(ctx, node); err == nil && found {
if !s.At.IsZero() {
since = s.At
}
g.Sent = map[string]inventory.SentDeclaration{node: s}
}
g.Since, g.Pending = &since, false
p.Note = fmt.Sprintf("%s was already sent the %d build(s) kept for it, at %s: judged from that send, nothing sent again",
node, len(g.Carried), since.Local().Format("15:04:05"))
fmt.Printf("%s: %s\n", p.ID, p.Note)
return keptAlready, nil
}
// alreadySent says a machine's last send carries every build of the moves: a send made and not kept.
func alreadySent(ctx context.Context, inv *inventory.Inventory, node string, moves []inventory.CarriedMove) bool {
if len(moves) == 0 {
return false
}
sent, known, err := inv.SentBuilds(ctx, node)
if err != nil || !known {
return false
}
for _, mv := range moves {
if !sameCommit(sent[mv.Module], mv.To) {
return false
}
}
return true
}
// advanceRest takes one step of a release plan's rest (novox/hq ADR 0353): the sends kept for a step made, every
// machine of the step judged by its own gate, one that fails put back by itself while the others go on to their
// verdicts, and the next step — or the end — once every gate of the step has its verdict or is silent.
func advanceRest(ctx context.Context, open *stores, p *inventory.Plan, now time.Time) (bool, error) {
r := p.Release
// The sends kept for this step, in the plan's order: the control node, last in it, last (§1).
var pending []string
for _, n := range r.Order {
if g := r.Gates[n]; g != nil && g.Pending {
pending = append(pending, n)
}
}
if len(pending) > 0 {
// **A kept send carries only what was kept** (ADR 0353 §2): a build that passed no gate and came to wait on
// a machine since its step was chosen could reach a second machine in it; the step is chosen again.
if grew, err := keptStepGrew(ctx, open, r, pending); err != nil {
return false, err
} else if grew != "" {
for _, n := range pending {
delete(r.Gates, n)
}
opened, waiting, err := openRestStep(ctx, open, p)
if err != nil {
return true, err
}
p.Note = "chosen again: " + grew
if !opened && waiting != "" {
p.Note += "; " + waiting
}
return true, nil
}
var sent, already, waiting []string
for _, node := range pending {
out, err := sendKept(ctx, open, p, node, r.Gates[node], now)
if err != nil {
return true, err
}
switch out {
case keptSent:
sent = append(sent, node)
case keptAlready:
already = append(already, node)
case keptNothing:
delete(r.Gates, node)
r.Done = append(r.Done, node)
case keptWaiting:
waiting = append(waiting, node)
}
}
var said []string
if len(sent) > 0 {
said = append(said, "sent "+strings.Join(sent, ", ")+" at once, each judged there")
}
if len(already) > 0 {
said = append(said, strings.Join(already, ", ")+" already sent it: judged from that send")
}
if len(waiting) > 0 {
said = append(said, "waiting before "+strings.Join(waiting, ", ")+": another walk judges a build it carries")
}
if len(said) > 0 {
p.Note = strings.Join(said, "; ")
}
return len(sent)+len(already) > 0 || len(waiting) < len(pending), nil
}
reports, err := open.inventory.LastReports(ctx)
if err != nil {
return false, err
}
byNode := map[string]inventory.Reported{}
for _, rep := range reports {
byNode[rep.Node] = rep
}
changed, noteWas := false, p.Note
var notes, judging []string
batched, back := batchingRollbacks(ctx)
for _, node := range r.Order {
g := r.Gates[node]
if g == nil || g.Verdict != "" || g.Silent {
continue
}
// What another machine's failure already put back is not this gate's to judge (ADR 0353 §3).
if len(g.Carried) > 0 && len(g.Returned) >= len(g.Carried) {
decide(g, inventory.GateFailed, "what it was sent was put back with another machine's failure", now)
r.Failed = append(r.Failed, node)
changed = true
continue
}
// **A machine that says nothing on its send is left silent, not failed** (ADR 0353 §4): it is not judged
// until it reports, and past ApplySilentAfter it is listed and left.
if !reportedOnSend(byNode[node], node, g) {
if now.Sub(*g.Since) > inventory.ApplySilentAfter {
g.Silent = true
r.Silent = append(r.Silent, node)
changed = true
notes = append(notes, fmt.Sprintf("%s said nothing on its send within %s: left silent", node,
inventory.ApplySilentAfter))
continue
}
judging = append(judging, node+": not yet reported on its send")
continue
}
verdict, err := judgeMoves(ctx, open, g, nil, now)
if err != nil {
return changed, err
}
switch verdict {
case "":
if putBackBroken(ctx, open, p, g, "", nil) {
changed = true
notes = append(notes, p.Note)
}
judging = append(judging, node+": "+gateLine(g))
case inventory.GatePassed:
passCarried(ctx, open, p, g, "")
r.Done = append(r.Done, node)
changed = true
case inventory.GateSuperseded:
// Another send moved a judged module there (novox/hq issue 352): nothing judged, nothing put back on
// it; what still waits is released again.
r.Done = append(r.Done, node)
changed = true
notes = append(notes, fmt.Sprintf("superseded on %s: %s", node, g.Why))
default:
// **Put back by itself** (ADR 0353 §3): what was found wanting goes back at once, and every other
// machine of the step goes on to its own verdict.
stateWas := p.State
p.Note = ""
failCarried(batched, open, p, g, "")
p.State = stateWas
r.Failed = append(r.Failed, node)
changed = true
notes = append(notes, fmt.Sprintf("failed its gate on %s: %s — %s", node, g.Why, p.Note))
returnedElsewhere(ctx, open, r, node, g)
}
}
p.Note = ""
sendRollbacks(ctx, open, p, back)
if p.Note != "" {
notes = append(notes, strings.TrimPrefix(p.Note, "; "))
}
if len(judging) > 0 {
note := strings.Join(append(notes, "judging "+strings.Join(judging, "; ")), "; ")
p.Note = note
return changed || noteWas != note, nil
}
// Every gate of the step has its verdict or is silent: the next step, unless a machine failed (§3).
if len(r.Failed) == 0 {
opened, waiting, err := openRestStep(ctx, open, p)
if err != nil {
return changed, err
}
if opened {
return true, nil
}
if waiting != "" {
note := strings.Join(append(notes, waiting), "; ")
p.Note = note
return changed || noteWas != note, nil
}
}
p.Tier = len(p.Tiers)
end := fmt.Sprintf("released on %s", orNone(strings.Join(r.Done, ", ")))
if len(r.Silent) > 0 {
end += "; silent on their send, left: " + strings.Join(r.Silent, ", ")
}
if len(r.Skipped) > 0 {
end += "; not heard from, left as they were: " + strings.Join(r.Skipped, ", ")
}
p.State = inventory.PlanDone
if len(r.Failed) > 0 {
p.State = inventory.PlanFailed
end = "failed on " + strings.Join(r.Failed, ", ") + "; " + end
} else if len(r.Silent) > 0 && len(r.Gates) == len(r.Silent) {
// **A rest that said nothing is no success** (ADR 0353 §4): failure is loud; the condition says it.
p.State = inventory.PlanFailed
end = "no machine of the rest reported on its send; " + end
}
p.Note = strings.Join(append(notes, end), "; ")
fmt.Printf("%s: %s\n", p.ID, p.Note)
return true, nil
}
// keptStepGrew says why a step's kept sends would now carry a build no gate passed that they were not kept with;
// empty when none would.
func keptStepGrew(ctx context.Context, open *stores, r *inventory.PlanRelease, pending []string) (string, error) {
f, err := readMoveFacts(ctx, open.inventory)
if err != nil {
return "", err
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return "", err
}
for _, node := range pending {
moves, err := gatedMoves(ctx, open, node, nil)
if errors.Is(err, errWalkedElsewhere) {
continue // sendKept says it waits
}
if err != nil {
return "", err
}
kept := r.Gates[node].Carried
for _, mv := range moves {
// The node-engine and the resolver keep one machine at a time, passed or not (ADR 0353 §9).
if (f.gated(mv.Module, mv.To) && !oneAtATime(mv.Module, shelf[mv.Module])) || slices.ContainsFunc(kept, func(c inventory.CarriedMove) bool {
return c.Module == mv.Module && sameCommit(c.To, mv.To)
}) {
continue
}
return fmt.Sprintf("%s's build %s, which this step did not keep for it, came to wait on %s", mv.Module, short(mv.To), node), nil
}
}
return "", nil
}
// reportedOnSend says a machine's last report is about the send its gate judges (novox/hq issue 352): the
// declaration kept on the gate, or one sequenced after it; a gate that kept none reads the send made last.
func reportedOnSend(r inventory.Reported, node string, g *inventory.PlanGate) bool {
if r.At == nil {
return false
}
if s, kept := g.Sent[node]; kept {
return s.ReportsOn(r)
}
return r.Current
}
// returnedElsewhere marks, on every other gate of the rest still judging, the builds a failed machine's
// rollback put back wherever they were sent (ADR 0236 §5): no longer theirs to judge (ADR 0353 §3).
func returnedElsewhere(ctx context.Context, open *stores, r *inventory.PlanRelease, failed string, g *inventory.PlanGate) {
for _, c := range g.Carried {
if c.Build == "" {
continue
}
if marked, err := open.inventory.GateFailed(ctx, c.Build); err != nil || !marked {
continue
}
for node, other := range r.Gates {
if node == failed || other.Verdict != "" || slices.Contains(other.Returned, c.Module) {
continue
}
for _, oc := range other.Carried {
if oc.Module == c.Module && sameCommit(oc.To, c.To) {
other.Returned = append(other.Returned, c.Module)
}
}
}
}
}
// openRestStep chooses the next step of a release plan's rest and keeps its sends (novox/hq ADR 0353 §1, §2):
// every machine left that is heard from and has something waiting, except that a build no gate has passed goes
// to one machine of the step only. Answers whether a step was kept, or why the walk waits.
func openRestStep(ctx context.Context, open *stores, p *inventory.Plan) (bool, string, error) {
r := p.Release
heard, err := releaseHeard(ctx, open)
if err != nil {
return false, "", err
}
f, err := readMoveFacts(ctx, open.inventory)
if err != nil {
return false, "", err
}
waiting := map[string][]inventory.CarriedMove{}
var nodes, walked []string
for _, node := range r.Order[min(r.Next, len(r.Order)):] {
if r.Gates[node] != nil || slices.Contains(r.Done, node) || slices.Contains(r.Skipped, node) ||
slices.Contains(r.Failed, node) || slices.Contains(r.Silent, node) {
continue
}
if !heard[node] {
r.Skipped = append(r.Skipped, node)
continue
}
moves, err := gatedMoves(ctx, open, node, nil)
if errors.Is(err, errWalkedElsewhere) {
walked = append(walked, fmt.Sprintf("%s (%v)", node, err))
continue
}
if err != nil {
return false, "", fmt.Errorf("reading what waits on %s: %w", node, err)
}
if len(moves) == 0 {
r.Done = append(r.Done, node)
continue
}
waiting[node] = moves
nodes = append(nodes, node)
}
shelf, err := open.inventory.Catalogue(ctx)
if err != nil {
return false, "", err
}
step := restStep(nodes, waiting, f.gated, func(module string) bool { return oneAtATime(module, shelf[module]) })
if len(step) == 0 {
if len(walked) > 0 {
return false, "waiting before " + strings.Join(walked, "; "), nil
}
return false, "", nil
}
if r.Gates == nil {
r.Gates = map[string]*inventory.PlanGate{}
}
for _, node := range step {
r.Gates[node] = &inventory.PlanGate{Machines: []string{node}, Carried: waiting[node], Pending: true}
}
p.Note = fmt.Sprintf("sending %s at once, each judged there", strings.Join(step, ", "))
if left := len(nodes) - len(step); left > 0 {
p.Note += fmt.Sprintf("; %d machine(s) wait for a build that passed no gate yet to pass on one", left)
}
return true, "", nil
}
// oneAtATime is a module whose builds go to the rest one machine at a time (novox/hq ADR 0353 §9): the
// node-engine, which applies every send, and a module holding the mesh's resolver, which names the bus.
func oneAtATime(module string, m catalogue.Manifest) bool {
if module == hostModule {
return true
}
return slices.ContainsFunc(m.Claims, func(c catalogue.Claim) bool { return c.Name == resolverSeat })
}
// restStep is the machines of one step of the rest (novox/hq ADR 0353 §2), in the order given: each machine, unless
// a build it would carry has passed no gate and an earlier machine of the step already carries it — that build's
// first machine. Pure.
//
// **The node-engine and the mesh's resolver still go one machine at a time** (ADR 0353 §9): a step carries a
// build of either to one machine only, judged before the next, since a fault that cuts a machine off the bus
// shows only as its silence and no rollback reaches it.
func restStep(nodes []string, waiting map[string][]inventory.CarriedMove, passed func(module, commit string) bool,
alone func(module string) bool) []string {
first := map[string]string{}
var out []string
for _, n := range nodes {
var unpassed []string
for _, mv := range waiting[n] {
if !passed(mv.Module, mv.To) || alone(mv.Module) {
unpassed = append(unpassed, mv.Module+"@"+mv.To)
}
}
if slices.ContainsFunc(unpassed, func(k string) bool { _, taken := first[k]; return taken }) {
continue
}
for _, k := range unpassed {
first[k] = n
}
out = append(out, n)
}
return out
}
// silentObservation is what the gate's probe says of the newest release whose rest went silent (novox/hq ADR 0353
// §4): machines that never reported on their send, left as they were — loud, never a success said by nothing.
func silentObservation(ctx context.Context, inv *inventory.Inventory) ([]conditions.Observation, error) {
recent, err := inv.RecentPlans(ctx, 20)
if err != nil {
return nil, err
}
for _, p := range recent {
if p.Release == nil || p.Release.Settings != nil || p.Open() {
continue
}
if len(p.Release.Silent) == 0 {
return nil, nil
}
machines := append([]string(nil), p.Release.Silent...)
sort.Strings(machines)
o := conditions.Observation{Scope: conditions.ScopeMesh, ID: "release", Token: "silent", Kind: "release-silent",
Severity: conditions.Warning, Resolver: conditions.ResolverOperator, Also: machines,
Summary: fmt.Sprintf("%s: %s said nothing on the send of the builds that waited for a gate within %s, and were "+
"left as they were", p.ID, strings.Join(machines, ", "), inventory.ApplySilentAfter)}
w := releaseSilentWords(machines)
o.Headline, o.Explanation, o.Resolved, o.Needs = w.Headline, w.Explanation, w.Resolved, w.Needs
return []conditions.Observation{o}, nil
}
return nil, nil
}
// backlogObservation is what the gate's probe says of a release held after a failure.
func backlogObservation() []conditions.Observation {
if backlogNow.held == "" || len(backlogNow.waiting) == 0 {
-273
View File
@@ -1,273 +0,0 @@
package main
import (
"context"
"errors"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/inventory"
"github.com/novox/mesh-controller/internal/link"
)
// **A pull request into a release branch is judged by that branch** (novox/hq ADR 0358, issue 704): a base that is
// not its repository's trunk, where mesh-controller has a branch of that name, is a release branch; the check then
// reads every repository beside it at its branch of that name where it has one, else at its trunk. Any other base is
// checked exactly as before.
const aRelease = "feat/57-config-and-provisioning"
// theCore is the mesh's core as a check reads it beside: the controller, the node-engine, the tool runner and the
// catalogue's bus, on the git seat.
func theCore() []inventory.Entry {
onSeat := func(module, repository string) inventory.Entry {
e := fromRepo(module, repository, "")
e.Source.Seat = "git"
return e
}
nats := onSeat("nats", "novox/mesh-catalog")
nats.Source.Path = "modules/nats"
return []inventory.Entry{onSeat("mesh-controller", "novox/mesh-controller"), onSeat("mesh-host", "novox/mesh-host"),
onSeat("node-tools", "novox/mesh-tools"), nats, onSeat("gitea", "novox/mesh-catalog")}
}
// aForgeWithBranches stands in for the forge: which branches each repository has, and how often it was asked.
func aForgeWithBranches(t *testing.T, branches map[string][]string, failing error) *int {
t.Helper()
asked := 0
was := forgeHasBranch
t.Cleanup(func() { forgeHasBranch = was })
forgeHasBranch = func(_ context.Context, owner, repo, branch string) (bool, error) {
asked++
if failing != nil {
return false, failing
}
for _, b := range branches[owner+"/"+repo] {
if b == branch {
return true, nil
}
}
return false, nil
}
return &asked
}
func TestABaseIsAReleaseBranchWhenTheControllerHasABranchOfThatName(t *testing.T) {
entries := theCore()
scopeOf := func(p link.PullUpdated) checkScope { return pullScope(p, entries, nil, nil) }
pull := func(repo, base string) link.PullUpdated {
return link.PullUpdated{Owner: "novox", Repo: repo, Number: 286, Base: base, Commit: "abc",
Paths: []string{"modules/gitea/index.ts"}}
}
asked := aForgeWithBranches(t, map[string][]string{"novox/mesh-controller": {"main", aRelease}}, nil)
for _, c := range []struct {
what string
p link.PullUpdated
release string
asks int
}{
{"a pull request into the trunk asks nothing and is no release", pull("mesh-catalog", "main"), "", 0},
{"a catalogue pull request into the release branch", pull("mesh-catalog", aRelease), aRelease, 1},
{"a controller pull request into the release branch", pull("mesh-controller", aRelease), aRelease, 1},
{"a base mesh-controller has no branch of is checked as today", pull("mesh-catalog", "feat/other"), "", 1},
{"no base: as today", pull("mesh-catalog", ""), "", 0},
} {
*asked = 0
got, err := releaseBranchOf(t.Context(), c.p, scopeOf(c.p), entries)
if err != nil {
t.Fatalf("%s: %v", c.what, err)
}
if got != c.release || *asked != c.asks {
t.Errorf("%s: release %q after %d question(s) to the forge, not %q after %d", c.what, got, *asked,
c.release, c.asks)
}
}
// The branch a module built from the repository follows is its trunk: never asked about.
photos := fromRepo("photos", "novox/photos", "")
photos.Source.Ref = "nox-mesh"
withPhotos := append(theCore(), photos)
p := link.PullUpdated{Owner: "novox", Repo: "photos", Base: "nox-mesh", Commit: "abc", Paths: []string{"x.js"}}
*asked = 0
if got, err := releaseBranchOf(t.Context(), p, pullScope(p, withPhotos, nil, nil), withPhotos); err != nil ||
got != "" || *asked != 0 {
t.Fatalf("a module's own trunk read as release %q (%v) after %d question(s)", got, err, *asked)
}
// A forge that cannot answer is an error, never "no release branch".
aForgeWithBranches(t, nil, errors.New("nothing serves gitea.gitea_api"))
q := pull("mesh-catalog", aRelease)
if _, err := releaseBranchOf(t.Context(), q, scopeOf(q), entries); err == nil ||
!strings.Contains(err.Error(), "cannot be told") {
t.Fatalf("an unanswered forge was read as an answer: %v", err)
}
}
func TestBesideAReleaseBranchCheckEachRepositoryIsReadAtThatBranchElseItsTrunk(t *testing.T) {
entries := theCore()
clone := func(s inventory.Source) (string, error) { return "http://git/" + s.Repository + ".git", nil }
running := func(module string) string { return "running-" + module }
beside, err := besideOf(entries, running, clone, aRelease)
if err != nil {
t.Fatal(err)
}
at := func(repo string, ref, orElse string) link.CheckedOut {
return link.CheckedOut{Repository: "http://git/novox/" + repo + ".git", Ref: ref, Else: orElse}
}
want := map[string]link.CheckedOut{
// The judge: mesh-controller's release branch, which it has.
"mesh-controller": at("mesh-controller", aRelease, ""),
"mesh-controller-main": at("mesh-controller", "main", ""),
// A controller pull request into the release branch reads mesh-host at that branch where it has one.
"mesh-host": at("mesh-host", aRelease, "main"),
"mesh-tools": at("mesh-tools", aRelease, "main"),
"mesh-catalog": at("mesh-catalog", aRelease, "main"),
// mesh-lab at its trunk, always: its replays run with the container runtime's socket.
"mesh-lab": at("mesh-lab", "main", ""),
"mesh-sdk": at("mesh-sdk", aRelease, "main"),
}
if !reflect.DeepEqual(beside, want) {
t.Fatalf("beside a release-branch check:\n got %v\nwant %v", beside, want)
}
// Any other base: exactly as before — what the mesh runs, the catalogue's and the lab's main.
beside, err = besideOf(entries, running, clone, "")
if err != nil {
t.Fatal(err)
}
want = map[string]link.CheckedOut{
"mesh-controller": at("mesh-controller", "running-mesh-controller", ""),
"mesh-controller-main": at("mesh-controller", "main", ""),
"mesh-host": at("mesh-host", "running-mesh-host", ""),
"mesh-tools": at("mesh-tools", "running-node-tools", ""),
"mesh-catalog": at("mesh-catalog", "main", ""),
"mesh-lab": at("mesh-lab", "main", ""),
"mesh-sdk": at("mesh-sdk", sdkPinned(), ""),
}
if !reflect.DeepEqual(beside, want) {
t.Fatalf("beside a trunk check:\n got %v\nwant %v", beside, want)
}
}
func TestAReleaseBranchVerdictNamesTheBranchTheJudgeAndWhatItRead(t *testing.T) {
asked := &link.CheckRequest{Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: aRelease, Release: aRelease}
c := checkedOf(link.BuildResult{ID: "b", Ref: "aaaaaaaaaaaa0000", Checked: asked, Check: &link.CheckOutcome{
Verdict: "pass", Head: "aaaaaaaaaaaa", Trunk: "111111111111", Gate: &link.CheckLayer{Verdict: "pass"},
RepoCheck: &link.CheckLayer{Verdict: "pass"}, Release: aRelease, JudgedBy: "fcdecb436117",
Beside: map[string]string{"mesh-host": "222222222222", "mesh-lab": "333333333333"},
BesideAt: map[string]string{"mesh-host": aRelease, "mesh-lab": "main"},
Validator: "mesh-host's validator of " + aRelease + " at 222222222222"}})
if c.Release != aRelease || c.JudgedBy != "fcdecb436117" {
t.Fatalf("the verdict names release %q judged by %q", c.Release, c.JudgedBy)
}
if c.BesideAt["mesh-host"] != aRelease || c.BesideAt["mesh-lab"] != "main" || c.Beside["mesh-host"] != "222222222222" ||
c.Validator != "mesh-host's validator of "+aRelease+" at 222222222222" {
t.Fatalf("the verdict names what it read beside it as %v at %v", c.Beside, c.BesideAt)
}
}
// **Clause 5** (ADR 0358 §5): a release branch's own pull request into its trunk is no release-branch check: the forge
// is asked nothing, the request names no release branch, and the repositories beside it are read as for any trunk
// pull request. Its delivery group's composed check is asked as for any other (askGroupCheck, unchanged).
func TestAReleaseBranchsPullRequestIntoItsTrunkIsCheckedAsAnyTrunkPullRequest(t *testing.T) {
entries := theCore()
asked := aForgeWithBranches(t, map[string][]string{"novox/mesh-controller": {"main", aRelease}}, nil)
for _, repo := range []string{"mesh-catalog", "mesh-controller", "mesh-host"} {
p := link.PullUpdated{Owner: "novox", Repo: repo, Number: 400, Head: aRelease, Base: "main", Commit: "abc",
Paths: []string{"modules/gitea/index.ts"}}
got, err := releaseBranchOf(t.Context(), p, pullScope(p, entries, nil, nil), entries)
if err != nil || got != "" || *asked != 0 {
t.Fatalf("%s's release branch into main read as release %q (%v) after %d question(s)", repo, got, err, *asked)
}
}
clone := func(s inventory.Source) (string, error) { return "http://git/" + s.Repository + ".git", nil }
beside, err := besideOf(entries, func(module string) string { return "running-" + module }, clone, "")
if err != nil {
t.Fatal(err)
}
for dir, b := range beside {
if b.Ref == aRelease || b.Else != "" {
t.Fatalf("beside a trunk pull request %s is read at %+v", dir, b)
}
}
}
// **A forge answer that names no branch is an error**, never "no release branch"; a 404 is "no such branch".
func TestTheForgesAnswerAboutABranchIsReadStrictly(t *testing.T) {
for _, c := range []struct {
what string
answer link.Answer
has bool
fails bool
}{
{"the branch named", link.Answer{Result: []byte(`{"result":{"name":"` + aRelease + `"}}`)}, true, false},
{"not found", link.Answer{Error: "Gitea API /repos/novox/mesh-controller/branches/x: 404 {\"message\":\"not found\"}"},
false, false},
{"an empty branch name", link.Answer{Result: []byte(`{"result":{"name":""}}`)}, false, true},
{"no result", link.Answer{Result: []byte(`{}`)}, false, true},
{"another branch named", link.Answer{Result: []byte(`{"result":{"name":"main"}}`)}, false, true},
{"unreadable", link.Answer{Result: []byte(`not json`)}, false, true},
{"another error", link.Answer{Error: "Gitea API: 500 internal"}, false, true},
} {
has, err := branchAnswered(c.answer, aRelease)
if has != c.has || (err != nil) != c.fails {
t.Errorf("%s: has %v, error %v", c.what, has, err)
}
}
}
// **The trunk is the repository's own, not a name**: a repository whose trunk is not main, one that follows main, and
// a controller whose trunk is not main.
func TestATrunkIsTheBranchTheRepositoryFollows(t *testing.T) {
asked := aForgeWithBranches(t, map[string][]string{"novox/mesh-controller": {"main", aRelease, "master"}}, nil)
legacy := fromRepo("legacy", "novox/legacy", "")
legacy.Source.Ref = "master"
onMain := fromRepo("photos", "novox/photos", "")
onMain.Source.Ref = "main"
entries := append(theCore(), legacy, onMain)
for _, c := range []struct {
what, repo, base, release string
asks int
}{
{"a repository whose trunk is master, into master", "legacy", "master", "", 0},
{"a repository whose trunk is master, into the release branch", "legacy", aRelease, aRelease, 1},
{"a repository whose module follows main, into main", "photos", "main", "", 0},
{"a repository whose module follows main, into the release branch", "photos", aRelease, aRelease, 1},
} {
*asked = 0
p := link.PullUpdated{Owner: "novox", Repo: c.repo, Base: c.base, Commit: "abc", Paths: []string{"x"}}
got, err := releaseBranchOf(t.Context(), p, pullScope(p, entries, nil, nil), entries)
if err != nil || got != c.release || *asked != c.asks {
t.Errorf("%s: release %q (%v) after %d question(s), not %q after %d", c.what, got, err, *asked, c.release, c.asks)
}
}
// A controller whose trunk is not main: main may be a release branch there, and its own trunk never is.
core := theCore()
core[0].Source.Ref = "trunk"
for _, c := range []struct {
base, release string
asks int
}{{"trunk", "", 0}, {"main", "main", 1}} {
*asked = 0
p := link.PullUpdated{Owner: "novox", Repo: "hq", Base: c.base, Commit: "abc", Paths: []string{"x"}}
got, err := releaseBranchOf(t.Context(), p, pullScope(p, core, nil, nil), core)
if err != nil || got != c.release || *asked != c.asks {
t.Errorf("hq into %s, the controller following trunk: release %q (%v) after %d question(s)", c.base, got, err,
*asked)
}
}
}
// trunkOf: the branch a source follows, and main when it records none or records a commit (issue 215's rule).
func TestATrunkIsMainWhenTheSourceRecordsNone(t *testing.T) {
for ref, want := range map[string]string{"": "main", "main": "main", "master": "master", "nox-mesh": "nox-mesh",
"0123456789abcdef0123456789abcdef01234567": "main", " ": "main"} {
if got := trunkOf(inventory.Source{Ref: ref}); got != want {
t.Errorf("a source recording %q follows %q, not %q", ref, got, want)
}
}
}
+18 -251
View File
@@ -9,7 +9,6 @@ import (
"slices"
"sort"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go"
@@ -252,20 +251,6 @@ func gates(p inventory.Plan, edges []inventory.Edge, rollsOut func(string) bool)
return out
}
// noMoveWhy is what a plan says of a module of its tier whose build was made from the source every machine
// running it runs (novox/hq issue 280): nothing is sent for it.
// noMovePrefix opens it, and is what noMove reads: one prefix for the writer and the reader.
const (
noMovePrefix = "no move"
noMoveWhy = noMovePrefix + ": made from the source every machine running it runs"
)
// noMove is whether the plan marked the module no move and sent it nothing. Read from what the plan says,
// so a plan kept before the mark had a name of its own is read the same (novox/hq issue 648).
func noMove(s *inventory.PlanModule) bool {
return s != nil && s.FirstAt == nil && s.Gate == nil && strings.HasPrefix(s.Why, noMovePrefix)
}
// applied says whether every machine running the module has reported since the module was built.
// appliedEach is applied with a moment of its own for each machine: the reports that count are the ones
// after that machine was sent the build (novox/hq issue 256).
@@ -387,73 +372,15 @@ var checkedBuildsOf = func(ctx context.Context, pulls []link.PullRef) ([]link.Ch
return link.ReadCheckBuilds(ctx, js, buildSeatHeld(ctx), pulls, time.Now().Add(-builder.CheckBuildsKept))
}
// mergedHeadsOf are the head commits the walk's merges merged, by pull request (novox/hq issue 703): a pull request
// merged twice in one walk, or whose head the announcement did not say, has none, and takes no check's build.
func mergedHeadsOf(p *inventory.Plan) map[link.PullRef]string {
heads := map[link.PullRef]string{}
if p.Delivery == nil {
return heads
}
seen := map[link.PullRef]bool{}
for _, m := range p.Delivery.Merges {
pull, ok := pullOf(m)
if !ok {
continue
}
if seen[pull] && heads[pull] != m.Head {
heads[pull] = ""
continue
}
seen[pull] = true
heads[pull] = m.Head
}
return heads
}
// walkVerdicts are the checks' builds a walk read, by the walk's plan: read once per walk (novox/hq issue 649), not
// once per module it asks for. Kept a day; a walk lasts minutes.
var walkVerdicts = struct {
sync.Mutex
read map[string]verdictsRead
}{read: map[string]verdictsRead{}}
type verdictsRead struct {
at time.Time
builds []link.CheckedBuild
}
// reuseWithVerdicts is a walk's leave with the builds its pull requests' verdicts name: the only builds the builder
// may take (ADR 0350 §2), and **only those of a check of the head each merge merged** (novox/hq issue 703): a build
// a check made at an earlier head of the same pull request is never registered. Read once per walk. When they
// cannot be read the leave names none, and the walk builds, saying why.
func reuseWithVerdicts(ctx context.Context, walk string, r *link.ReuseAsk, heads map[link.PullRef]string) *link.ReuseAsk {
walkVerdicts.Lock()
read, known := walkVerdicts.read[walk]
walkVerdicts.Unlock()
if !known || walk == "" {
builds, err := checkedBuildsOf(ctx, r.Pulls)
if err != nil {
fmt.Printf(" the checks' builds of %d pull request(s) could not be read, so the walk builds: %v\n", len(r.Pulls), err)
return r
}
read = verdictsRead{at: time.Now(), builds: builds}
if walk != "" {
walkVerdicts.Lock()
for id, w := range walkVerdicts.read {
if time.Since(w.at) > 24*time.Hour {
delete(walkVerdicts.read, id)
}
}
walkVerdicts.read[walk] = read
walkVerdicts.Unlock()
}
}
r.Builds = nil
for _, b := range read.builds {
if link.SameHead(b.Head, heads[b.Pull]) {
r.Builds = append(r.Builds, b)
}
// may take (ADR 0350 §2). When they cannot be read the leave names none, and the walk builds, saying why.
func reuseWithVerdicts(ctx context.Context, r *link.ReuseAsk) *link.ReuseAsk {
builds, err := checkedBuildsOf(ctx, r.Pulls)
if err != nil {
fmt.Printf(" the checks' builds of %d pull request(s) could not be read, so the walk builds: %v\n", len(r.Pulls), err)
return r
}
r.Builds = builds
return r
}
@@ -496,8 +423,7 @@ func askModule(ctx context.Context, p *inventory.Plan, name string, byName map[s
}
// **The walk takes the check's build of the same key** (novox/hq ADR 0350 §2): its ask says so, with the pull
// requests it answers, whose checks name the part of a differing key.
id, err := askABuild(withReuse(ctx, reuseWithVerdicts(ctx, p.ID, reuseAskOf(p), mergedHeadsOf(p))), source,
e.Source.Path, ref)
id, err := askABuild(withReuse(ctx, reuseWithVerdicts(ctx, reuseAskOf(p))), source, e.Source.Path, ref)
if err != nil {
state.State = "failed"
state.Why = err.Error()
@@ -669,7 +595,7 @@ func advancePlans(ctx context.Context, open *stores) {
return
}
defer release()
// What waits for a gate, released one machine first, then the rest at once (ADR 0236, ADR 0353).
// What waits for a gate, released one machine at a time (ADR 0236).
if _, err := releaseBacklog(ctx, open, ""); err != nil {
fmt.Printf("plans: what waits for a gate could not be looked at: %v\n", err)
}
@@ -879,7 +805,7 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
} else if same {
now := time.Now().UTC()
state.SentAt = &now
state.Why = noMoveWhy
state.Why = "no move: made from the source every machine running it runs"
fmt.Printf("%s: %s built from the source every machine running it runs: no move, nothing sent\n", p.ID, m)
continue
}
@@ -1069,25 +995,6 @@ func advanceOnce(ctx context.Context, open *stores, p *inventory.Plan,
if state.BuiltAt != nil {
since = *state.BuiltAt
}
// **A build that is no move is waited on by the send that last carried it** (novox/hq issue 648).
// Nothing was sent for it, so no report follows its marking or its build: a machine that does not
// happen to report again held the walk for ever. Each machine runs it once it answered its last
// send — the one whose builds made it no move — whenever that was.
if noMove(state) {
byNode := map[string]inventory.Reported{}
for _, r := range reports {
byNode[r.Node] = r
}
if ok, on := appliedEach(m, func(node string) time.Time {
if s := byNode[node].Sent; s != nil {
return *s
}
return time.Time{}
}, running, reports); !ok {
waiting = append(waiting, fmt.Sprintf("%s on %s", m, strings.Join(on, ", ")))
}
continue
}
// **Each machine from its own send** (novox/hq issue 256). With one machine first (ADR 0218)
// a module is sent twice — the first machine, then the rest — and SentAt is the second.
// Asked of every machine, the first machine's report, made between the two sends, read as
@@ -1149,36 +1056,10 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
runningOf map[string][]string) error {
inv := open.inventory
before, beforeKnown, _ := inv.SentBuilds(ctx, node)
// **A machine already sent what this send is for is not sent again** (novox/hq ADR 0353 §6): sent by this plan
// before a controller was replaced and before it was kept, or by another walk. Judged from that send, and from
// the readings of a gate already judging it there.
// Only a send the machine reported it applied is one it has; one it never reported on is sent, as before
// (issue 280: what it ran before is then that build, and a failure leaves it as it was).
if beforeKnown && slices.IndexFunc(modules, func(m string) bool { return !sameCommit(before[m], p.Modules[m].Commit) }) < 0 &&
appliedLastSend(ctx, inv, node) {
if others, err := gatedMoves(ctx, open, node, nil); err == nil && len(others) == 0 {
adoptSend(ctx, open, p, node, modules, before, runningOf)
return nil
}
}
// **What the machine ran before is kept before the send is made** (novox/hq ADR 0353 §6): a controller replaced
// between the send and keeping it reads it here, and a failure still puts the machine back.
kept := false
for _, m := range modules {
s := p.Modules[m]
if beforeKnown && (s.Previous == "" || !sameCommit(before[m], s.Commit)) && s.Previous != before[m] {
s.Previous, kept = before[m], true
}
}
if kept {
if err := inv.SavePlan(ctx, p); err != nil {
return fmt.Errorf("what %s ran before is not kept, so nothing is sent: %w", node, err)
}
}
owns := make([]inventory.CarriedMove, 0, len(modules))
for _, m := range modules {
s := p.Modules[m]
owns = append(owns, inventory.CarriedMove{Module: m, Node: node, From: s.Previous, To: s.Commit, Build: s.Build})
owns = append(owns, inventory.CarriedMove{Module: m, Node: node, From: before[m], To: s.Commit, Build: s.Build})
}
// A gated send (ADR 0236): everything waiting on the machine goes with the tier, and the gate judges
// all of it there.
@@ -1193,7 +1074,10 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
sentWhat := sentNow(ctx, inv, sent)
for _, m := range modules {
s := p.Modules[m]
// What the first machine ran before (ADR 0236), kept above before the send.
// What the first machine ran before: what a failed gate puts back (ADR 0236).
if beforeKnown {
s.Previous = before[m]
}
s.First, s.FirstAt = sent, &now
s.Gate = &inventory.PlanGate{Component: coreComponent(m), Machines: firstRunning(sent, runningOf[m]),
From: s.Previous, To: s.Commit, Since: &now, Sent: sentWhat}
@@ -1222,113 +1106,6 @@ func firstSend(ctx context.Context, open *stores, p *inventory.Plan, node string
return nil
}
// adoptSend records a first send the machine already had (novox/hq ADR 0353 §6, §7): nothing is sent; the gate counts
// from the machine's last send, and takes over the readings of an open gate of another walk judging the same build
// there — whose carried move also says what the machine ran before, for a rollback.
func adoptSend(ctx context.Context, open *stores, p *inventory.Plan, node string, modules []string,
before map[string]string, runningOf map[string][]string) {
inv := open.inventory
since := time.Now().UTC()
last, found, err := inv.SentTo(ctx, node)
if err == nil && found && !last.At.IsZero() {
since = last.At
}
lead := modules[0]
other, id, from := judgingThere(ctx, inv, p.ID, lead, node, p.Modules[lead].Commit, p.Modules[lead].Build)
for _, m := range modules {
s := p.Modules[m]
// What it ran before: kept by this walk before its send was made (§6); else what the other walk's send
// moved it from; else the build itself, so a failure leaves it as it was (issue 280).
switch {
case s.Previous != "":
case m == lead && from != "":
s.Previous = from
default:
s.Previous = before[m]
}
at := since
s.First, s.FirstAt = []string{node}, &at
s.Gate = &inventory.PlanGate{Component: coreComponent(m), Machines: firstRunning([]string{node}, runningOf[m]),
From: s.Previous, To: s.Commit, Since: &at}
if found {
s.Gate.Sent = map[string]inventory.SentDeclaration{node: last}
}
s.GatedBy = ""
if m == lead {
s.Gate.TakeReadings(other, modules, id)
} else {
s.GatedBy = lead
}
}
p.State = inventory.PlanRolling
p.Note = fmt.Sprintf("tier %d built; %s already runs %s's build: nothing sent again, judged from that send",
p.Tier, node, strings.Join(modules, ", "))
if id != "" {
p.Note += ", going on from the readings of " + id
}
fmt.Printf("%s: %s\n", p.ID, p.Note)
}
// appliedLastSend says a machine reported it applied the declaration it was sent last.
func appliedLastSend(ctx context.Context, inv *inventory.Inventory, node string) bool {
reports, err := inv.LastReports(ctx)
if err != nil {
return false
}
for _, r := range reports {
if r.Node == node {
return r.Current && r.At != nil && r.Outcome == inventory.OutcomeApplied
}
}
return false
}
// judgingThere is an open gate of another walk judging a module's build on a machine (novox/hq ADR 0353 §7), the
// walk's id, and what the machine ran before; nil when none does.
func judgingThere(ctx context.Context, inv *inventory.Inventory, self, module, node, commit, build string) (*inventory.PlanGate, string, string) {
plans, err := inv.OpenPlans(ctx)
if err != nil {
return nil, "", ""
}
// The same build, by its id where both name one, else by its commit.
same := func(b, c string) bool {
if b != "" && build != "" {
return b == build
}
return sameCommit(c, commit)
}
usable := func(g *inventory.PlanGate) bool {
return g != nil && !g.Pending && g.Since != nil && (g.Verdict == "" || g.Verdict == inventory.GatePassed) &&
slices.Contains(g.Machines, node)
}
for _, q := range plans {
if q.ID == self {
continue
}
if r := q.Release; r != nil {
gates := []*inventory.PlanGate{r.Gate}
for _, g := range r.Gates {
gates = append(gates, g)
}
for _, g := range gates {
if !usable(g) {
continue
}
for _, c := range g.Carried {
if c.Module == module && c.Node == node && same(c.Build, c.To) {
return g, q.ID, c.From
}
}
}
continue
}
if s := q.Modules[module]; s != nil && usable(s.Gate) && same(s.Build, s.Commit) {
return s.Gate, q.ID, s.Previous
}
}
return nil, "", ""
}
// passedWith keeps, on a module sent with others, the verdict of the gate that judged the send: the gate
// on the first of them passed, and its pass was kept for every build it carried (passCarried).
func passedWith(module string, s *inventory.PlanModule, lead string, g *inventory.PlanGate) {
@@ -1817,19 +1594,9 @@ func plansCommand(ctx context.Context, args []string) error {
// A release plan's walk (ADR 0236): machines done, the one judged, those to come.
fmt.Printf(" machines in order: %s; done: %s; skipped: %s\n", strings.Join(r.Order, ", "),
orNone(strings.Join(r.Done, ", ")), orNone(strings.Join(r.Skipped, ", ")))
if len(r.Failed)+len(r.Silent) > 0 {
fmt.Printf(" failed: %s; silent: %s\n", orNone(strings.Join(r.Failed, ", ")), orNone(strings.Join(r.Silent, ", ")))
}
gates := []*inventory.PlanGate{r.Gate}
for _, n := range r.Order {
gates = append(gates, r.Gates[n])
}
for _, g := range gates {
if g == nil {
continue
}
fmt.Printf(" %s\n", gateLine(g))
for _, c := range g.Carried {
if r.Gate != nil {
fmt.Printf(" %s\n", gateLine(r.Gate))
for _, c := range r.Gate.Carried {
fmt.Printf(" %-22s %s → %s\n", c.Module, short(c.From), short(c.To))
if c.Recreates != "" {
fmt.Printf(" %-22s %s\n", "", c.Recreates)
-372
View File
@@ -1,372 +0,0 @@
package main
import (
"context"
"slices"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// novox/hq ADR 0353: after its first machine passes, a build goes to every other machine at once, each judged by
// its own gate and a failing one put back by itself; and a gate's readings outlive the controller that took them —
// a machine already sent what a send is for is not sent again, and its judging goes on from what was read.
//
// Seen 2026-10-14: release-1791930194184991192 sent ace, g14, shanks and the control node one after another in
// eleven minutes; the new controller its last send started sent the control node again the controller build it
// had just applied, and judged it from its first reading.
// aBacklogOnThree is the backlog fixture with a third machine, desk, running app at c1 like the others.
func aBacklogOnThree(t *testing.T) *backlogMesh {
t.Helper()
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
if _, err := inv.AddNode(ctx, "desk"); err != nil {
t.Fatal(err)
}
if _, err := inv.Assign(ctx, "desk", "app"); err != nil {
t.Fatal(err)
}
if err := inv.RecordSent(ctx, nodeID(t, b.open, "desk"), "d-desk", map[string]string{"app": "c1"}); err != nil {
t.Fatal(err)
}
wasHeard := releaseHeard
releaseHeard = func(context.Context, *stores) (map[string]bool, error) {
return map[string]bool{"anchor": true, "laptop": true, "desk": true}, nil
}
t.Cleanup(func() { releaseHeard = wasHeard })
return b
}
// events records every send and every judging, in order.
func events(t *testing.T) *[]string {
t.Helper()
var said []string
wasSend, wasGather := sendRollout, gatherGateFacts
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
said = append(said, "send "+strings.Join(names, ","))
return wasSend(ctx, open, names)
}
gatherGateFacts = func(ctx context.Context, open *stores, component string) (gateFacts, error) {
said = append(said, "judge")
return wasGather(ctx, open, component)
}
t.Cleanup(func() { sendRollout, gatherGateFacts = wasSend, wasGather })
return &said
}
// Once the first machine passed, the rest are both sent before either is judged (ADR 0353 §1); before, the second
// was judged to its verdict before the third was sent.
func TestAReleaseSendsTheRestAtOnceAfterTheFirstPasses(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
said := events(t)
gateEvery = 0
for i := 0; i < 12; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State != inventory.PlanDone || len(p.Release.Done) != 3 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
desk, laptop := slices.Index(*said, "send desk"), slices.Index(*said, "send laptop")
if desk < 0 || laptop < 0 || slices.Index(*said, "send anchor") > min(desk, laptop) {
t.Fatalf("sends and judgings: %v", *said)
}
if slices.Contains((*said)[min(desk, laptop):max(desk, laptop)], "judge") {
t.Fatalf("the rest were sent one at a time, each judged before the next: %v", *said)
}
if v, found, err := b.open.inventory.GateOf(ctx, "build-late-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("late's pass on laptop was not kept: %+v %v %v", v, found, err)
}
}
// A machine of the rest whose gate fails is put back at once, and the other machine of the step goes on to its own
// verdict (ADR 0353 §3); before, the walk stopped at the failure and the machine after it was never sent.
func TestARestMachineThatFailsIsPutBackAloneAndTheOthersAreJudgedOn(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
inv := b.open.inventory
gateEvery = 0
b.broken["desk"] = true // desk fails what it is sent in the rest's step
advancePlans(ctx, b.open)
for i := 0; i < 12 && b.release(t).Open(); i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
if p.State != inventory.PlanFailed || !slices.Equal(p.Release.Failed, []string{"desk"}) {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
if !slices.ContainsFunc(b.sent, func(s []string) bool { return slices.Equal(s, []string{"laptop"}) }) {
t.Fatalf("laptop was never sent: %v", b.sent)
}
if !slices.Contains(p.Release.Done, "laptop") {
t.Fatalf("laptop has no verdict of its own: %+v", p.Release)
}
// late waited on laptop alone and was healthy there: its pass is kept, whatever desk did.
if v, found, err := inv.GateOf(ctx, "build-late-c2"); err != nil || !found || v.Verdict != inventory.GatePassed {
t.Fatalf("late's pass on laptop was not kept: %+v %v %v", v, found, err)
}
// app failed on desk: marked, and put back wherever it was sent (ADR 0236).
if failed, _ := inv.GateFailed(ctx, "build-app-c2"); !failed {
t.Fatal("app's build is not marked failed")
}
if sent, _, _ := inv.SentBuilds(ctx, "desk"); sent["app"] != "c1" {
t.Fatalf("desk was not put back: %v", sent)
}
}
// A walk whose first machine already runs its build under another walk's gate sends nothing and goes on from that
// gate's readings (ADR 0353 §7); before, it sent the machine again and judged from its first reading.
func TestAWalkTakesOverTheReadingsOfAGateJudgingTheSameBuildThere(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
advancePlans(ctx, b.open) // the release sends anchor app c2, and reads it once
release := b.release(t)
judging := release.Release.Gate
if judging == nil || judging.Pending || judging.Passes == 0 {
t.Fatalf("the release is not judging anchor: %+v", judging)
}
built := time.Now().UTC()
plan := inventory.Plan{ID: "plan-app", Repository: "novox/mesh-catalog", Branch: "main", Commit: "c2",
Created: built, State: inventory.PlanBuilding, Tiers: [][]string{{"app"}},
Modules: map[string]*inventory.PlanModule{"app": {State: "built", BuiltAt: &built, Commit: "c2",
Build: "build-app-c2"}}}
if err := inv.SavePlan(ctx, &plan); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
if len(b.sent) != 1 {
t.Fatalf("anchor was sent again what it runs: %v", b.sent)
}
p, err := inv.PlanByID(ctx, "plan-app")
if err != nil {
t.Fatal(err)
}
g := p.Modules["app"].Gate
if g == nil || g.Since == nil || !g.Since.Equal(*judging.Since) || g.Passes < judging.Passes || g.TookOver != release.ID {
t.Fatalf("the walk's gate starts from zero: %+v (the release's: %+v)", g, judging)
}
if p.Modules["app"].Previous != "c1" {
t.Fatalf("what anchor ran before is %q, not c1", p.Modules["app"].Previous)
}
}
// A walk's first send that was made and not kept — the controller replaced between the two — is not made again
// by the controller that takes over: the gate counts from that send (ADR 0353 §5, §6).
func TestAFirstSendMadeAndNotKeptIsNotSentAgain(t *testing.T) {
judgeLeavesAsTheCore(t)
g := aGateMesh(t)
ctx := t.Context()
// The old controller sends anchor app c2 and is replaced before it keeps the send.
made := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
if _, err := made(ctx, open, names); err != nil {
return nil, err
}
return nil, context.Canceled
}
advancePlans(ctx, g.open)
sendRollout = made
if len(g.sent) != 1 || g.plan(t).Modules["app"].FirstAt != nil {
t.Fatalf("the fixture's send was not made, or was kept: %v", g.sent)
}
last, _, err := g.open.inventory.SentTo(ctx, "anchor")
if err != nil {
t.Fatal(err)
}
advancePlans(ctx, g.open)
if len(g.sent) != 1 {
t.Fatalf("the send was made again: %v", g.sent)
}
s := g.plan(t).Modules["app"]
if s.FirstAt == nil || s.Gate == nil || s.Gate.Since == nil || !s.Gate.Since.Equal(last.At) {
t.Fatalf("the gate does not count from the send made: %+v %+v", s, s.Gate)
}
// What anchor ran before was kept before the send, so a failure still puts it back (§5).
if s.Previous != "c1" || s.Gate.From != "c1" {
t.Fatalf("what anchor ran before is %q (gate from %q), not c1", s.Previous, s.Gate.From)
}
}
// A release's send kept before it was made, and made, is judged by the controller that takes over without being
// made again (ADR 0353 §8).
func TestAReleaseSendKeptAndMadeIsNotMadeAgain(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
p, err := releaseBacklog(ctx, b.open, "")
if err != nil || p == nil {
t.Fatalf("no release opened: %v", err)
}
if _, err := advanceRelease(ctx, b.open, p); err != nil || p.Release.Gate == nil || !p.Release.Gate.Pending {
t.Fatalf("the send was not kept before it was made: %+v %v", p.Release, err)
}
if err := inv.SavePlan(ctx, p); err != nil {
t.Fatal(err)
}
// The send is made, and the controller is replaced before it keeps it.
if _, err := sendRollout(ctx, b.open, []string{"anchor"}); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
if len(b.sent) != 1 {
t.Fatalf("anchor was sent again: %v", b.sent)
}
g := b.release(t).Release.Gate
if g == nil || g.Pending || g.Since == nil || len(g.Carried) == 0 {
t.Fatalf("anchor is not judged on the send made: %+v", g)
}
}
// A build no gate has passed goes to one machine of a step, its first; a machine that would be its second waits
// for the next step, and one whose every build passed goes now (ADR 0353 §2).
func TestABuildThatPassedNoGateGoesToOneMachineOfAStep(t *testing.T) {
waiting := map[string][]inventory.CarriedMove{
"desk": {{Module: "app", To: "c2"}, {Module: "new", To: "n1"}},
"laptop": {{Module: "app", To: "c2"}, {Module: "new", To: "n1"}},
"shanks": {{Module: "app", To: "c2"}},
"novox": {{Module: "app", To: "c2"}, {Module: "ctl", To: "k1"}},
}
passed := func(module, commit string) bool { return module == "app" && commit == "c2" }
never := func(string) bool { return false }
step := restStep([]string{"desk", "laptop", "shanks", "novox"}, waiting, passed, never)
if !slices.Equal(step, []string{"desk", "shanks", "novox"}) {
t.Fatalf("the step is %v: laptop would be new's second machine before it passed", step)
}
if step := restStep([]string{"laptop"}, waiting, func(string, string) bool { return true }, never); !slices.Equal(step, []string{"laptop"}) {
t.Fatalf("a machine whose every build passed waits: %v", step)
}
}
// The node-engine and the mesh's resolver go to the rest one machine at a time, even passed (ADR 0353 §9): a step
// carries either to one machine only, while what else passed goes everywhere at once.
func TestTheNodeEngineAndTheResolverGoToTheRestOneAtATime(t *testing.T) {
waiting := map[string][]inventory.CarriedMove{
"desk": {{Module: hostModule, To: "h2"}},
"laptop": {{Module: hostModule, To: "h2"}, {Module: "app", To: "c2"}},
"shanks": {{Module: "app", To: "c2"}},
"novox": {{Module: "dns", To: "d2"}},
"ace": {{Module: "dns", To: "d2"}},
}
resolver := catalogue.Manifest{Module: "dns", Claims: []catalogue.Claim{{Name: resolverSeat}}}
alone := func(m string) bool {
if m == "dns" {
return oneAtATime(m, resolver)
}
return oneAtATime(m, catalogue.Manifest{Module: m})
}
step := restStep([]string{"desk", "laptop", "shanks", "novox", "ace"}, waiting,
func(string, string) bool { return true }, alone)
if !slices.Equal(step, []string{"desk", "shanks", "novox"}) {
t.Fatalf("the step is %v: the node-engine and the resolver each reach one machine of it", step)
}
if oneAtATime("app", catalogue.Manifest{Module: "app"}) {
t.Fatal("a leaf module is held to one machine at a time")
}
}
// A first machine that fails still stops the walk (ADR 0236 §5, kept by ADR 0353 §3): the rest are never sent.
func TestAFirstMachineThatFailsStillStopsTheWalk(t *testing.T) {
b := aBacklogOnThree(t)
ctx := t.Context()
gateEvery = 0
b.broken["anchor"] = true
advancePlans(ctx, b.open)
for i := 0; i < 6 && b.release(t).Open(); i++ {
advancePlans(ctx, b.open)
}
if p := b.release(t); p.State != inventory.PlanFailed || p.Release.Rest {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
for _, s := range b.sent {
if !slices.Equal(s, []string{"anchor"}) {
t.Fatalf("sent %v: the rest after a failed first machine", b.sent)
}
}
}
// The control node is sent last in a step of the rest (ADR 0353 §1): the machine holding the controller is
// ordered after every other.
func TestTheControlNodeIsSentLastInAStep(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklogOnThree(t)
ctx := t.Context()
register(t, b.open, catalogue.Manifest{Module: catalogue.ControllerSeatName, Version: "1"})
if _, err := b.open.inventory.Assign(ctx, "anchor", catalogue.ControllerSeatName); err != nil {
t.Fatal(err)
}
said := events(t)
gateEvery = 0
for i := 0; i < 12 && (len(b.sent) == 0 || b.release(t).Open()); i++ {
advancePlans(ctx, b.open)
}
if p := b.release(t); !slices.Equal(p.Release.Order, []string{"desk", "laptop", "anchor"}) {
t.Fatalf("the order is %v", p.Release.Order)
}
laptop, anchor := slices.Index(*said, "send laptop"), slices.Index(*said, "send anchor")
if laptop < 0 || anchor < laptop || slices.Contains((*said)[laptop:anchor], "judge") {
t.Fatalf("the control node is not sent last in the rest's step: %v", *said)
}
}
// A machine of the rest that says nothing on its send is left silent after fifteen minutes: not failed, not put
// back; a rest that went silent whole ends failed, said as a condition (ADR 0353 §4).
func TestARestMachineSilentPastItsBoundIsLeftAndSaid(t *testing.T) {
judgeLeavesAsTheCore(t)
b := aBacklog(t)
ctx := t.Context()
inv := b.open.inventory
reporting := sendRollout
sendRollout = func(ctx context.Context, open *stores, names []string) ([]string, error) {
if slices.Equal(names, []string{"laptop"}) { // laptop sleeps: sent, and says nothing
b.sent = append(b.sent, names)
current, _ := open.inventory.CurrentBuilds(ctx)
return names, open.inventory.RecordSent(ctx, nodeID(t, open, "laptop"), "d-laptop-asleep",
map[string]string{"app": current["app"].Commit, "late": current["late"].Commit, "same": current["same"].Commit})
}
return reporting(ctx, open, names)
}
t.Cleanup(func() { sendRollout = reporting })
gateEvery = 0
for i := 0; i < 8; i++ {
advancePlans(ctx, b.open)
}
p := b.release(t)
g := p.Release.Gates["laptop"]
if !p.Open() || g == nil || g.Verdict != "" || g.Silent {
t.Fatalf("laptop is judged before fifteen minutes: %s %s %+v", p.State, p.Note, g)
}
since := time.Now().Add(-inventory.ApplySilentAfter - time.Minute)
g.Since = &since
if err := inv.SavePlan(ctx, &p); err != nil {
t.Fatal(err)
}
advancePlans(ctx, b.open)
p = b.release(t)
if p.State != inventory.PlanFailed || !slices.Equal(p.Release.Silent, []string{"laptop"}) || len(p.Release.Failed) > 0 {
t.Fatalf("the release is %s: %s %+v", p.State, p.Note, p.Release)
}
if failed, _ := inv.GateFailed(ctx, "build-late-c2"); failed {
t.Fatal("a silent machine's build was marked failed")
}
if sent, _, _ := inv.SentBuilds(ctx, "laptop"); sent["late"] != "c2" {
t.Fatalf("laptop was put back: %v", sent)
}
obs, err := silentObservation(ctx, inv)
if err != nil {
t.Fatal(err)
}
if len(obs) != 1 || obs[0].Kind != "release-silent" || !slices.Equal(obs[0].Also, []string{"laptop"}) {
t.Fatalf("the silence is not said: %+v", obs)
}
}
-28
View File
@@ -457,28 +457,6 @@ func (a *verbArguments) commandLine() ([]string, error) {
return nil, err
}
return []string{"forget-checks", str("owner"), str("repo"), str("number")}, nil
case "try":
// A branch tried before its pull request (novox/hq issue 686): asked, or read by the id it answered.
if id := str("id"); id != "" {
return []string{"try", "--id", id}, nil
}
if a.given["repository"] == "" && a.given["ref"] == "" {
// Neither shape named: the command says its usage, which names both, as `build` does.
return []string{"try"}, nil
}
if err := need("repository", "ref"); err != nil {
return nil, err
}
argv := []string{"try", str("repository"), str("ref")}
for _, flag := range []string{"base", "tests"} {
if v := str(flag); v != "" {
argv = append(argv, "--"+flag, v)
}
}
if on("store-only") {
argv = append(argv, "--store-only")
}
return argv, nil
case "cancel", "kill":
if err := need("id"); err != nil {
return nil, err
@@ -1040,8 +1018,6 @@ var jsonVerbs = map[string]bool{"status": true, "seats": true, "plan": true, "co
"delivery": true,
// The build queue, each ask with its position among the waiting (novox/hq issue 455).
"queue": true,
// A branch tried before its pull request (novox/hq issue 686).
"try": true,
// A plan followed (novox/hq issue 229): `plans wait` answers JSON; its other forms say text, and so answer no data.
"plans": true}
@@ -1332,10 +1308,6 @@ func seatToolHandlers() (map[string]link.ToolHandler, []string, error) {
link.AnswersWithin(ctx, bound+planWaitStart)
}
}
if verb == "try" && len(argv) > 1 && argv[1] == "--id" {
// Reading a try holds its answer for the verdict, up to its bound (novox/hq issue 686, 229).
link.AnswersWithin(ctx, tryWait+planWaitStart)
}
if answer, read := readHere(ctx, argv); read {
return answer, nil
}
+2 -58
View File
@@ -153,15 +153,11 @@ var signalsTable = []signalRow{
return newestOf(f.standings, func(c conditions.Condition) time.Time { return c.LastObserved })
}},
{Row: "S9", Signal: "bus advisories: maximum deliveries, consumer deleted; the controller's own slow " +
"consumer and refused subjects; every user's connection the server closed as a slow consumer or for " +
"breaking the bus's rules (ADR 0357)",
Emitter: "bus server's advisory subjects; the controller's connection; the bus module's watch of its " +
"server (bus.slow-consumer, bus.connection-closed)",
"consumer and refused subjects", Emitter: "bus server's advisory subjects; the controller's connection",
Trigger: "any", Bound: "any occurrence; clears after an hour without another, a deleted consumer " +
"once it exists again or the mesh no longer expects it, and a message given up on once DEAD_LETTERS " +
"no longer holds it (novox/hq issue 330)",
Kind: "slow-consumer, connection-closed, max-deliveries, refused, consumer-lost", Severity: conditions.Warning,
Phase: 1,
Kind: "slow-consumer, max-deliveries, refused, consumer-lost", Severity: conditions.Warning, Phase: 1,
needs: func(f *signalFacts) error { return f.advisoriesErr }, watch: watchAdvisories,
newest: func(f *signalFacts) time.Time {
return newestOf(f.advisories, func(a link.Advisory) time.Time { return a.Last })
@@ -573,9 +569,6 @@ func watchAdvisories(f *signalFacts) []conditions.Observation {
// The advisory is remembered and read again on every look for an hour: the condition counts
// what the bus said and when, not the looks (novox/hq issue 402).
Happened: a.Last, Times: a.Count}
if a.User != "" && a.Reason != "" {
busUserClosed(&o, a, f.host)
}
if a.Kind == link.AdvisoryMaxDeliveries && a.Token == link.AdvisoryNotKept {
o.Machine = consumerMachine(a.Stream, a.Consumer)
o.Headline = clip(conditions.Capital(fmt.Sprintf("%s gave up on a message, not kept",
@@ -586,58 +579,9 @@ func watchAdvisories(f *signalFacts) []conditions.Observation {
}
out = append(out, o)
}
if f.busFaultsErr != nil {
out = append(out, busFaultsUnread(f.busFaultsErr))
}
return append(out, watchDeadLetters(f)...)
}
// kindBusFaultsUnread is the bus module's events not read back (novox/hq ADR 0357): every other user's
// slow consumer is unheard while it lasts, and the controller's own signals are still said.
const kindBusFaultsUnread = "bus-faults-unread"
func busFaultsUnread(err error) conditions.Observation {
return conditions.Observation{Scope: conditions.ScopeBus, ID: "bus-module", Kind: kindBusFaultsUnread,
Severity: conditions.Warning, Confirm: true,
Summary: "what the bus module said of its users' connections could not be read back, so a slow consumer " +
"or a connection the bus closed is not said until it can",
Said: firstLine(err.Error())}
}
// busUserClosed words an advisory about a bus user's connection the server closed (novox/hq ADR 0357): the
// user as the operator says it and the machine it is on, the dotted user name kept in the evidence only.
func busUserClosed(o *conditions.Observation, a link.Advisory, host string) {
who, machine := busUserWords(a.User)
if machine == "" && a.User == "controller" {
machine = host
}
o.Machine = machine
times, count := "once", "once"
if a.Count > 1 {
times = fmt.Sprintf("%d times since %s", a.Count, a.First.UTC().Format("15:04 MST"))
count = fmt.Sprintf("%d times", a.Count)
}
switch a.Kind {
case link.AdvisorySlowConsumer:
o.Summary = fmt.Sprintf("the bus closed the connection of %s as a slow consumer %s in the last hour, "+
"newest at %s: it could not take what it was sent fast enough, and what was on its way to it was lost",
who, times, a.Last.UTC().Format("15:04 MST"))
o.Headline = clip(conditions.Capital(who)+" was too slow for the bus", 60)
o.Explanation = conditions.Capital(fmt.Sprintf("%s could not keep up with what the bus sent it, so the bus "+
"closed its connection (%s in the last hour) and what was on its way was lost. It connects again "+
"by itself; this clears an hour after the last time.", who, count))
o.Resolved = "Resolved: " + who + " kept up for an hour"
default:
o.Summary = fmt.Sprintf("the bus closed the connection of %s for breaking its rules %s in the last hour, "+
"newest at %s: %s", who, times, a.Last.UTC().Format("15:04 MST"), strings.ToLower(a.Reason))
o.Headline = clip(conditions.Capital(who)+" was cut off by the bus", 60)
o.Explanation = conditions.Capital(fmt.Sprintf("The bus closed the connection of %s because it broke "+
"the bus's rules (%s, %s in the last hour), so what it was sending was lost. This clears an hour "+
"after the last time.", who, strings.ToLower(a.Reason), count))
o.Resolved = "Resolved: " + who + " kept to the bus's rules for an hour"
}
}
// watchDeadLetters says each consumer that DEAD_LETTERS holds a message for (novox/hq issue 330): open
// while it holds any, so it clears when they are delivered again or dropped, never because the server
// stopped saying it.
-353
View File
@@ -1,353 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"net/url"
"os"
"regexp"
"strings"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// A branch is tried before its pull request (novox/hq issue 686).
//
// A repository's store tests skip without a store, and no agent may start a database on its machine. So a
// builder pushed, a reviewer read, and only the pull request's merge check ran those tests for the first time:
// each failure there cost a round of fifteen to thirty minutes (mesh-controller #283, #310, #312).
//
// `try <owner>/<repository> <ref>` asks the build seat for the repository's own check of a pushed branch or
// commit, as the merge check runs it (internal/builder/check.go): the ref merged into the trunk's head, the
// trunk's merge-check.sh, each part in its toolchain's container, against a throwaway store and bus. It is a
// check and nothing more:
//
// - **no pull request, so no status**: its outcome is never said as `checked`, which is what the forge's
// holder sets statuses from (isTry, in Built);
// - **nothing registered, nothing built**: it names no module of the graph, so the merge gate does not run and
// no build is made for a merge (novox/hq ADR 0350's scratch builds are a pull request's check's alone);
// - **a check in the build queue**: after deliveries (link.KindOf, novox/hq issue 558), and bounded by the
// check's own bound (checkTimeout);
// - **the code a pull request's check would run anyway**: only a repository of the mesh, whose every pull
// request the build seat already checks, in containers that hold no runtime socket.
//
// A check outlasts a seat verb's one answer, so the ask answers its id at once and `try --id <id>` reads what
// came of it from the build's log, where the build seat says it whole (link.StepTried): holding its answer up
// to tryWait for a verdict, like `plans wait` (novox/hq issue 229). Nothing is kept beside the log.
// tryWait is how long reading a try holds its answer for the verdict. A variable so a test need not wait.
var tryWait = 20 * time.Second
// A ref as the forge is asked for it — a branch or a commit, never an option — and a `go test -run` pattern.
var (
tryRef = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._/-]*$`)
tryTests = regexp.MustCompile(`^[A-Za-z0-9_^$|().*+?/\[\]-]+$`)
)
// tryAnswer is what `try` answers, asking or reading.
type tryAnswer struct {
ID string `json:"id"`
// State is asked (just now), waiting (nothing said of it yet: in the build queue), running, or the verdict:
// pass, warning (the repository declares no merge-check.sh), fail, or error (it could not run).
State string `json:"state"`
Said string `json:"said"`
Repository string `json:"repository,omitempty"`
Ref string `json:"ref,omitempty"`
Commit string `json:"commit,omitempty"`
Base string `json:"base,omitempty"`
Trunk string `json:"trunk,omitempty"`
On string `json:"on,omitempty"`
Took string `json:"took,omitempty"`
Failed []string `json:"failed,omitempty"`
// Last is the last thing the build seat said of one still running.
Last string `json:"last,omitempty"`
}
// isTry says a check's outcome is a try's: marked so, or — echoed by a build seat from before the mark — a
// check of no pull request and no group, which is nobody's status either.
func isTry(result link.BuildResult) bool {
c := result.Checked
return c != nil && (c.Try || (c.Number == 0 && c.Group == ""))
}
// asATry makes a check's ask a try's: the repository's own script alone, for no pull request.
func asATry(request link.BuildRequest, tests string, storeOnly bool) link.BuildRequest {
c := *request.Check
c.Number, c.Try, c.Tests, c.StoreOnly = 0, true, tests, storeOnly
c.Paths, c.Modules, c.New, c.Dependents, c.Manifests, c.Builds = nil, nil, nil, nil, nil, nil
c.Judge, c.Plan, c.Group, c.Members = "", nil, "", nil
request.Check = &c
return request
}
// maxTries is how many tries may wait or run at once, mesh-wide: each holds a build machine for minutes.
const maxTries = 4
// triesIn reads the build queue for a new try of one commit: the id of a try of that commit already waiting or
// running — the answer to a second ask — and whether there is room for another.
func triesIn(q link.Queue, owner, repo, commit string) (same string, room bool) {
n := 0
for _, a := range q.Asks {
if !link.IsTry(a.Request) || (a.State != link.AskWaiting && a.State != link.AskInFlight) {
continue
}
n++
c := a.Request.Check
if same == "" && strings.EqualFold(c.Owner, owner) && strings.EqualFold(c.Repo, repo) && a.Request.Ref == commit {
same = a.ID
}
}
return same, n < maxTries
}
// forgeCommit asks the forge, through its module's tools on the bus, which commit a branch or commit of a
// repository is, and the repository's default branch. A variable so a test needs no forge.
var forgeCommit = func(ctx context.Context, owner, repo, ref string) (commit, trunk string, err error) {
js, err := aBus()
if err != nil {
return "", "", err
}
defer js.Close()
bus := link.OverNATS{Conn: js.Conn()}
ask := func(path string, into any) error {
raw, _ := json.Marshal(map[string]any{"path": path})
answer, err := link.Ask(ctx, bus, "gitea", "gitea_api", raw, 30*time.Second)
if err != nil {
return err
}
if answer.Error != "" {
return errors.New(answer.Error)
}
return json.Unmarshal(answer.Result, into)
}
var commits struct {
Result []struct {
SHA string `json:"sha"`
} `json:"result"`
}
if err := ask("/repos/"+owner+"/"+repo+"/commits?limit=1&stat=false&verification=false&files=false&sha="+
url.QueryEscape(ref), &commits); err != nil {
return "", "", err
}
if len(commits.Result) == 0 || commits.Result[0].SHA == "" {
return "", "", errors.New("the forge named no commit for it")
}
var found struct {
Result struct {
DefaultBranch string `json:"default_branch"`
} `json:"result"`
}
if err := ask("/repos/"+owner+"/"+repo, &found); err != nil {
return "", "", err
}
return commits.Result[0].SHA, found.Result.DefaultBranch, nil
}
// tryCommand is `try <owner>/<repository> <ref> [--base <branch>] [--tests <pattern>] [--store-only]`, and
// `try --id <id>` to read what came of one.
func tryCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("try", flag.ContinueOnError)
base := set.String("base", "", "the branch it is merged into (the repository's default branch when none)")
tests := set.String("tests", "", "only the tests this `go test -run` pattern names")
storeOnly := set.Bool("store-only", false, "only the repository's store tests")
id := set.String("id", "", "a try's id: what came of it")
positionals, err := parseAround(set, args)
if err != nil {
return err
}
usage := errors.New("try <owner>/<repository> <branch or commit> [--base <branch>] [--tests <pattern>] " +
"[--store-only], or try --id <id>")
if *id != "" {
if len(positionals) != 0 {
return usage
}
// Before it is a token of a bus subject: an id, never a pattern.
if !link.SafeBuildID(*id) {
return fmt.Errorf("%q is not a try's id", *id)
}
return tryRead(ctx, *id)
}
if len(positionals) != 2 {
return usage
}
owner, repo, isPath := strings.Cut(positionals[0], "/")
ref := positionals[1]
switch {
case !isPath || !tryRef.MatchString(owner) || !tryRef.MatchString(repo) || strings.Contains(repo, "/"):
return fmt.Errorf("%q is not a repository on the forge, owner/name; nothing was asked", positionals[0])
case !tryRef.MatchString(ref):
return fmt.Errorf("%q is not a branch or a commit; nothing was asked", ref)
case *base != "" && !tryRef.MatchString(*base):
return fmt.Errorf("%q is not a branch; nothing was asked", *base)
case *tests != "" && !tryTests.MatchString(*tests):
return fmt.Errorf("%q is not a `go test -run` pattern of test names; nothing was asked", *tests)
}
// **An unknown ref is refused here, loudly** — not sent to a machine to fail at its clone minutes later.
commit, trunk, err := forgeCommit(ctx, owner, repo, ref)
if err != nil {
return fmt.Errorf("the forge holds no branch or commit %s in %s/%s — is it pushed? — so nothing was asked: %w",
ref, owner, repo, err)
}
if *base != "" {
trunk = *base
}
if trunk == "" {
return fmt.Errorf("the forge named no default branch for %s/%s, and no --base says what %s is merged into; "+
"nothing was asked", owner, repo, ref)
}
open, err := openStores(ctx)
if err != nil {
return err
}
defer open.Close()
entries, read, edges, err := theGraph(ctx, open.inventory)
if err != nil {
return err
}
p := link.PullUpdated{Owner: owner, Repo: repo, Base: trunk, Commit: commit}
// Whose repository it is, as a pull request's check reckons it: only the mesh's is tried, the code the
// build seat runs for every pull request already.
mesh := pullScope(p, entries, read, edges)
if !mesh.Mesh {
return fmt.Errorf("%s/%s is no repository of the mesh — no module is built from it and its owner is not "+
"the core's — and only the mesh's are tried on the build seat; nothing was asked", owner, repo)
}
request, err := checkRequestFor(ctx, open, p, checkScope{Mesh: true, From: mesh.From}, entries)
if err != nil {
return err
}
request = asATry(request, *tests, *storeOnly)
seat := buildSeatHeld(ctx)
// **Bounded in number** (novox/hq issue 686): one try of a commit at a time, and maxTries in all.
js, err := dialTheBus()
if err != nil {
return err
}
queue, err := link.ReadQueue(ctx, js, seat)
js.Close()
if err != nil {
return err
}
same, room := triesIn(queue, owner, repo, commit)
if same != "" {
return printTry(tryAnswer{ID: same, State: "asked", Repository: owner + "/" + repo, Ref: ref, Commit: commit,
On: seat, Said: fmt.Sprintf("a try of %s/%s at %.12s already waits or runs, as %s: nothing new was asked. "+
"Read it with try id=%s", owner, repo, commit, same, same)})
}
if !room {
return fmt.Errorf("%d tries already wait or run on the build seat, the most at once (`queue` lists them); "+
"nothing was asked — read one of yours, or ask again when one has ended", maxTries)
}
ask, err := askOverOn(seat)
if err != nil {
return err
}
defer ask.Close()
if err := ask.Ask(ctx, request); err != nil {
return err
}
return printTry(tryAnswer{ID: request.ID, State: "asked", Repository: owner + "/" + repo, Ref: ref, Commit: commit,
Base: trunk, On: seat, Said: fmt.Sprintf("asked %s to try %s/%s at %.12s (%s) merged into %s: its own "+
"merge-check.sh, as a check in the build queue, bounded at %s; no status is set and nothing is registered. "+
"Read it with try id=%s — a full run of this kind takes minutes", seat, owner, repo, commit, ref, trunk,
checkTimeout, request.ID)})
}
// tryRead answers what came of a try, holding the answer up to tryWait for its verdict.
func tryRead(ctx context.Context, id string) error {
until := time.Now().Add(tryWait)
for {
lines, err := buildLines(ctx, id)
if err != nil {
return err
}
a := tryAnswerOf(id, lines)
if (a.State != "waiting" && a.State != "running") || !time.Now().Add(2*time.Second).Before(until) {
return printTry(a)
}
if err := sleepOrDone(ctx, 2*time.Second); err != nil {
return err
}
}
}
// tryAnswerOf reads a try's answer from what the build seat said of it: its outcome under StepTried; `failed`
// for one that could not run — an error, never a pass; else how far it is.
func tryAnswerOf(id string, lines []link.BuildLine) tryAnswer {
a := tryAnswer{ID: id, State: "waiting", Said: "nothing is said of " + id + " yet: it waits in the build queue " +
"(`queue` says where), or no try has that id; ask again"}
for _, l := range lines {
switch l.Step {
case link.StepTried:
var t link.Tried
if err := json.Unmarshal([]byte(l.Message), &t); err != nil || t.Verdict == "" {
return tryAnswer{ID: id, State: "error", Said: "the build seat's word on it cannot be read: " + l.Message}
}
return tryAnswer{ID: id, State: t.Verdict, Said: t.Summary, Took: t.Took, Commit: t.Head, Trunk: t.Trunk,
Failed: t.Failed}
case "failed":
return tryAnswer{ID: id, State: "error", Said: "it could not run: " + l.Message}
case "checked":
// A check that is no try, or a build seat from before the verb: its verdict is not a try's.
return tryAnswer{ID: id, State: "error", Said: "the build seat checked it and said no try's outcome — " +
"it predates the verb (novox/hq issue 686), or " + id + " is no try: " + l.Message + "; `builds` with " +
"the id says every line"}
}
a.State, a.Last = "running", l.Message
a.Said = "still running; ask again. `builds` with the id says every line so far"
}
return a
}
func printTry(a tryAnswer) error {
fmt.Fprintln(os.Stderr, a.Said)
for _, l := range a.Failed {
fmt.Fprintln(os.Stderr, " "+l)
}
out, err := json.MarshalIndent(a, "", " ")
if err != nil {
return err
}
fmt.Println(string(out))
return nil
}
// buildLines is everything the build seat said of one build, as the bus keeps it, in order.
func buildLines(ctx context.Context, id string) ([]link.BuildLine, error) {
js, err := aBus()
if err != nil {
return nil, fmt.Errorf("cannot reach the bus to read a build's log: %w", err)
}
defer js.Close()
subject := link.BuildLogOf("*", id)
sub, err := js.Context().PullSubscribe(subject, "",
nats.BindStream(broker.EventsStream), nats.DeliverAll(), nats.AckNone())
if err != nil {
return nil, fmt.Errorf("cannot read %s from the bus: %w", subject, err)
}
defer func() { _ = sub.Unsubscribe() }()
var lines []link.BuildLine
for {
batch, err := sub.Fetch(200, nats.MaxWait(2*time.Second))
if err != nil && !errors.Is(err, nats.ErrTimeout) && !errors.Is(err, context.DeadlineExceeded) {
return nil, fmt.Errorf("reading a build's log: %w", err)
}
for _, msg := range batch {
var line link.BuildLine
if json.Unmarshal(msg.Data, &line) == nil {
lines = append(lines, line)
}
}
if len(batch) < 200 {
return lines, nil
}
}
}
-194
View File
@@ -1,194 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"reflect"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/link"
)
// A branch is tried before its pull request (novox/hq issue 686): the repository's own check asked for a pushed
// branch or commit, with no pull request.
// What a try asks is a check — after deliveries in the build queue, bounded as one — of the repository's own
// script alone: no module of the graph, so no gate; nothing built for a merge, so no reusable image.
func TestATryAsksACheckOfTheRepositorysOwnScriptAlone(t *testing.T) {
request := link.BuildRequest{ID: "build-1", Ref: "c0ffee", Check: &link.CheckRequest{Owner: "novox", Repo: "a",
Number: 7, Base: "main", Modules: []string{"a"}, New: []string{"b"}, Manifests: []string{"module.json"},
Dependents: []string{"c"}, Builds: []link.CheckBuild{{Module: "a"}}, Judge: link.JudgeSelf,
Plan: &link.ChangePlan{}}}
tried := asATry(request, "^TestStore", true)
c := tried.Check
if !c.Try || c.Number != 0 || c.Tests != "^TestStore" || !c.StoreOnly {
t.Fatalf("not marked a try, or it kept a pull request's number: %+v", c)
}
if len(c.Modules)+len(c.New)+len(c.Manifests)+len(c.Dependents)+len(c.Builds) != 0 || c.Judge != "" || c.Plan != nil {
t.Fatalf("a try runs the gate or builds for a merge: %+v", c)
}
if c.Base != "main" || c.Owner != "novox" || tried.Ref != "c0ffee" {
t.Fatalf("a try lost what it merges into what: %+v", tried)
}
if link.KindOf(tried) != link.KindCheck {
t.Fatal("a try does not count as a check in the build queue")
}
}
// A try's outcome is heard and left there: no `checked` is said, so the forge's holder sets no status, and
// nothing is recorded or registered (the handler holds no store here, and would fail reaching for one).
func TestATrysOutcomeSetsNoStatusAndRegistersNothing(t *testing.T) {
said := &saidOnTheBus{}
checkEvents = said
defer func() { checkEvents = nil }()
for name, result := range map[string]link.BuildResult{
"a try": {ID: "build-1", Ref: "c0ffee", Checked: &link.CheckRequest{Owner: "novox", Repo: "a", Try: true},
Check: &link.CheckOutcome{Verdict: "pass", RepoCheck: &link.CheckLayer{Verdict: "fail", Summary: "failed"}}},
"a try that could not run": {ID: "build-2", Checked: &link.CheckRequest{Owner: "novox", Repo: "a", Try: true},
Failed: "cannot clone"},
// A build seat from before the mark echoes the ask without it: a check of no pull request and no
// group is still nobody's status.
"a try echoed by an older build seat": {ID: "build-3", Checked: &link.CheckRequest{Owner: "novox", Repo: "a"},
Check: &link.CheckOutcome{Verdict: "pass"}},
} {
if err := (builds{}).Built(context.Background(), result); err != nil {
t.Fatalf("%s: %v", name, err)
}
}
if len(said.events) != 0 {
t.Fatalf("a try's outcome was said for the forge: %v", said.events)
}
// A pull request's check is still said.
_ = (builds{}).Built(context.Background(), link.BuildResult{ID: "build-4", Checked: &link.CheckRequest{
Owner: "novox", Repo: "a", Number: 7}, Check: &link.CheckOutcome{Verdict: "pass"}})
if len(said.events) != 1 {
t.Fatalf("a pull request's check was said %d time(s)", len(said.events))
}
}
// saidOnTheBus keeps the seat events said, and nothing else.
type saidOnTheBus struct {
link.Bus
events []string
}
func (s *saidOnTheBus) PublishSeatEvent(_ context.Context, seat, event string, _ []byte) error {
s.events = append(s.events, seat+" "+event)
return nil
}
// A ref the forge does not hold is refused before anything is asked, naming it.
func TestATryOfAnUnknownRefIsRefusedLoudly(t *testing.T) {
was := forgeCommit
defer func() { forgeCommit = was }()
forgeCommit = func(_ context.Context, owner, repo, ref string) (string, string, error) {
return "", "", errors.New("404 object does not exist [id: " + ref + "]")
}
err := tryCommand(context.Background(), []string{"novox/mesh-controller", "fix/no-such-branch"})
if err == nil {
t.Fatal("a try of a ref the forge does not hold was asked")
}
for _, want := range []string{"fix/no-such-branch", "novox/mesh-controller", "nothing was asked", "404"} {
if !strings.Contains(err.Error(), want) {
t.Errorf("the refusal does not say %q: %v", want, err)
}
}
for _, args := range [][]string{{"mesh-controller", "main"}, {"novox/mesh-controller"}, {"novox/a", "--upload-pack=x"},
{"novox/a", "main", "--tests", "x; rm -rf /"}, {"novox/a", "main", "--id", "build-1"}} {
if err := tryCommand(context.Background(), args); err == nil {
t.Errorf("try %v was taken", args)
}
}
}
// The answer of a try carries the lines that name what failed (novox/hq issue 460's shape) and how long it took.
func TestATrysAnswerCarriesTheFailingLines(t *testing.T) {
tried, _ := json.Marshal(link.Tried{Verdict: "fail", Summary: "its merge-check.sh failed: TestX", Took: "5m3s",
Head: "c0ffee", Trunk: "ca3f43c", Failed: []string{"--- FAIL: TestX (0.01s)", " x_test.go:12: got 1, want 2",
"FAIL\tgithub.com/novox/a/cmd\t3.1s"}})
lines := []link.BuildLine{{Seq: 1, Step: "check", Message: "novox/a#0 at c0ffee"},
{Seq: 2, Step: "output", Message: "ok github.com/novox/a/internal 1.2s"},
{Seq: 3, Step: link.StepTried, Message: string(tried)}, {Seq: 4, Step: "checked", Message: "pass: x"}}
a := tryAnswerOf("build-1", lines)
if a.State != "fail" || a.Took != "5m3s" || len(a.Failed) != 3 || a.Failed[0] != "--- FAIL: TestX (0.01s)" ||
a.Trunk != "ca3f43c" || a.ID != "build-1" {
t.Fatalf("the answer lost the verdict, the time or the failing lines: %+v", a)
}
// Not answered yet: said as running, with how far it is — never as a pass.
if a := tryAnswerOf("build-1", lines[:2]); a.State != "running" || a.Last == "" {
t.Fatalf("a try still running: %+v", a)
}
if a := tryAnswerOf("build-1", nil); a.State != "waiting" {
t.Fatalf("a try nothing was said of: %+v", a)
}
// One that could not run is an error with the build seat's word, never a pass.
if a := tryAnswerOf("build-1", append(lines[:1:1], link.BuildLine{Seq: 2, Step: "failed",
Message: "cannot clone"})); a.State != "error" || !strings.Contains(a.Said, "cannot clone") {
t.Fatalf("a try that could not run: %+v", a)
}
}
// One try of a commit at a time, and four in all (novox/hq issue 686): a second ask is answered the first's id,
// a fifth is refused; a try that ended, and a pull request's check, count for nothing.
func TestTriesAreBoundedInNumber(t *testing.T) {
ask := func(id, repo, ref, state string, try bool) link.QueuedAsk {
return link.QueuedAsk{ID: id, State: state, Request: link.BuildRequest{ID: id, Ref: ref,
Check: &link.CheckRequest{Owner: "novox", Repo: repo, Try: try}}}
}
q := link.Queue{Asks: []link.QueuedAsk{
ask("t1", "a", "c1", link.AskInFlight, true), ask("t2", "a", "c2", link.AskWaiting, true),
ask("t3", "b", "c1", link.AskDead, true), ask("p1", "a", "c9", link.AskWaiting, false)}}
if same, room := triesIn(q, "novox", "a", "c2"); same != "t2" || !room {
t.Fatalf("a second try of a commit: %q %v", same, room)
}
if same, room := triesIn(q, "novox", "b", "c1"); same != "" || !room {
t.Fatalf("a dead try, or another repository's, was taken for this one: %q %v", same, room)
}
if same, _ := triesIn(q, "novox", "a", "c9"); same != "" {
t.Fatalf("a pull request's check was taken for a try: %q", same)
}
q.Asks = append(q.Asks, ask("t4", "c", "c3", link.AskWaiting, true), ask("t5", "d", "c4", link.AskWaiting, true))
if _, room := triesIn(q, "novox", "e", "c5"); room {
t.Fatal("a fifth try was let in")
}
// An id is checked before it is a token of a bus subject.
for _, id := range []string{"x.>", "*", "a b"} {
if err := tryCommand(context.Background(), []string{"--id", id}); err == nil ||
!strings.Contains(err.Error(), "not a try's id") {
t.Errorf("try --id %q: %v", id, err)
}
}
}
// The verb runs its command, and is declared with the shell habit it replaces (novox/hq ADR 0245).
func TestTheTryVerbRunsItsCommand(t *testing.T) {
for _, c := range []struct {
args map[string]any
want []string
}{
{map[string]any{"repository": "novox/a", "ref": "fix/x"}, []string{"try", "novox/a", "fix/x"}},
{map[string]any{"repository": "novox/a", "ref": "fix/x", "base": "main", "tests": "^TestX$", "store-only": "true"},
[]string{"try", "novox/a", "fix/x", "--base", "main", "--tests", "^TestX$", "--store-only"}},
{map[string]any{"id": "build-1"}, []string{"try", "--id", "build-1"}},
} {
got, err := argvFor("try", c.args)
if err != nil || !reflect.DeepEqual(got, c.want) {
t.Errorf("try %v: %v %v, want %v", c.args, got, err, c.want)
}
}
if _, err := argvFor("try", map[string]any{"repository": "novox/a"}); err == nil {
t.Error("a try of no ref was taken")
}
for _, v := range catalogue.ControllerVerbs {
if v.Name == "try" {
if len(v.Replaces) == 0 {
t.Error("try names no shell habit it replaces")
}
return
}
}
t.Error("try is not a verb of the controller seat")
}
+1 -1
View File
@@ -79,7 +79,7 @@ func (f following) Upgraded(ctx context.Context, u link.Upgraded) error {
// `rebuild`, a `replay` registered — was sent here one machine after another without any judging; it
// now waits for a gate like any other build, and the release plan walks it through the machines, one
// at a time, each judged.
fmt.Printf("%s moved to %s outside a plan; a release plan sends it to %s, one machine first and then the rest at once, each judged "+
fmt.Printf("%s moved to %s outside a plan; a release plan sends it to %s, one machine at a time, each judged "+
"(`upgrade backlog` lists what waits)\n", u.Module, shortCommit(u.Commit), readableList(on))
return nil
}
-6
View File
@@ -177,12 +177,6 @@ func walkReached(p inventory.Plan, running func(string) []string) map[string][]s
reached = appendNew(reached, r.Gate.Machines...)
carried(r.Gate)
}
for _, g := range r.Gates {
if g != nil && !g.Pending {
reached = appendNew(reached, g.Machines...)
carried(g)
}
}
for _, tier := range p.Tiers {
for _, m := range tier {
for _, n := range running(m) {
-11
View File
@@ -92,8 +92,6 @@ type signalFacts struct {
// consumers: the condition counts the messages given up on, not the looks (novox/hq issue 440).
deadLettersNewest map[string]time.Time
advisoriesErr error
// busFaultsErr is why the bus module's events could not be read this tick (novox/hq ADR 0357).
busFaultsErr error
selfCheck selfCheckFacts
@@ -360,15 +358,6 @@ func (w *watchdogs) gather(ctx context.Context) *signalFacts {
}
f.advisories = link.Advisories.Since(now.Add(-advisoryQuiet))
f.lostConsumers, f.advisoriesErr = w.lostConsumers(ctx, f.advisories)
// And what the bus did to every other user's connection, as the bus module said it (novox/hq ADR 0357):
// read back for the same hour, and one advisory with the controller's own where both are its.
// A read that fails is its own condition, never S9 blind: the controller's own signals are still said.
if w.server != nil && w.js != nil {
var faults []link.Advisory
if faults, f.busFaultsErr = w.server.BusFaults(ctx, now.Add(-advisoryQuiet)); f.busFaultsErr == nil {
f.advisories = link.MergeAdvisories(f.advisories, faults)
}
}
if f.advisoriesErr == nil && w.js != nil {
f.deadLetters, f.advisoriesErr = link.HeldDeadLetters(w.js.Context())
}
-34
View File
@@ -1,34 +0,0 @@
package broker
import (
"slices"
"testing"
)
// **The bus module's watch speaks through the runtime that carries it** (novox/hq ADR 0357): the node's
// runtime is granted the events the bus module declares, under the module's name, while the bus's own
// account keeps the snapshot API and nothing else (ADR 0235).
func TestTheRuntimeCarryingTheBusModuleMayPublishItsEventsAndTheBussAccountMayNot(t *testing.T) {
events := []string{"bus.slow-consumer", "bus.connection-closed"}
runtime := Principal{Kind: KindNodeTools, Node: "anchor", Module: RuntimeModule,
Carries: []Declared{{Module: "nats", Emits: events}}}
p, err := PermissionsFor(runtime)
if err != nil {
t.Fatal(err)
}
for _, e := range events {
if !slices.Contains(p.Publish, "mesh.mod.nats.event."+e) {
t.Errorf("the runtime carrying the bus module may not publish %s: %v", e, p.Publish)
}
}
account := Principal{Kind: KindModule, Node: "anchor", Module: "nats", Emits: events, SnapshotsTheBus: true}
a, err := PermissionsFor(account)
if err != nil {
t.Fatal(err)
}
for _, s := range a.Publish {
if s == "mesh.mod.nats.event.bus.slow-consumer" || s == "mesh.mod.nats.event.bus.connection-closed" {
t.Errorf("the bus's own account is granted %s", s)
}
}
}
+58 -2
View File
@@ -40,6 +40,39 @@ var (
AskedBucket = BucketName(ControllerSeat, "asked")
)
// The controller's record mirrors (novox/hq ADR 0289, design 54, design 57 package F3): a bucket per kind of
// record, one key per record, written from the store by the lease holder only when the record changed, with
// one event on the controller's seat per write. Read by whatever names them in `reads` as
// `mesh-controller.<mirror>` (ADR 0201 §1), written by nobody else. **No secret in either** (ADR 0201): an
// assignment's value names its module and node; a module's declared config is what its manifest, which is
// public, declares.
//
// AssignmentsBucket holds which module is assigned to which node, keyed `<node>.<module>`; and
// DeclaredConfigBucket each registered module's config declarations as its manifest holds them, keyed by
// the module: what mesh-config resolves a config value from (ADR 0348 clauses 2 and 4).
var (
AssignmentsBucket = BucketName(ControllerSeat, MirrorAssignments)
DeclaredConfigBucket = BucketName(ControllerSeat, MirrorDeclaredConfig)
)
// The mirrors' local names, as a reader names them in `reads` after `mesh-controller.`.
const (
MirrorAssignments = "assignments"
MirrorDeclaredConfig = "declared-config"
)
// ControllerMirrors are the local names of the controller's record mirrors, which a module may read.
func ControllerMirrors() []string { return []string{MirrorAssignments, MirrorDeclaredConfig} }
// The mirrors' caps (design 54 §5): one value per key, no age, 64 KiB a value, 64 MiB a bucket.
const (
MirrorValueBytes = 64 << 10
MirrorBytes = 64 << 20
// MirrorMarkerTTL is the longest a purge's marker is kept (design 54 §2): the bucket allows a
// per-message age for it, and the writer gives each marker a day.
MirrorMarkerTTL = 24 * time.Hour
)
// AskedKeptFor is how long an ask is kept after it was made: a month, as the router keeps its own.
const AskedKeptFor = 30 * 24 * time.Hour
@@ -66,12 +99,14 @@ const (
// IsControllerBucket says a bucket is the controller's own, not a module's state nothing declares.
func IsControllerBucket(bucket string) bool {
return bucket == CallsBucket || bucket == HandActsBucket || bucket == ConditionsBucket ||
bucket == ConditionHistoryBucket || bucket == LeaseBucket || bucket == AskedBucket
bucket == ConditionHistoryBucket || bucket == LeaseBucket || bucket == AskedBucket ||
bucket == AssignmentsBucket || bucket == DeclaredConfigBucket
}
// ControllerBuckets are the controller's own buckets, in the order they are asserted.
func ControllerBuckets() []string {
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket, AskedBucket}
return []string{LeaseBucket, CallsBucket, HandActsBucket, ConditionsBucket, ConditionHistoryBucket, AskedBucket,
AssignmentsBucket, DeclaredConfigBucket}
}
// ControllerBucketsAsserter is what raising the controller's buckets needs of a connection.
@@ -168,6 +203,27 @@ func (j *JetStream) EnsureControllerBuckets() error {
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", AskedBucket, err)
}
// **The record mirrors: no age** (design 54 §5). A key is removed when its record is, by a purge whose
// marker lives a day; a key that expired would be a record the bus forgot while the store still holds it.
for _, m := range []struct{ bucket, why string }{
{AssignmentsBucket, "which module is assigned to which node, keyed <node>.<module> (novox/hq ADR 0289, " +
"design 54): written by the controller alone from its store when an assignment changes, read through `reads`"},
{DeclaredConfigBucket, "each registered module's config declarations as its manifest holds them, keyed by " +
"the module (novox/hq ADR 0348, design 57 F3): written by the controller alone from its store when a " +
"registration changes them, read through `reads`"},
} {
if _, err := js.CreateOrUpdateKeyValue(ctx, jetstream.KeyValueConfig{
Bucket: m.bucket,
Description: m.why,
History: 1,
MaxValueSize: MirrorValueBytes,
MaxBytes: MirrorBytes,
Storage: jetstream.FileStorage,
LimitMarkerTTL: MirrorMarkerTTL,
}); err != nil {
return fmt.Errorf("asserting bucket %s: %w", m.bucket, err)
}
}
return nil
}
+83 -1
View File
@@ -21,7 +21,8 @@ func TestTheControllerMayWriteEveryBucketItWrites(t *testing.T) {
if err != nil {
t.Fatal(err)
}
want := []string{"$KV." + CallsBucket + ".>", "$KV." + HandActsBucket + ".>"}
want := []string{"$KV." + CallsBucket + ".>", "$KV." + HandActsBucket + ".>",
"$KV." + AssignmentsBucket + ".>", "$KV." + DeclaredConfigBucket + ".>"}
for _, seat := range seatsTheControllerAsks {
if hasCancelledSet(seat) {
want = append(want, "$KV."+CancelledSetName(seat)+".>")
@@ -37,6 +38,34 @@ func TestTheControllerMayWriteEveryBucketItWrites(t *testing.T) {
}
}
// **A module reads a record mirror through `reads`, and never writes it** (ADR 0201 §1, design 54 §5,
// design 57 F3): granted the direct reads and watches of the bucket, and refused at composition a grant to
// write it, whose writer is the controller.
func TestAModuleReadsTheControllersMirrorsAndWritesNone(t *testing.T) {
reader := Principal{Kind: KindModule, Node: "one", Module: "mesh-config", PasswordHash: "x",
Reads: []string{"mesh-controller." + MirrorAssignments, "mesh-controller." + MirrorDeclaredConfig}}
p, err := PermissionsFor(reader)
if err != nil {
t.Fatal(err)
}
for _, bucket := range []string{AssignmentsBucket, DeclaredConfigBucket} {
for _, subject := range []string{"$JS.API.STREAM.INFO.KV_" + bucket, "$JS.API.DIRECT.GET.KV_" + bucket + ".>",
"$JS.API.CONSUMER.CREATE.KV_" + bucket + ".>"} {
if !slices.Contains(p.Publish, subject) {
t.Errorf("a module reading %s may not publish %s", bucket, subject)
}
}
for _, subject := range p.Publish {
if SubjectsOverlap(subject, "$KV."+bucket+".>") {
t.Errorf("a module reading %s may write it through %s", bucket, subject)
}
}
if err := CheckWriters(reader, []string{"$KV." + bucket + ".>"}); err == nil {
t.Errorf("a module granted a write of %s would compose", bucket)
}
}
}
// **A machine's node tools may say they are there, as that machine and no other** (novox/hq to-be 45
// S11), and the controller may hear the bus's advisories and ask who answers — read-only, named.
func TestTheWatchedSignalsMayBeSaidAndHeard(t *testing.T) {
@@ -96,3 +125,56 @@ func TestWhatTheControllerAskedIsBounded(t *testing.T) {
t.Errorf("history %d, age %s, bytes %d", status.History(), status.TTL(), info.Config.MaxBytes)
}
}
// **A listed mirror is no undeclared bucket** (ADR 0356 §3, ADR 0201 §16): the controller's list of its mirrors
// stands for their declaration, so the report of state nothing declares never names one.
func TestAListedMirrorIsNotReportedAsUndeclared(t *testing.T) {
names := []string{"gone_state"}
for _, m := range ControllerMirrors() {
names = append(names, BucketName(ControllerSeat, m))
}
undeclared, err := RaiseBuckets(fakeBuckets{names: names}, nil)
if err != nil {
t.Fatal(err)
}
if !slices.Equal(undeclared, []string{"gone_state"}) {
t.Fatalf("the report of undeclared state names %v", undeclared)
}
}
// **The mirrors' list and the writers table agree** (ADR 0356 §3, to-be 45 §1): every listed mirror is a row of
// the table whose one writer is the controller, on its bucket alone; and every row that writes a bucket of the
// controller's named as a mirror is on the list.
func TestTheMirrorsListAndTheWritersTableAgree(t *testing.T) {
controller := Principal{Kind: KindController}
module := Principal{Kind: KindModule, Module: ControllerSeat, Node: "one"}
rowed := map[string]bool{}
for _, row := range WritersTable {
if !strings.HasSuffix(row.State, " mirror") {
continue
}
for _, subject := range row.Subjects {
bucket := strings.TrimSuffix(strings.TrimPrefix(subject, "$KV."), ".>")
rowed[bucket] = true
if row.Writes == nil || !row.Writes(controller, subject) || row.Writes(module, subject) {
t.Errorf("%q is not written by the controller alone", row.State)
}
}
}
listed := map[string]bool{}
for _, m := range ControllerMirrors() {
bucket := BucketName(ControllerSeat, m)
listed[bucket] = true
if !rowed[bucket] {
t.Errorf("the mirror %s is listed and no row of the writers table writes it", bucket)
}
if !IsControllerBucket(bucket) || !slices.Contains(ControllerBuckets(), bucket) {
t.Errorf("the mirror %s is listed and not asserted as the controller's own", bucket)
}
}
for bucket := range rowed {
if !listed[bucket] {
t.Errorf("the writers table writes the mirror %s, which the list does not name", bucket)
}
}
}
+6 -2
View File
@@ -172,7 +172,9 @@ func TestRaisingLogsReportsWhatNothingDeclares(t *testing.T) {
// **No path of the mesh deletes or purges a log or a bucket, or recreates one** (novox/hq ADR 0297 §2,
// ADR 0201 §15–16): no code of this repository outside its tests calls the client's stream or bucket
// delete, or purges a stream, but for the controller lease's own stream, which purges its own history
// below a sequence (internal/lease). A new call is a decision, not a refactor.
// below a sequence (internal/lease), and the controller's record mirrors, which purge the key of a record
// the store no longer holds (internal/recordmirror; novox/hq ADR 0289, design 54 §2). A new call is a decision,
// not a refactor.
func TestNoPathDeletesALogOrABucket(t *testing.T) {
removers := map[string]bool{"DeleteStream": true, "DeleteKeyValue": true, "PurgeStream": true,
"DeleteObjectStore": true}
@@ -197,7 +199,9 @@ func TestNoPathDeletesALogOrABucket(t *testing.T) {
return true
}
name := sel.Sel.Name
if removers[name] || (name == "Purge" && !strings.Contains(filepath.ToSlash(path), "internal/lease/")) {
decided := strings.Contains(filepath.ToSlash(path), "internal/lease/") ||
strings.Contains(filepath.ToSlash(path), "internal/recordmirror/")
if removers[name] || (name == "Purge" && !decided) {
found = append(found, path+": "+name)
}
return true
+7
View File
@@ -34,6 +34,8 @@ func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
states = append(states, link.KeyChecked)
// And a walk kept in a new state, for the delivery it walks (novox/hq ADR 0239).
states = append(states, link.KeyPlanMoved)
// And a key of a record mirror written or removed (novox/hq ADR 0289, design 54 §3; design 57 F3).
states = append(states, link.KeyAssignmentChanged, link.KeyDeclaredConfigChanged)
for _, event := range states {
if !slices.Contains(broker.ControllerStates, event) {
t.Errorf("the mesh states %q and its account may not publish it", event)
@@ -42,6 +44,11 @@ func TestTheFactsTheGrantPermitsAreTheFactsTheMeshStates(t *testing.T) {
if len(broker.ControllerStates) != len(states) {
t.Errorf("the grant permits %v, which is more than the mesh states", broker.ControllerStates)
}
// The record mirrors a module may read are the ones the bus holds (design 54 §5, design 57 F3).
if !slices.Equal(catalogue.ControllerMirrors, broker.ControllerMirrors()) {
t.Errorf("the catalogue lets a module read the controller's mirrors %v and the bus holds %v",
catalogue.ControllerMirrors, broker.ControllerMirrors())
}
// **And the seat says it.** A seat carries the protocol of its role (novox/hq ADR 0129), so the
// facts the control plane states are the seat's `emits` — which is what lets anything else declare
// that it consumes them, and what the subject-agreement check reads to know they have an owner.
+4 -1
View File
@@ -313,7 +313,10 @@ var ControllerStates = []string{"applied", "refused", "built-before",
// the pull request's status, and anybody else may read.
"checked",
// And a walk kept in a new state (novox/hq ADR 0239): what the delivery it walks reads its steps from.
"plan-moved"}
"plan-moved",
// And a key of a record mirror written or removed (novox/hq ADR 0289, design 54 §3): the assignments and
// each registered module's declared config (design 57 F3).
"assignment-changed", "declared-config-changed"}
// BusAdvisories are what the bus server says about the mesh's own account that the controller
// reads (novox/hq to-be 45 §3, S9): a durable consumer that handed a message over as often as it
+1 -1
View File
@@ -24,7 +24,7 @@ accounts {
jetstream: enabled
users = [
{ user: "controller", password: "$2a$11$cccccccccccccccccccccc", permissions: {
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*"] }
publish: { allow: ["$JS.ACK.CONTROL.controller.>", "$JS.ACK.DEAD_LETTER_NOTICES.controller.>", "$JS.ACK.EVENTS.controller.>", "$JS.API.>", "$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>", "$KV.SEAT_NODE_BUILD_AGENT_cancelled.>", "$KV.mesh-controller_asked.>", "$KV.mesh-controller_assignments.>", "$KV.mesh-controller_calls.>", "$KV.mesh-controller_condition-history.>", "$KV.mesh-controller_conditions.>", "$KV.mesh-controller_declared-config.>", "$KV.mesh-controller_hand-acts.>", "$KV.mesh-controller_lease.>", "$SRV.INFO", "_INBOX.enrol.>", "mesh.again.>", "mesh.assignment.>", "mesh.events.dead.>", "mesh.mod.*.tool.>", "mesh.node.>", "mesh.seat.mesh-build-machine.accept.>", "mesh.seat.mesh-build-machine.tool.>", "mesh.seat.mesh-controller.event.applied", "mesh.seat.mesh-controller.event.assignment-changed", "mesh.seat.mesh-controller.event.built-before", "mesh.seat.mesh-controller.event.checked", "mesh.seat.mesh-controller.event.condition-changed", "mesh.seat.mesh-controller.event.condition-cleared", "mesh.seat.mesh-controller.event.condition-raised", "mesh.seat.mesh-controller.event.declared-config-changed", "mesh.seat.mesh-controller.event.doctor-heartbeat", "mesh.seat.mesh-controller.event.healer-acted", "mesh.seat.mesh-controller.event.plan-moved", "mesh.seat.mesh-controller.event.refused", "mesh.seat.mesh-controller.event.rolled-back", "mesh.seat.mesh-controller.event.secret-replaced", "mesh.seat.mesh-controller.tool.plans", "mesh.seat.mesh-delivery.tool.close", "mesh.seat.mesh-delivery.tool.groups", "mesh.seat.mesh-delivery.tool.release", "mesh.seat.mesh-delivery.tool.stalled", "mesh.seat.mesh-delivery.tool.stop", "mesh.seat.mesh-delivery.tool.times", "mesh.seat.node-backup.tool.backed-up.*", "mesh.seat.node-backup.tool.now.*", "mesh.seat.node-build-agent.accept.>", "mesh.seat.node-build-agent.tool.>", "mesh.seat.node-intrusion-prevention.tool.banned.*", "mesh.seat.node-launcher.tool.secret.*", "mesh.seat.node-service-manager.tool.restart.*"] }
subscribe: { allow: ["$JS.API.>", "$JS.EVENT.ADVISORY.CONSUMER.DELETED.>", "$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>", "$SRV.INFO", "$SRV.INFO.mesh-controller", "$SRV.INFO.mesh-controller.>", "$SRV.PING", "$SRV.PING.mesh-controller", "$SRV.PING.mesh-controller.>", "$SRV.STATS", "$SRV.STATS.mesh-controller", "$SRV.STATS.mesh-controller.>", "_DELIVER.controller", "_DELIVER.controller.>", "_INBOX.controller.>", "mesh.control.>", "mesh.mod.*.event.provisioner.failing", "mesh.mod.*.event.provisioner.recovered", "mesh.mod.*.event.provisioner.retirement", "mesh.mod.gitea.event.pull.merged", "mesh.mod.gitea.event.pull.updated", "mesh.mod.mesh-catalog.event.catching-up", "mesh.mod.mesh-catalog.event.upgraded", "mesh.seat.mesh-build-machine.event.built", "mesh.seat.mesh-controller.tool.>", "mesh.seat.node-build-agent.event.built", "mesh.seat.operator-channel.event.decided.mesh-controller"] }
allow_responses: { max: 1, ttl: "1m" }
} }
+9
View File
@@ -158,6 +158,15 @@ var WritersTable = []WriterRow{
{State: "a message a consumer gave up on", Writer: "controller", KeptIn: "the bus, the stream " + DeadLettersStream,
Others: "read, delivered again or dropped through the controller's dead-letters verb",
Subjects: []string{deadLetterPrefix + ">", againPrefix + ">"}, Writes: isController},
// The record mirrors (novox/hq ADR 0289 §9, design 54 §2): rows of this table, so a grant that lets
// anything else write them is refused at composition. Built so far: the assignments and each registered
// module's declared config (design 57 F3).
{State: "the assignments mirror", Writer: "controller (lease holder), from its store, only on a change",
KeptIn: "key-value " + AssignmentsBucket, Others: "read through `reads` as mesh-controller." + MirrorAssignments,
Subjects: kvOf(AssignmentsBucket), Writes: isController},
{State: "the declared-config mirror", Writer: "controller (lease holder), from its store, only on a change",
KeptIn: "key-value " + DeclaredConfigBucket, Others: "read through `reads` as mesh-controller." + MirrorDeclaredConfig,
Subjects: kvOf(DeclaredConfigBucket), Writes: isController},
}
// CheckWriters refuses a grant that lets a principal publish on a subject the writers table gives
+3
View File
@@ -34,6 +34,9 @@ var designRows = []string{
"the operator-channel's open messages",
"the facts snapshot",
"a message a consumer gave up on",
// The record mirrors, rows of to-be 45 §1's table by design 54 §2 (ADR 0289 §9).
"the assignments mirror",
"the declared-config mirror",
}
func TestTheWritersTableIsTheDesigns(t *testing.T) {
+1 -74
View File
@@ -162,10 +162,6 @@ type Reuse struct {
// keyFirst is a check's build (§1): its key is taken before anything is built, and a module with none is not
// built at all — nothing could ever take it.
keyFirst bool
// pull and head are the check's own pull request and the head it judged (novox/hq issue 703): it keeps only a
// build an earlier check of the same pull request at the same head made.
pull PullRef
head string
}
// errNoKey is a check's build of a module whose source does not pin it: not built (ADR 0350 §1).
@@ -378,7 +374,7 @@ func build(ctx context.Context, run Runner, publish Publisher,
// only when asked — a check, a dry run, `build --ref` and `rebuild` never are. Its key is taken by the same
// steps a build takes, stopped before anything is built; what it takes is copied by digest.
if reuse != nil && reuse.keyFirst {
keyed, key, err := keyFor(ctx, run, publish, manifest, within, workspace, commit, credentials, artifacts, args,
_, key, err := keyFor(ctx, run, publish, manifest, within, workspace, commit, credentials, artifacts, args,
held, npmrcPath, npmrc, seatBases, src, say)
if err == nil && key.Key == "" {
err = fmt.Errorf("%w: %s", errNoKey, key.Why)
@@ -387,11 +383,6 @@ func build(ctx context.Context, run Runner, publish Publisher,
say("reuse", "not built for the merge: %v", err)
return Result{Commit: commit, NoKey: err.Error()}, err
}
// **A check keeps a build of the same key a verdict names** (novox/hq issue 703): asked again when only
// the trunk moved, it builds nothing the earlier check built of an unchanged source.
if kept := keptCheckBuild(ctx, *reuse, manifest.Module, key.Key, len(artifacts), say); kept != nil {
built, src, reused = kept.Made, keyed, kept
}
} else if reuse != nil && reuse.Scratch != nil {
taken, keyed, why := takeACheckBuild(ctx, run, publish, *reuse, manifest, within, workspace, commit,
credentials, artifacts, args, held, npmrcPath, npmrc, seatBases, src, onTrunk, say)
@@ -501,47 +492,6 @@ func takeACheckBuild(ctx context.Context, run Runner, publish Publisher, reuse R
return taken, keyed, ""
}
// keptCheckBuild is the build a check keeps instead of building (novox/hq issue 703): the one kept under the
// module's key by an earlier check of the same pull request at the same head — only the trunk moved — when a verdict
// of the build seat names its key and every digest — the record alone is never trusted (ADR 0350 §2) — and it holds
// every artifact the module declares. Another pull request's build, or an earlier head's, is never kept. Nil, and
// the check builds, otherwise.
func keptCheckBuild(ctx context.Context, reuse Reuse, module, key string, artifacts int,
say func(step, format string, args ...any)) *Reused {
if reuse.Scratch == nil || reuse.head == "" {
return nil
}
var same []NamedBuild
for _, n := range reuse.Named {
if n.Pull == reuse.pull {
same = append(same, n)
}
}
if len(same) == 0 {
return nil
}
found, ok, err := reuse.Scratch.Find(ctx, module, key)
switch {
case err != nil:
say("reuse", "the scratch namespace could not be read, so the check builds: %v", err)
return nil
case !ok:
return nil
case found.Pull != reuse.pull || found.Head == "" || twelve(found.Head) != twelve(reuse.head):
say("reuse", "the build kept under %s is %s, not this pull request's at this head: built", twelve(key[3:]),
found.Said())
return nil
case unnamed(same, module, key, found) != "":
say("reuse", "a build is kept under %s and no verdict names it with its digests: built", twelve(key[3:]))
return nil
case len(found.Made) != artifacts:
return nil
}
kept := &Reused{From: found, Made: found.Made}
say("reuse", "kept, not built: %s", strings.Replace(kept.Words(), "reused from", "the build of", 1))
return kept
}
// keyFor takes a build's reuse key by the steps a build takes, each artifact stopped before anything is built or
// published (ADR 0350 §2): its own copy of the source inputs, so the build's are untouched.
func keyFor(ctx context.Context, run Runner, publish Publisher, manifest catalogue.Manifest,
@@ -1126,16 +1076,6 @@ func one(ctx context.Context, run Runner, publish Publisher,
return catalogue.Built{}, fmt.Errorf("%s: building %s failed: %w", module, a.Name, err)
}
say("image", "built, publishing")
// Its bases linked into its repository first, so the push needs none of their blobs from this
// machine's store, which may not hold them (novox/hq issue 694). Said and not fatal: a push
// from a store that holds them works without, and one that fails says which blob it lacked.
if lender, can := publish.(BaseLender); can {
if n, err := lender.LendBases(ctx, module+"/"+a.Name, buildArgValues(args)); err != nil {
say("image", "its bases could not all be linked into %s/%s before the push: %v", module, a.Name, err)
} else if n > 0 {
say("image", "%d blob(s) of its bases linked into %s/%s before the push", n, module, a.Name)
}
}
reference, err := publish.PublishImage(ctx, local, module+"/"+a.Name)
if err != nil {
return catalogue.Built{}, err
@@ -1487,19 +1427,6 @@ func standingOn(ctx context.Context, manifest catalogue.Manifest, held map[strin
return args, resolved, nil
}
// buildArgValues is what each --build-arg of a build was handed: the bases, as this mesh holds them.
func buildArgValues(args []string) []string {
var values []string
for i := 0; i+1 < len(args); i += 2 {
if args[i] == "--build-arg" {
if _, value, has := strings.Cut(args[i+1], "="); has {
values = append(values, value)
}
}
}
return values
}
// compile runs a module's own code through its toolchain, and says where the result is.
//
// **In the module's own directory, under the path the toolchain expects.** A module is compiled
+25 -582
View File
@@ -13,7 +13,6 @@ import (
"os/exec"
"path/filepath"
"regexp"
"sort"
"strings"
"time"
@@ -88,19 +87,6 @@ type CheckSpec struct {
// Judge is who judges the gate: "" the controller the mesh runs, "self" the change's own, "validator"
// the running one with the change's validator.
Judge string
// Release is the release branch the pull request merges into (novox/hq ADR 0358), empty for any other base. The
// gate is then judged by the controller beside it as mesh-controller — mesh-controller's release branch, never
// the controller the mesh runs nor its main — with mesh-host's validator from beside it when mesh-host was read at
// the release branch too; a change to the controller still judges itself and a change to the node-engine brings
// its own validator. The verdict names the branch, the judge's commit and the branch each beside was read at.
Release string
// JudgeCache is the judge cache (novox/hq ADR 0358 §6): a directory of the build agent's own, outside the
// workspace, that only judge builds mount — never a check's containers, so no pull request's script can write it.
// It holds the Go build cache every judge is built with (judges stay warm), the release-branch judges kept by the
// commits they are built from (judges/, one per release-branch head), and each judge build's fresh home (homes/,
// removed after the build). Empty: each judge is built cold under the check's own root, which every check empties
// before it clones, and nothing is kept.
JudgeCache string
// Toolchain is the image a check's Go runs in: the mesh's own Go toolchain, as it holds it.
// Toolchains is every toolchain the mesh holds, by language, for a script that declares another.
Toolchain string
@@ -117,23 +103,6 @@ type CheckSpec struct {
// for the walk to take. None for a group's composed check, whose members' own checks build.
Builds []CheckBuildSpec
BuildWith *CheckBuilding
// Tests and StoreOnly narrow what the repository's own script runs, for a branch tried before its pull
// request (novox/hq issue 686): handed to the script as EnvTests and EnvStoreOnly, which it may read.
Tests string
StoreOnly bool
}
// selection is what a check hands the repository's script to narrow what it runs (novox/hq issue 686): nothing
// for a pull request's check, which always runs whole.
func selection(spec CheckSpec) []string {
var env []string
if spec.Tests != "" {
env = append(env, EnvTests+"="+spec.Tests)
}
if spec.StoreOnly {
env = append(env, EnvStoreOnly+"=1")
}
return env
}
// GroupHead is one other head of a delivery group's composed check.
@@ -167,12 +136,10 @@ func ToolchainsOf(held map[string]string) map[string]string {
return out
}
// Beside is one repository cloned next to the one checked: at Ref, or at Else when Else is said and the repository
// has no branch Ref (novox/hq ADR 0358 §3).
// Beside is one repository cloned next to the one checked.
type Beside struct {
Repository string
Ref string
Else string
}
// CheckVerdict is what came of one check. Verdict and Summary are the gate's; Gate and Repo each layer.
@@ -195,16 +162,6 @@ type CheckVerdict struct {
Members []JudgedHead
// Built are the builds it made for its merge (novox/hq ADR 0350 §1), each with its key, or why not.
Built []CheckBuilt
// Release is the release branch it was judged for (novox/hq ADR 0358 §4), JudgedBy the commit of the controller
// that judged its gate (twelve characters), and BesideAt the branch each repository beside it was read at, by
// directory. Empty for a check of any other base.
Release string
JudgedBy string
BesideAt map[string]string
// Validator says which node-engine validator the judge of a release-branch check was built with (ADR 0358 §2):
// mesh-host's of the release branch with its commit, the one the release branch's controller vendors, the
// change's own, or the one the change's own controller vendors. Empty for a check of any other base.
Validator string
}
// Layer is one layer of a check, judged.
@@ -236,11 +193,6 @@ const (
// EnvGroup is a delivery group's other heads, as JSON, for the gate (novox/hq ADR 0239); empty for a
// pull request checked alone.
EnvGroup = "MESH_CHECK_GROUP"
// EnvTests is a `go test -run` pattern and EnvStoreOnly is 1 when a branch tried before its pull request
// asks for only those tests, or only the store tests (novox/hq issue 686). Never set for a pull request's
// check. A script that reads neither runs whole.
EnvTests = "MESH_CHECK_TESTS"
EnvStoreOnly = "MESH_CHECK_STORE_ONLY"
)
// CheckTimeout bounds one check; a check that runs past it is an error, not a pass.
@@ -329,39 +281,13 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
if err := forgetLeftCredentials(workspace); err != nil {
return CheckVerdict{}, err
}
// **mesh-lab's replays run where no pull request reaches** (novox/hq issue 709, ADR 0358 §6): they hold the
// container runtime's socket, so their sources, home, Go cache and module cache are none of the workspace's.
replay, err := replayDirs(spec.JudgeCache, workspace, root, spec.ID)
if err != nil {
return CheckVerdict{}, err
}
if spec.JudgeCache != "" {
defer os.RemoveAll(replay.base)
}
defer func() {
if replay.volumeMade {
removing, done := context.WithTimeout(context.Background(), time.Minute)
defer done()
_, _ = run(removing, root, "docker", "volume", "rm", "--force", replay.volume)
}
}()
// cloneAt clones a repository and checks out ref — or, when the repository has no branch ref and orElse is said,
// orElse (novox/hq ADR 0358 §3); readAt keeps which, by directory.
readAt := map[string]string{}
cloneAt := func(repository, ref, orElse, dir string) error {
clone := func(repository, ref, dir string) error {
if _, err := run(ctx, root, "git", cloneWith(credentials, "clone", "--quiet", repository, dir)...); err != nil {
return fmt.Errorf("cannot clone %s: %s", redact.redact(repository), redact.redact(err.Error()))
}
if err := recordedWithoutUserinfo(ctx, run, filepath.Join(root, dir), repository); err != nil {
return err
}
if orElse != "" && ref != "" {
if _, err := run(ctx, filepath.Join(root, dir), "git", "rev-parse", "--verify", "--quiet",
"refs/remotes/origin/"+ref); err != nil {
ref = orElse
}
}
readAt[dir] = ref
if ref != "" {
if _, err := run(ctx, filepath.Join(root, dir), "git", "checkout", "--quiet", ref); err != nil {
return fmt.Errorf("%s has no %s: %w", redact.redact(repository), ref, err)
@@ -369,9 +295,6 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
return nil
}
clone := func(repository, ref, dir string) error {
return cloneAt(repository, ref, "", dir)
}
name := spec.Repo
if name == "" || !safeName.MatchString(name) {
name = "checked"
@@ -407,25 +330,8 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
named := func(v CheckVerdict) CheckVerdict {
v.Head, v.Trunk, v.Merge, v.Beside, v.Members = judged.Head, judged.Trunk, judged.Merge, judged.Beside, judged.Members
v.Release, v.JudgedBy, v.BesideAt, v.Validator = spec.Release, judged.JudgedBy, judged.BesideAt, judged.Validator
return v
}
// **What the merge would move is built from a checkout nothing of the check touches** (novox/hq issue 703): the
// gate and the repository's own check run in containers that mount the workspace read-write, so they can change
// its tree, its objects and its caches. The builds are made from a copy of the merge's objects, cloned before
// anything of the change runs, outside the workspace — so what is kept under a key is what the key describes.
building, buildingTree := "", ""
if len(spec.Builds) > 0 && spec.BuildWith != nil && len(spec.Group) == 0 {
at := judged.Merge
if at == "" {
at = spec.Ref
}
building, buildingTree, err = checkoutForTheBuilds(ctx, run, tree, at, workspace)
if err != nil {
return CheckVerdict{}, err
}
defer os.RemoveAll(building)
}
// **The base branch's check judges the change** (novox/hq issue 310), never the one the change brings.
judging, err := TheCheckThatJudges(ctx, tree, spec.Base)
if err != nil {
@@ -448,35 +354,9 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
if dir == name || !safeName.MatchString(dir) {
continue
}
if dir == "mesh-lab" {
// **mesh-lab is never read at a release branch** (novox/hq ADR 0358): its replays run with the container
// runtime's socket, so a branch of a release's name — which anyone may create — is never what runs there,
// whether it is asked with a fallback or without.
if b.Else != "" || (spec.Release != "" && b.Ref == spec.Release) {
b.Ref = b.Else
if b.Ref == "" {
b.Ref = "main"
}
}
b.Else = ""
}
if err := cloneAt(b.Repository, b.Ref, b.Else, dir); err != nil {
if err := clone(b.Repository, b.Ref, dir); err != nil {
return CheckVerdict{}, fmt.Errorf("beside it, %w", err)
}
if dir == "mesh-lab" {
// The copy the replays run from, made before anything of the change runs, outside every check container
// (novox/hq issue 709) — and whether it holds replays, read now, so nothing written later decides it.
if err := replay.copyLab(ctx, run, spec, root, workspace, filepath.Join(root, dir)); err != nil {
return CheckVerdict{}, fmt.Errorf("mesh-lab could not be copied for its replays: %w", err)
}
}
if spec.Release != "" {
// The branch each was read at is named with its commit (ADR 0358 §4).
if judged.BesideAt == nil {
judged.BesideAt = map[string]string{}
}
judged.BesideAt[dir] = readAt[dir]
}
// The commit each one read is named in the verdict (ADR 0323 §3): its move is issue 568's question.
if at := headAt(ctx, run, filepath.Join(root, dir)); at != "" {
if judged.Beside == nil {
@@ -484,7 +364,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
judged.Beside[dir] = at
}
say("check", "beside it %s at %s", dir, refWords(readAt[dir]))
say("check", "beside it %s at %s", dir, short(b.Ref))
}
// A delivery group's other heads (novox/hq ADR 0239), each at its head, for the gate to compose with this.
@@ -598,61 +478,21 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
user = fmt.Sprintf("%d:%d", os.Getuid(), os.Getgid())
}
in := func(image, dir string, env []string, command ...string) []string {
home := workspace
for _, e := range env {
if strings.HasPrefix(e, "HOME=") {
home = strings.TrimPrefix(e, "HOME=")
}
}
// As the builder itself: what a check writes into the workspace is the builder's to remove.
args := []string{"run", "--rm", "--network", "host", "--volume", workspace + ":" + workspace, "--workdir", dir,
"--user", user, "--env", "HOME=" + home,
"--user", user, "--env", "HOME=" + workspace,
// The check's own checkouts, whoever cloned them: git in a container of another user than the
// one that cloned refuses a repository it does not own ("dubious ownership"), and Go's build
// stamps the version from git — a judge that would not build for want of it.
"--env", "GIT_CONFIG_COUNT=1", "--env", "GIT_CONFIG_KEY_0=safe.directory", "--env", "GIT_CONFIG_VALUE_0=*"}
for _, e := range env {
if !strings.HasPrefix(e, "HOME=") {
args = append(args, "--env", e)
}
args = append(args, "--env", e)
}
return append(append(args, image), command...)
}
inToolchain := func(dir string, env []string, command ...string) []string {
return in(spec.Toolchain, dir, env, command...)
}
// **Every judge is built where no pull request can reach** (novox/hq issue 482, ADR 0358 review): the workspace's
// Go cache, HOME and git configuration are written by pull requests' own scripts, and git runs during a Go build
// to stamp its version — a planted hook would rewrite the judge's sources before they compile. So a judge is built
// with a HOME and configuration made fresh for this check and a Go cache only judge builds mount (judgeBuildDirs),
// no global or system git configuration, no Go environment file, and no version stamping from git.
judgeHome, judgeGoCache, err := judgeBuildDirs(spec.JudgeCache, workspace, root, spec.ID)
if err != nil {
return CheckVerdict{}, err
}
defer os.RemoveAll(judgeHome)
inJudgeBuild := func(dir string, _ []string, command ...string) []string {
args := in(spec.Toolchain, dir, JudgeBuildEnv(judgeHome, judgeGoCache), command...)
if spec.JudgeCache == "" {
return args // under the check's root, in the workspace mounted already
}
// The judge cache's parts this build needs, mounted into judge builds alone (ADR 0358 §6).
return append([]string{args[0], "--volume", judgeHome + ":" + judgeHome, "--volume",
judgeGoCache + ":" + judgeGoCache}, args[1:]...)
}
inReplays := func(env []string, command ...string) []string {
args := in(spec.Toolchain, filepath.Join(replay.lab, "replays"), append(ReplayEnv(replay.home, replay.goCache,
replay.modCache), env...), command...)
// The socket goes to the replays alone, never to the change's own code.
mounts := []string{"--volume", "/var/run/docker.sock:/var/run/docker.sock"}
if replay.volume != "" {
mounts = append(mounts, "--volume", replay.volume+":"+replay.base)
} else {
mounts = append(mounts, "--volume", replay.base+":"+replay.base, "--volume", replay.goCache+":"+replay.goCache,
"--volume", replay.modCache+":"+replay.modCache)
}
return append(append([]string{args[0]}, mounts...), args[1:]...)
}
var out tail
// running runs one container of the check, its output into the report, in a process group of its own.
running := func(args []string) error {
@@ -664,23 +504,10 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
// The judge. Its failing to build is the change's fault when the change is the judge or its validator,
// and the check's when it is the controller the mesh runs.
gate, judgeFault, err := judgeFor(ctx, labelled, run, spec, root, tree, judged.BesideAt, inJudgeBuild, say)
gate, judgeFault, err := judgeFor(ctx, labelled, run, spec, root, tree, inToolchain, say)
if err != nil {
return CheckVerdict{}, err
}
if spec.Release != "" {
// **A release-branch judgement says so** (novox/hq ADR 0358 §4): the branch, the controller's commit that
// judged, and each repository read beside it at its branch and commit.
judgedBy := filepath.Join(root, "mesh-controller")
if spec.Judge == "self" {
judgedBy = tree
}
judged.JudgedBy = headAt(ctx, run, judgedBy)
judged.Validator = releaseValidator(spec, judged.BesideAt, judged.Beside)
said := releaseWords(spec.Release, judged.JudgedBy, judged.Validator, judged.BesideAt, judged.Beside)
fmt.Fprintf(&out, "--- %s\n", said)
say("check", "%s", said)
}
verdictFile := filepath.Join(root, "verdict.json")
env := []string{EnvFacts + "=" + factsFile, EnvGate + "=" + gate, EnvGateStore + "=" + storeURL,
EnvTestStore + "=" + storeURL, EnvTestBus + "=nats://" + bus, EnvRepository + "=" + spec.Owner + "/" + spec.Repo,
@@ -688,7 +515,6 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
EnvModules + "=" + strings.Join(append(append([]string{}, spec.Modules...), spec.New...), ","),
EnvGroup + "=" + groupFile,
"GOCACHE=" + filepath.Join(workspace, "go-cache"), "GOMODCACHE=" + filepath.Join(workspace, "go-modules")}
env = append(env, selection(spec)...)
v := CheckVerdict{}
modules := append(append([]string{}, spec.Modules...), prefixed("new:", spec.New)...)
@@ -706,7 +532,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
say("check", "the gate: %d module(s) of the graph touched — %s", len(modules), strings.Join(modules, ", "))
fmt.Fprintf(&out, "--- the gate: %s\n", strings.Join(modules, ", "))
v.Gate.Verdict, v.Gate.Summary = gateLayer(ctx, spec, tree, root, gate, verdictFile, env, inToolchain,
running, &out, bus, name, replay.hasReplays, inReplays, say)
running, &out, bus, workspace, name, say)
}
if timedOut() {
v.Gate.Verdict, v.Gate.Summary = "error", fmt.Sprintf("the check ran past %s and was ended", CheckTimeout)
@@ -746,11 +572,6 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
}
}
if spec.Release != "" {
// The statuses name the release branch and the judge first, so the forge's short description keeps them.
mark := releaseMark(spec.Release, judged.JudgedBy)
v.Gate.Summary, v.Repo.Summary = mark+v.Gate.Summary, mark+v.Repo.Summary
}
// What the check printed travels in the verdict to the forge and the controller: redacted as the log is.
v.Gate.Summary, v.Repo.Summary, v.Repo.Failed = redact.redact(v.Gate.Summary), redact.redact(v.Repo.Summary),
redact.redact(v.Repo.Failed)
@@ -766,7 +587,7 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
say("check", "%s", why)
} else {
began := time.Now()
v.Built = buildForTheMerge(ctx, run, spec, *spec.BuildWith, building, buildingTree, judged.Merge, judged, forge,
v.Built = buildForTheMerge(ctx, run, spec, *spec.BuildWith, workspace, tree, judged.Merge, judged, forge,
func(step, message string) {
if log != nil {
log(step, redact.redact(message))
@@ -779,39 +600,6 @@ func Check(ctx context.Context, run Runner, spec CheckSpec, workspace, registry
return named(v), nil
}
// checkoutForTheBuilds is a clone of the commit a check builds for its merge, made before anything of the check runs,
// outside the workspace its containers mount (novox/hq issue 703): the directory to remove, and the clone in it.
// Cloned over git's own transport, never by linking the objects: what the check's containers later write into the
// workspace's objects cannot reach it.
func checkoutForTheBuilds(ctx context.Context, run Runner, tree, at, workspace string) (string, string, error) {
dir, err := os.MkdirTemp("", "mesh-check-builds-")
if err != nil {
return "", "", err
}
if withinDir(workspace, dir) {
os.RemoveAll(dir)
return "", "", fmt.Errorf("the temporary directory %s is inside the workspace %s, which a check's containers "+
"mount: the merge is not built from where its tests could change it", dir, workspace)
}
commit, err := run(ctx, tree, "git", "rev-parse", "--verify", at+"^{commit}")
if err != nil {
os.RemoveAll(dir)
return "", "", fmt.Errorf("the commit to build for the merge, %s: %w", at, err)
}
commit = strings.TrimSpace(commit)
if _, err := run(ctx, tree, "git", "update-ref", "refs/heads/mesh-check-builds", commit); err != nil {
os.RemoveAll(dir)
return "", "", err
}
cloned := filepath.Join(dir, "merged")
if _, err := run(ctx, dir, "git", "clone", "--quiet", "--no-local", "--no-checkout", "--branch",
"mesh-check-builds", tree, cloned); err != nil {
os.RemoveAll(dir)
return "", "", fmt.Errorf("cannot copy the merge to build it: %w", err)
}
return dir, cloned, nil
}
// builtCount is how many of a check's builds for its merge were built.
func builtCount(built []CheckBuilt) int {
n := 0
@@ -983,9 +771,8 @@ const SomeManifestsEnv = "MESH_MODULE_CHECK_SOME=1"
// gateLayer runs the gate: the touched manifests through `module check`, every machine composed with the
// change, and the replays of what the mesh runs. It answers the gate's verdict and summary.
func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFile string, env []string,
inToolchain func(string, []string, ...string) []string, running func([]string) error, out *tail, bus,
name string, hasReplays bool, inReplays func([]string, ...string) []string,
say func(step, format string, args ...any)) (string, string) {
inToolchain func(string, []string, ...string) []string, running func([]string) error, out *tail, bus, workspace,
name string, say func(step, format string, args ...any)) (string, string) {
// 1. The manifests the change touches, as the judge reads them: a manifest it cannot read, or one with a
// problem the change brings, fails here. **What was already so on the base branch is said and fails
// nothing** — the gate's own rule: a module whose manifest the running controller already finds fault
@@ -1077,18 +864,19 @@ func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFil
// 3. **The replays of what the mesh runs** (to-be 45 §9, M9): mesh-lab's, from its main — reviewed code,
// so given the container runtime the resolver replay raises containers with — against the bus of the
// release the mesh runs and the change's own catalogue when the change is to the catalogue.
if ctx.Err() == nil {
if hasReplays {
if lab := filepath.Join(root, "mesh-lab", "replays"); ctx.Err() == nil {
if _, err := os.Stat(lab); err == nil {
catalogue := filepath.Join(root, "mesh-catalog")
if name == "mesh-catalog" {
catalogue = tree
}
say("check", "the replays of what the mesh runs, from mesh-lab")
fmt.Fprintln(out, "--- the replays (mesh-lab replays/)")
// In the replays' own environment (inReplays): reviewed code, the socket, and nothing the workspace holds
// but the catalogue they read.
args := inReplays([]string{EnvTestBus + "=nats://" + bus, "MESH_REPLAY_CATALOGUE=" + catalogue},
args := inToolchain(lab, []string{EnvTestBus + "=nats://" + bus, "MESH_REPLAY_CATALOGUE=" + catalogue,
"GOCACHE=" + filepath.Join(workspace, "go-cache"), "GOMODCACHE=" + filepath.Join(workspace, "go-modules")},
"go", "test", "-count=1", "./...")
// The socket goes to the replays alone, never to the change's own code above.
args = append([]string{args[0], "--volume", "/var/run/docker.sock:/var/run/docker.sock"}, args[1:]...)
if err := running(args); err != nil {
if ctx.Err() != nil {
return "error", "the check was ended during the replays"
@@ -1108,32 +896,21 @@ func gateLayer(ctx context.Context, spec CheckSpec, tree, root, gate, verdictFil
// the mesh runs. It answers the judge's path; or, when the change makes its own judge unbuildable, why —
// the change's fault, a failing gate; or an error when the check cannot build a judge at all. A check
// whose gate does not run builds a judge only to hand its scripts one, and goes on without when it cannot.
//
// **A pull request into a release branch is judged by that branch's controller** (novox/hq ADR 0358 §2): the one
// beside it as mesh-controller, read at the release branch — with mesh-host's validator from beside it when mesh-host
// has that branch too, or with the change's own validator for a change to the node-engine — never the controller the
// mesh runs. besideAt is the branch each repository beside it was read at.
func judgeFor(ctx context.Context, labelled, run Runner, spec CheckSpec, root, tree string, besideAt map[string]string,
func judgeFor(ctx context.Context, labelled, run Runner, spec CheckSpec, root, tree string,
inToolchain func(string, []string, ...string) []string, say func(step, format string, args ...any)) (string, string, error) {
gated := spec.Gated()
switch spec.Judge {
case "self":
bin := filepath.Join(root, "bin", "judge-of-itself")
if _, err := labelled(ctx, root, "docker", inToolchain(tree, nil,
if _, err := labelled(ctx, root, "docker", inToolchain(tree,
[]string{"CGO_ENABLED=0", "GOFLAGS=-mod=vendor", "GOPROXY=off", "GOCACHE=" + filepath.Join(filepath.Dir(root), "go-cache")},
"go", "build", "-o", bin, "./cmd/mesh-controller")...); err != nil {
return "", "the change's controller, which judges itself, does not build: " + firstLine(err.Error()), nil
}
say("check", "judged by the change's own controller")
return bin, "", nil
}
var bin, judgeTree string
var err error
if spec.Release != "" {
withHost := spec.Judge != "validator" && besideAt["mesh-host"] == spec.Release
bin, judgeTree, err = releaseJudge(ctx, labelled, run, spec, root, withHost, inToolchain, say)
} else {
bin, judgeTree, err = judge(ctx, labelled, run, root, inToolchain, say)
}
bin, judgeTree, err := judge(ctx, labelled, run, root, inToolchain, say)
if err != nil {
if !gated {
say("check", "no judge for its script: %v", err)
@@ -1152,350 +929,15 @@ func judgeFor(ctx context.Context, labelled, run Runner, spec CheckSpec, root, t
}
}
withValidator := filepath.Join(root, "bin", "judge-with-this-validator")
if _, err := labelled(ctx, root, "docker", inToolchain(filepath.Join(root, judgeTree), nil,
if _, err := labelled(ctx, root, "docker", inToolchain(filepath.Join(root, judgeTree),
[]string{"CGO_ENABLED=0", "GOFLAGS=-mod=vendor", "GOPROXY=off", "GOCACHE=" + filepath.Join(filepath.Dir(root), "go-cache")},
"go", "build", "-o", withValidator, "./cmd/mesh-controller")...); err != nil {
if spec.Release != "" {
return "", spec.Release + "'s controller does not build with this change's validator: " +
firstLine(err.Error()), nil
}
return "", "the controller the mesh runs does not build with this change's validator: " + firstLine(err.Error()), nil
}
if spec.Release != "" {
say("check", "judged by %s's controller, with this change's validator", spec.Release)
return withValidator, "", nil
}
say("check", "judged by the controller the mesh runs, with this change's validator")
return withValidator, "", nil
}
// releaseJudge builds the controller that judges a pull request into a release branch (novox/hq ADR 0358 §2): the
// one beside the check as mesh-controller, read at that branch — with mesh-host's validator from beside it in place
// of the one it vendors when withHost. **Built once per release-branch head**: kept in the spec's judge cache by the
// commits it is built from and taken from there while they stand. It answers the binary and the directory it was
// built from.
func releaseJudge(ctx context.Context, run, plain Runner, spec CheckSpec, root string, withHost bool,
inToolchain func(string, []string, ...string) []string, say func(step, format string, args ...any)) (string, string, error) {
const dir = "mesh-controller"
if _, err := os.Stat(filepath.Join(root, dir)); err != nil {
return "", "", fmt.Errorf("no controller of %s beside the check to judge it with", spec.Release)
}
key := fullHeadAt(ctx, plain, filepath.Join(root, dir))
if key == "" {
return "", "", fmt.Errorf("the controller of %s beside the check names no commit", spec.Release)
}
with := ""
if withHost {
host := fullHeadAt(ctx, plain, filepath.Join(root, "mesh-host"))
if host == "" {
return "", "", fmt.Errorf("mesh-host of %s beside the check names no commit", spec.Release)
}
key += "-with-mesh-host-" + host
with = ", with mesh-host's validator of " + spec.Release + " at " + twelve(host)
}
bin := filepath.Join(root, "bin", "mesh-controller")
if took := judgeKept(keptJudges(spec.JudgeCache), key, bin); took {
say("check", "judged by %s's controller at %s%s, built for an earlier check of that head", spec.Release,
twelve(key), with)
return bin, dir, nil
}
if withHost {
vendored := filepath.Join(root, dir, "vendor", "github.com", "novox", "mesh-host")
for _, pkg := range []string{"validate", filepath.Join("internal", "declaration")} {
if err := replaceGoFiles(filepath.Join(root, "mesh-host", pkg), filepath.Join(vendored, pkg)); err != nil {
return "", "", fmt.Errorf("mesh-host's validator of %s could not be put in its controller: %w", spec.Release, err)
}
}
}
// Built where no pull request reaches (JudgeBuildEnv): a judge kept for this head is the head's, nothing else.
if _, err := run(ctx, root, "docker", inToolchain(filepath.Join(root, dir), nil,
"go", "build", "-o", bin, "./cmd/mesh-controller")...); err != nil {
return "", "", fmt.Errorf("the controller of %s does not build%s: %w", spec.Release, with, err)
}
// Kept before anything of the change runs: nothing a pull request's script writes can reach the cache.
keepJudge(keptJudges(spec.JudgeCache), key, bin, say)
say("check", "judged by %s's controller at %s%s", spec.Release, twelve(key), with)
return bin, dir, nil
}
// JudgeBuildEnv is the environment every judge is built in (novox/hq ADR 0358 §6, issue 482): home, made fresh for the
// build, as HOME and configuration; goCache, the judge cache's Go cache that no check's container mounts; no global
// or system git configuration; no Go environment file; and no version stamped from git, so no git runs at all.
func JudgeBuildEnv(home, goCache string) []string {
return []string{"HOME=" + home, "XDG_CONFIG_HOME=" + filepath.Join(home, ".config"),
"GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_NOSYSTEM=1", "GOENV=off",
"GOCACHE=" + goCache, "GOPATH=" + filepath.Join(home, "go"),
"CGO_ENABLED=0", "GOFLAGS=-mod=vendor -buildvcs=false", "GOPROXY=off"}
}
// judgeBuildDirs are the home and the Go cache a check's judge builds use (ADR 0358 §6). With a judge cache: a home
// made fresh under its homes/ for this check, and its go-cache/, kept warm between checks — both on the host path the
// build agent declares, so the container runtime mounts what the agent sees, and nothing is left in a temporary
// directory of the host. Without one: both under the check's root, which every check empties before it clones. A
// judge cache inside the workspace is refused: a check's containers mount the workspace.
func judgeBuildDirs(cache, workspace, root, id string) (string, string, error) {
if cache == "" {
home, goCache := filepath.Join(root, "judge-home"), filepath.Join(root, "judge-go-cache")
for _, d := range []string{home, goCache} {
if err := os.MkdirAll(d, 0o700); err != nil {
return "", "", err
}
}
return home, goCache, nil
}
if withinDir(workspace, cache) {
return "", "", fmt.Errorf("the judge cache %s is inside the workspace %s, which a check's containers mount: "+
"no judge is built where a pull request could reach it", cache, workspace)
}
if !safeName.MatchString(id) {
id = "check"
}
home, goCache := filepath.Join(cache, "homes", id), filepath.Join(cache, "go-cache")
if err := os.RemoveAll(home); err != nil {
return "", "", err
}
for _, d := range []string{home, goCache} {
if err := os.MkdirAll(d, 0o700); err != nil {
return "", "", err
}
}
return home, goCache, nil
}
// replayPlace is where mesh-lab's replays run from and with (novox/hq issue 709), never anything a pull request can
// write: base, the place made for this check; lab, the copy of mesh-lab they run; home, their HOME and configuration;
// and their Go build and module caches. With a judge cache, base is a directory under its homes/ (the paths the same
// on the host and in the build agent) and the caches are the judge cache's. Without one, base is where a container
// volume of this check's own is mounted (volume), holding the copy, the home and throwaway caches, removed after the
// check: the build agent declares no other host path outside the workspace. hasReplays says whether mesh-lab held
// replays when it was copied, before anything of the change ran.
type replayPlace struct {
base, lab, home, goCache, modCache string
volume string
volumeMade, hasReplays bool
}
// ReplayMount is where a check's own replay volume is mounted in the replays' container when there is no judge cache.
var ReplayMount = "/mesh-replays"
// replayDirs makes the replays' place (issue 709, ADR 0358 §6). The replays' build is as trusted as a judge's: it
// compiles mesh-lab's trunk, reviewed code, and nothing of the pull request, which hands them only its catalogue, as
// data. Never the workspace: the gate runs before the replays, a pull request's own code among its containers.
func replayDirs(cache, workspace, root, id string) (replayPlace, error) {
if !safeName.MatchString(id) {
id = "check"
}
if cache == "" {
base := ReplayMount
return replayPlace{base: base, lab: filepath.Join(base, "mesh-lab"), home: filepath.Join(base, "home"),
goCache: filepath.Join(base, "go-cache"), modCache: filepath.Join(base, "go-modules"),
volume: "mesh-check-" + strings.ToLower(id) + "-replays"}, nil
}
if withinDir(workspace, cache) {
return replayPlace{}, fmt.Errorf("the judge cache %s is inside the workspace %s, which a check's containers "+
"mount: no replay is run where a pull request could reach it", cache, workspace)
}
base := filepath.Join(cache, "homes", id+"-replays")
p := replayPlace{base: base, lab: filepath.Join(base, "mesh-lab"), home: filepath.Join(base, "home"),
goCache: filepath.Join(cache, "go-cache"), modCache: filepath.Join(cache, "go-modules")}
if err := os.RemoveAll(base); err != nil {
return replayPlace{}, err
}
for _, d := range []string{p.home, p.goCache, p.modCache} {
if err := os.MkdirAll(d, 0o700); err != nil {
return replayPlace{}, err
}
}
return p, nil
}
// copyLab copies mesh-lab's checkout, freshly cloned and before anything of the change runs, to where the replays
// run it. With a judge cache, over git's own transport into its directory. Without one, into a volume of this check's
// own, made now: a container reads the checkout from the workspace, mounted read-only, and copies it with a fresh home
// and empty caches, owned by the user the check's containers run as.
func (p *replayPlace) copyLab(ctx context.Context, run Runner, spec CheckSpec, root, workspace, from string) error {
_, err := os.Stat(filepath.Join(from, "replays"))
p.hasReplays = err == nil
if p.volume == "" {
return copyCheckout(ctx, run, from, p.lab)
}
if spec.Toolchain == "" {
return errors.New("the mesh holds no Go toolchain to copy mesh-lab with")
}
user := spec.User
if user == "" {
user = fmt.Sprintf("%d:%d", os.Getuid(), os.Getgid())
}
_, _ = run(ctx, root, "docker", "volume", "rm", "--force", p.volume) // one an earlier delivery left
if _, err := run(ctx, root, "docker", "volume", "create", "--label", BuildLabel+"="+spec.ID, p.volume); err != nil {
return err
}
p.volumeMade = true
script := fmt.Sprintf("mkdir -p %s %s %s %s && cp -a %s/. %s/ && chown -R %s %s",
p.lab, p.home, p.goCache, p.modCache, from, p.lab, user, p.base)
_, err = run(ctx, root, "docker", LabelledArgs("docker", []string{"run", "--rm", "--network", "none",
"--volume", p.volume + ":" + p.base, "--volume", workspace + ":" + workspace + ":ro", "--workdir", "/",
spec.Toolchain, "sh", "-c", script}, spec.ID)...)
return err
}
// ReplayEnv is the environment mesh-lab's replays run in (novox/hq issue 709): home as HOME and configuration, the Go
// build and module caches given, no global or system git configuration, no Go environment file, and no version
// stamped from git — as a judge is built (JudgeBuildEnv), with the modules the replays fetch.
func ReplayEnv(home, goCache, modCache string) []string {
return []string{"HOME=" + home, "XDG_CONFIG_HOME=" + filepath.Join(home, ".config"),
"GIT_CONFIG_GLOBAL=/dev/null", "GIT_CONFIG_NOSYSTEM=1", "GOENV=off",
"GOCACHE=" + goCache, "GOMODCACHE=" + modCache, "GOPATH=" + filepath.Join(home, "go"),
"GOFLAGS=-buildvcs=false", "GOWORK=off"}
}
// copyCheckout copies a checkout at the commit it stands at into dir, over git's own transport — never by linking
// objects — so what is written into the source afterwards does not reach the copy.
func copyCheckout(ctx context.Context, run Runner, from, dir string) error {
commit := fullHeadAt(ctx, run, from)
if commit == "" {
return fmt.Errorf("%s names no commit", from)
}
if _, err := run(ctx, filepath.Dir(dir), "git", "clone", "--quiet", "--no-local", "--no-checkout", from, dir); err != nil {
return err
}
_, err := run(ctx, dir, "git", "checkout", "--quiet", "--detach", commit)
return err
}
// keptJudges is where a judge cache keeps the release-branch judges, empty when there is no judge cache.
func keptJudges(cache string) string {
if cache == "" {
return ""
}
return filepath.Join(cache, "judges")
}
// judgesKept is how many release-branch judges a judge cache keeps: a head moved past is not built from again.
const judgesKept = 4
// judgeKept puts the judge kept under key in the cache at bin, and says whether there was one.
func judgeKept(cache, key, bin string) bool {
if cache == "" {
return false
}
body, err := os.ReadFile(filepath.Join(cache, key, "mesh-controller"))
if err != nil {
return false
}
if err := os.MkdirAll(filepath.Dir(bin), 0o755); err != nil {
return false
}
if err := os.WriteFile(bin, body, 0o755); err != nil {
return false
}
now := time.Now()
_ = os.Chtimes(filepath.Join(cache, key), now, now)
return true
}
// keepJudge keeps a judge just built under key, and lets go of all but the judgesKept most recently used. A judge
// that cannot be kept is said and built again next time: the cache only saves time.
func keepJudge(cache, key, bin string, say func(step, format string, args ...any)) {
if cache == "" {
return
}
body, err := os.ReadFile(bin)
if err == nil {
err = os.MkdirAll(filepath.Join(cache, key), 0o755)
}
if err == nil {
kept := filepath.Join(cache, key, "mesh-controller")
if err = os.WriteFile(kept+".part", body, 0o755); err == nil {
err = os.Rename(kept+".part", kept)
}
}
if err != nil {
say("check", "the judge could not be kept for the next check, which builds it again: %v", err)
return
}
entries, err := os.ReadDir(cache)
if err != nil || len(entries) <= judgesKept {
return
}
type used struct {
name string
at time.Time
}
var all []used
for _, e := range entries {
if info, err := e.Info(); err == nil && e.IsDir() {
all = append(all, used{e.Name(), info.ModTime()})
}
}
sort.Slice(all, func(i, j int) bool { return all[i].at.After(all[j].at) })
for _, u := range all[min(judgesKept, len(all)):] {
if u.name != key {
_ = os.RemoveAll(filepath.Join(cache, u.name))
}
}
}
// fullHeadAt is the whole commit a checkout stands at, empty when it names none.
func fullHeadAt(ctx context.Context, run Runner, tree string) string {
out, err := run(ctx, tree, "git", "rev-parse", "--verify", "--quiet", "HEAD^{commit}")
if err != nil {
return ""
}
return strings.TrimSpace(out)
}
// releaseValidator says which node-engine validator a release-branch check's judge is built with (novox/hq ADR 0358
// §2), by the rule judgeFor builds it by.
func releaseValidator(spec CheckSpec, at, commits map[string]string) string {
switch {
case spec.Judge == "self":
return "the validator the change's own controller vendors"
case spec.Judge == "validator":
return "the change's own validator"
case at["mesh-host"] == spec.Release:
return fmt.Sprintf("mesh-host's validator of %s at %s", spec.Release, commits["mesh-host"])
}
return fmt.Sprintf("the validator %s's controller vendors (mesh-host has no branch %s)", spec.Release, spec.Release)
}
// releaseWords is how a release-branch check says what judged it and what it read (novox/hq ADR 0358 §4).
func releaseWords(release, judgedBy, validator string, at, commits map[string]string) string {
var read []string
for _, dir := range sortedKeys(at) {
read = append(read, fmt.Sprintf("%s %s at %s", dir, refWords(at[dir]), commits[dir]))
}
said := fmt.Sprintf("judged for the release branch %s by its controller at %s, with %s", release, judgedBy, validator)
if len(read) > 0 {
said += "; read beside it: " + strings.Join(read, ", ")
}
return said
}
// releaseMark opens each layer's summary of a release-branch check, so a status says it first.
func releaseMark(release, judgedBy string) string {
return fmt.Sprintf("[release %s, judged by its controller %s] ", release, judgedBy)
}
// refWords is a ref as a line says it: a commit by its first eight characters, a branch whole.
func refWords(ref string) string {
if commitLike.MatchString(ref) {
return short(ref)
}
return ref
}
// commitLike is a ref that names a commit.
var commitLike = regexp.MustCompile(`^[0-9a-f]{7,40}$`)
func sortedKeys(m map[string]string) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
// replaceGoFiles puts a package's Go files — never its tests — in place of another copy's.
func replaceGoFiles(from, into string) error {
old, err := filepath.Glob(filepath.Join(into, "*.go"))
@@ -1730,7 +1172,8 @@ func judge(ctx context.Context, run, plain Runner, root string, inToolchain func
say func(step, format string, args ...any)) (string, string, error) {
bin := filepath.Join(root, "bin", "mesh-controller")
build := func(dir string) error {
_, err := run(ctx, root, "docker", inToolchain(filepath.Join(root, dir), nil,
_, err := run(ctx, root, "docker", inToolchain(filepath.Join(root, dir),
[]string{"CGO_ENABLED=0", "GOFLAGS=-mod=vendor", "GOPROXY=off", "GOCACHE=" + filepath.Join(filepath.Dir(root), "go-cache")},
"go", "build", "-o", bin, "./cmd/mesh-controller")...)
return err
}
+2 -22
View File
@@ -37,11 +37,6 @@ type CheckBuilding struct {
Held map[string]string
Npmrc Npmrc
Seats map[string]string
// Named are the builds the build seat's verdicts of earlier checks of this pull request at this head name
// (novox/hq issue 703): a build kept under a module's key is kept again, instead of built, only when one of them
// names its key and every digest, and the build is this pull request's at this head. The record alone is never
// enough.
Named []NamedBuild
}
// CheckBuilt is one module a check built for its merge, as its verdict lists it (§1): its key and its artifacts, or
@@ -53,9 +48,6 @@ type CheckBuilt struct {
// NotBuilt says why it was not: no key, a build that failed, or a module after one of those.
NotBuilt string
Took time.Duration
// Kept is the check whose build this check kept instead of building, its key unchanged (novox/hq issue 703):
// the verdict names that build, by key and digest, as that check's. Empty when this check built it.
Kept string
}
// skipsTheBuilds is why a check builds nothing for its merge; empty when it builds.
@@ -108,9 +100,8 @@ func buildForTheMerge(ctx context.Context, run Runner, spec CheckSpec, with Chec
say("build", "for the merge: %s (tier %d)", b.Module, b.Tier)
var mirrored []string
result, err := build(ctx, run, scratchPublisher{to: with.Publish}, repository, b.Path, at,
filepath.Join(workspace, "for-the-merge"), held, with.Npmrc, forge, log, &mirrored,
&Reuse{keyFirst: true, Scratch: with.Scratch, Named: with.Named,
pull: PullRef{Owner: spec.Owner, Repo: spec.Repo, Number: spec.Number}, head: judged.Head}, with.Seats)
filepath.Join(workspace, "for-the-merge"), held, with.Npmrc, forge, log, &mirrored, &Reuse{keyFirst: true},
with.Seats)
built := CheckBuilt{Module: b.Module, Key: result.ReuseKey, Took: time.Since(began)}
switch {
case errors.Is(err, errNoKey):
@@ -132,17 +123,6 @@ func buildForTheMerge(ctx context.Context, run Runner, spec CheckSpec, with Chec
for _, m := range result.Built {
held[b.Module+"/"+m.Name] = m.Reference
}
if result.Reused != nil {
// **Kept, not built** (novox/hq issue 703): the earlier check's build, named as that check's, so the
// walk takes it as the verdict names it; this check's document names it too, so it stays while this
// check does (ADR 0350 §5).
built.Kept = result.Reused.From.Check
say("build", "%s kept for the merge under %s: %s", b.Module, twelve(result.ReuseKey[3:]),
result.Reused.From.Said())
out = append(out, built)
kept = append(kept, result.Reused.From)
continue
}
say("build", "%s built for the merge under %s (%s)", b.Module, twelve(result.ReuseKey[3:]), built.Took.Round(time.Second))
out = append(out, built)
if len(result.Built) > 0 {
-50
View File
@@ -210,53 +210,3 @@ func TestACheckMergeRunsNoHook(t *testing.T) {
t.Fatalf("a hook ran when the check made its merge: %s", said)
}
}
// What a check's tests change in the workspace never reaches what it builds for the merge (novox/hq issue 703): the
// builds are made from a copy of the merge, outside the workspace, cloned before anything of the change runs.
func TestWhatTheTestsChangeInTheWorkspaceNeverReachesTheBuild(t *testing.T) {
repo, head, _ := aTrunkAndABranch(t, map[string]string{"main.go": "package main // the merge\n"},
map[string]string{"trunk-only": "t"}, map[string]string{"head-only": "h"})
workspace := t.TempDir()
tree := filepath.Join(workspace, "check", "checked")
if out, err := exec.Command("git", "clone", "--quiet", repo, tree).CombinedOutput(); err != nil {
t.Fatalf("%v\n%s", err, out)
}
dir, cloned, err := checkoutForTheBuilds(t.Context(), Command, tree, head, workspace)
if err != nil {
t.Fatal(err)
}
defer os.RemoveAll(dir)
if withinDir(workspace, cloned) {
t.Fatalf("the builds' checkout %s is inside the workspace %s the tests mount", cloned, workspace)
}
// The tests rewrite the tree, its branch and every object it holds.
if err := os.WriteFile(filepath.Join(tree, "main.go"), []byte("package main // planted\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := filepath.WalkDir(filepath.Join(tree, ".git", "objects"), func(path string, d os.DirEntry, err error) error {
if err != nil || d.IsDir() {
return err
}
_ = os.Chmod(path, 0o644)
return os.WriteFile(path, []byte("planted"), 0o644)
}); err != nil {
t.Fatal(err)
}
// The build clones the merge from its own checkout, as buildForTheMerge does, and reads what the merge holds.
build := filepath.Join(t.TempDir(), "source")
if out, err := exec.Command("git", "clone", "--quiet", cloned, build).CombinedOutput(); err != nil {
t.Fatalf("%v\n%s", err, out)
}
if out, err := exec.Command("git", "-C", build, "checkout", "--quiet", head).CombinedOutput(); err != nil {
t.Fatalf("%v\n%s", err, out)
}
body, err := os.ReadFile(filepath.Join(build, "main.go"))
if err != nil {
t.Fatal(err)
}
if string(body) != "package main // the merge\n" {
t.Fatalf("the build read %q: what the tests wrote reached it", body)
}
}
-12
View File
@@ -590,15 +590,3 @@ func TestARepositoryInTwoLanguagesIsCheckedInBoth(t *testing.T) {
t.Errorf("a part in a toolchain the mesh does not hold answered %+v — never a pass", layer)
}
}
// A branch tried before its pull request may narrow what the repository's script runs (novox/hq issue 686); a
// pull request's check is handed nothing, and runs whole.
func TestOnlyATrysSelectionReachesTheScript(t *testing.T) {
if env := selection(CheckSpec{Number: 7}); len(env) != 0 {
t.Fatalf("a pull request's check was handed a selection: %v", env)
}
env := selection(CheckSpec{Tests: "^TestX$", StoreOnly: true})
if len(env) != 2 || env[0] != "MESH_CHECK_TESTS=^TestX$" || env[1] != "MESH_CHECK_STORE_ONLY=1" {
t.Fatalf("a try's selection: %v", env)
}
}
-178
View File
@@ -1,178 +0,0 @@
package builder
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"runtime"
"strings"
)
// **A built image's bases are linked into its repository before it is pushed** (novox/hq issue 694).
//
// A push reads every blob of the image out of this machine's store, unless the repository it goes to
// already holds that blob: the runtime asks the registry first, blob by blob, and reads locally only
// what the registry lacks. A runtime keeping its images in containerd's store — the default for a
// fresh install from docker 29 on — does not always hold the compressed layers of the base an image
// was built FROM: a layer whose unpacked form it already had is never fetched. The push of such an
// image then fails on a layer this machine never had ("not all of them are available locally"), while
// the very bytes sit in this registry, under the repository the base was mirrored into.
//
// So the base's blobs are mounted into the image's repository first. A mount moves no bytes, and once
// the repository holds a blob the push asks this machine for nothing of it — whichever store the
// runtime keeps. A runtime that holds every layer (the classic store) is not changed by this: its push
// finds the same blobs already there.
// BaseLender links what a build's bases hold into the repository its image is about to be pushed to.
type BaseLender interface {
// LendBases mounts every blob of each base this registry holds — for the platform this machine
// builds — into `repository`, and answers how many it mounted.
LendBases(ctx context.Context, repository string, bases []string) (int, error)
}
// platformed is an index's entry: a manifest and the platform it is for.
type platformed struct {
descriptor
Platform *struct {
OS string `json:"os"`
Architecture string `json:"architecture"`
} `json:"platform"`
}
// LendBases mounts each base's blobs into the repository. A base this registry does not hold under its
// repository — named by tag, or held somewhere else — lends nothing and is no error: the push reads it
// from this machine's store as before.
func (r Registry) LendBases(ctx context.Context, repository string, bases []string) (int, error) {
mounted := 0
for _, base := range bases {
name, digest, pinned := strings.Cut(base, "@")
if !pinned || !strings.HasPrefix(digest, "sha256:") {
continue
}
where, err := parseReference(name + "@" + digest)
if err != nil {
continue
}
from := where.repository
if from == repository {
continue
}
blobs, held, err := r.blobsOf(ctx, from, digest)
if err != nil {
return mounted, fmt.Errorf("reading %s: %w", base, err)
}
if !held {
continue
}
for _, blob := range blobs {
did, err := r.mount(ctx, repository, from, blob)
if err != nil {
return mounted, fmt.Errorf("linking %s of %s into %s: %w", blob, base, repository, err)
}
if did {
mounted++
}
}
}
return mounted, nil
}
// blobsOf is the config and layers of the manifest under repository@digest — for an index, of each
// manifest it names for this machine's platform. False when the registry holds no such manifest there.
func (r Registry) blobsOf(ctx context.Context, repository, digest string) ([]string, bool, error) {
body, held, err := r.manifest(ctx, repository, digest)
if err != nil || !held {
return nil, held, err
}
var document struct {
Manifests []platformed `json:"manifests"`
Config *descriptor `json:"config"`
Layers []descriptor `json:"layers"`
}
if err := json.Unmarshal(body, &document); err != nil {
return nil, false, fmt.Errorf("%s@%s is not a manifest: %w", repository, digest, err)
}
var blobs []string
if document.Config != nil {
blobs = append(blobs, document.Config.Digest)
}
for _, l := range document.Layers {
blobs = append(blobs, l.Digest)
}
for _, m := range document.Manifests {
if m.Platform == nil || m.Platform.OS != runtime.GOOS || m.Platform.Architecture != runtime.GOARCH {
continue
}
more, held, err := r.blobsOf(ctx, repository, m.Digest)
if err != nil {
return nil, false, err
}
if held {
blobs = append(blobs, more...)
}
}
return blobs, true, nil
}
// manifest reads one manifest document from this registry; false when it is not there.
func (r Registry) manifest(ctx context.Context, repository, digest string) ([]byte, bool, error) {
request, err := http.NewRequestWithContext(ctx, http.MethodGet,
"http://"+r.Address+"/v2/"+repository+"/manifests/"+digest, nil)
if err != nil {
return nil, false, err
}
request.Header.Set("Accept", manifestAccept)
response, err := r.client().Do(request)
if err != nil {
return nil, false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
}
defer response.Body.Close()
if response.StatusCode == http.StatusNotFound {
return nil, false, nil
}
if response.StatusCode != http.StatusOK {
return nil, false, fmt.Errorf("%s/%s@%s: %s", r.Address, repository, digest, response.Status)
}
body, err := io.ReadAll(io.LimitReader(response.Body, 4<<20))
return body, err == nil, err
}
// mount links one blob from one repository into another; false when the repository already held it.
func (r Registry) mount(ctx context.Context, repository, from, digest string) (bool, error) {
base := "http://" + r.Address + "/v2/" + repository
if there, err := r.has(ctx, base+"/blobs/"+digest); err != nil {
return false, err
} else if there {
return false, nil
}
start, err := http.NewRequestWithContext(ctx, http.MethodPost,
base+"/blobs/uploads/?mount="+digest+"&from="+from, nil)
if err != nil {
return false, err
}
begun, err := r.client().Do(start)
if err != nil {
return false, fmt.Errorf("cannot reach the registry at %s: %w", r.Address, err)
}
begun.Body.Close()
if begun.StatusCode == http.StatusCreated {
return true, nil
}
if begun.StatusCode == http.StatusAccepted {
// Not mounted, and an upload begun in its place: let go of it, so nothing is left half-open.
if location := begun.Header.Get("Location"); location != "" {
if strings.HasPrefix(location, "/") {
location = "http://" + r.Address + location
}
if cancel, err := http.NewRequestWithContext(ctx, http.MethodDelete, location, nil); err == nil {
if done, err := r.client().Do(cancel); err == nil {
done.Body.Close()
}
}
}
return false, fmt.Errorf("%s did not mount it from %s", r.Address, from)
}
return false, fmt.Errorf("%s answered %s when asked to mount it from %s", r.Address, begun.Status, from)
}
-140
View File
@@ -1,140 +0,0 @@
package builder
import (
"context"
"encoding/json"
"net/http/httptest"
"runtime"
"strings"
"testing"
)
// A built image's bases are linked into its repository before it is pushed (novox/hq issue 694): a
// runtime keeping containerd's image store may not hold a base layer it never had to fetch, and its push
// asks this machine only for what the repository lacks.
// aBaseForThisMachine puts an index into the registry under `repository`: one manifest for this
// machine's platform, one for another, and answers the index's digest and each manifest's blobs.
func aBaseForThisMachine(t *testing.T, dst *aRegistryOfRepositories, repository string) (string, []string, []string) {
t.Helper()
put := func(arch string) (string, []string) {
config := []byte(`{"architecture":"` + arch + `"}`)
layer := []byte(arch + " layer bytes")
for _, b := range [][]byte{config, layer} {
dst.blobs[digestOf(b)] = b
dst.link(repository, digestOf(b))
}
m, _ := json.Marshal(map[string]any{
"schemaVersion": 2, "mediaType": mediaManifestOCI,
"config": map[string]any{"mediaType": "application/vnd.oci.image.config.v1+json", "digest": digestOf(config), "size": len(config)},
"layers": []map[string]any{{"mediaType": "application/vnd.oci.image.layer.v1.tar+gzip", "digest": digestOf(layer), "size": len(layer)}},
})
dst.manifests[repository+"@"+digestOf(m)] = m
return digestOf(m), []string{digestOf(config), digestOf(layer)}
}
other := "s390x"
if runtime.GOARCH == other {
other = "amd64"
}
mine, mineBlobs := put(runtime.GOARCH)
theirs, theirBlobs := put(other)
index, _ := json.Marshal(map[string]any{
"schemaVersion": 2, "mediaType": mediaIndexOCI,
"manifests": []map[string]any{
{"mediaType": mediaManifestOCI, "digest": mine, "size": 1, "platform": map[string]string{"os": runtime.GOOS, "architecture": runtime.GOARCH}},
{"mediaType": mediaManifestOCI, "digest": theirs, "size": 1, "platform": map[string]string{"os": "linux", "architecture": other}},
},
})
dst.manifests[repository+"@"+digestOf(index)] = index
return digestOf(index), mineBlobs, theirBlobs
}
func TestABuildsBaseIsLinkedIntoItsRepositoryBeforeThePush(t *testing.T) {
dst := newRegistryOfRepositories()
server := httptest.NewServer(dst.handler())
defer server.Close()
address := strings.TrimPrefix(server.URL, "http://")
r := Registry{Address: address}
const from = "upstream/docker.io/library/golang"
index, mine, theirs := aBaseForThisMachine(t, dst, from)
bases := []string{
address + "/" + from + "@" + index,
// Not bases this registry holds: a package version, a tag, an image held nowhere here.
"@novox/mesh-sdk@1.2.3",
"golang:1-alpine",
"elsewhere.invalid/thing@sha256:" + strings.Repeat("e", 64),
}
n, err := r.LendBases(context.Background(), "mesh-tools-go/build", bases)
if err != nil {
t.Fatal(err)
}
// Every blob of this machine's platform is in the image's repository, so a push asks this machine's
// store for none of them; another platform's are not, since nothing built here stands on them.
for _, blob := range mine {
if !dst.links["mesh-tools-go/build"][blob] {
t.Fatalf("the base's blob %s is not in the image's repository", blob)
}
}
for _, blob := range theirs {
if dst.links["mesh-tools-go/build"][blob] {
t.Fatalf("another platform's blob %s was linked", blob)
}
}
if n != len(mine) || dst.mounts != len(mine) || dst.uploads != 0 {
t.Fatalf("%d linked, %d mounts, %d uploads; want %d mounts and no bytes moved", n, dst.mounts, dst.uploads, len(mine))
}
// Asked again, nothing is mounted twice.
if n, err := r.LendBases(context.Background(), "mesh-tools-go/build", bases); err != nil || n != 0 {
t.Fatalf("a second lending mounted %d: %v", n, err)
}
}
// A registry that will not mount is said, not hidden: the push is still tried, from the store.
func TestABaseTheRegistryWillNotMountIsAnError(t *testing.T) {
dst := newRegistryOfRepositories()
dst.noMount = true
server := httptest.NewServer(dst.handler())
defer server.Close()
address := strings.TrimPrefix(server.URL, "http://")
index, _, _ := aBaseForThisMachine(t, dst, "upstream/thing")
_, err := Registry{Address: address}.LendBases(context.Background(), "hello/server",
[]string{address + "/upstream/thing@" + index})
if err == nil || !strings.Contains(err.Error(), "did not mount") {
t.Fatalf("a refused mount passed silently: %v", err)
}
}
// lending records what a build lent, and when, against the push.
type lending struct {
*mirroring
order []string
}
func (l *lending) LendBases(_ context.Context, repository string, bases []string) (int, error) {
l.order = append(l.order, "lend "+repository+" "+strings.Join(bases, " "))
return len(bases), nil
}
func (l *lending) PublishImage(ctx context.Context, localTag, repository string) (string, error) {
l.order = append(l.order, "push "+repository)
return l.mirroring.PublishImage(ctx, localTag, repository)
}
func TestAnImageIsPushedOnlyAfterItsBasesAreLent(t *testing.T) {
r, workspace := aRepository(t, anImage, map[string]string{"modules/bus/Dockerfile": "ARG BASE\nFROM ${BASE}"})
r.contents["modules/bus/module.json"] = anImage
l := &lending{mirroring: &mirroring{recorded: r, at: "127.0.0.1:5000"}}
if _, err := Build(context.Background(), r.run, l, "https://forge.invalid/catalogue.git", "modules/bus", "", workspace,
nil, Npmrc{}, GitCredential{}, nil); err != nil {
t.Fatal(err)
}
want := []string{
"lend bus/server 127.0.0.1:5000/upstream/docker.io/vendor/server@sha256:" + strings.Repeat("1", 64),
"push bus/server",
}
if strings.Join(l.order, "\n") != strings.Join(want, "\n") {
t.Fatalf("the build did:\n%s\nwant:\n%s", strings.Join(l.order, "\n"), strings.Join(want, "\n"))
}
}
-826
View File
@@ -1,826 +0,0 @@
package builder
import (
"fmt"
"net"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
// **A pull request into a release branch is judged by the controller of that branch** (novox/hq ADR 0358, issue
// 704): the build seat's check builds the judge from mesh-controller read at the release branch — with mesh-host's
// validator of that branch when mesh-host has one — reads every repository beside it at that branch where it has one
// and at its trunk otherwise, and names the branch, the judge's commit and each repository read with its commit.
const releaseBranch = "feat/57-config-and-provisioning"
// gitInRepo runs git in a repository made for a test.
func gitInRepo(t *testing.T, dir string, args ...string) string {
t.Helper()
cmd := exec.Command("git", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GIT_AUTHOR_NAME=t", "GIT_AUTHOR_EMAIL=t@example.org",
"GIT_COMMITTER_NAME=t", "GIT_COMMITTER_EMAIL=t@example.org")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatalf("git %v: %v\n%s", args, err, out)
}
return strings.TrimSpace(string(out))
}
// commitFiles writes files into a repository and commits them on the branch checked out.
func commitFiles(t *testing.T, dir string, files map[string]string) string {
t.Helper()
for name, body := range files {
if err := os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(dir, name), []byte(body), 0o755); err != nil {
t.Fatal(err)
}
}
gitInRepo(t, dir, "add", "-A")
gitInRepo(t, dir, "commit", "--quiet", "-m", "x")
return gitInRepo(t, dir, "rev-parse", "HEAD")
}
// aRepositoryWithARelease is a repository whose main holds mainFiles and, when releaseFiles is not nil, a release
// branch over them holding releaseFiles. It answers the repository, main's head and the release branch's head.
func aRepositoryWithARelease(t *testing.T, mainFiles, releaseFiles map[string]string) (string, string, string) {
t.Helper()
dir := t.TempDir()
gitInRepo(t, dir, "init", "--quiet", "-b", "main")
main := commitFiles(t, dir, mainFiles)
if releaseFiles == nil {
return dir, main, ""
}
gitInRepo(t, dir, "checkout", "--quiet", "-b", releaseBranch)
head := commitFiles(t, dir, releaseFiles)
gitInRepo(t, dir, "checkout", "--quiet", "main")
return dir, main, head
}
// aJudgeOf is a controller that judges as told: its gate passes, saying who it is and the word of the node-engine
// validator it vendors — the one it was built with.
func aJudgeOf(who string) map[string]string {
return map[string]string{
"go.mod": "module example.org/judge\n\ngo 1.22\n\nrequire github.com/novox/mesh-host v0.0.0\n",
"vendor/modules.txt": "# github.com/novox/mesh-host v0.0.0\n## explicit; go 1.22\ngithub.com/novox/mesh-host/validate\n",
"vendor/github.com/novox/mesh-host/validate/v.go": "package validate\n\nconst Word = \"the vendored validator\"\n",
"vendor/github.com/novox/mesh-host/internal/declaration/d.go": "package declaration\n",
"cmd/mesh-controller/main.go": `package main
import (
"fmt"
"os"
"github.com/novox/mesh-host/validate"
)
func main() {
if len(os.Args) == 2 && os.Args[1] == "merge-gate" {
fmt.Println("usage: merge-gate --facts <file> --store <postgres>")
os.Exit(2)
}
fmt.Printf("{\"verdict\":\"pass\",\"summary\":\"%s with %s\"}\n", "` + who + `", validate.Word)
}
`,
}
}
// aHostSaying is the node-engine's validator, saying a word.
func aHostSaying(word string) map[string]string {
return map[string]string{
"validate/v.go": "package validate\n\nconst Word = \"" + word + "\"\n",
"internal/declaration/d.go": "package declaration\n",
"validate/v_test.go": "package validate\n",
"internal/declaration/x_test.go": "package declaration\n",
}
}
// aRuntimeThatBuildsTheJudge puts a container runtime on PATH that runs every container's command on this machine,
// in the directory and with the environment the check gives it — the judge's build too — and a store and a bus that
// answer. It answers the file each `go` the check runs is written to.
func aRuntimeThatBuildsTheJudge(t *testing.T) string {
t.Helper()
if _, err := exec.LookPath("go"); err != nil {
t.Skip("no Go toolchain on this machine to build a judge with")
}
bus, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { bus.Close() })
go func() {
for {
c, err := bus.Accept()
if err != nil {
return
}
c.Close()
}
}()
dir := t.TempDir()
built := filepath.Join(dir, "built")
runtime := `#!/bin/sh
cmd="$1"; shift
case "$cmd" in
ps|rm|exec) exit 0 ;;
volume)
if [ -n "$VOLUMES" ]; then echo "$*" >> "$VOLUMES"; fi
if [ "$1" = rm ] && [ -n "$REPLAY_MOUNT" ]; then rm -rf "$REPLAY_MOUNT"; fi
exit 0 ;;
port) echo "` + bus.Addr().String() + `"; exit 0 ;;
run)
if [ -n "$RUNS" ]; then echo "$*" >> "$RUNS"; fi
workdir=""; detached=""
while [ $# -gt 0 ]; do
case "$1" in
-d) detached=1; shift ;;
--rm) shift ;;
--workdir) workdir="$2"; shift 2 ;;
--env) export "$2"; shift 2 ;;
--network|--volume|--user|--label|--name|-p|-e) shift 2 ;;
*) break ;;
esac
done
if [ -n "$detached" ]; then echo a-container; exit 0; fi
shift
if [ "$1" = go ]; then echo "GOCACHE=$GOCACHE GOENV=$GOENV $*" >> "` + built + `"; fi
if [ "$1" = go ] && [ "$2" = build ] && [ -n "$JUDGE_HOMES" ]; then echo "$HOME" >> "$JUDGE_HOMES"; fi
if [ "$1" = go ] && [ "$2" = test ] && [ -n "$REPLAYS" ]; then
echo "HOME=$HOME GOCACHE=$GOCACHE GOMODCACHE=$GOMODCACHE GOENV=$GOENV GOFLAGS=$GOFLAGS DIR=$workdir" >> "$REPLAYS"
fi
cd "$workdir" && exec "$@" ;;
esac
exit 1
`
if err := os.WriteFile(filepath.Join(dir, "docker"), []byte(runtime), 0o755); err != nil {
t.Fatal(err)
}
t.Setenv("PATH", dir+string(os.PathListSeparator)+os.Getenv("PATH"))
t.Setenv("GOTOOLCHAIN", "local")
return built
}
// builds is how many times the check ran go to build a judge.
func builds(t *testing.T, file string) int {
t.Helper()
body, err := os.ReadFile(file)
if err != nil {
return 0
}
return strings.Count(string(body), "go build")
}
// theReleasedMesh is the core beside a release-branch check: a controller whose main and release branch judge
// differently, a node-engine with a validator of each, and a lab with no release branch.
func theReleasedMesh(t *testing.T) (map[string]Beside, string, string, string) {
t.Helper()
controller, _, controllerRelease := aRepositoryWithARelease(t, aJudgeOf("main's controller"),
aJudgeOf("the release branch's controller"))
host, _, hostRelease := aRepositoryWithARelease(t, aHostSaying("main's validator"),
aHostSaying("the release branch's validator"))
lab, labMain, _ := aRepositoryWithARelease(t, map[string]string{"README": "the lab"}, nil)
return map[string]Beside{
"mesh-controller": {Repository: controller, Ref: releaseBranch},
"mesh-controller-main": {Repository: controller, Ref: "main"},
"mesh-host": {Repository: host, Ref: releaseBranch, Else: "main"},
"mesh-lab": {Repository: lab, Ref: releaseBranch, Else: "main"},
}, controllerRelease, hostRelease, labMain
}
// aPullRequestInto is a repository whose pull request's head is cut from base, with a change.
func aPullRequestInto(t *testing.T, files map[string]string, base string, change map[string]string) (string, string) {
t.Helper()
repo, _, _ := aRepositoryWithARelease(t, files, map[string]string{"released": "r"})
gitInRepo(t, repo, "checkout", "--quiet", "-b", "change", base)
head := commitFiles(t, repo, change)
gitInRepo(t, repo, "checkout", "--quiet", "main")
return repo, head
}
func TestACataloguePullRequestIntoAReleaseBranchIsJudgedByThatBranchsController(t *testing.T) {
built := aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
beside, controllerRelease, hostRelease, labMain := theReleasedMesh(t)
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
cache := t.TempDir()
check := func(id string) CheckVerdict {
t.Helper()
v, err := Check(t.Context(), Command, CheckSpec{ID: id, Repository: catalogue, Ref: head, Owner: "novox",
Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch, JudgeCache: cache,
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside, Toolchain: "golang"},
t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
return v
}
v := check("check-704-catalogue")
// The release branch's controller judged, with mesh-host's validator of the release branch.
if v.Gate == nil || v.Gate.Verdict != "pass" ||
!strings.HasSuffix(v.Gate.Summary, "the release branch's controller with the release branch's validator") {
t.Fatalf("the gate was not judged by the release branch's controller with its node-engine's validator: %+v\n%s",
v.Gate, v.Report)
}
// It names the release branch, the judge's commit, and each repository read beside it at its branch and commit.
mark := "[release " + releaseBranch + ", judged by its controller " + controllerRelease[:12] + "] "
if !strings.HasPrefix(v.Gate.Summary, mark) || !strings.HasPrefix(v.Repo.Summary, mark) {
t.Fatalf("the statuses do not name the release branch and its judge first:\n%s\n%s", v.Gate.Summary, v.Repo.Summary)
}
if v.Release != releaseBranch || v.JudgedBy != controllerRelease[:12] {
t.Fatalf("the verdict names release %q judged by %q, not %s by %s", v.Release, v.JudgedBy, releaseBranch,
controllerRelease[:12])
}
if v.BesideAt["mesh-host"] != releaseBranch || v.Beside["mesh-host"] != hostRelease[:12] {
t.Fatalf("mesh-host was read at %q (%s), not its release branch at %s", v.BesideAt["mesh-host"],
v.Beside["mesh-host"], hostRelease[:12])
}
// A repository with no release branch is read at its trunk, and says so.
if v.BesideAt["mesh-lab"] != "main" || v.Beside["mesh-lab"] != labMain[:12] {
t.Fatalf("the lab, with no release branch, was read at %q (%s), not main at %s", v.BesideAt["mesh-lab"],
v.Beside["mesh-lab"], labMain[:12])
}
if !strings.Contains(v.Report, "read beside it: ") || !strings.Contains(v.Report, "mesh-host "+releaseBranch+" at "+hostRelease[:12]) {
t.Fatalf("the report does not say what was read beside it:\n%s", v.Report)
}
// It says which validator judged: mesh-host's of the release branch, with its commit.
if want := "mesh-host's validator of " + releaseBranch + " at " + hostRelease[:12]; v.Validator != want ||
!strings.Contains(v.Report, want) {
t.Fatalf("the verdict says the validator was %q, not %q:\n%s", v.Validator, want, v.Report)
}
// Each repository read beside it is named with the branch and the commit read.
for dir := range beside {
if v.BesideAt[dir] == "" || v.Beside[dir] == "" ||
!strings.Contains(v.Report, dir+" "+v.BesideAt[dir]+" at "+v.Beside[dir]) {
t.Fatalf("%s is not named with its branch and commit: at %q, commit %q\n%s", dir, v.BesideAt[dir],
v.Beside[dir], v.Report)
}
}
if n := builds(t, built); n != 1 {
t.Fatalf("the judge was built %d times for one check", n)
}
// **The judge is built with the judge cache's Go cache** (ADR 0358 §6), which no check's container mounts, and no
// Go environment file.
said, _ := os.ReadFile(built)
if !strings.Contains(string(said), "GOCACHE="+filepath.Join(cache, "go-cache")+" GOENV=off go build") {
t.Fatalf("the release judge was not built with the judge cache's Go cache:\n%s", said)
}
// **Clause 5**: the head was merged onto the release branch's head, which the verdict names as its trunk commit,
// so the release branch's move puts it out of date as a trunk's move does (ADR 0323 §4, the forge's holder).
if releaseHead := gitInRepo(t, catalogue, "rev-parse", releaseBranch); v.Trunk != releaseHead[:12] {
t.Fatalf("the verdict names trunk %q, not the release branch's head %s", v.Trunk, releaseHead[:12])
}
// **Built once per release-branch head**: a second check while that head stands takes the judge kept.
v = check("check-704-catalogue-again")
if !strings.HasSuffix(v.Gate.Summary, "the release branch's controller with the release branch's validator") {
t.Fatalf("the kept judge did not judge: %+v\n%s", v.Gate, v.Report)
}
if n := builds(t, built); n != 1 {
t.Fatalf("the judge of the same release-branch head was built again: %d builds", n)
}
}
func TestAControllerPullRequestIntoAReleaseBranchReadsTheNodeEngineAtThatBranch(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
beside, _, hostRelease, _ := theReleasedMesh(t)
delete(beside, "mesh-controller")
controller, head := aPullRequestInto(t, aJudgeOf("the change's controller"), releaseBranch,
map[string]string{"cmd/x.txt": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-controller", Repository: controller, Ref: head,
Owner: "novox", Repo: "mesh-controller", Number: 315, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"cmd/x.txt"}, Modules: []string{"mesh-controller"}, Judge: "self",
Beside: beside, Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || !strings.HasSuffix(v.Gate.Summary, "the change's controller with the vendored validator") {
t.Fatalf("a controller change into a release branch did not judge itself: %+v\n%s", v.Gate, v.Report)
}
if v.BesideAt["mesh-host"] != releaseBranch || v.Beside["mesh-host"] != hostRelease[:12] {
t.Fatalf("mesh-host was read beside it at %q (%s), not at the release branch's %s", v.BesideAt["mesh-host"],
v.Beside["mesh-host"], hostRelease[:12])
}
if v.Release != releaseBranch || v.JudgedBy == "" || v.JudgedBy != v.Merge[:12] {
t.Fatalf("the verdict names release %q judged by %q, not its own merge %s", v.Release, v.JudgedBy, v.Merge)
}
}
func TestAPullRequestIntoTheTrunkIsJudgedAsBefore(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("main's controller"),
aJudgeOf("the release branch's controller"))
host, hostMain, _ := aRepositoryWithARelease(t, aHostSaying("main's validator"),
aHostSaying("the release branch's validator"))
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-trunk", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 1, Base: "main", JudgeCache: t.TempDir(),
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Toolchain: "golang",
Beside: map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain},
"mesh-host": {Repository: host, Ref: hostMain}}}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Summary != "main's controller with the vendored validator" {
t.Fatalf("a trunk pull request was not judged by the controller beside it as before: %+v\n%s", v.Gate, v.Report)
}
if v.Release != "" || v.JudgedBy != "" || v.BesideAt != nil || strings.Contains(v.Report, "release branch") {
t.Fatalf("a trunk pull request's verdict speaks of a release branch: %q %q %v", v.Release, v.JudgedBy, v.BesideAt)
}
if v.Beside["mesh-host"] != hostMain[:12] {
t.Fatalf("mesh-host was read at %s, not the commit asked %s", v.Beside["mesh-host"], hostMain[:12])
}
}
// **Without mesh-host's release branch, the release branch's controller keeps the validator it vendors** (ADR 0358
// §2), and the verdict says so.
func TestWithoutTheNodeEnginesReleaseBranchTheJudgeKeepsItsVendoredValidator(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
beside, controllerRelease, _, _ := theReleasedMesh(t)
host, hostMain, _ := aRepositoryWithARelease(t, aHostSaying("main's validator"), nil)
beside["mesh-host"] = Beside{Repository: host, Ref: releaseBranch, Else: "main"}
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-vendored", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || !strings.HasSuffix(v.Gate.Summary, "the release branch's controller with the vendored validator") {
t.Fatalf("without mesh-host's release branch the judge did not keep its vendored validator: %+v\n%s", v.Gate, v.Report)
}
if v.JudgedBy != controllerRelease[:12] || v.BesideAt["mesh-host"] != "main" || v.Beside["mesh-host"] != hostMain[:12] {
t.Fatalf("judged by %q, mesh-host read at %q (%s)", v.JudgedBy, v.BesideAt["mesh-host"], v.Beside["mesh-host"])
}
if !strings.Contains(v.Validator, "controller vendors") || !strings.Contains(v.Report, v.Validator) {
t.Fatalf("the verdict does not say the vendored validator judged: %q\n%s", v.Validator, v.Report)
}
}
// **Clause 7** (ADR 0358 §7): a release branch's own pull request into its trunk is checked as any pull request into a
// trunk — by the controller beside it as the mesh runs it, the repositories at what a trunk check reads — and its
// verdict names no release branch.
func TestAReleaseBranchsPullRequestIntoItsTrunkIsJudgedByTheControllerTheMeshRuns(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"),
aJudgeOf("the release branch's controller"))
host, hostMain, _ := aRepositoryWithARelease(t, aHostSaying("main's validator"),
aHostSaying("the release branch's validator"))
catalogue, _, releaseHead := aRepositoryWithARelease(t, map[string]string{
"modules/gitea/module.json": `{"module":"gitea"}`, CheckScript: "true\n"},
map[string]string{"modules/gitea/index.ts": "released"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-release-into-trunk", Repository: catalogue,
Ref: releaseHead, Owner: "novox", Repo: "mesh-catalog", Number: 300, Base: "main", JudgeCache: t.TempDir(),
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Toolchain: "golang",
Beside: map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain},
"mesh-host": {Repository: host, Ref: hostMain}}}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Summary != "the controller the mesh runs with the vendored validator" {
t.Fatalf("the release branch's pull request into its trunk was not judged by the controller the mesh runs: %+v\n%s",
v.Gate, v.Report)
}
if v.Release != "" || v.JudgedBy != "" || v.BesideAt != nil || v.Validator != "" ||
strings.Contains(v.Report, "release branch") || strings.Contains(v.Gate.Summary, "[release") {
t.Fatalf("its verdict names a release branch: %q %q %v %q", v.Release, v.JudgedBy, v.BesideAt, v.Validator)
}
if v.Beside["mesh-host"] != hostMain[:12] || v.Beside["mesh-controller"] != controllerMain[:12] {
t.Fatalf("beside it were read %v, not the trunk's %s and %s", v.Beside, controllerMain[:12], hostMain[:12])
}
}
// **mesh-lab is read at its trunk, always** (ADR 0358): its replays run with the container runtime's socket, so a
// branch of the release's name in mesh-lab — which anyone may create — is never what a check reads.
func TestAReleaseBranchCheckReadsMeshLabAtItsTrunkEvenWhenItHasThatBranch(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
beside, _, _, _ := theReleasedMesh(t)
lab, labMain, labRelease := aRepositoryWithARelease(t, map[string]string{"README": "the lab"},
map[string]string{"README": "a branch anyone could push"})
beside["mesh-lab"] = Beside{Repository: lab, Ref: releaseBranch, Else: "main"}
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-lab", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.BesideAt["mesh-lab"] != "main" || v.Beside["mesh-lab"] != labMain[:12] || v.Beside["mesh-lab"] == labRelease[:12] {
t.Fatalf("mesh-lab was read at %q (%s), not its trunk main at %s", v.BesideAt["mesh-lab"], v.Beside["mesh-lab"],
labMain[:12])
}
}
// aPlantedWorkspace is a workspace a pull request's own script has written into, as it can: a git configuration in its
// HOME whose fsmonitor hook says each directory git runs in, a git configuration under XDG_CONFIG_HOME doing the same,
// and a Go environment file that would hand the judge's build a tool of its own. It answers the workspace and the
// file the hook writes to.
func aPlantedWorkspace(t *testing.T) (string, string) {
t.Helper()
workspace, logs := t.TempDir(), t.TempDir()
ran := filepath.Join(logs, "hook-ran")
hook := filepath.Join(workspace, "hook.sh")
files := map[string]string{
hook: "#!/bin/sh\necho \"$PWD\" >> " + ran + "\n",
filepath.Join(workspace, ".gitconfig"): "[core]\n\tfsmonitor = " + hook + "\n",
filepath.Join(workspace, ".config", "git", "config"): "[core]\n\tfsmonitor = " + hook + "\n",
filepath.Join(workspace, ".config", "go", "env"): "GOFLAGS=-toolexec=" + hook + "\n",
}
for name, body := range files {
if err := os.MkdirAll(filepath.Dir(name), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(name, []byte(body), 0o755); err != nil {
t.Fatal(err)
}
}
return workspace, ran
}
// judgeRanTheHook says whether the planted hook ran in a judge's tree: the controller beside the check, or the
// change's own when it judges itself.
func judgeRanTheHook(t *testing.T, ran string) string {
t.Helper()
body, _ := os.ReadFile(ran)
for _, line := range strings.Split(string(body), "\n") {
if strings.HasSuffix(line, "/check/mesh-controller") {
return string(body)
}
}
return ""
}
// **Every judge is built where no pull request reaches** (issue 482, ADR 0358 review): a git configuration with a
// hook, and a Go environment file, planted in the workspace by an earlier pull request's script, do not run during
// the release judge's build — while the pull request's own script, in the workspace, still meets them.
func TestAHookPlantedInTheWorkspaceDoesNotRunDuringTheReleaseJudgesBuild(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
homes := filepath.Join(t.TempDir(), "homes")
t.Setenv("JUDGE_HOMES", homes)
registry := aFactsRegistry(t)
workspace, ran := aPlantedWorkspace(t)
beside, _, _, _ := theReleasedMesh(t)
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "git status >/dev/null 2>&1; true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-planted", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
Toolchain: "golang"}, workspace, registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Verdict != "pass" {
t.Fatalf("the check did not judge: %+v\n%s", v.Gate, v.Report)
}
if body, _ := os.ReadFile(ran); !strings.Contains(string(body), "/check/mesh-catalog") {
t.Fatalf("the planted hook never ran, so this test proves nothing: %q", body)
}
if said := judgeRanTheHook(t, ran); said != "" {
t.Fatalf("the hook planted in the workspace ran during the judge's build:\n%s", said)
}
if body, _ := os.ReadFile(homes); len(body) == 0 || strings.Contains(string(body), workspace) {
t.Fatalf("the judge was built with a HOME in the workspace: %q", body)
}
}
// The same for the judge built with a node-engine change's own validator, into a release branch (ADR 0358 §2).
func TestAHookPlantedInTheWorkspaceDoesNotRunDuringTheValidatorJudgesBuild(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
homes := filepath.Join(t.TempDir(), "homes")
t.Setenv("JUDGE_HOMES", homes)
registry := aFactsRegistry(t)
workspace, ran := aPlantedWorkspace(t)
beside, _, _, _ := theReleasedMesh(t)
delete(beside, "mesh-host")
files := aHostSaying("the release branch's validator")
files["module.json"] = `{"module":"mesh-host"}`
files[CheckScript] = "git status >/dev/null 2>&1; true\n"
host, head := aPullRequestInto(t, files, releaseBranch, aHostSaying("this change's validator"))
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-planted-validator", Repository: host, Ref: head,
Owner: "novox", Repo: "mesh-host", Number: 111, Base: releaseBranch, Release: releaseBranch, Judge: "validator",
JudgeCache: t.TempDir(), Paths: []string{"validate/v.go"}, Modules: []string{"mesh-host"}, Beside: beside,
Toolchain: "golang"}, workspace, registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || !strings.HasSuffix(v.Gate.Summary, "the release branch's controller with this change's validator") {
t.Fatalf("a node-engine change into a release branch was not judged by that branch's controller with its own "+
"validator: %+v\n%s", v.Gate, v.Report)
}
if body, _ := os.ReadFile(ran); !strings.Contains(string(body), "/check/mesh-host") {
t.Fatalf("the planted hook never ran, so this test proves nothing: %q", body)
}
if said := judgeRanTheHook(t, ran); said != "" {
t.Fatalf("the hook planted in the workspace ran during a judge's build:\n%s", said)
}
if body, _ := os.ReadFile(homes); strings.Count(string(body), "\n") != 2 || strings.Contains(string(body), workspace) {
t.Fatalf("the judges were not both built with a HOME outside the workspace: %q", body)
}
}
// **mesh-lab is never read at a release branch**, asked with a fallback or without (ADR 0358).
func TestTheBuildSeatNeverReadsMeshLabAtAReleaseBranch(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
registry := aFactsRegistry(t)
lab, labMain, _ := aRepositoryWithARelease(t, map[string]string{"README": "the lab"},
map[string]string{"README": "a branch anyone could push"})
for _, asked := range []Beside{{Repository: lab, Ref: releaseBranch}, {Repository: lab, Ref: releaseBranch, Else: "main"}} {
beside, _, _, _ := theReleasedMesh(t)
beside["mesh-lab"] = asked
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, releaseBranch, map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-704-lab-asked", Repository: catalogue, Ref: head,
Owner: "novox", Repo: "mesh-catalog", Number: 286, Base: releaseBranch, Release: releaseBranch,
JudgeCache: t.TempDir(), Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Beside: beside,
Toolchain: "golang"}, t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.BesideAt["mesh-lab"] != "main" || v.Beside["mesh-lab"] != labMain[:12] {
t.Fatalf("mesh-lab asked as %+v was read at %q (%s), not main at %s", asked, v.BesideAt["mesh-lab"],
v.Beside["mesh-lab"], labMain[:12])
}
}
}
// **Judges are built warm and where no pull request reaches** (ADR 0358 §6): every judge build uses the judge cache's
// one Go cache, kept between checks; its home is made fresh under the judge cache and removed after; nothing is left in
// a temporary directory of the host; and no container a check runs for the pull request mounts the judge cache.
func TestJudgesAreBuiltWarmFromAJudgeCacheNoCheckContainerMounts(t *testing.T) {
built := aRuntimeThatBuildsTheJudge(t)
runs := filepath.Join(t.TempDir(), "runs")
t.Setenv("RUNS", runs)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
registry := aFactsRegistry(t)
cache := t.TempDir()
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
beside := map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain}}
for i := range 2 {
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": fmt.Sprint(i)})
v, err := Check(t.Context(), Command, CheckSpec{ID: fmt.Sprintf("check-704-warm-%d", i), Repository: catalogue,
Ref: head, Owner: "novox", Repo: "mesh-catalog", Number: 1, Base: "main", JudgeCache: cache,
Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"}, Toolchain: "golang", Beside: beside},
t.TempDir(), registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Summary != "the controller the mesh runs with the vendored validator" {
t.Fatalf("check %d was not judged: %+v\n%s", i, v.Gate, v.Report)
}
if i == 0 {
if entries, _ := os.ReadDir(filepath.Join(cache, "go-cache")); len(entries) == 0 {
t.Fatal("the judge's build left nothing in the judge cache's Go cache to build the next judge warm")
}
}
}
// Both judges built with the one Go cache of the judge cache.
said, _ := os.ReadFile(built)
if n := strings.Count(string(said), "GOCACHE="+filepath.Join(cache, "go-cache")+" GOENV=off go build"); n != 2 {
t.Fatalf("%d of 2 judge builds used the judge cache's Go cache:\n%s", n, said)
}
// Nothing left behind: no home under the judge cache, nothing in the host's temporary directory.
if left, _ := os.ReadDir(filepath.Join(cache, "homes")); len(left) != 0 {
t.Fatalf("judge builds left %d home(s) under the judge cache", len(left))
}
if left, _ := os.ReadDir(tmp); len(left) != 0 {
t.Fatalf("a check left %d entr(ies) in the temporary directory, e.g. %s", len(left), left[0].Name())
}
// Only a judge build mounts the judge cache: no container the check runs for the pull request does.
body, _ := os.ReadFile(runs)
judgeBuilds := 0
for _, line := range strings.Split(strings.TrimSpace(string(body)), "\n") {
judging := strings.Contains(line, " go build ")
if judging {
judgeBuilds++
if !strings.Contains(line, "--volume "+filepath.Join(cache, "go-cache")+":") {
t.Fatalf("a judge build did not mount the judge cache's Go cache: %s", line)
}
continue
}
if strings.Contains(line, cache) {
t.Fatalf("a container the check runs for the pull request names the judge cache: %s", line)
}
}
if judgeBuilds != 2 {
t.Fatalf("%d judge builds were run, not 2", judgeBuilds)
}
}
// Without a judge cache, a judge is built cold under the check's own root — never in a temporary directory of the
// host — and a judge cache inside the workspace is refused.
func TestWithoutAJudgeCacheAJudgeIsBuiltColdUnderTheChecksRoot(t *testing.T) {
built := aRuntimeThatBuildsTheJudge(t)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
registry := aFactsRegistry(t)
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
beside := map[string]Beside{"mesh-controller": {Repository: controller, Ref: controllerMain}}
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
workspace := t.TempDir()
spec := CheckSpec{ID: "check-704-cold", Repository: catalogue, Ref: head, Owner: "novox", Repo: "mesh-catalog",
Number: 1, Base: "main", Paths: []string{"modules/gitea/index.ts"}, Modules: []string{"gitea"},
Toolchain: "golang", Beside: beside}
v, err := Check(t.Context(), Command, spec, workspace, registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Verdict != "pass" {
t.Fatalf("the check was not judged: %+v\n%s", v.Gate, v.Report)
}
said, _ := os.ReadFile(built)
if !strings.Contains(string(said), "GOCACHE="+filepath.Join(workspace, "check", "judge-go-cache")+" GOENV=off go build") {
t.Fatalf("without a judge cache the judge was not built under the check's root:\n%s", said)
}
if left, _ := os.ReadDir(tmp); len(left) != 0 {
t.Fatalf("a check left %d entr(ies) in the temporary directory", len(left))
}
spec.JudgeCache = filepath.Join(workspace, "judge-cache")
if _, err := Check(t.Context(), Command, spec, workspace, registry, GitCredential{}, nil); err == nil ||
!strings.Contains(err.Error(), "inside the workspace") {
t.Fatalf("a judge cache inside the workspace was taken: %v", err)
}
}
// aLabWithReplays is mesh-lab with replays: a Go module of its own whose one test passes, needing nothing fetched.
func aLabWithReplays(t *testing.T) (string, string) {
t.Helper()
lab, labMain, _ := aRepositoryWithARelease(t, map[string]string{
"replays/go.mod": "module example.org/replays\n\ngo 1.22\n",
"replays/replay_test.go": "package replays\n\nimport \"testing\"\n\nfunc TestAReplay(t *testing.T) {}\n",
}, nil)
return lab, labMain
}
// **mesh-lab's replays run where no pull request reaches** (novox/hq issue 709, ADR 0358 §6): they hold the container
// runtime's socket, so a Go environment file with a -toolexec hook and a Go cache and module cache planted in the
// workspace by a pull request's script are neither run nor read — the replays run from a copy of mesh-lab outside
// the workspace with a fresh home and the judge cache's caches, all removed or kept outside every check container.
func TestTheReplaysRunWhereNoPullRequestReaches(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
replays := filepath.Join(t.TempDir(), "replays")
t.Setenv("REPLAYS", replays)
registry := aFactsRegistry(t)
workspace, ran := aPlantedWorkspace(t)
for _, poisoned := range []string{filepath.Join(workspace, "go-cache", "README"),
filepath.Join(workspace, "go-modules", "cache", "download", "planted")} {
if err := os.MkdirAll(filepath.Dir(poisoned), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(poisoned, []byte("planted by a pull request"), 0o644); err != nil {
t.Fatal(err)
}
}
cache := t.TempDir()
controller, controllerMain, _ := aRepositoryWithARelease(t, aJudgeOf("the controller the mesh runs"), nil)
lab, labMain := aLabWithReplays(t)
catalogue, head := aPullRequestInto(t, map[string]string{"modules/gitea/module.json": `{"module":"gitea"}`,
CheckScript: "true\n"}, "main", map[string]string{"modules/gitea/index.ts": "x"})
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-709", Repository: catalogue, Ref: head, Owner: "novox",
Repo: "mesh-catalog", Number: 1, Base: "main", JudgeCache: cache, Paths: []string{"modules/gitea/index.ts"},
Modules: []string{"gitea"}, Toolchain: "golang", Beside: map[string]Beside{
"mesh-controller": {Repository: controller, Ref: controllerMain}, "mesh-lab": {Repository: lab, Ref: labMain}}},
workspace, registry, GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Verdict != "pass" || !strings.Contains(v.Report, "--- the replays") {
t.Fatalf("the replays did not run and pass: %+v\n%s", v.Gate, v.Report)
}
body, _ := os.ReadFile(replays)
said := strings.TrimSpace(string(body))
if said == "" || strings.Count(said, "\n") != 0 {
t.Fatalf("the replays ran %q, not once", said)
}
for _, want := range []string{"GOCACHE=" + filepath.Join(cache, "go-cache") + " ",
"GOMODCACHE=" + filepath.Join(cache, "go-modules") + " ", "GOENV=off ", "GOFLAGS=-buildvcs=false ",
"HOME=" + filepath.Join(cache, "homes", "check-709-replays", "home") + " ",
"DIR=" + filepath.Join(cache, "homes", "check-709-replays", "mesh-lab", "replays")} {
if !strings.Contains(said, want) {
t.Fatalf("the replays did not run with %q: %s", want, said)
}
}
if strings.Contains(said, workspace) {
t.Fatalf("the replays ran with something of the workspace: %s", said)
}
if body, _ := os.ReadFile(ran); strings.Contains(string(body), "replays") || strings.Contains(string(body), "mesh-lab") {
t.Fatalf("a hook planted in the workspace ran during the replays:\n%s", body)
}
if left, _ := os.ReadDir(filepath.Join(cache, "homes")); len(left) != 0 {
t.Fatalf("the replays left %d entr(ies) under the judge cache's homes", len(left))
}
}
// aSelfJudgingControllerThatWritesIntoTheReplays is a controller pull request whose own judge, while the gate runs it,
// writes a test into the replays beside the check — the reviewer's case for issue 709: code of the pull request's,
// run before the replays, reaching what they run.
func aSelfJudgingControllerThatWritesIntoTheReplays(t *testing.T, marker string) (string, string) {
t.Helper()
files := aJudgeOf("the change's controller")
evil := "package replays\n\nimport (\n\t\"os\"\n\t\"testing\"\n)\n\nfunc TestEvil(t *testing.T) {\n\t_ = os.WriteFile(" +
fmt.Sprintf("%q", marker) + ", []byte(\"ran\"), 0o644)\n}\n"
files["cmd/mesh-controller/main.go"] = strings.Replace(files["cmd/mesh-controller/main.go"], "func main() {",
"func main() {\n\tif len(os.Args) > 2 && os.Args[1] == \"merge-gate\" {\n\t\t_ = os.WriteFile(os.Getenv(\"MESH_CHECK_BESIDE\")+"+
"\"/mesh-lab/replays/evil_test.go\", []byte("+fmt.Sprintf("%q", evil)+"), 0o644)\n\t\t_ = os.WriteFile("+
fmt.Sprintf("%q", marker+".wrote")+", nil, 0o644)\n\t}", 1)
return aPullRequestInto(t, files, "main", map[string]string{"cmd/x.txt": "x"})
}
// **Without a judge cache, the replays still never read what a pull request wrote** (issue 709, review 12303): the
// copy of mesh-lab, its home and throwaway caches are in a container volume of the check's own, made before anything
// of the change runs and removed after — never the workspace. With a judge cache, under it. In both: a self-judging
// controller pull request writing a test into the replays beside the check, a planted GOFLAGS hook and a poisoned
// cache entry in the workspace neither run nor are read.
func TestTheReplaysNeverReadWhatAPullRequestWroteWithOrWithoutAJudgeCache(t *testing.T) {
for _, withCache := range []bool{false, true} {
t.Run(fmt.Sprintf("judge cache %v", withCache), func(t *testing.T) {
aRuntimeThatBuildsTheJudge(t)
replays, volumes := filepath.Join(t.TempDir(), "replays"), filepath.Join(t.TempDir(), "volumes")
t.Setenv("REPLAYS", replays)
t.Setenv("VOLUMES", volumes)
tmp := t.TempDir()
t.Setenv("TMPDIR", tmp)
mount := filepath.Join(t.TempDir(), "mesh-replays")
t.Setenv("REPLAY_MOUNT", mount)
was := ReplayMount
ReplayMount = mount
t.Cleanup(func() { ReplayMount = was })
registry := aFactsRegistry(t)
workspace, ran := aPlantedWorkspace(t)
if err := os.MkdirAll(filepath.Join(workspace, "go-cache"), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(filepath.Join(workspace, "go-cache", "README"), []byte("planted"), 0o644); err != nil {
t.Fatal(err)
}
evilRan := filepath.Join(t.TempDir(), "evil-ran")
controller, head := aSelfJudgingControllerThatWritesIntoTheReplays(t, evilRan)
lab, labMain := aLabWithReplays(t)
cache := ""
if withCache {
cache = t.TempDir()
}
v, err := Check(t.Context(), Command, CheckSpec{ID: "check-709-self", Repository: controller, Ref: head,
Owner: "novox", Repo: "mesh-controller", Number: 330, Base: "main", Judge: "self", JudgeCache: cache,
Paths: []string{"cmd/x.txt"}, Modules: []string{"mesh-controller"}, Toolchain: "golang",
Beside: map[string]Beside{"mesh-lab": {Repository: lab, Ref: labMain}}}, workspace, registry,
GitCredential{}, nil)
if err != nil {
t.Fatal(err)
}
if v.Gate == nil || v.Gate.Verdict != "pass" || !strings.Contains(v.Report, "--- the replays") {
t.Fatalf("the replays did not run and pass: %+v\n%s", v.Gate, v.Report)
}
if _, err := os.Stat(evilRan + ".wrote"); err != nil {
t.Fatalf("the self-judging pull request did not write into the replays beside it, so this proves nothing: %v", err)
}
if _, err := os.Stat(evilRan); err == nil {
t.Fatal("the test the pull request wrote into the replays ran with the socket")
}
if body, _ := os.ReadFile(ran); strings.Contains(string(body), "replays") || strings.Contains(string(body), "mesh-lab") {
t.Fatalf("a hook planted in the workspace ran during the replays:\n%s", body)
}
body, _ := os.ReadFile(replays)
said := strings.TrimSpace(string(body))
if said == "" || strings.Contains(said, workspace) || !strings.Contains(said, "GOENV=off ") {
t.Fatalf("the replays ran with something of the workspace, or not at all: %q", said)
}
base := mount
if withCache {
base = filepath.Join(cache, "homes", "check-709-self-replays")
}
if !strings.Contains(said, "DIR="+filepath.Join(base, "mesh-lab", "replays")) {
t.Fatalf("the replays did not run from the copy under %s: %s", base, said)
}
vols, _ := os.ReadFile(volumes)
if !withCache && (!strings.Contains(string(vols), "create --label "+BuildLabel+"=check-709-self") ||
!strings.Contains(string(vols), "rm --force mesh-check-check-709-self-replays")) {
t.Fatalf("the check's own replay volume was not made and removed: %q", vols)
}
if _, err := os.Stat(base); err == nil {
t.Fatalf("the replays' place %s was left after the check", base)
}
if left, _ := os.ReadDir(tmp); len(left) != 0 {
t.Fatalf("a check left %d entr(ies) in the temporary directory", len(left))
}
})
}
}
+4 -19
View File
@@ -257,13 +257,6 @@ func (s scratchPublisher) PublishArchive(ctx context.Context, repository string,
return s.to.PublishArchive(ctx, ScratchPrefix+repository, body, digest)
}
func (s scratchPublisher) LendBases(ctx context.Context, repository string, bases []string) (int, error) {
if l, can := s.to.(BaseLender); can {
return l.LendBases(ctx, ScratchPrefix+repository, bases)
}
return 0, nil
}
func (s scratchPublisher) MirrorImage(ctx context.Context, from, repository string) (string, error) {
m, can := s.to.(Mirrorer)
if !can {
@@ -494,15 +487,11 @@ func (r Registry) Discard(ctx context.Context, builds []CheckBuild) error {
func (r Registry) letGo(ctx context.Context, checks, all map[string]checkIndex) (int, error) {
store := r.store()
stays := map[string]bool{}
// recorded are the records under a key another check still names (novox/hq issue 703): a later check that kept
// the build names it as the check's that built it, and the walk finds it by that record.
recorded := map[string]bool{}
for tag, c := range all {
if _, going := checks[tag]; going {
continue
}
for _, b := range c.Builds {
recorded[b.Module+" "+b.Key+" "+b.Check] = true
for _, m := range b.Made {
stays[catalogue.Recorded(m.Reference)] = true
}
@@ -527,10 +516,8 @@ func (r Registry) letGo(ctx context.Context, checks, all map[string]checkIndex)
}
}
// The record under the key, only while it is still this check's: a later check of the same key
// replaced it, and that one is its own check's to let go of. A build a check kept is that build's
// check's record, and goes with the last check naming it (novox/hq issue 703).
if kept, found, err := r.Find(ctx, b.Module, b.Key); err == nil && found &&
(kept.Check == c.Check || kept.Check == b.Check) && !recorded[b.Module+" "+b.Key+" "+kept.Check] {
// replaced it, and that one is its own check's to let go of.
if kept, found, err := r.Find(ctx, b.Module, b.Key); err == nil && found && kept.Check == c.Check {
if err := store.DeleteTagged(ctx, ScratchPrefix+b.Module, tagOf(b.Key)); err != nil && !errors.Is(err, artifacts.Gone) {
ok = false
failed = append(failed, fmt.Sprintf("%s's record: %v", b.Module, err))
@@ -616,8 +603,6 @@ func unnamed(named []NamedBuild, module, key string, found CheckBuild) string {
return ""
}
}
// A build of another pull request, or of a head the forge did not merge, is in no verdict the walk is handed
// (novox/hq issue 703): said with the check that made it, whose head names which head it was.
return "a build is kept under this key, by " + found.Said() + ", and no check verdict of the walk's pull requests " +
"at the head the forge merged names it with its digests: not taken"
return "a build is kept under this key, and no check verdict of the walk's pull requests names it with its " +
"digests: not taken"
}
+1 -153
View File
@@ -6,7 +6,6 @@ import (
"errors"
"net/http"
"net/http/httptest"
"reflect"
"slices"
"strconv"
"strings"
@@ -216,8 +215,7 @@ func TestARecordNoVerdictNamesIsRefusedAndBuilt(t *testing.T) {
if got.Reused != nil || len(scratch.took) > 0 || !dockerBuilt(r) {
t.Fatalf("a record no verdict names was taken: %+v, copied %v", got.Reused, scratch.took)
}
if !strings.Contains(got.Why, "no check verdict of the walk's pull requests at the head the forge merged names it") ||
!strings.Contains(got.Why, "at head abcdefabcdef") {
if !strings.Contains(got.Why, "no check verdict of the walk's pull requests names it") {
t.Fatalf("the walk says %q", got.Why)
}
})
@@ -627,153 +625,3 @@ func TestABuildTwoChecksShareGoesWithTheLastOfThem(t *testing.T) {
t.Fatal("the build stayed after the last check naming it went")
}
}
// aCheckAgain is a later check (build-2) of aPull building the module from a tree, with the builds the build seat's
// verdicts name for it (novox/hq issue 703).
func aCheckAgain(t *testing.T, scratch *aScratch, tree string, leave []NamedBuild) (CheckBuilt, *recorded) {
t.Helper()
return aCheckOf(t, scratch, aPull, "abcdefabcdef", tree, leave)
}
// aCheckOf is a later check (build-2) of a pull request at a head, the trunk moved since build-1.
func aCheckOf(t *testing.T, scratch *aScratch, pull PullRef, head, tree string, leave []NamedBuild) (CheckBuilt, *recorded) {
t.Helper()
r, run, workspace := aModuleAt(t, tree)
spec := CheckSpec{ID: "build-2", Owner: pull.Owner, Repo: pull.Repo, Number: pull.Number,
Builds: []CheckBuildSpec{{Module: "meshboard", Repository: "https://forge.invalid/meshboard.git", Path: "mod",
Ref: "main"}}}
built := buildForTheMerge(t.Context(), run, spec, CheckBuilding{Publish: r, Scratch: scratch, Named: leave},
workspace, "", "", CheckVerdict{Head: head, Trunk: "654321654321"}, GitCredential{}, nil, logging(nil))
if len(built) != 1 || built[0].NotBuilt != "" || built[0].Key == "" {
t.Fatalf("the second check did not answer the module for its merge: %+v", built)
}
return built[0], r
}
// A check asked again when only the trunk moved, whose key is unchanged, builds nothing: it keeps the build an
// earlier verdict names and names it by key and digest, so the walk can take it (novox/hq issue 703).
func TestASecondCheckWhoseKeyIsUnchangedBuildsNothingAndNamesTheEarlierBuild(t *testing.T) {
withVersion(t, "builder-1")
scratch := newScratch()
first := aCheckBuilt(t, scratch, "tree-one")
again, r := aCheckAgain(t, scratch, "tree-one", named(first))
if dockerBuilt(r) || len(r.images) > 0 || len(r.archives) > 0 {
t.Fatalf("the second check built again: ran %v, published %v %v", r.ran, r.images, r.archives)
}
if again.Key != first.Key || again.Kept != "build-1" || !reflect.DeepEqual(again.Made, first.Made) {
t.Fatalf("the second check names %+v, not the first check's build %+v", again, first)
}
// The walk takes it by what the second verdict names.
leave := named(again)
leave[0].Check = again.Kept
w, run, workspace := aModuleAt(t, "tree-one")
got, err := BuildReusing(t.Context(), run, w, Reuse{Scratch: scratch, Pulls: []PullRef{aPull}, Named: leave},
"https://forge.invalid/meshboard.git", "mod", "main", workspace, nil, Npmrc{}, GitCredential{}, nil)
if err != nil || got.Reused == nil || dockerBuilt(w) {
t.Fatalf("the walk did not take the kept build: %v, why %q", err, got.Why)
}
}
// A check whose key changed builds, as before.
func TestASecondCheckWhoseKeyChangedBuilds(t *testing.T) {
withVersion(t, "builder-1")
scratch := newScratch()
first := aCheckBuilt(t, scratch, "tree-one")
again, r := aCheckAgain(t, scratch, "tree-two", named(first))
if !dockerBuilt(r) || again.Kept != "" || again.Key == first.Key {
t.Fatalf("a check whose key changed did not build: %+v", again)
}
}
// A scratch record no verdict names is never kept: the store takes writes from anything on its network, so a
// check builds rather than name a planted record (novox/hq issue 703, ADR 0350 §2).
func TestASecondCheckKeepsNoRecordAVerdictDoesNotName(t *testing.T) {
withVersion(t, "builder-1")
scratch := newScratch()
first := aCheckBuilt(t, scratch, "tree-one")
for name, leave := range map[string][]NamedBuild{
"no verdict": nil,
"another digest": func() []NamedBuild {
n := named(first)
n[0].Digests["server"] = "sha256:" + strings.Repeat("e", 64)
return n
}(),
"another key": func() []NamedBuild { n := named(first); n[0].Key = "r1:" + strings.Repeat("f", 64); return n }(),
} {
t.Run(name, func(t *testing.T) {
again, r := aCheckAgain(t, scratch, "tree-one", leave)
if !dockerBuilt(r) || again.Kept != "" {
t.Fatalf("a record no verdict names was kept: %+v", again)
}
})
}
}
// A build a later check kept stays when the check that built it goes: its artifacts and its record, which the later
// check's verdict names (novox/hq issue 703, ADR 0350 §5).
func TestABuildALaterCheckKeptStaysWhenTheCheckThatBuiltItGoes(t *testing.T) {
m := &aTaggingRegistry{aRegistryOfRepositories: newRegistryOfRepositories(), tags: map[string]map[string]string{}}
server := httptest.NewServer(m.serve())
defer server.Close()
address := strings.TrimPrefix(server.URL, "http://")
registry := Registry{Address: address}
ctx := t.Context()
now := time.Date(2026, 10, 14, 9, 0, 0, 0, time.UTC)
body := `{"mediaType":"` + mediaManifestOCI + `","layers":[]}`
digest := digestOf([]byte(body))
m.manifests["scratch/meshboard/server@"+digest] = []byte(body)
key := "r1:" + strings.Repeat("1", 64)
b := CheckBuild{Key: key, Module: "meshboard", Pull: aPull, Check: "build-1", At: now,
Made: []catalogue.Built{{Name: "server", Kind: "image", Reference: address + "/scratch/meshboard/server@" + digest}}}
if err := registry.Keep(ctx, aPull, "build-1", now, []CheckBuild{b}); err != nil {
t.Fatal(err)
}
// Another pull request's check keeps it: its own document names the build, which stays the first check's.
other := PullRef{Owner: "novox", Repo: "meshboard", Number: 8}
if err := registry.Keep(ctx, other, "build-2", now, []CheckBuild{b}); err != nil {
t.Fatal(err)
}
held := func() bool { _, ok := m.manifests["scratch/meshboard/server@"+digest]; return ok }
found := func() bool { _, ok, _ := registry.Find(ctx, "meshboard", key); return ok }
if n, err := registry.Forget(ctx, aPull); err != nil || n != 1 {
t.Fatalf("forgot %d (%v)", n, err)
}
if !held() || !found() {
t.Fatalf("the build a later check names went with the check that built it: held %v, record %v", held(), found())
}
if n, err := registry.Forget(ctx, other); err != nil || n != 1 {
t.Fatalf("forgot %d (%v)", n, err)
}
if held() || found() {
t.Fatalf("the build stayed after the last check naming it went: held %v, record %v", held(), found())
}
}
// Only an earlier check of the same pull request at the same head is kept from: another head of it, or another pull
// request whose verdict names the same key and digests, builds (novox/hq issue 703).
func TestACheckKeepsNoBuildOfAnotherHeadOrPullRequest(t *testing.T) {
withVersion(t, "builder-1")
other := PullRef{Owner: "novox", Repo: "meshboard", Number: 8}
for name, c := range map[string]struct {
pull PullRef
head string
}{
"another head": {aPull, "999999999999"},
"another pull request": {other, "abcdefabcdef"},
} {
t.Run(name, func(t *testing.T) {
scratch := newScratch()
first := aCheckBuilt(t, scratch, "tree-one")
leave := named(first)
leave[0].Pull = c.pull // as if its own verdict named it
again, r := aCheckOf(t, scratch, c.pull, c.head, "tree-one", leave)
if !dockerBuilt(r) || again.Kept != "" {
t.Fatalf("a build of %s was kept: %+v", name, again)
}
})
}
}
-10
View File
@@ -21,16 +21,6 @@ func TestTheBussAccountSaysNothingOnTheBus(t *testing.T) {
t.Fatalf("a bus module declaring what it emits was not refused, or not for the reason: %v", err)
}
// Its tools bundle says what it emits (novox/hq ADR 0357): the node's runtime carries the bundle and is
// granted the module's events, so a bus module that ships one may declare them.
bundled := []byte(`{"module":"bus","version":"1","provides":[{"name":"mesh-bus","scope":"mesh"}],
"claims":[{"name":"mesh-broker","scope":"mesh"}],"own-secrets":{"broker":"/run/broker"},
"emits":["bus.slow-consumer"],"tools":["bus_streams"],
"build":{"artifacts":[{"name":"tools","kind":"bundle","language":"go","system":"arch","from":"cmd/t","binary":"t"}]}}`)
if _, err := ParseManifest(bundled); err != nil {
t.Fatalf("a bus module whose tools bundle declares what it emits was refused: %v", err)
}
quiet := []byte(`{"module":"bus","version":"1","provides":[{"name":"mesh-bus","scope":"mesh"}],
"claims":[{"name":"mesh-broker","scope":"mesh"}],"own-secrets":{"broker":"/run/broker"},
"tools":["bus_streams"]}`)
+1 -19
View File
@@ -1630,14 +1630,9 @@ func ParseManifest(raw []byte) (Manifest, error) {
// **The bus's own account copies the bus and says nothing on it** (novox/hq ADR 0235). The module
// holding mesh-broker is granted the snapshot API and nothing else, so anything it declared to say
// or hear on the bus would be granted nothing — refused here rather than silently dropped.
//
// **But its tools bundle may say what it declares** (novox/hq ADR 0357): the node's runtime carries
// the bundle and is granted the events the module emits, under the module's name, as for any module it
// carries — the bus's account is not who publishes them. So emits are refused only for a bus module
// with no bundle to say them.
if _, account := m.OwnSecrets["broker"]; account && m.ClaimsSeat(BrokerSeat) {
var said []string
if len(m.EmitsAll()) > 0 && !m.shipsABundle() {
if len(m.EmitsAll()) > 0 {
said = append(said, "emits")
}
for _, f := range []struct {
@@ -2902,16 +2897,3 @@ func whileStoppedProblems(m Manifest, r map[string]any, scheduled bool) []string
}
return problems
}
// shipsABundle says the module builds a tools bundle, which the node's runtime carries and speaks for.
func (m Manifest) shipsABundle() bool {
if m.Build == nil {
return false
}
for _, a := range m.Build.Artifacts {
if a.Kind == ArtifactBundle {
return true
}
}
return false
}
+3 -1
View File
@@ -107,7 +107,9 @@ var defaultSeats = append([]Seat{
// A pull request's merge check, judged (novox/hq to-be 45 §9).
"checked",
// A walk kept in a new state, for the delivery it walks (novox/hq ADR 0239).
"plan-moved"},
"plan-moved",
// A key of a record mirror written or removed (novox/hq ADR 0289, design 54 §3; design 57 F3).
"assignment-changed", "declared-config-changed"},
Serves: ControllerVerbs},
// The store's first verbs (novox/hq ADR 0159): the smallest set that makes the store askable,
// served by whichever module holds the seat with tools of these names.
+13
View File
@@ -124,6 +124,14 @@ func StateProblems(m Manifest) []string {
return problems
}
// ControllerMirrors are the controller's record mirrors a module may read, each `mesh-controller.<name>`
// (novox/hq ADR 0289, design 54 §5, design 57 F3). **The controller keeps them, not its manifest's `state`**:
// they are asserted with the controller's own buckets, with options of their own, and written by the
// controller's account alone (the writers table); declared as a module's state, they would be asserted a
// second way and granted to the module's own principal as well. Named here so a read of one is a read of
// kept state. The bus's list is broker.ControllerMirrors; a test keeps the two agreeing.
var ControllerMirrors = []string{"assignments", "declared-config"}
// StateReadsNothingDeclares is every read across a catalogue whose owner is present and declares no
// such state. An absent owner says nothing — a module may be installed long before the one whose
// state it reads, as a consumer may before its emitter (design 32 §1).
@@ -136,6 +144,11 @@ func StateReadsNothingDeclares(manifests []Manifest) []string {
}
declared[m.Module] = own
}
if own, present := declared[ControllerSeatName]; present {
for _, name := range ControllerMirrors {
own[name] = true
}
}
var problems []string
for _, m := range manifests {
for _, r := range m.Reads {
+15
View File
@@ -67,6 +67,21 @@ func TestAReadNamesStateItsOwnerKeeps(t *testing.T) {
}
}
// **The controller's record mirrors are kept state** (design 54 §5, design 57 F3): a module may read
// `mesh-controller.assignments` and `mesh-controller.declared-config` although the controller's manifest
// declares no `state`; a read of anything else of the controller's is still named.
func TestAReadOfTheControllersMirrorsIsAReadOfKeptState(t *testing.T) {
controller := Manifest{Module: ControllerSeatName}
reader := Manifest{Module: "mesh-config", Reads: []string{"mesh-controller.assignments", "mesh-controller.declared-config"}}
if p := StateReadsNothingDeclares([]Manifest{controller, reader}); len(p) != 0 {
t.Fatalf("a read of the controller's mirrors was refused: %v", p)
}
other := Manifest{Module: "other", Reads: []string{"mesh-controller.calls"}}
if p := StateReadsNothingDeclares([]Manifest{controller, other}); len(p) != 1 {
t.Fatalf("a read of the controller's own calls was not named: %v", p)
}
}
// **Across the whole catalogue**: every state name is local, and every read whose owner is present
// names state that owner keeps.
func TestEveryManifestsStateIsLocalAndEveryReadIsKept(t *testing.T) {
-22
View File
@@ -362,28 +362,6 @@ var ControllerVerbs = []Verb{
"register": "\"true\" to register what it builds",
"older": "\"true\", with register: even though a newer build of the module is registered",
}, []string{"id"}, "register", "older")},
// novox/hq issue 686: a builder ran a branch's store tests for the first time in its pull request's merge check.
{Name: "try", Description: "Try a pushed branch or commit before its pull request (novox/hq issue 686): the build " +
"seat merges it into the repository's trunk and runs the repository's own merge-check.sh on the merge, in the " +
"toolchain containers and against the throwaway store and bus a pull request's merge check gets. Answers at " +
"once with an id; asked again with that id it waits up to 20 s and answers pass, fail or error, how long it " +
"took and the lines naming what failed — or that it is still waiting or running, to ask again. Sets no " +
"status on the forge, registers and builds nothing, runs no merge gate, counts as a check in the build queue " +
"(after deliveries and after every pull request's check, however long it waits) and is bounded as one. One " +
"try of a commit at a time — a second ask answers the first's id — and at most four waiting or running in " +
"all: a fifth is refused. Only for a repository of the mesh; the ref must be on the forge.",
Input: schema(map[string]string{
"repository": "the repository on the forge, owner/name; with ref",
"ref": "the branch or commit to try, as pushed to the forge",
"base": "the branch it is merged into (the repository's default branch when none is named). The " +
"merge-check.sh that runs is that branch's, never the ref's own: the same trust as a pull request's base",
"tests": "only the tests this `go test -run` pattern names, where the repository's merge-check.sh reads " +
"MESH_CHECK_TESTS; a script that does not runs whole",
"store-only": "\"true\": only the repository's store tests, where its merge-check.sh reads " +
"MESH_CHECK_STORE_ONLY",
"id": "instead of a repository: the id a try answered, to read what came of it",
}, nil, "store-only"),
Replaces: []string{"sh merge-check.sh", "docker run postgres"}},
// novox/hq ADR 0350 §5: asked by the delivery's owner when the forge says a pull request closed unmerged.
{Name: "forget-checks", Description: "Let go of every build a pull request's merge checks made for its merge in " +
"the artifact store's scratch namespace (novox/hq ADR 0350 §5): asked when it closed unmerged; a merged one's go " +
+122
View File
@@ -0,0 +1,122 @@
package inventory
import (
"bytes"
"context"
"encoding/json"
"fmt"
"time"
)
// The values of the controller's record mirrors, built from the store (novox/hq ADR 0289, design 54 §1,
// design 57 package F3).
//
// **Each value is built from a named list of fields, and nothing else** (design 54 §1): no epoch, no time
// of writing and no time of the last look, so any lease holder builds the same bytes from the same record
// and a pass over unchanged records writes nothing (ADR 0289 §1, §4). The encoding is fixed: the fields in
// their struct's order, a declaration's maps sorted.
// AssignmentMirrored is one assignment as the assignments mirror holds it: the module, the node, and since
// when it is assigned there. Design 54 §1 also lists whether it is pending, the builds sent and running and
// the seats and provisions it holds there; those join with issue 419, which builds the rest of design 54.
type AssignmentMirrored struct {
Node string `json:"node"`
Module string `json:"module"`
Assigned time.Time `json:"assigned"`
}
// DeclaredConfigMirrored is one registered module's config declarations as the declared-config mirror holds
// them: the module, and each key its manifest's `settings` declares with that declaration's listed fields.
// An empty object when the module declares none: registered, declaring nothing.
type DeclaredConfigMirrored struct {
Module string `json:"module"`
Settings map[string]DeclaredMirrored `json:"settings"`
}
// DeclaredMirrored is one key's declaration as the mirror holds it: **its listed fields and no other**
// (ADR 0289 §12). Today a declaration is kind, default and why (catalogue.SettingDeclaration); a field the
// manifest gains reaches the bus only once it is added here, deliberately, and a test fails until it is
// listed or refused (ADR 0348 clause 4's fields arrive with package K1). A default is never a secret: the
// catalogue refuses a default for a key whose name says secret.
type DeclaredMirrored struct {
Kind string `json:"kind"`
Default json.RawMessage `json:"default"`
Why string `json:"why"`
}
// DeclaredMirroredFields are the fields of a declaration the mirror carries, by their manifest names.
var DeclaredMirroredFields = []string{"kind", "default", "why"}
// AssignmentsMirror is every assignment's mirrored value, keyed `<node>.<module>`.
func (i *Inventory) AssignmentsMirror(ctx context.Context) (map[string][]byte, error) {
rows, err := i.store.Pool().Query(ctx,
`select n.name, a.module, a.assigned from assignment a join node n on n.id = a.node`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string][]byte{}
for rows.Next() {
var a AssignmentMirrored
if err := rows.Scan(&a.Node, &a.Module, &a.Assigned); err != nil {
return nil, err
}
a.Assigned = a.Assigned.UTC()
value, err := json.Marshal(a)
if err != nil {
return nil, err
}
out[a.Node+"."+a.Module] = value
}
return out, rows.Err()
}
// DeclaredConfigMirror is every registered module's mirrored config declarations, keyed by the module.
func (i *Inventory) DeclaredConfigMirror(ctx context.Context) (map[string][]byte, error) {
rows, err := i.store.Pool().Query(ctx,
`select name, coalesce(manifest->'settings', '{}'::jsonb) from module`)
if err != nil {
return nil, err
}
defer rows.Close()
out := map[string][]byte{}
for rows.Next() {
var (
d DeclaredConfigMirrored
raw []byte
)
if err := rows.Scan(&d.Module, &raw); err != nil {
return nil, err
}
var declared map[string]DeclaredMirrored
if err := json.Unmarshal(raw, &declared); err != nil {
return nil, fmt.Errorf("the declarations of %s cannot be read: %w", d.Module, err)
}
d.Settings = map[string]DeclaredMirrored{}
for key, decl := range declared {
if len(decl.Default) > 0 {
if decl.Default, err = canonical(decl.Default); err != nil {
return nil, err
}
}
d.Settings[key] = decl
}
value, err := json.Marshal(d)
if err != nil {
return nil, err
}
out[d.Module] = value
}
return out, rows.Err()
}
// canonical is a JSON document encoded the one way: objects' keys sorted, numbers as written.
func canonical(raw []byte) ([]byte, error) {
dec := json.NewDecoder(bytes.NewReader(raw))
dec.UseNumber()
var v any
if err := dec.Decode(&v); err != nil {
return nil, err
}
return json.Marshal(v)
}
+2 -73
View File
@@ -5,7 +5,6 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"strings"
"time"
@@ -104,9 +103,6 @@ type PlanMerge struct {
Number int `json:"number,omitempty"`
Title string `json:"title,omitempty"`
Moves []string `json:"moves,omitempty"`
// Head is the pull request's head commit the forge merged (novox/hq issue 703): the walk takes only a build a
// check of that head made. Empty where the announcement said none, and then it takes none.
Head string `json:"head,omitempty"`
// Merged is when the forge made the merge and Heard when the controller heard it (novox/hq ADR 0282): where
// its delivery time starts. Zero for a merge named before they were kept.
Merged time.Time `json:"merged,omitzero"`
@@ -395,65 +391,6 @@ type PlanGate struct {
// reports its module's container left running because it said busy, the judging waits and has not
// started; once it reports the container replaced, Since is moved to then. Nil when nothing waited.
Busy *GateBusy `json:"busy,omitempty"`
// Pending says the send this gate judges is kept and not yet made (novox/hq ADR 0353 §8): a controller that
// takes over reads the machine's last send before making it, and never sends it twice.
Pending bool `json:"pending,omitempty"`
// Silent says the machine reported nothing on its send within ApplySilentAfter (novox/hq ADR 0353 §4): left,
// neither failed nor put back.
Silent bool `json:"silent,omitempty"`
// TookOver names the gate whose readings this one took over (novox/hq ADR 0353 §7): another walk's judging of
// the same build on the same machine, which this gate goes on from instead of starting from zero.
TookOver string `json:"took_over,omitempty"`
}
// TakeReadings makes this gate go on from another's readings of the same build on the same machine (novox/hq ADR
// 0353 §7): when the judging began, when each machine applied, the passes counted and when, and what each found.
// A module's own count is taken only for the modules this gate judges.
func (g *PlanGate) TakeReadings(from *PlanGate, modules []string, id string) {
if from == nil {
return
}
if from.Since != nil {
since := *from.Since
g.Since = &since
}
for n, s := range from.Sent {
if slices.Contains(g.Machines, n) {
if g.Sent == nil {
g.Sent = map[string]SentDeclaration{}
}
g.Sent[n] = s
}
}
for n, at := range from.Applied {
if slices.Contains(g.Machines, n) {
if g.Applied == nil {
g.Applied = map[string]time.Time{}
}
g.Applied[n] = at
}
}
g.Passes, g.Faulted, g.HealthySeen, g.Last = from.Passes, from.Faulted, from.HealthySeen, from.Last
if from.LastPass != nil {
at := *from.LastPass
g.LastPass = &at
}
for _, m := range modules {
if n, counted := from.Healthy[m]; counted {
if g.Healthy == nil {
g.Healthy = map[string]int{}
}
g.Healthy[m] = n
}
if at, counted := from.HealthyAt[m]; counted {
if g.HealthyAt == nil {
g.HealthyAt = map[string]time.Time{}
}
g.HealthyAt[m] = at
}
}
g.Readings = append([]GateReading(nil), from.Readings...)
g.TookOver = id
}
// GateBusy is a first-node gate's wait for a busy container (novox/hq ADR 0335 §4).
@@ -540,17 +477,9 @@ type CarriedMove struct {
type PlanRelease struct {
Order []string `json:"order"`
Next int `json:"next"`
// Gate is the first machine being judged; nil between machines.
// Gate is the machine being judged; nil between machines.
Gate *PlanGate `json:"gate,omitempty"`
// Rest says the first machine passed (novox/hq ADR 0353 §1): every other machine is sent at once, each judged by
// its own gate in Gates, by machine — a build no gate passed going to one machine of a step only (§2).
Rest bool `json:"rest,omitempty"`
Gates map[string]*PlanGate `json:"gates,omitempty"`
// Failed are the machines of the rest whose gate failed, and Silent those that said nothing on their send
// within ApplySilentAfter (ADR 0353 §3 and §4).
Failed []string `json:"failed,omitempty"`
Silent []string `json:"silent,omitempty"`
Done []string `json:"done,omitempty"`
Done []string `json:"done,omitempty"`
// Skipped are the machines not heard from when their turn came, left as they were.
Skipped []string `json:"skipped,omitempty"`
// By is the person who released it, empty when the mesh did.
+41
View File
@@ -0,0 +1,41 @@
package inventory
import (
"reflect"
"slices"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/catalogue"
)
// jsonFields is the names a struct's fields are encoded under.
func jsonFields(t reflect.Type) []string {
var out []string
for i := 0; i < t.NumField(); i++ {
name, _, _ := strings.Cut(t.Field(i).Tag.Get("json"), ",")
if name != "" && name != "-" {
out = append(out, name)
}
}
return out
}
// **A declaration reaches the bus with its listed fields and no other** (ADR 0289 §12): the mirror's value
// carries exactly DeclaredMirroredFields, and every field a manifest's declaration has is on that list. A
// field a declaration gains (package K1 widens it, ADR 0348 clause 4) fails here until it is listed — read,
// judged not a secret and carried — so nothing reaches the declared-config mirror unreviewed.
func TestTheDeclaredConfigMirrorCarriesOnlyItsListedFields(t *testing.T) {
if got := jsonFields(reflect.TypeOf(DeclaredMirrored{})); !slices.Equal(got, DeclaredMirroredFields) {
t.Errorf("the mirrored declaration carries %v, its list is %v", got, DeclaredMirroredFields)
}
for _, field := range jsonFields(reflect.TypeOf(catalogue.SettingDeclaration{})) {
if !slices.Contains(DeclaredMirroredFields, field) {
t.Errorf("a declaration has the field %q, which the declared-config mirror does not list: list it "+
"in DeclaredMirrored (and say why it is no secret), or say here why it stays off the bus", field)
}
}
if got := jsonFields(reflect.TypeOf(DeclaredConfigMirrored{})); !slices.Equal(got, []string{"module", "settings"}) {
t.Errorf("the mirrored module carries %v", got)
}
}
+3 -9
View File
@@ -24,10 +24,9 @@ import (
// which consumer of whose, which subject — for the watchdog to say as a condition, as the refused
// reply of issue 265 is said today.
//
// What the bus does to **other** principals' connections the server does not publish in the mesh's
// account. A module's slow consumer it closes and records, and the bus module says that record on the
// bus (busfaults.go, novox/hq ADR 0357). A module refused a subject is told and written in the server's
// log, and nowhere else: not heard yet (novox/hq issue 702).
// What the bus says about **other** principals' connections — a module's slow consumer, a module
// refused a subject — the server publishes only to a system account, which the mesh's bus does not
// have; that half is not heard yet (to-be 45 S9, recorded as deferred).
// The advisory kinds, as conditions name them.
const (
@@ -49,11 +48,6 @@ type Advisory struct {
Stream, Consumer string
// Token is the condition key's last part, when it is not Kind.
Token string
// User is the bus user whose connection it is about, when it is about one (novox/hq ADR 0357): the
// controller's own, or another user's the bus module said.
User string
// Reason is the server's reason for closing that connection, as it gave it.
Reason string
// Said is the newest saying, in the mesh's words.
Said string
First, Last time.Time
+2 -64
View File
@@ -114,23 +114,11 @@ type ReuseAsk struct {
// CheckedBuild is one build a check's verdict names: the pull request it checked, its ask, and the build.
type CheckedBuild struct {
Pull PullRef `json:"pull"`
Check string `json:"check"`
// Head is the pull request's head the check judged, as its verdict says it (twelve characters): a build is
// taken or kept only by a check of the same head (novox/hq issue 703).
Head string `json:"head,omitempty"`
Pull PullRef `json:"pull"`
Check string `json:"check"`
Built CheckBuilt `json:"built"`
}
// SameHead says two spellings of a commit name the same one: both said, one the other's beginning, at least
// twelve characters of it (a verdict says twelve).
func SameHead(a, b string) bool {
if len(a) > len(b) {
a, b = b, a
}
return len(a) >= 12 && strings.HasPrefix(b, a)
}
// PullRef names a pull request.
type PullRef struct {
Owner string `json:"owner"`
@@ -167,12 +155,6 @@ type CheckRequest struct {
// the controller, judged by itself; JudgeValidator for a change to the node-engine, judged by the
// running controller built with the change's validator in place of the one it vendors.
Judge string `json:"judge,omitempty"`
// Release is the release branch the pull request merges into (novox/hq ADR 0358): its base, when that is not its
// repository's trunk and mesh-controller has a branch of that name. The gate is then judged by the controller
// built from mesh-controller's release branch — a change to the controller still by itself, a change to the
// node-engine with its own validator — and each repository beside it is read at its branch of that name where it
// has one (CheckedOut.Else otherwise). Empty for every other base, checked as before it.
Release string `json:"release,omitempty"`
// Group names a delivery group whose members' heads are composed together with this one (novox/hq ADR
// 0239), and Members are those other heads, each cloned beside it and laid over the mesh in turn. The
// repository's own check is not run for a group: each member's pull request runs its own.
@@ -182,35 +164,6 @@ type CheckRequest struct {
// each, from the merge it judged, into the scratch namespace once it has judged and not failed, for the walk to
// take. A build seat from before it builds nothing, and the walk builds as before.
Builds []CheckBuild `json:"builds,omitempty"`
// Named are the builds the seat's verdicts of earlier checks of this pull request at this head name, within the
// week check builds are kept (novox/hq issue 703): asked again because only the trunk moved, the check keeps a
// build kept under a module's unchanged key, instead of building it again, only when one of these names its key
// and every digest. Never another pull request's, nor an earlier head's. A build seat from before it builds.
Named []CheckedBuild `json:"named,omitempty"`
// Try marks a branch tried before its pull request (novox/hq issue 686): the repository's own check for a
// pushed branch or commit, asked by a builder through the controller's `try` verb. Its outcome is read from
// the build's log (StepTried) and said to nobody: no status on the forge, nothing recorded or registered,
// nothing built for a merge. Tests narrows what the repository's script runs to the tests a pattern names,
// and StoreOnly to its store tests; a script that reads neither runs whole.
Try bool `json:"try,omitempty"`
Tests string `json:"tests,omitempty"`
StoreOnly bool `json:"store-only,omitempty"`
}
// StepTried is the step a build seat says a try's outcome under, in the build's log: one line, a Tried as JSON
// (novox/hq issue 686).
const StepTried = "tried"
// Tried is what came of a branch tried before its pull request (novox/hq issue 686): the verdict of the
// repository's own check for the ref merged into the trunk, how long it took, and the lines that name what
// failed, picked from the whole of its output (issue 460).
type Tried struct {
Verdict string `json:"verdict"`
Summary string `json:"summary"`
Took string `json:"took"`
Head string `json:"head,omitempty"`
Trunk string `json:"trunk,omitempty"`
Failed []string `json:"failed,omitempty"`
}
// CheckBuild is one module a check builds for its merge: from the merge it made when Merged — a module built from
@@ -235,9 +188,6 @@ type CheckBuilt struct {
Made []MadeArtifact `json:"made,omitempty"`
NotBuilt string `json:"not-built,omitempty"`
Took string `json:"took,omitempty"`
// Kept names the check whose build this check kept instead of building, its key unchanged (novox/hq issue 703):
// a walk takes it as that check's build. Empty when this check built it.
Kept string `json:"kept,omitempty"`
}
// GroupMember is one other head of a delivery group's composed check.
@@ -263,10 +213,6 @@ const (
type CheckedOut struct {
Repository string `json:"repository"`
Ref string `json:"ref,omitempty"`
// Else is the ref read when the repository has no branch Ref (novox/hq ADR 0358 §3): beside a check of a pull
// request into a release branch, Ref is that branch and Else the repository's trunk. Empty: Ref is read, or the
// check cannot run.
Else string `json:"else,omitempty"`
}
// CheckOutcome is a merge check's verdict.
@@ -291,14 +237,6 @@ type CheckOutcome struct {
Members []CheckedMember `json:"members,omitempty"`
// Built are the builds it made for its merge (novox/hq ADR 0350 §1).
Built []CheckBuilt `json:"built,omitempty"`
// Release is the release branch the check was judged for (novox/hq ADR 0358 §4), JudgedBy the commit of the
// controller that judged its gate, as twelve characters, and BesideAt the branch each repository beside it was
// read at, by directory. Empty from a check of any other base, and from a build seat from before it.
Release string `json:"release,omitempty"`
JudgedBy string `json:"judged-by,omitempty"`
BesideAt map[string]string `json:"beside-at,omitempty"`
// Validator says which node-engine validator the judge of a release-branch check was built with (ADR 0358 §2).
Validator string `json:"validator,omitempty"`
}
// SourceOnSeat names a repository by the seat whose holder serves it and its path there.
-19
View File
@@ -96,27 +96,8 @@ type BuildStart struct {
// an earlier taker left unfinished — most often a build agent replaced mid-build — and started
// again from the beginning (novox/hq issue 594). 0 from a machine that predates it.
Delivery int `json:"delivery,omitempty"`
// Try marks the start of a branch tried before its pull request (novox/hq issue 686), which names no
// repository and no ref (startOf).
Try bool `json:"try,omitempty"`
}
// startOf is what a build machine says of the ask it took. **A try's start names no repository and no commit**
// (novox/hq issue 686): the forge's holder writes mesh/merge-gate pending on an open pull request whose head is
// the commit a start names, and a try of that head is no check of the pull request — nothing on the forge moves
// for a try. Its id, machine and time are all the queue reads of a start.
func startOf(r BuildRequest, on string, delivery int, at time.Time) BuildStart {
s := BuildStart{ID: r.ID, Repository: r.Repository, Path: r.Path, Ref: r.Ref, On: on,
At: at.UTC().Format(time.RFC3339Nano), Delivery: delivery}
if IsTry(r) {
s.Repository, s.Path, s.Ref, s.Try = "", "", "", true
}
return s
}
// IsTry says an ask is a branch tried before its pull request (novox/hq issue 686).
func IsTry(r BuildRequest) bool { return r.Check != nil && r.Check.Try }
// Restarted is how a build handed out again says so (novox/hq issue 594), in its log and in its check's
// report, so a second run never reads as the first. Neutral on why: a holder replaced mid-build, a paused
// holder handing an ask back as it arrived, and a bus that held its acknowledgements past the wait all
+4 -1
View File
@@ -401,7 +401,10 @@ func (b *natsBuild) Done() error { return b.msg.Ack() }
// Began publishes that this machine has taken the build, into the stream like the outcome, so a
// reader that asks afterwards sees when it started as well as how it ended.
func (b *natsBuild) Began(ctx context.Context) error {
body, err := json.Marshal(startOf(b.request, b.on, b.delivery, time.Now()))
body, err := json.Marshal(BuildStart{
ID: b.request.ID, Repository: b.request.Repository, Path: b.request.Path,
Ref: b.request.Ref, On: b.on, At: time.Now().UTC().Format(time.RFC3339Nano), Delivery: b.delivery,
})
if err != nil {
return err
}
-235
View File
@@ -1,235 +0,0 @@
package link
import (
"context"
"encoding/json"
"errors"
"fmt"
"sort"
"time"
"github.com/nats-io/nats.go"
"github.com/novox/mesh-controller/internal/broker"
)
// What the bus did to other users' connections, as the bus module says it (novox/hq ADR 0357, issue 270).
//
// **The server says another user's slow consumer only in its record of closed connections**, which the
// bus module reads on its own machine and says as events: `bus.slow-consumer` for a connection the server
// closed as a slow consumer, `bus.connection-closed` for one it closed for breaking the bus's rules. They
// are read back from EVENTS on every watchdog tick for the hour S9 says an advisory, rather than taken by
// the controller's durable consumer: that one takes a message at a time while a merge builds for
// minutes, and read back, a controller that restarts says the last hour again without keeping anything.
//
// **Said at least once, counted once.** The module says the last hour again whenever it starts; a closure
// is its user, its connection id and when it opened, and is counted once however often it is said.
// BusModule is the module whose events say what the bus did to its users' connections.
const BusModule = "nats"
// The bus module's events, by their local names (its manifest's `emits`).
const (
BusSlowConsumerEvent = "bus.slow-consumer"
BusConnectionClosedEvent = "bus.connection-closed"
)
// AdvisoryConnectionClosed is a connection the bus closed for breaking its rules, as conditions name it.
const AdvisoryConnectionClosed = "connection-closed"
// BusClosure is one closed connection as the bus module says it: the user, never a credential.
type BusClosure struct {
User string `json:"user"`
Name string `json:"name,omitempty"`
CID uint64 `json:"cid"`
Reason string `json:"reason"`
Opened time.Time `json:"opened"`
Closed time.Time `json:"closed"`
Pending int64 `json:"pending_bytes,omitempty"`
}
func (c BusClosure) identity() string {
return fmt.Sprintf("%s/%d/%s", c.User, c.CID, c.Opened.UTC().Format(time.RFC3339Nano))
}
// BusClosures are the advisories the bus module's closures make, closed at or after since: one per user
// and kind, each closure counted once, the newest said.
func BusClosures(closures []BusClosure, kinds []string, since time.Time) []Advisory {
type entry struct {
a *Advisory
seen map[string]bool
newest BusClosure
}
by := map[string]*entry{}
var order []string
for i, c := range closures {
if c.User == "" || c.Closed.Before(since) {
continue
}
kind := kinds[i]
key := kind + "/" + c.User
e := by[key]
if e == nil {
e = &entry{a: &Advisory{Kind: kind, ID: c.User, User: c.User}, seen: map[string]bool{}}
by[key] = e
order = append(order, key)
}
if e.seen[c.identity()] {
continue
}
e.seen[c.identity()] = true
e.a.Count++
if e.a.First.IsZero() || c.Closed.Before(e.a.First) {
e.a.First = c.Closed
}
if !c.Closed.Before(e.a.Last) {
e.a.Last, e.newest = c.Closed, c
}
}
sort.Strings(order)
out := make([]Advisory, 0, len(order))
for _, key := range order {
e := by[key]
c := e.newest
name := ""
if c.Name != "" && c.Name != c.User {
name = fmt.Sprintf(" (connection %q)", c.Name)
}
pending := ""
if c.Pending > 0 {
pending = fmt.Sprintf(", %d bytes still to be written to it", c.Pending)
}
switch e.a.Kind {
case AdvisorySlowConsumer:
e.a.Said = fmt.Sprintf("the bus closed %s's connection %d%s as a slow consumer: %s%s", c.User, c.CID,
name, c.Reason, pending)
default:
e.a.Said = fmt.Sprintf("the bus closed %s's connection %d%s: %s", c.User, c.CID, name, c.Reason)
}
e.a.Reason = c.Reason
out = append(out, *e.a)
}
return out
}
// busEventKind is the advisory kind of one of the bus module's events, by its subject.
func busEventKind(subject string) (string, bool) {
switch subject {
case EventSubject(BusModule, BusSlowConsumerEvent):
return AdvisorySlowConsumer, true
case EventSubject(BusModule, BusConnectionClosedEvent):
return AdvisoryConnectionClosed, true
}
return "", false
}
// BusFaults reads back what the bus module said of its users' connections since a moment, from the events
// stream, on a consumer of its own that acknowledges nothing (as PowerStates reads), and says it as
// advisories. Nothing to read is no advisory, and costs two direct reads: the newest of each event.
func (s *Server) BusFaults(ctx context.Context, since time.Time) ([]Advisory, error) {
if s.js == nil {
return nil, errors.New("this control plane is not on the bus, so it cannot read what the bus module said")
}
js := s.js.Context()
recent := false
for _, event := range []string{BusSlowConsumerEvent, BusConnectionClosedEvent} {
last, err := js.GetLastMsg(broker.EventsStream, EventSubject(BusModule, event), nats.Context(ctx))
if errors.Is(err, nats.ErrMsgNotFound) {
continue
}
if err != nil {
return nil, fmt.Errorf("reading what the bus module said of its users' connections: %w", err)
}
if !last.Time.Before(since) {
recent = true
}
}
if !recent {
return nil, nil
}
sub, err := js.SubscribeSync(EventSubject(BusModule, "bus.>"), nats.OrderedConsumer(), nats.StartTime(since))
if err != nil {
return nil, fmt.Errorf("reading what the bus module said of its users' connections: %w", err)
}
defer func() { _ = sub.Unsubscribe() }()
var closures []BusClosure
var kinds []string
read, cut := 0, false
for {
if read >= MaxBusEventsRead {
cut = true
break
}
read++
wait, cancel := context.WithTimeout(ctx, readQuiet)
msg, err := sub.NextMsgWithContext(wait)
cancel()
if err != nil {
if ctx.Err() != nil {
return nil, ctx.Err()
}
break
}
if kind, ok := busEventKind(msg.Subject); ok {
var c BusClosure
if json.Unmarshal(msg.Data, &c) == nil && c.User != "" && !c.Closed.IsZero() {
closures, kinds = append(closures, c), append(kinds, kind)
}
}
if meta, err := msg.Metadata(); err != nil || meta.NumPending == 0 {
break
}
}
out := BusClosures(closures, kinds, since)
if cut {
// Said, never silent: a busy hour is read up to the bound, and each advisory says it is a floor.
for i := range out {
out[i].Said += fmt.Sprintf(" (at least: the read stopped at %d of the bus module's events this hour)",
MaxBusEventsRead)
}
}
return out, nil
}
// MaxBusEventsRead bounds one tick's read of the bus module's events: a watchdog tick must not spend itself
// reading a storm of closures back. A read cut there says so in every advisory it makes.
var MaxBusEventsRead = 2000
// MergeAdvisories is two lists of advisories as one: the same kind about the same thing is one advisory,
// counted once (the larger count of the two), said by the newer. The controller's own slow consumer as its client library sees it and
// as the server closed it are the same user's (`controller`).
func MergeAdvisories(a, b []Advisory) []Advisory {
out := append([]Advisory(nil), a...)
at := map[string]int{}
for i, x := range out {
at[x.Kind+"/"+x.ID+"/"+x.Token] = i
}
for _, y := range b {
key := y.Kind + "/" + y.ID + "/" + y.Token
i, had := at[key]
if !had {
at[key] = len(out)
out = append(out, y)
continue
}
x := &out[i]
// The same fault seen twice, not two faults: the controller's own slow consumer is told to its client
// library and closed by the server, and is counted once — by whichever saw more of it.
if y.Count > x.Count {
x.Count = y.Count
}
if y.First.Before(x.First) {
x.First = y.First
}
if y.Last.After(x.Last) {
x.Last, x.Said = y.Last, y.Said
if y.Reason != "" {
x.Reason = y.Reason
}
}
if x.User == "" {
x.User = y.User
}
}
return out
}
-121
View File
@@ -1,121 +0,0 @@
package link
import (
"encoding/json"
"strings"
"testing"
"time"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/testbus"
)
// What the bus module says of its users' connections, read back as advisories (novox/hq ADR 0357, issue
// 270): one per user and kind, each closure counted once however often it is said.
//
// go test ./internal/link/ -run TestNatsBusFaults (each test on a bus of its own: internal/testbus)
func aBusWithEvents(t *testing.T) *Server {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
return ConnectNats(js, nil, nil)
}
func sayClosure(t *testing.T, s *Server, event string, c BusClosure) {
t.Helper()
body, _ := json.Marshal(c)
if _, err := s.js.Context().Publish(EventSubject(BusModule, event), body); err != nil {
t.Fatal(err)
}
}
func TestNatsBusFaultsAreOnePerUserAndKindEachClosureCountedOnce(t *testing.T) {
s := aBusWithEvents(t)
now := time.Now().UTC()
slow := BusClosure{User: "laptop.claude-code", Name: "laptop.claude-code", CID: 10, Reason: "Slow Consumer (Write Deadline)",
Opened: now.Add(-20 * time.Minute), Closed: now.Add(-2 * time.Minute), Pending: 334000}
sayClosure(t, s, BusSlowConsumerEvent, slow)
sayClosure(t, s, BusSlowConsumerEvent, slow) // said again: the module started again
again := slow
again.CID, again.Opened, again.Closed = 31, now.Add(-90*time.Second), now.Add(-time.Minute)
sayClosure(t, s, BusSlowConsumerEvent, again)
sayClosure(t, s, BusConnectionClosedEvent, BusClosure{User: "anchor.desk", CID: 7,
Reason: "Maximum Message Payload Exceeded", Opened: now.Add(-time.Hour / 2), Closed: now.Add(-30 * time.Second)})
// Older than the hour: S9 no longer says it.
sayClosure(t, s, BusSlowConsumerEvent, BusClosure{User: "anchor.media", CID: 3, Reason: "Slow Consumer (Pending Bytes)",
Opened: now.Add(-3 * time.Hour), Closed: now.Add(-2 * time.Hour)})
got, err := s.BusFaults(t.Context(), now.Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
if len(got) != 2 {
t.Fatalf("read %d advisories, want one for each user within the hour: %+v", len(got), got)
}
for _, a := range got {
switch a.User {
case "laptop.claude-code":
if a.Kind != AdvisorySlowConsumer || a.ID != "laptop.claude-code" || a.Count != 2 ||
!a.Last.Equal(again.Closed) || !a.First.Equal(slow.Closed) || !strings.Contains(a.Said, "connection 31") {
t.Errorf("the slow consumer read as %+v: two closures, the one said twice counted once", a)
}
case "anchor.desk":
if a.Kind != AdvisoryConnectionClosed || a.Count != 1 || a.Reason != "Maximum Message Payload Exceeded" {
t.Errorf("the closed connection read as %+v", a)
}
default:
t.Errorf("an advisory for %s", a.User)
}
}
}
func TestNatsBusFaultsReadNothingWhenTheBusModuleSaidNothing(t *testing.T) {
s := aBusWithEvents(t)
started := time.Now()
got, err := s.BusFaults(t.Context(), time.Now().Add(-time.Hour))
if err != nil || len(got) != 0 {
t.Fatalf("read %v, %v from a stream the bus module never wrote", got, err)
}
if took := time.Since(started); took > time.Second {
t.Fatalf("reading nothing took %s: a tick must not wait on an empty stream", took)
}
}
func TestTheControllersOwnSlowConsumerAndItsClosedConnectionAreOneAdvisory(t *testing.T) {
at := time.Date(2026, 10, 14, 12, 0, 0, 0, time.UTC)
own := []Advisory{{Kind: AdvisorySlowConsumer, ID: "controller", Said: "the controller fell behind", First: at, Last: at, Count: 1}}
closed := BusClosures([]BusClosure{{User: "controller", CID: 5, Reason: "Slow Consumer (Pending Bytes)",
Opened: at.Add(-time.Hour), Closed: at.Add(time.Minute)}}, []string{AdvisorySlowConsumer}, at.Add(-time.Hour))
got := MergeAdvisories(own, closed)
if len(got) != 1 || got[0].Count != 1 || !got[0].Last.Equal(at.Add(time.Minute)) || got[0].User != "controller" {
t.Fatalf("merged as %+v: one fall behind, told to the client and closed by the server, is counted once", got)
}
}
// **A busy hour is read up to a bound, and says it was cut** (review of #326): a storm of closures must not
// spend a watchdog tick, and a count cut short is a floor, said as one.
func TestNatsBusFaultsReadAtMostTheBoundAndSaySo(t *testing.T) {
s := aBusWithEvents(t)
was := MaxBusEventsRead
MaxBusEventsRead = 2
t.Cleanup(func() { MaxBusEventsRead = was })
now := time.Now().UTC()
for i := range 3 {
sayClosure(t, s, BusSlowConsumerEvent, BusClosure{User: "laptop.claude-code", CID: uint64(10 + i),
Reason: "Slow Consumer (Pending Bytes)", Opened: now.Add(-time.Hour / 2), Closed: now.Add(-time.Duration(3-i) * time.Minute)})
}
got, err := s.BusFaults(t.Context(), now.Add(-time.Hour))
if err != nil {
t.Fatal(err)
}
if len(got) != 1 || got[0].Count != 2 || !strings.Contains(got[0].Said, "at least: the read stopped at 2") {
t.Fatalf("a read past its bound read as %+v", got)
}
}
-48
View File
@@ -37,51 +37,3 @@ func TestNatsTheBuildsAWalkMayTakeAreTheChecksOutcomesOfItsPullRequests(t *testi
t.Fatalf("read %+v", got)
}
}
// A build is read with the head its check judged, and a kept build is named as the check's that built it (novox/hq
// issue 703).
func TestNatsAKeptBuildIsTheBuildersCheckAndEveryBuildSaysItsHead(t *testing.T) {
js := aBus(t)
since := time.Now().Add(-time.Minute)
say := func(r BuildResult) {
body, _ := json.Marshal(r)
if _, err := js.Context().Publish(BuildOutcomeOf(TheBuildMachine), body); err != nil {
t.Fatal(err)
}
}
mine := &CheckRequest{Owner: "novox", Repo: "shop", Number: 7}
app := CheckBuilt{Module: "app", Key: "r1:1", Made: []MadeArtifact{{Name: "server", Reference: "s/app/server@sha256:a"}}}
say(BuildResult{ID: "build-1", Checked: mine, Check: &CheckOutcome{Verdict: "pass", Head: "aaaaaaaaaaaa",
Built: []CheckBuilt{app}}})
kept := app
kept.Kept = "build-1"
say(BuildResult{ID: "build-2", Checked: mine, Check: &CheckOutcome{Verdict: "pass", Head: "aaaaaaaaaaaa",
Built: []CheckBuilt{kept}}})
got, err := ReadCheckBuilds(t.Context(), js, TheBuildMachine, []PullRef{{Owner: "novox", Repo: "shop", Number: 7}}, since)
if err != nil {
t.Fatal(err)
}
if len(got) != 2 || got[0].Check != "build-1" || got[0].Built.Kept != "build-1" || got[0].Head != "aaaaaaaaaaaa" ||
got[1].Check != "build-1" || got[1].Head != "aaaaaaaaaaaa" {
t.Fatalf("read %+v", got)
}
}
// Two spellings of a head are the same head only when both say at least twelve characters of it.
func TestSameHeadNeedsTwelveCharactersOfBoth(t *testing.T) {
for _, c := range []struct {
a, b string
same bool
}{
{"aaaaaaaaaaaa", "aaaaaaaaaaaabbbb", true},
{"aaaaaaaaaaaabbbb", "aaaaaaaaaaaa", true},
{"aaaaaaaaaaaa", "aaaaaaaaaaab", false},
{"", "", false},
{"aaaa", "aaaaaaaaaaaa", false},
} {
if SameHead(c.a, c.b) != c.same {
t.Errorf("SameHead(%q, %q) is %v", c.a, c.b, !c.same)
}
}
}
+9 -15
View File
@@ -81,6 +81,11 @@ const (
// steps from. Said after every save of the serving controller; a delivery's holder also asks for the
// walks it follows, so a save made elsewhere is found by comparison, never lost.
KeyPlanMoved = "plan-moved"
// KeyAssignmentChanged and KeyDeclaredConfigChanged: a key of the controller's assignments or declared-config
// mirror was written or removed (novox/hq ADR 0289, design 54 §3, design 57 F3): which key, how and at which
// revision of the bucket — never the value, which the bucket holds.
KeyAssignmentChanged = "assignment-changed"
KeyDeclaredConfigChanged = "declared-config-changed"
)
// Applied is what a machine now runs, as the mesh states it.
@@ -172,13 +177,10 @@ type Announcement struct {
// which branch, producing which commit. The mesh matches it against every module's recorded
// source and builds what moved, bases first.
type SourceMoved struct {
Owner string `json:"owner"`
Repo string `json:"repo"`
Base string `json:"base"`
Head string `json:"head"`
// HeadSHA is the head commit the forge merged (novox/hq issue 703): a walk takes only a build a check of that
// head made. Empty from a forge module from before it, and then the walk takes no check's build.
HeadSHA string `json:"head_sha,omitempty"`
Owner string `json:"owner"`
Repo string `json:"repo"`
Base string `json:"base"`
Head string `json:"head"`
Commit string `json:"merge_commit_sha"`
CloneURL string `json:"clone_url"`
HTMLURL string `json:"html_url"`
@@ -293,14 +295,6 @@ type Checked struct {
Trunk string `json:"trunk,omitempty"`
Beside map[string]string `json:"beside,omitempty"`
Took string `json:"took,omitempty"`
// Release is the release branch the check judged for (novox/hq ADR 0358 §4), JudgedBy the commit of the
// controller that judged its gate (twelve characters), and BesideAt the branch each repository beside it was
// read at, by directory: so a reader sees a release-branch judgement apart from a trunk one. Empty otherwise.
Release string `json:"release,omitempty"`
JudgedBy string `json:"judged-by,omitempty"`
BesideAt map[string]string `json:"beside-at,omitempty"`
// Validator says which node-engine validator the judge of a release-branch check was built with (ADR 0358 §2).
Validator string `json:"validator,omitempty"`
// Built are the builds the check made for its merge, each with its reuse key and digests, or why not
// (novox/hq ADR 0350 §1): what the walk of the merge takes instead of building.
Built []CheckBuilt `json:"built,omitempty"`
+5 -16
View File
@@ -375,13 +375,6 @@ func ReadCheckBuilds(ctx context.Context, js *broker.JetStream, seat string, pul
for _, p := range pulls {
wanted[p] = true
}
return readCheckBuilds(ctx, js, seat, since, func(p PullRef, _ CheckBuilt) bool { return wanted[p] })
}
// readCheckBuilds is every build the seat's checks said they made, or kept, that wanted takes. A kept build is named
// as the check's that built it (novox/hq issue 703): the record under its key is that check's.
func readCheckBuilds(ctx context.Context, js *broker.JetStream, seat string, since time.Time,
wanted func(PullRef, CheckBuilt) bool) ([]CheckedBuild, error) {
subject := BuildOutcomeOf(seat)
sub, err := js.Context().PullSubscribe(subject, "",
nats.BindStream(broker.EventsStream), nats.StartTime(since), nats.AckNone())
@@ -409,13 +402,12 @@ func readCheckBuilds(ctx context.Context, js *broker.JetStream, seat string, sin
continue
}
pull := PullRef{Owner: r.Checked.Owner, Repo: r.Checked.Repo, Number: r.Checked.Number}
if !wanted[pull] {
continue
}
for _, b := range r.Check.Built {
if b.NotBuilt == "" && b.Key != "" && wanted(pull, b) {
check := r.ID
if b.Kept != "" {
check = b.Kept
}
out = append(out, CheckedBuild{Pull: pull, Check: check, Head: r.Check.Head, Built: b})
if b.NotBuilt == "" && b.Key != "" {
out = append(out, CheckedBuild{Pull: pull, Check: r.ID, Built: b})
}
}
}
@@ -435,9 +427,6 @@ func readCheckBuilds(ctx context.Context, js *broker.JetStream, seat string, sin
// safeKey is an id that can be a key, and a subject token, as it is: a build id, never a pattern.
var safeKey = regexp.MustCompile(`^[A-Za-z0-9_-]+$`)
// SafeBuildID says an id can be a subject token as it is: a build's id, never a pattern.
func SafeBuildID(id string) bool { return safeKey.MatchString(id) }
// Cancelled is what the controller writes for an ask it cancelled.
type Cancelled struct {
At string `json:"at"`
+2 -9
View File
@@ -26,8 +26,6 @@ import (
// - a check that has waited CheckWaitsAtMost goes before every delivery asked after it, so no check
// waits for ever behind a stream of deliveries;
// - within a kind, first come, first served, by when it was asked;
// - a branch tried before its pull request goes after everything else, however long it has waited: it is
// never promoted ahead of a delivery, and never ahead of a pull request's check (novox/hq issue 686);
// - an ask whose kind cannot be told counts as a check: a delivery taken late breaks a budget, a check
// taken late breaks nothing.
//
@@ -66,12 +64,8 @@ func (a QueuedAsk) askedOrStored() time.Time {
// QueueOrder is the order the waiting asks are to be handed out in, now: the deliveries and every check
// that has waited CheckWaitsAtMost, by when each was asked, then the other checks, by when each was asked.
func QueueOrder(waiting []QueuedAsk, now time.Time) []QueuedAsk {
var first, then, tries []QueuedAsk
var first, then []QueuedAsk
for _, a := range waiting {
if IsTry(a.Request) {
tries = append(tries, a)
continue
}
if KindOf(a.Request) == KindCheck && now.Sub(a.askedOrStored()) < CheckWaitsAtMost {
then = append(then, a)
continue
@@ -88,8 +82,7 @@ func QueueOrder(waiting []QueuedAsk, now time.Time) []QueuedAsk {
}
byAsking(first)
byAsking(then)
byAsking(tries)
return append(append(first, then...), tries...)
return append(first, then...)
}
// toMove is the asks to move to the tail, in this order, for the queue to stand as order says: everything
-55
View File
@@ -1,55 +0,0 @@
package link
import (
"encoding/json"
"reflect"
"strings"
"testing"
"time"
)
// The start of a try names no repository and no commit (novox/hq issue 686): the forge's holder writes
// mesh/merge-gate pending on the open pull request whose head a start names, and nothing on the forge moves
// for a try. A pull request's check still names both.
func TestATrysStartNamesNoCommitTheForgeCouldMatch(t *testing.T) {
sha := strings.Repeat("c0ffee", 7)[:40]
at := time.Date(2026, 10, 14, 0, 0, 0, 0, time.UTC)
try := BuildRequest{ID: "build-1", Repository: "https://forge/novox/a.git", Ref: sha,
Check: &CheckRequest{Owner: "novox", Repo: "a", Try: true}}
body, _ := json.Marshal(startOf(try, "ace", 1, at))
if strings.Contains(string(body), sha) || strings.Contains(string(body), "novox/a") {
t.Fatalf("a try's start names what the forge matches a pull request by: %s", body)
}
if s := startOf(try, "ace", 2, at); s.ID != "build-1" || s.On != "ace" || !s.Try || s.Delivery != 2 || s.At == "" {
t.Fatalf("a try's start lost what the queue reads: %+v", s)
}
pull := try
pull.Check = &CheckRequest{Owner: "novox", Repo: "a", Number: 7}
if s := startOf(pull, "ace", 1, at); s.Ref != sha || s.Repository == "" || s.Try {
t.Fatalf("a pull request's check's start: %+v", s)
}
}
// A try stays behind every delivery and every pull request's check, however long it has waited: the ten
// minutes that put a check ahead of later deliveries never promote a try (novox/hq issue 686).
func TestATryIsNeverHandedOutBeforeADeliveryOrAPullRequestsCheck(t *testing.T) {
now := time.Date(2026, 10, 12, 12, 0, 0, 0, time.UTC)
try := func(seq uint64, id string, asked time.Time) QueuedAsk {
a := anAsk(seq, id, asked, true)
a.Request.Check.Try = true
return a
}
waiting := []QueuedAsk{
try(1, "try-old", now.Add(-40*time.Minute)),
try(2, "try-young", now.Add(-time.Minute)),
anAsk(3, "check-young", now.Add(-2*time.Minute), true),
anAsk(4, "delivery", now.Add(-time.Minute), false),
}
want := []string{"delivery", "check-young", "try-old", "try-young"}
if got := idsOf(QueueOrder(waiting, now)); !reflect.DeepEqual(got, want) {
t.Fatalf("handed out as %v, want %v", got, want)
}
if !SafeBuildID("build-20261014T000000-1a2b") || SafeBuildID("x.>") || SafeBuildID("*") || SafeBuildID("") {
t.Fatal("an id that is a pattern passes for a build's id")
}
}
+344
View File
@@ -0,0 +1,344 @@
// Package recordmirror writes the controller's record mirrors (novox/hq ADR 0289, design 54): a bucket per kind
// of record, one key per record, written from the store only when the record changed, and one event on the
// controller's seat per write.
//
// **The store stays the record.** A mirror is built from what the store committed, so a change rolled back
// is never written. Every pass reads what the bucket holds and compares it byte for byte with what the store
// says: a pass over unchanged records writes nothing and says nothing (ADR 0289 §1), and a value somebody
// else wrote behind the writer's back (a former lease holder) differs from the store and is put right on
// the next pass. Each write names the revision it read (compare-and-set), so a write in between is noticed,
// never overwritten blind.
//
// **Only the lease holder writes** (ADR 0289 §9, design 54 §2): a pass begins by asking whether this
// controller holds the lease, and asks again before every write; without it, it writes nothing.
//
// **The bucket is the authority, the event a notice** (design 54 §3): the event is sent after the write
// succeeded, so there is no event for a value the bucket does not hold. A refused event is tried once more
// and then counted as lost; a watcher that must not miss a change watches the bucket.
//
// What this package does not do yet is design 54 §4's audit (probe D17 and its healer H6), which compares a
// mirror with its store behind the writer's back; novox/hq issue 419 carries it.
package recordmirror
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"regexp"
"sort"
"time"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
)
// A Mirror is one kind of record the controller keeps on the bus: its local name, its bucket and the event
// each write raises on the controller's seat.
type Mirror struct {
Name string
Bucket string
Event string
}
// The mirrors this package writes (novox/hq design 57, package F3). The assignments mirror is design 54's,
// keyed `<node>.<module>`; the declared config is each registered module's config declarations as its
// manifest holds them, keyed by the module, which mesh-config resolves a module's config values from
// (ADR 0348 clauses 2 and 4).
var (
Assignments = Mirror{Name: broker.MirrorAssignments, Bucket: broker.AssignmentsBucket,
Event: link.KeyAssignmentChanged}
DeclaredConfig = Mirror{Name: broker.MirrorDeclaredConfig, Bucket: broker.DeclaredConfigBucket,
Event: link.KeyDeclaredConfigChanged}
)
// Mirrors is every mirror this package writes, in the order they are written.
func Mirrors() []Mirror { return []Mirror{Assignments, DeclaredConfig} }
// The changes an event says.
const (
Added = "added"
Changed = "changed"
Removed = "removed"
)
// Event is what one write says on the controller's seat (design 54 §3): which mirror and key, what kind of
// change, the names of the top-level fields that changed, the bucket's revision of the write, the lease's
// epoch and when. Never the value: a watcher reads it from the bucket at that revision or later.
type Event struct {
Mirror string `json:"mirror"`
Key string `json:"key"`
Change string `json:"change"`
Fields []string `json:"fields,omitempty"`
Revision uint64 `json:"revision"`
Epoch uint64 `json:"epoch,omitempty"`
At time.Time `json:"at"`
}
// Announcer says an event on a seat; link.OverNATS is one, and so is link.Bus.
type Announcer interface {
PublishSeatEvent(ctx context.Context, seat, event string, body []byte) error
}
// MarkerTTL is how long the marker a purge leaves is kept (design 54 §2): a day, so the markers of removed
// records do not pile up in a bucket that has no age.
const MarkerTTL = 24 * time.Hour
// ErrNotHeld is a pass refused because this controller does not hold the lease: it wrote nothing.
var ErrNotHeld = errors.New("this controller does not hold the lease, so it writes no mirror")
// Writer writes one mirror. Not safe for use by two goroutines at once: one controller's one loop owns it.
type Writer struct {
Mirror Mirror
KV jetstream.KeyValue
Announce Announcer
// Epoch is the lease's epoch the events carry; nil carries none.
Epoch func() uint64
// Holds says this controller holds the lease. Asked before every pass and every write; a writer
// without it writes nothing (design 54 §2). Required: nil holds nothing.
Holds func() bool
// Now is the clock the events are dated by; nil is time.Now.
Now func() time.Time
known map[string]kept
// LostEvents counts the events refused twice since the writer was made.
LostEvents int
}
type kept struct {
value []byte
revision uint64
}
// Result is what one pass did: keys written, keys removed, and keys refused with why.
type Result struct {
Written int
Removed int
Refused int
}
// Sync makes the bucket hold want, one key per record: writes each value that differs from what the key
// holds, purges each key no record is left for, and raises one event per write. A key the bus cannot hold
// is refused alone, and the others are written; the error then names it.
func (w *Writer) Sync(ctx context.Context, want map[string][]byte) (Result, error) {
var r Result
if !w.holds() {
return r, ErrNotHeld
}
// What the bucket holds now, every pass: a value written behind this writer's back differs from the
// store's and is written again below.
if err := w.fill(ctx); err != nil {
return r, fmt.Errorf("reading the %s mirror: %w", w.Mirror.Name, err)
}
var problems []error
keys := make([]string, 0, len(want))
for k := range want {
keys = append(keys, k)
}
sort.Strings(keys)
for _, key := range keys {
value := want[key]
before, had := w.known[key]
if had && bytes.Equal(before.value, value) {
continue
}
if err := validKey(key); err != nil {
r.Refused++
problems = append(problems, err)
continue
}
if !w.holds() {
return r, errors.Join(append(problems, ErrNotHeld)...)
}
revision, wrote, err := w.put(ctx, key, value, before, had)
if err != nil {
problems = append(problems, fmt.Errorf("writing %s: %w", key, err))
continue
}
w.known[key] = kept{value: append([]byte(nil), value...), revision: revision}
if !wrote {
continue
}
r.Written++
change, fields := Added, []string(nil)
if had {
change, fields = Changed, changedFields(before.value, value)
}
w.say(ctx, key, change, fields, revision)
}
gone := make([]string, 0)
for key := range w.known {
if _, ok := want[key]; !ok {
gone = append(gone, key)
}
}
sort.Strings(gone)
for _, key := range gone {
k := w.known[key]
if !w.holds() {
return r, errors.Join(append(problems, ErrNotHeld)...)
}
err := w.KV.Purge(ctx, key, jetstream.LastRevision(k.revision), jetstream.PurgeTTL(MarkerTTL))
if refusedByRevision(err) {
// Read it again: purged already, or moved by another writer.
if e, gerr := w.KV.Get(ctx, key); errors.Is(gerr, jetstream.ErrKeyNotFound) {
delete(w.known, key)
continue
} else if gerr == nil && w.holds() {
err = w.KV.Purge(ctx, key, jetstream.LastRevision(e.Revision()), jetstream.PurgeTTL(MarkerTTL))
}
}
if err != nil {
problems = append(problems, fmt.Errorf("removing %s: %w", key, err))
continue
}
delete(w.known, key)
r.Removed++
revision := uint64(0)
if status, err := w.KV.Status(ctx); err == nil {
// The purge's own sequence is the stream's last one only when nothing wrote in between, which no
// other writer may; said as the bucket's revision after it.
if s, ok := status.(interface{ StreamInfo() *jetstream.StreamInfo }); ok {
revision = s.StreamInfo().State.LastSeq
}
}
w.say(ctx, key, Removed, nil, revision)
}
return r, errors.Join(problems...)
}
// put writes one value by compare-and-set and answers its revision, and whether this call wrote it (false
// when the bucket already held it).
func (w *Writer) put(ctx context.Context, key string, value []byte, before kept, had bool) (uint64, bool, error) {
var (
revision uint64
err error
)
if had {
revision, err = w.KV.Update(ctx, key, value, before.revision)
} else {
revision, err = w.KV.Create(ctx, key, value)
}
if !refusedByRevision(err) {
return revision, err == nil, err
}
// Refused: another writer in between, or this writer's own write whose acknowledgement was lost.
e, gerr := w.KV.Get(ctx, key)
switch {
case errors.Is(gerr, jetstream.ErrKeyNotFound):
if !w.holds() {
return 0, false, errors.New("the key moved and this controller no longer holds the lease")
}
revision, err = w.KV.Create(ctx, key, value)
return revision, err == nil, err
case gerr != nil:
return 0, false, gerr
case bytes.Equal(e.Value(), value):
return e.Revision(), false, nil
case !w.holds():
return 0, false, errors.New("the key moved and this controller no longer holds the lease")
}
revision, err = w.KV.Update(ctx, key, value, e.Revision())
return revision, err == nil, err
}
// say raises one write's event, tried once more when refused and then counted as lost.
func (w *Writer) say(ctx context.Context, key, change string, fields []string, revision uint64) {
if w.Announce == nil {
return
}
e := Event{Mirror: w.Mirror.Name, Key: key, Change: change, Fields: fields, Revision: revision, At: w.now().UTC()}
if w.Epoch != nil {
e.Epoch = w.Epoch()
}
body, err := json.Marshal(e)
if err != nil {
w.LostEvents++
return
}
for try := 0; try < 2; try++ {
if err = w.Announce.PublishSeatEvent(ctx, broker.ControllerSeat, w.Mirror.Event, body); err == nil {
return
}
}
w.LostEvents++
}
// fill reads what the bucket holds now, so a controller started again writes only what differs.
func (w *Writer) fill(ctx context.Context) error {
w.known = map[string]kept{}
watch, err := w.KV.WatchAll(ctx, jetstream.IgnoreDeletes())
if err != nil {
return err
}
defer func() { _ = watch.Stop() }()
for {
select {
case <-ctx.Done():
return ctx.Err()
case e := <-watch.Updates():
if e == nil {
return nil
}
w.known[e.Key()] = kept{value: append([]byte(nil), e.Value()...), revision: e.Revision()}
}
}
}
func (w *Writer) holds() bool { return w.Holds != nil && w.Holds() }
func (w *Writer) now() time.Time {
if w.Now == nil {
return time.Now()
}
return w.Now()
}
// refusedByRevision is the server refusing a compare-and-set: the key is not at the revision named.
func refusedByRevision(err error) bool {
if err == nil {
return false
}
if errors.Is(err, jetstream.ErrKeyExists) {
return true
}
var apiErr *jetstream.APIError
return errors.As(err, &apiErr) && apiErr.ErrorCode == jetstream.JSErrCodeStreamWrongLastSequence
}
// keyChars is what a bucket's key may be made of.
var keyChars = regexp.MustCompile(`^[-/_=.a-zA-Z0-9]+$`)
// validKey refuses a key the bus cannot hold, naming it.
func validKey(key string) error {
if key == "" || !keyChars.MatchString(key) || key[0] == '.' || key[len(key)-1] == '.' {
return fmt.Errorf("%q cannot be a key of a bucket: letters, digits, '-', '_', '=', '/' and '.', not at "+
"either end", key)
}
return nil
}
// changedFields names the top-level fields whose values differ between two encoded objects, sorted; nil
// when either is not an object.
func changedFields(before, after []byte) []string {
var a, b map[string]json.RawMessage
if json.Unmarshal(before, &a) != nil || json.Unmarshal(after, &b) != nil || a == nil || b == nil {
return nil
}
var out []string
for k, v := range b {
if old, ok := a[k]; !ok || !bytes.Equal(old, v) {
out = append(out, k)
}
}
for k := range a {
if _, ok := b[k]; !ok {
out = append(out, k)
}
}
sort.Strings(out)
return out
}
+178
View File
@@ -0,0 +1,178 @@
package recordmirror
import (
"encoding/json"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/catalogue"
"github.com/novox/mesh-controller/internal/inventory"
)
// **Design 57's F3, end to end** (novox/hq issue 652): from the controller's store to a real bus. An
// assignment changes exactly one key of the assignments mirror and raises exactly one `assignment-changed`;
// a registration changes exactly one key of the declared-config mirror and raises exactly one
// `declared-config-changed`; and the same records written again change nothing and say nothing — a module
// registered again unchanged included, although the store's registration time moved.
func TestNatsAnAssignmentAndARegistrationEachChangeOneKeyAndRaiseOneEvent(t *testing.T) {
inv := inventory.ForTest(t)
ctx := t.Context()
js, api := aMirrorBus(t)
assignments := aWriter(t, js, api, Assignments)
declared := aWriter(t, js, api, DeclaredConfig)
heardAssignments := listen(t, js.Conn(), "mesh.seat."+broker.ControllerSeat+".event."+Assignments.Event)
heardDeclared := listen(t, js.Conn(), "mesh.seat."+broker.ControllerSeat+".event."+DeclaredConfig.Event)
pass := func() (Result, Result) {
t.Helper()
a, err := inv.AssignmentsMirror(ctx)
if err != nil {
t.Fatal(err)
}
ra, err := assignments.Sync(ctx, a)
if err != nil {
t.Fatal(err)
}
d, err := inv.DeclaredConfigMirror(ctx)
if err != nil {
t.Fatal(err)
}
rd, err := declared.Sync(ctx, d)
if err != nil {
t.Fatal(err)
}
return ra, rd
}
if _, err := inv.AddNode(ctx, "anchor"); err != nil {
t.Fatal(err)
}
ui := catalogue.Manifest{Module: "ui", Version: "1", Settings: map[string]catalogue.SettingDeclaration{
"font-size": {Kind: catalogue.KindPreference, Default: 14, Why: "readable on the desktop"}}}
if err := inv.RegisterModule(ctx, ui, inventory.Source{}); err != nil {
t.Fatal(err)
}
pass()
assignmentWrites, declaredWrites := writes(t, api, Assignments.Bucket), writes(t, api, DeclaredConfig.Bucket)
assignmentsSaid, declaredSaid := len(settle(heardAssignments, 0)), len(settle(heardDeclared, 1))
// **A registration**: a module new to the catalogue.
if err := inv.RegisterModule(ctx, catalogue.Manifest{Module: "notes", Version: "1"}, inventory.Source{}); err != nil {
t.Fatal(err)
}
ra, rd := pass()
if ra != (Result{}) || rd != (Result{Written: 1}) {
t.Errorf("a registration wrote assignments %+v and declared config %+v", ra, rd)
}
if got := writes(t, api, DeclaredConfig.Bucket) - declaredWrites; got != 1 {
t.Errorf("a registration changed %d keys of the declared config", got)
}
if got := writes(t, api, Assignments.Bucket) - assignmentWrites; got != 0 {
t.Errorf("a registration changed %d keys of the assignments", got)
}
said := settle(heardDeclared, declaredSaid+1)[declaredSaid:]
if len(said) != 1 || said[0].Key != "notes" || said[0].Change != Added {
t.Errorf("a registration raised %+v", said)
}
declaredWrites, declaredSaid = writes(t, api, DeclaredConfig.Bucket), declaredSaid+len(said)
// **An assignment.**
if _, err := inv.Assign(ctx, "anchor", "ui"); err != nil {
t.Fatal(err)
}
ra, rd = pass()
if ra != (Result{Written: 1}) || rd != (Result{}) {
t.Errorf("an assignment wrote assignments %+v and declared config %+v", ra, rd)
}
if got := writes(t, api, Assignments.Bucket) - assignmentWrites; got != 1 {
t.Errorf("an assignment changed %d keys of the assignments", got)
}
said = settle(heardAssignments, assignmentsSaid+1)[assignmentsSaid:]
if len(said) != 1 || said[0].Key != "anchor.ui" || said[0].Change != Added {
t.Errorf("an assignment raised %+v", said)
}
assignmentWrites, assignmentsSaid = writes(t, api, Assignments.Bucket), assignmentsSaid+len(said)
e, err := assignments.KV.Get(ctx, "anchor.ui")
if err != nil {
t.Fatal(err)
}
var value inventory.AssignmentMirrored
if err := json.Unmarshal(e.Value(), &value); err != nil || value.Node != "anchor" || value.Module != "ui" ||
value.Assigned.IsZero() {
t.Errorf("the assignment is mirrored as %s (%v)", e.Value(), err)
}
// **The same records written again change nothing**: an assignment made again, and a module registered
// again with the manifest it had.
if _, err := inv.Assign(ctx, "anchor", "ui"); err != nil {
t.Fatal(err)
}
if err := inv.RegisterModule(ctx, ui, inventory.Source{}); err != nil {
t.Fatal(err)
}
if ra, rd := pass(); ra != (Result{}) || rd != (Result{}) {
t.Errorf("unchanged records wrote assignments %+v and declared config %+v", ra, rd)
}
if writes(t, api, Assignments.Bucket) != assignmentWrites || writes(t, api, DeclaredConfig.Bucket) != declaredWrites {
t.Error("unchanged records changed a key")
}
if a, d := settle(heardAssignments, assignmentsSaid+1), settle(heardDeclared, declaredSaid+1); len(a) != assignmentsSaid ||
len(d) != declaredSaid {
t.Errorf("unchanged records raised %+v and %+v", a[assignmentsSaid:], d[declaredSaid:])
}
// **A declaration changed by a registration**: one key, one event, naming the field that moved.
ui.Version = "2"
ui.Settings["font-size"] = catalogue.SettingDeclaration{Kind: catalogue.KindPreference, Default: 16,
Why: "readable on the desktop"}
if err := inv.RegisterModule(ctx, ui, inventory.Source{}); err != nil {
t.Fatal(err)
}
if ra, rd := pass(); ra != (Result{}) || rd != (Result{Written: 1}) {
t.Errorf("a changed declaration wrote assignments %+v and declared config %+v", ra, rd)
}
said = settle(heardDeclared, declaredSaid+1)[declaredSaid:]
if len(said) != 1 || said[0].Key != "ui" || said[0].Change != Changed || strings.Join(said[0].Fields, ",") != "settings" {
t.Errorf("a changed declaration raised %+v", said)
}
got, err := declared.KV.Get(ctx, "ui")
if err != nil {
t.Fatal(err)
}
if want := `{"module":"ui","settings":{"font-size":{"kind":"preference","default":16,"why":"readable on the desktop"}}}`; string(got.Value()) != want {
t.Errorf("the declaration is mirrored as %s, want %s", got.Value(), want)
}
// **An unassignment** removes its key, said once.
if err := inv.Unassign(ctx, "anchor", "ui"); err != nil {
t.Fatal(err)
}
if ra, _ := pass(); ra != (Result{Removed: 1}) {
t.Errorf("an unassignment did %+v", ra)
}
said = settle(heardAssignments, assignmentsSaid+1)[assignmentsSaid:]
if len(said) != 1 || said[0].Key != "anchor.ui" || said[0].Change != Removed {
t.Errorf("an unassignment raised %+v", said)
}
// **A module no longer registered** has its key removed, said once as `removed` (ADR 0356 §4).
if err := inv.ForgetModule(ctx, "notes"); err != nil {
t.Fatal(err)
}
declaredWrites = writes(t, api, DeclaredConfig.Bucket)
declaredSaid = len(heardDeclared.all())
if _, rd := pass(); rd != (Result{Removed: 1}) {
t.Errorf("a module forgotten did %+v", rd)
}
if got := writes(t, api, DeclaredConfig.Bucket) - declaredWrites; got != 1 {
t.Errorf("a module forgotten changed %d keys of the declared config", got)
}
if _, err := declared.KV.Get(ctx, "notes"); err == nil {
t.Error("a module forgotten keeps its key")
}
said = settle(heardDeclared, declaredSaid+1)[declaredSaid:]
if len(said) != 1 || said[0].Key != "notes" || said[0].Change != Removed {
t.Errorf("a module forgotten raised %+v", said)
}
}
+356
View File
@@ -0,0 +1,356 @@
package recordmirror
import (
"encoding/json"
"errors"
"slices"
"sync"
"testing"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"github.com/novox/mesh-controller/internal/broker"
"github.com/novox/mesh-controller/internal/link"
"github.com/novox/mesh-controller/internal/testbus"
)
// A record mirror against a real bus (novox/hq design 54 §2, §3; ADR 0289): a change of a record changes
// exactly one key and raises exactly one event; a pass over unchanged records writes nothing and says
// nothing; a record gone is a purge said as `removed`.
// aMirrorBus is a bus of this test alone with the mesh's streams and the controller's buckets, as the
// controller asserts them at its start, and a counter of the writes made to one bucket.
func aMirrorBus(t *testing.T) (*broker.JetStream, jetstream.JetStream) {
t.Helper()
js, err := broker.Dial(testbus.URL(t))
if err != nil {
t.Fatal(err)
}
t.Cleanup(js.Close)
if err := broker.AssertMeshStreams(js); err != nil {
t.Fatal(err)
}
if err := js.EnsureControllerBuckets(); err != nil {
t.Fatal(err)
}
api, err := jetstream.New(js.Conn())
if err != nil {
t.Fatal(err)
}
return js, api
}
// heard counts what is said on one subject, each body kept.
type heard struct {
mu sync.Mutex
bodies []Event
}
func (h *heard) all() []Event {
h.mu.Lock()
defer h.mu.Unlock()
return append([]Event(nil), h.bodies...)
}
func listen(t *testing.T, conn *nats.Conn, subject string) *heard {
t.Helper()
h := &heard{}
sub, err := conn.Subscribe(subject, func(m *nats.Msg) {
var e Event
if err := json.Unmarshal(m.Data, &e); err != nil {
t.Errorf("an event that is not one: %s", m.Data)
return
}
h.mu.Lock()
h.bodies = append(h.bodies, e)
h.mu.Unlock()
})
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { _ = sub.Unsubscribe() })
if err := conn.Flush(); err != nil {
t.Fatal(err)
}
return h
}
// writes is how many messages the bucket's stream has taken: every put and every purge is one.
func writes(t *testing.T, api jetstream.JetStream, bucket string) uint64 {
t.Helper()
s, err := api.Stream(t.Context(), "KV_"+bucket)
if err != nil {
t.Fatal(err)
}
info, err := s.Info(t.Context())
if err != nil {
t.Fatal(err)
}
return info.State.LastSeq
}
// settle waits until n events are heard or a second passed, and answers what was heard.
func settle(h *heard, n int) []Event {
deadline := time.Now().Add(time.Second)
for time.Now().Before(deadline) && len(h.all()) < n {
time.Sleep(10 * time.Millisecond)
}
// And a moment more, so an event too many is heard too.
time.Sleep(100 * time.Millisecond)
return h.all()
}
func aWriter(t *testing.T, js *broker.JetStream, api jetstream.JetStream, m Mirror) *Writer {
t.Helper()
kv, err := api.KeyValue(t.Context(), m.Bucket)
if err != nil {
t.Fatal(err)
}
return &Writer{Mirror: m, KV: kv, Announce: link.OverNATS{Conn: js.Conn(), JS: js.Context()},
Epoch: func() uint64 { return 7 }, Holds: func() bool { return true }}
}
func TestNatsAMirrorChangesOneKeyAndRaisesOneEventPerChange(t *testing.T) {
for _, m := range []Mirror{Assignments, DeclaredConfig} {
t.Run(m.Name, func(t *testing.T) {
js, api := aMirrorBus(t)
events := listen(t, js.Conn(), "mesh.seat."+broker.ControllerSeat+".event."+m.Event)
w := aWriter(t, js, api, m)
ctx := t.Context()
first := map[string][]byte{"anchor.gitea": []byte(`{"a":1}`)}
if _, err := w.Sync(ctx, first); err != nil {
t.Fatal(err)
}
before := writes(t, api, m.Bucket)
heardBefore := len(settle(events, 1))
// One record changes, one is added: two keys, two events, nothing else.
second := map[string][]byte{"anchor.gitea": []byte(`{"a":2}`), "anchor.forgejo": []byte(`{"a":1}`)}
r, err := w.Sync(ctx, second)
if err != nil {
t.Fatal(err)
}
if r.Written != 2 || r.Removed != 0 {
t.Errorf("a pass over one changed and one new record wrote %+v", r)
}
if got := writes(t, api, m.Bucket) - before; got != 2 {
t.Errorf("the bucket took %d writes for two changed records", got)
}
said := settle(events, heardBefore+2)[heardBefore:]
if len(said) != 2 {
t.Fatalf("two changes raised %d events: %+v", len(said), said)
}
changes := map[string]Event{}
for _, e := range said {
changes[e.Key] = e
}
if e := changes["anchor.gitea"]; e.Change != Changed || !slices.Equal(e.Fields, []string{"a"}) ||
e.Mirror != m.Name || e.Epoch != 7 || e.Revision == 0 {
t.Errorf("the change of anchor.gitea was said as %+v", e)
}
if e := changes["anchor.forgejo"]; e.Change != Added {
t.Errorf("the new anchor.forgejo was said as %+v", e)
}
// **An unchanged pass writes nothing**, and says nothing (ADR 0289 §1).
heardBefore = len(events.all())
before = writes(t, api, m.Bucket)
r, err = w.Sync(ctx, second)
if err != nil {
t.Fatal(err)
}
if r != (Result{}) {
t.Errorf("a pass over unchanged records did %+v", r)
}
if got := writes(t, api, m.Bucket) - before; got != 0 {
t.Errorf("a pass over unchanged records wrote the bucket %d times", got)
}
if said := settle(events, heardBefore+1)[heardBefore:]; len(said) != 0 {
t.Errorf("a pass over unchanged records raised %+v", said)
}
// **A record gone is a purge**, one write and one `removed`.
before = writes(t, api, m.Bucket)
r, err = w.Sync(ctx, map[string][]byte{"anchor.gitea": []byte(`{"a":2}`)})
if err != nil {
t.Fatal(err)
}
if r.Removed != 1 || r.Written != 0 {
t.Errorf("a record gone did %+v", r)
}
if got := writes(t, api, m.Bucket) - before; got != 1 {
t.Errorf("a record gone wrote the bucket %d times", got)
}
if _, err := w.KV.Get(ctx, "anchor.forgejo"); !errors.Is(err, jetstream.ErrKeyNotFound) {
t.Errorf("the key of a record gone is still read: %v", err)
}
said = settle(events, heardBefore+1)[heardBefore:]
if len(said) != 1 || said[0].Change != Removed || said[0].Key != "anchor.forgejo" {
t.Errorf("a record gone was said as %+v", said)
}
})
}
}
// **A controller started again writes nothing for what the bucket already holds** (design 54 §2: the writer's
// memory is filled from the bucket), and a key nobody holds a record for is removed.
func TestNatsAWriterStartedAgainWritesOnlyWhatDiffers(t *testing.T) {
js, api := aMirrorBus(t)
ctx := t.Context()
first := aWriter(t, js, api, Assignments)
held := map[string][]byte{"anchor.gitea": []byte(`{"a":1}`), "anchor.forgejo": []byte(`{"a":1}`)}
if _, err := first.Sync(ctx, held); err != nil {
t.Fatal(err)
}
before := writes(t, api, Assignments.Bucket)
again := aWriter(t, js, api, Assignments)
r, err := again.Sync(ctx, held)
if err != nil {
t.Fatal(err)
}
if r != (Result{}) || writes(t, api, Assignments.Bucket) != before {
t.Errorf("a writer started again over unchanged records did %+v", r)
}
}
// **Another writer in between is no lost value** (design 54 §2, compare-and-set): a write refused because the
// key moved reads it again, takes the revision when it already holds the value, and otherwise writes again.
func TestNatsAWriteRefusedByCompareAndSetReadsAgain(t *testing.T) {
js, api := aMirrorBus(t)
ctx := t.Context()
w := aWriter(t, js, api, Assignments)
if _, err := w.Sync(ctx, map[string][]byte{"anchor.gitea": []byte(`{"a":1}`)}); err != nil {
t.Fatal(err)
}
// Behind its back, as a former lease holder would.
if _, err := w.KV.Put(ctx, "anchor.gitea", []byte(`{"a":9}`)); err != nil {
t.Fatal(err)
}
r, err := w.Sync(ctx, map[string][]byte{"anchor.gitea": []byte(`{"a":2}`)})
if err != nil {
t.Fatal(err)
}
if r.Written != 1 {
t.Errorf("a refused write was not made again: %+v", r)
}
e, err := w.KV.Get(ctx, "anchor.gitea")
if err != nil || string(e.Value()) != `{"a":2}` {
t.Fatalf("the bucket holds %v, %v", e, err)
}
}
// **A value written behind the writer's back is put right on the next pass** (ADR 0289 §10): a former lease
// holder's write differs from the store, and the store's value is written again, said as one change.
func TestNatsAForeignWriteIsCorrectedOnTheNextPass(t *testing.T) {
js, api := aMirrorBus(t)
ctx := t.Context()
events := listen(t, js.Conn(), "mesh.seat."+broker.ControllerSeat+".event."+Assignments.Event)
w := aWriter(t, js, api, Assignments)
store := map[string][]byte{"anchor.gitea": []byte(`{"a":1}`)}
if _, err := w.Sync(ctx, store); err != nil {
t.Fatal(err)
}
heardBefore := len(settle(events, 1))
if _, err := w.KV.Put(ctx, "anchor.gitea", []byte(`{"a":9}`)); err != nil {
t.Fatal(err)
}
r, err := w.Sync(ctx, store)
if err != nil {
t.Fatal(err)
}
if r != (Result{Written: 1}) {
t.Errorf("a pass after a foreign write did %+v", r)
}
e, err := w.KV.Get(ctx, "anchor.gitea")
if err != nil || string(e.Value()) != `{"a":1}` {
t.Fatalf("the bucket still holds %v, %v", e, err)
}
if said := settle(events, heardBefore+1)[heardBefore:]; len(said) != 1 || said[0].Change != Changed {
t.Errorf("the correction was said as %+v", said)
}
// And put right, it stays: the next pass writes nothing.
if r, err := w.Sync(ctx, store); err != nil || r != (Result{}) {
t.Errorf("a pass after the correction did %+v, %v", r, err)
}
}
// **A writer without the lease writes nothing** (ADR 0289 §9, design 54 §2): not a new key, not a change, not
// a removal, and it says so.
func TestNatsAWriterWithoutTheLeaseWritesNothing(t *testing.T) {
js, api := aMirrorBus(t)
ctx := t.Context()
held := aWriter(t, js, api, Assignments)
if _, err := held.Sync(ctx, map[string][]byte{"anchor.gitea": []byte(`{"a":1}`)}); err != nil {
t.Fatal(err)
}
before := writes(t, api, Assignments.Bucket)
for _, holds := range []func() bool{nil, func() bool { return false }} {
w := aWriter(t, js, api, Assignments)
w.Holds = holds
r, err := w.Sync(ctx, map[string][]byte{"anchor.gitea": []byte(`{"a":2}`), "anchor.forgejo": []byte(`{"a":1}`)})
if !errors.Is(err, ErrNotHeld) || r != (Result{}) {
t.Errorf("a writer without the lease did %+v, %v", r, err)
}
r, err = w.Sync(ctx, map[string][]byte{})
if !errors.Is(err, ErrNotHeld) || r != (Result{}) {
t.Errorf("a writer without the lease removed: %+v, %v", r, err)
}
}
// And one that loses it mid-pass stops before its next write.
asked := 0
w := aWriter(t, js, api, Assignments)
w.Holds = func() bool { asked++; return asked <= 2 }
r, err := w.Sync(ctx, map[string][]byte{"anchor.a": []byte(`{}`), "anchor.b": []byte(`{}`), "anchor.gitea": []byte(`{"a":1}`)})
if !errors.Is(err, ErrNotHeld) || r.Written != 1 {
t.Errorf("a writer that lost the lease mid-pass did %+v, %v", r, err)
}
if got := writes(t, api, Assignments.Bucket) - before; got != 1 {
t.Errorf("writers without the lease wrote the bucket %d times", got)
}
}
// **A removal asks for the lease too** (ADR 0289 §9): a writer that held the lease when its pass began and lost
// it before its first removal purges nothing, and says it does not hold the lease.
func TestNatsARemovalWithoutTheLeaseRemovesNothing(t *testing.T) {
js, api := aMirrorBus(t)
ctx := t.Context()
held := aWriter(t, js, api, Assignments)
if _, err := held.Sync(ctx, map[string][]byte{"anchor.gitea": []byte(`{"a":1}`)}); err != nil {
t.Fatal(err)
}
before := writes(t, api, Assignments.Bucket)
asked := 0
w := aWriter(t, js, api, Assignments)
w.Holds = func() bool { asked++; return asked == 1 }
r, err := w.Sync(ctx, map[string][]byte{})
if !errors.Is(err, ErrNotHeld) || r != (Result{}) || asked != 2 {
t.Errorf("a removal without the lease did %+v, %v (the lease asked %d times)", r, err, asked)
}
if got := writes(t, api, Assignments.Bucket) - before; got != 0 {
t.Errorf("a removal without the lease wrote the bucket %d times", got)
}
if _, err := w.KV.Get(ctx, "anchor.gitea"); err != nil {
t.Errorf("the key was removed without the lease: %v", err)
}
}
// **A key the bus cannot hold is refused alone**, and the others are written.
func TestAKeyTheBusCannotHoldIsRefusedAlone(t *testing.T) {
if validKey("anchor.gitea") != nil || validKey("bad key") == nil || validKey(".lead") == nil ||
validKey("") == nil || validKey("trail.") == nil {
t.Error("the keys a bucket may hold are not told apart")
}
}
func TestTheFieldsThatChangedAreNamed(t *testing.T) {
got := changedFields([]byte(`{"a":1,"b":{"x":1},"c":"same"}`), []byte(`{"a":2,"b":{"x":1},"d":true,"c":"same"}`))
if !slices.Equal(got, []string{"a", "d"}) {
t.Errorf("the fields that changed are %v", got)
}
if got := changedFields(nil, []byte(`{"a":1}`)); got != nil {
t.Errorf("an added key names fields %v", got)
}
}
+181
View File
@@ -0,0 +1,181 @@
package recordmirror
import (
"go/ast"
"go/parser"
"go/token"
"io/fs"
"os"
"path/filepath"
"slices"
"strconv"
"strings"
"testing"
"github.com/novox/mesh-controller/internal/broker"
)
// mayNameAMirror are the files of the controller that may name a record mirror's bucket outside their tests:
// where the buckets are defined, asserted and granted (internal/broker), the list a reader names in `reads`
// (internal/catalogue/state.go), and each mirror's own writer (this package and the loop that runs it). The
// audit of a mirror against its store (ADR 0289 §4, issue 419) joins this list when it is built.
var mayNameAMirror = []string{
"cmd/mesh-controller/record_mirrors.go",
"internal/broker/controller_buckets.go",
"internal/broker/writers.go",
"internal/catalogue/state.go",
}
// The packages whose names stand for a mirror, by import path, and the names in each.
const (
brokerPath = "github.com/novox/mesh-controller/internal/broker"
recordmirrorPath = "github.com/novox/mesh-controller/internal/recordmirror"
)
var mirrorNames = map[string][]string{
brokerPath: {"AssignmentsBucket", "DeclaredConfigBucket", "MirrorAssignments", "MirrorDeclaredConfig",
"ControllerMirrors"},
recordmirrorPath: {"Assignments", "DeclaredConfig", "Mirrors"},
}
// **Nothing in the controller reads a mirror to decide anything** (ADR 0289 §19 as ADR 0356 §2 changes it): no
// file of the controller's packages outside its tests names a mirror's bucket but the audit and a mirror's own
// writer. A reader elsewhere fails here, naming the file and what it named.
//
// **What it catches**: a name that stands for a mirror, qualified by the package it comes from under any
// local name the file imports it as (`broker.AssignmentsBucket`, `b.AssignmentsBucket`); the same names bare
// in a file of package broker or package recordmirror; and a string literal holding a mirror's bucket name.
// **What it does not catch**: a bucket name built at run time, such as `BucketName(ControllerSeat,
// "assignments")` or literals concatenated into one, and a mirror reached through a value passed in from an
// allowed file. Those are deliberate evasions rather than ordinary code; a reviewer judges them.
func TestOnlyTheAuditAndAMirrorsWriterReadAMirror(t *testing.T) {
var buckets []string
for _, m := range broker.ControllerMirrors() {
buckets = append(buckets, broker.BucketName(broker.ControllerSeat, m))
}
root := filepath.Join("..", "..")
found := mirrorReaders(t, root, buckets)
if len(found) > 0 {
t.Fatalf("a package of the controller reads a record mirror, which only the audit and the mirror's "+
"writer may (ADR 0289 §19, ADR 0356 §2):\n %s", strings.Join(found, "\n "))
}
}
// **The reader test sees what it says it sees**: in a planted tree, each planted reader is named, and an allowed
// file and a selector on a value are not.
func TestTheReaderTestFindsPlantedReaders(t *testing.T) {
root := t.TempDir()
plant := func(path, src string) {
full := filepath.Join(root, path)
if err := os.MkdirAll(filepath.Dir(full), 0o755); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(full, []byte(src), 0o644); err != nil {
t.Fatal(err)
}
}
plant("cmd/mesh-controller/qualified.go", "package main\n\nimport \""+brokerPath+"\"\n\nvar a = broker.AssignmentsBucket\n")
plant("internal/x/aliased.go", "package x\n\nimport b \""+brokerPath+"\"\n\nvar a = b.DeclaredConfigBucket\n")
plant("internal/broker/zz_reader.go", "package broker\n\nfunc zzR() string { return AssignmentsBucket }\n")
plant("internal/inventory/literal.go", "package inventory\n\nvar a = \"mesh-controller_declared-config\"\n")
plant("internal/y/writer.go", "package y\n\nimport \""+recordmirrorPath+"\"\n\nvar a = recordmirror.Assignments\n")
// Allowed: a mirror's definition and its writer.
plant("internal/broker/controller_buckets.go", "package broker\n\nvar AssignmentsBucket = \"x\"\n")
plant("internal/broker/other.go", "package broker\n\nfunc other() string { return \"nothing\" }\n")
// A selector on a value (v.AssignmentsBucket) is no use of the mirror, but declaring a field of a mirror's
// name in package broker is named, on the safe side; and a bare use inside a call under a selector is found.
plant("internal/broker/field.go", "package broker\n\ntype s struct{ AssignmentsBucket string }\n\nfunc f(v s) string { return v.AssignmentsBucket }\n")
plant("internal/broker/nested.go", "package broker\n\nfunc g() int { return len(h(DeclaredConfigBucket).x) }\n")
found := mirrorReaders(t, root, []string{"mesh-controller_assignments", "mesh-controller_declared-config"})
want := []string{
"cmd/mesh-controller/qualified.go: broker.AssignmentsBucket",
"internal/broker/field.go: AssignmentsBucket",
"internal/broker/nested.go: DeclaredConfigBucket",
"internal/broker/zz_reader.go: AssignmentsBucket",
"internal/inventory/literal.go: \"mesh-controller_declared-config\"",
"internal/x/aliased.go: b.DeclaredConfigBucket",
"internal/y/writer.go: recordmirror.Assignments",
}
slices.Sort(found)
if !slices.Equal(found, want) {
t.Errorf("the planted readers were found as\n %s\nwant\n %s", strings.Join(found, "\n "),
strings.Join(want, "\n "))
}
}
// mirrorReaders is every place under root's cmd and internal, outside tests and the allowed files, that names a
// mirror.
func mirrorReaders(t *testing.T, root string, buckets []string) []string {
t.Helper()
var found []string
for _, dir := range []string{"cmd", "internal"} {
err := filepath.WalkDir(filepath.Join(root, dir), func(path string, e fs.DirEntry, err error) error {
if err != nil || e.IsDir() || !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") {
return err
}
rel, err := filepath.Rel(root, path)
if err != nil {
return err
}
rel = filepath.ToSlash(rel)
if strings.HasPrefix(rel, "internal/recordmirror/") || slices.Contains(mayNameAMirror, rel) {
return nil
}
f, err := parser.ParseFile(token.NewFileSet(), path, nil, 0)
if err != nil {
return err
}
// Each import's local name, mapped to the package it stands for.
imported := map[string]string{}
for _, imp := range f.Imports {
p, _ := strconv.Unquote(imp.Path.Value)
if _, ok := mirrorNames[p]; !ok {
continue
}
local := filepath.Base(p)
if imp.Name != nil {
local = imp.Name.Name
}
imported[local] = p
}
// A file of the package that defines the names may use them bare.
var bare []string
switch f.Name.Name {
case "broker":
bare = mirrorNames[brokerPath]
case "recordmirror":
bare = mirrorNames[recordmirrorPath]
}
ast.Inspect(f, func(n ast.Node) bool {
switch n := n.(type) {
case *ast.SelectorExpr:
pkg, ok := n.X.(*ast.Ident)
if !ok {
return true
}
if p, imp := imported[pkg.Name]; imp && slices.Contains(mirrorNames[p], n.Sel.Name) {
found = append(found, rel+": "+pkg.Name+"."+n.Sel.Name)
}
// Neither a selector's own name nor its package is a bare use.
return false
case *ast.Ident:
if slices.Contains(bare, n.Name) {
found = append(found, rel+": "+n.Name)
}
case *ast.BasicLit:
for _, b := range buckets {
if n.Kind == token.STRING && strings.Contains(n.Value, b) {
found = append(found, rel+": "+n.Value)
}
}
}
return true
})
return nil
})
if err != nil {
t.Fatal(err)
}
}
return found
}
+4 -19
View File
@@ -54,26 +54,11 @@ if [ -n "$unformatted" ]; then
fi
CGO_ENABLED=0 go vet ./...
suite() {
if command -v gcc >/dev/null 2>&1; then
CGO_ENABLED=1 go test -race -count=1 -timeout 30m "$@"
else
echo "NOT RACE-CHECKED: the toolchain holds no C compiler; the suite runs without the race detector"
CGO_ENABLED=0 go test -count=1 -timeout 30m "$@"
fi
}
# **A branch tried before its pull request may ask for a part** (novox/hq issue 686): the controller's `try` verb
# hands MESH_CHECK_TESTS, a `go test -run` pattern, or MESH_CHECK_STORE_ONLY=1. A pull request's merge check is
# handed neither and always runs whole; a part that passes is said as a part, never as the check.
if [ -n "${MESH_CHECK_TESTS:-}" ]; then
echo "TRIED IN PART: only the tests matching $MESH_CHECK_TESTS, in every package"
suite -run "$MESH_CHECK_TESTS" ./...
exit 0
fi
if [ "${MESH_CHECK_STORE_ONLY:-}" = 1 ]; then
echo "TRIED IN PART: only the store tests listed in testdata/store-tests"
if command -v gcc >/dev/null 2>&1; then
CGO_ENABLED=1 go test -race -count=1 -timeout 30m ./...
else
suite ./...
echo "NOT RACE-CHECKED: the toolchain holds no C compiler; the suite runs without the race detector"
CGO_ENABLED=0 go test -count=1 -timeout 30m ./...
fi
# **The store tests listed ran** (novox/hq issue 459): a test that needs the store skips without one, and a skip
-1
View File
@@ -77,7 +77,6 @@
"replay",
"kill",
"forget-checks",
"try",
"pause",
"resume",
"hand-act",
+1 -1
View File
@@ -8,7 +8,7 @@ private addresses. That is acceptable while mesh-controller is private; if it ev
a second place to clean besides the catalogue itself.
mesh-catalog e0bc919d4945bb5f13f971f74643a0b714cf45da modules/*/module.json, modules/nats/Dockerfile
mesh-host 7a41e120635b3f7b6cb94cca8b3fc4093e4e6110 examples/foundation-first-node.lock
mesh-host c8920622516548978d92e734f3991a262f98a742 examples/foundation-first-node.lock
mesh-sdk 5933eb757d443293631be2bed27a96288c2348d8 conformance/events/module-event.json
To move them, from this repository's root, with the two repositories checked out beside it:
+79 -29
View File
@@ -28,11 +28,12 @@
// image, and this file is applied on a machine with no mesh to ask about anything (ADR 0006). The
// registry prefix is the lab's; the digests are upstream's.
//
// The store's data is a NAMED VOLUME, not a directory on the machine. A directory the host
// creates is owned by root, and the database runs as somebody else inside the container — so it
// could not write, and the container crash-looped. A named volume lets the image set up its own
// ownership, and outlives the container, which is what you want for the thing holding the mesh's
// state.
// **The store is raised where the `postgres` module keeps it** (novox/hq ADR 0338): the container
// `postgres`, on the module's pgvector image, its data in the directory /var/lib/mesh-store owned by
// the image's postgres user, its superuser password read from the module's own-secret file — so the
// store step adopts it as it runs. Pinned upstream by the module's own digest, not through the lab's
// registry prefix, because the two must be the same reference (mesh-host internal/bootstrap holds
// them to each other against the module's manifest).
{
"declaration": 1,
@@ -83,17 +84,28 @@
"boot": "enabled",
"restart-on": ["base-filter"]
},
// The store's data is a directory owned by the image's postgres user (999), made before the
// container starts: a directory the runtime made would be root's, and the database could not
// write it. Measured on the module's image 2026-10-13: it initialises a new store on a directory
// made so (novox/hq ADR 0338 §5).
{
"id": "store-data",
"type": "directory",
"path": "/var/lib/mesh-store",
"mode": "0700",
"owner": "999:70"
},
{
"id": "store",
"type": "container",
"name": "mesh-store",
"image": "192.0.2.250:5000/postgres@sha256:7abf537131b66ed5af448d90653abf1679b0c7e9a1f07efdd4c3108a401b259a",
"name": "postgres",
"image": "pgvector/pgvector@sha256:cf134a767f474095eeba57e0117be8e568e011a63f33fbf252f14c9b760f8e6f",
"env": {
"POSTGRES_PASSWORD": "bootstrap",
"PGDATA": "/var/lib/postgresql/data/pgdata"
},
"ports": ["5432:5432"],
"volumes": ["mesh-store-data:/var/lib/postgresql/data"]
"volumes": ["/var/lib/mesh-store:/var/lib/postgresql/data"]
},
// Over TCP, not the socket. While the store initialises it runs a temporary server on the
// socket ONLY, then stops it and starts the real one — so a socket check passes, the action
@@ -102,21 +114,21 @@
{
"id": "store-ready",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "for i in $(seq 1 180); do pg_isready -h 127.0.0.1 -U postgres >/dev/null 2>&1 && exit 0; sleep 1; done; echo 'the store did not answer within 180s; its own last words follow'; pg_isready -h 127.0.0.1 -U postgres; tail -n 20 /var/lib/postgresql/data/log/*.log 2>/dev/null; exit 1"],
"verify": ["pg_isready", "-h", "127.0.0.1", "-U", "postgres"]
},
{
"id": "inventory-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE inventory'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw inventory"]
},
{
"id": "identity-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE identity'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw identity"]
},
@@ -126,20 +138,44 @@
{
"id": "licences-database",
"type": "action",
"in": "mesh-store",
"in": "postgres",
"command": ["sh", "-c", "psql -U postgres -c 'CREATE DATABASE licences'"],
"verify": ["sh", "-c", "psql -U postgres -lqt | cut -d'|' -f1 | grep -qw licences"]
},
{
"id": "context-schemas",
"type": "action",
"command": ["docker", "run", "--rm", "--network", "container:mesh-store",
"command": ["docker", "run", "--rm", "--network", "container:postgres",
"-e", "MESH_STORE_INVENTORY=postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"-e", "MESH_STORE_IDENTITY=postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
"-e", "MESH_STORE_LICENCES=postgres://postgres:bootstrap@127.0.0.1:5432/licences?sslmode=disable",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"migrate"],
"verify": ["sh", "-c", "docker exec mesh-store psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec mesh-store psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec mesh-store psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
"verify": ["sh", "-c", "docker exec postgres psql -U postgres -d inventory -tAc \"select to_regclass('public.node')\" | grep -qx node && docker exec postgres psql -U postgres -d identity -tAc \"select to_regclass('public.signing_key')\" | grep -qx signing_key && docker exec postgres psql -U postgres -d licences -tAc \"select to_regclass('public.licence')\" | grep -qx licence"]
},
// **The bus is raised where the `nats` module keeps it** (novox/hq ADR 0333 §1): the container
// `mesh-broker-nats`, its configuration and user list in /var/lib/nats-module/conf, its streams in
// /var/lib/mesh-broker-nats and its certificate in /var/lib/mesh-broker-tls — the names and paths the
// catalogue's module declares. So adopting it as that module at the broker step keeps it: the
// module's built image takes the place of the upstream one below, once, on the same streams, users
// and certificate, and nothing else moves. mesh-host's internal/bootstrap holds the two to each other
// against the module's own manifest (busLayout), at the broker step and in its tests.
//
// Directories on the machine, not volumes the runtime names: the module mounts these paths, and the
// control plane, once it runs as the mesh's process, reads the certificate at its path on the machine.
// The certificate's directory is readable by everyone because the certificate is public and the control
// plane runs as nobody; its key is written 0600 by the program that makes it.
{
"id": "bus-tls-dir",
"type": "directory",
"path": "/var/lib/mesh-broker-tls",
"mode": "0755"
},
{
"id": "bus-data-dir",
"type": "directory",
"path": "/var/lib/mesh-broker-nats",
"mode": "0700"
},
{
"id": "bus-certificate",
@@ -151,49 +187,63 @@
// it, and asking nothing of the image it writes into. Self-signed on purpose: a host pins
// this server's exact certificate (novox/hq ADR 0004), and at this moment there is no mesh
// to ask an authority of.
// `--user 0:0` because the volume is root's and this image runs as nobody, which is right
// for the long-running control plane and wrong for a one-shot writing into a fresh volume.
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
// `--user 0:0` because the directory is root's and this image runs as nobody, which is right
// for the long-running control plane and wrong for a one-shot writing into a fresh directory.
"command": ["docker", "run", "--rm", "--user", "0:0", "-v", "/var/lib/mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--into", "/tls"],
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "mesh-broker-tls:/tls",
"verify": ["docker", "run", "--rm", "--user", "0:0", "-v", "/var/lib/mesh-broker-tls:/tls",
"192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"broker", "certificate", "--check", "--into", "/tls"]
},
// **Seeds, written only where nothing is** (create-once). The bus's configuration and user list are
// the bundle's until the bus is adopted, and the module's from then on (ADR 0333 §4): an installer
// run again after the adoption applies this bundle again, and an ordinary file would put the
// bundle's one user back over every user the mesh composed, and leave the node unable to reach its
// bus. Until the adoption the installer places the composed list over the seed (enrol.go).
{
"id": "bus-conf-dir",
"type": "directory",
"path": "/var/lib/mesh-bus-conf",
"path": "/var/lib/nats-module/conf",
"mode": "0700"
},
{
"id": "bus-conf",
"type": "file",
"path": "/var/lib/mesh-bus-conf/nats.conf",
"path": "/var/lib/nats-module/conf/nats.conf",
"mode": "0644",
"content": "port: 4222\nhttp: 127.0.0.1:8222\n\ntls {\n cert_file: \"/tls/tls.crt\"\n key_file: \"/tls/tls.key\"\n}\n\njetstream {\n store_dir: \"/data\"\n}\n\ninclude accounts.conf\n"
"create-once": true,
"content": "port: 4222\nhttp: 0.0.0.0:8222\n\ntls {\n cert_file: \"/tls/tls.crt\"\n key_file: \"/tls/tls.key\"\n}\n\njetstream {\n store_dir: \"/data\"\n}\n\ninclude accounts.conf\n"
},
{
"id": "bus-accounts",
"type": "file",
"path": "/var/lib/mesh-bus-conf/accounts.conf",
"path": "/var/lib/nats-module/conf/accounts.conf",
"mode": "0600",
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
"create-once": true,
"content": "// The first user list, carried by the installer because at genesis there is no mesh to\n// compose one. A bootstrap credential, rotated with the store's and replaced by the\n// controller's own composition from its first start onward.\naccounts {\n MESH {\n jetstream: enabled\n users = [\n { user: \"controller\", password: \"$2a$10$AHqJgOifIVbU41KmATiMhuXFs8xa7Wl2HuN4UVBCXdN2jIQzjqApy\", permissions: {\n publish: { allow: [\"$JS.ACK.CONTROL.controller.>\", \"$JS.ACK.EVENTS.controller.>\", \"$JS.ACK.DEAD_LETTER_NOTICES.controller.>\", \"$JS.API.>\", \"$KV.mesh-controller_asked.>\", \"$KV.mesh-controller_assignments.>\", \"$KV.mesh-controller_calls.>\", \"$KV.mesh-controller_hand-acts.>\", \"$KV.mesh-controller_conditions.>\", \"$KV.mesh-controller_condition-history.>\", \"$KV.mesh-controller_declared-config.>\", \"$KV.mesh-controller_lease.>\", \"$KV.SEAT_MESH_BUILD_MACHINE_cancelled.>\", \"$KV.SEAT_NODE_BUILD_AGENT_cancelled.>\", \"_INBOX.enrol.>\", \"mesh.assignment.>\", \"mesh.mod.*.tool.>\", \"mesh.node.>\", \"mesh.seat.mesh-build-machine.accept.>\", \"mesh.seat.node-build-agent.accept.>\", \"mesh.seat.mesh-build-machine.tool.>\", \"mesh.seat.node-build-agent.tool.>\", \"mesh.seat.mesh-controller.event.applied\", \"mesh.seat.mesh-controller.event.built-before\", \"mesh.seat.mesh-controller.event.refused\", \"mesh.seat.mesh-controller.event.rolled-back\", \"mesh.seat.mesh-controller.event.checked\", \"mesh.seat.mesh-controller.event.plan-moved\", \"mesh.seat.mesh-controller.event.assignment-changed\", \"mesh.seat.mesh-controller.event.declared-config-changed\", \"mesh.seat.mesh-controller.event.condition-raised\", \"mesh.seat.mesh-controller.event.condition-changed\", \"mesh.seat.mesh-controller.event.condition-cleared\", \"mesh.seat.mesh-controller.event.doctor-heartbeat\", \"mesh.seat.mesh-controller.event.secret-replaced\", \"mesh.seat.mesh-controller.event.healer-acted\", \"$SRV.INFO\", \"mesh.seat.node-intrusion-prevention.tool.banned.*\", \"mesh.seat.node-backup.tool.backed-up.*\", \"mesh.seat.node-backup.tool.now.*\", \"mesh.seat.mesh-delivery.tool.stalled\", \"mesh.seat.mesh-delivery.tool.close\", \"mesh.seat.node-launcher.tool.secret.*\", \"mesh.seat.mesh-delivery.tool.release\", \"mesh.seat.mesh-delivery.tool.stop\", \"mesh.seat.mesh-delivery.tool.times\", \"mesh.seat.mesh-delivery.tool.groups\", \"mesh.seat.node-service-manager.tool.restart.*\", \"mesh.seat.mesh-controller.tool.plans\", \"mesh.events.dead.>\", \"mesh.again.>\"] }\n subscribe: { allow: [\"$JS.API.>\", \"$JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.>\", \"$JS.EVENT.ADVISORY.CONSUMER.DELETED.>\", \"_DELIVER.controller\", \"_DELIVER.controller.>\", \"_INBOX.controller.>\", \"mesh.control.>\", \"mesh.mod.*.event.provisioner.failing\", \"mesh.mod.*.event.provisioner.recovered\", \"mesh.mod.*.event.provisioner.retirement\", \"mesh.mod.gitea.event.pull.merged\", \"mesh.mod.gitea.event.pull.updated\", \"mesh.mod.mesh-catalog.event.catching-up\", \"mesh.mod.mesh-catalog.event.upgraded\", \"mesh.seat.mesh-build-machine.event.built\", \"mesh.seat.node-build-agent.event.built\", \"mesh.seat.operator-channel.event.decided.mesh-controller\", \"mesh.seat.mesh-controller.tool.>\", \"$SRV.PING\", \"$SRV.INFO\", \"$SRV.PING.mesh-controller\", \"$SRV.PING.mesh-controller.>\", \"$SRV.INFO.mesh-controller\", \"$SRV.INFO.mesh-controller.>\", \"$SRV.STATS\", \"$SRV.STATS.mesh-controller\", \"$SRV.STATS.mesh-controller.>\"] }\n allow_responses: { max: 1, ttl: \"1m\" }\n } }\n ]\n }\n}\n"
},
// The upstream server, until the broker step adopts it: the `nats` module's own image, built by the
// mesh, takes its place there once, on these same directories (ADR 0333 §3), as the bus's planned
// step (ADR 0236). The release is the one the module's image is built on (its NATS_BASE, 2.11.17),
// pinned upstream by that same digest: the adoption changes no server release, so it can be undone
// by putting the old image back and is said reversible. The published port is the
// node's bus port, 5671 unless given, which every enrolment token names; the installer gives the
// module the setting that maps its declared 4222 to it (ADR 0333 §2).
{
"id": "broker",
"type": "container",
"name": "mesh-broker",
"image": "192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927",
"name": "mesh-broker-nats",
"image": "nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00",
"ports": ["5671:4222", "127.0.0.1:8222:8222"],
"volumes": ["mesh-broker-data:/data", "mesh-broker-tls:/tls:ro", "/var/lib/mesh-bus-conf:/etc/nats:ro"],
"volumes": ["/var/lib/mesh-broker-nats:/data", "/var/lib/nats-module/conf:/etc/nats:ro", "/var/lib/mesh-broker-tls:/tls:ro"],
"args": ["-c", "/etc/nats/nats.conf", "-js"]
},
{
"id": "broker-ready",
"type": "action",
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh 192.0.2.250:5000/nats@sha256:b83efabe3e7def1e0a4a31ec6e078999bb17c80363f881df35edc70fcb6bb927 -c 'nc -z 127.0.0.1 5671'"]
"command": ["sh", "-c", "for i in $(seq 1 60); do docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671' >/dev/null 2>&1 && exit 0; sleep 1; done; exit 1"],
"verify": ["sh", "-c", "docker run --rm --network host --entrypoint sh nats@sha256:e4bf19f15fd3218814a4e3c9e0064e1334bd8aa20d5984b9f1a0afd084f8cc00 -c 'nc -z 127.0.0.1 5671'"]
},
{
"id": "control-plane",
@@ -202,7 +252,7 @@
"image": "192.0.2.250:5000/mesh-controller@sha256:c67db38439ff0aee242b467486765467bb95801f52175fc5727cc4e437338ace",
"network": "host",
"args": ["serve"],
"volumes": ["mesh-broker-tls:/broker-tls:ro"],
"volumes": ["/var/lib/mesh-broker-tls:/broker-tls:ro"],
"env": {
"MESH_STORE_INVENTORY": "postgres://postgres:bootstrap@127.0.0.1:5432/inventory?sslmode=disable",
"MESH_STORE_IDENTITY": "postgres://postgres:bootstrap@127.0.0.1:5432/identity?sslmode=disable",
+1 -11
View File
@@ -49,6 +49,7 @@
./internal/inventory TestACredentialIssuedOutsideIsNeverMade
./internal/inventory TestOneAcceptForAProviderIsSealedToEveryConsumer
./internal/inventory TestASharedOwnSecretIssuedOutsideIsNeverMade
./internal/recordmirror TestNatsAnAssignmentAndARegistrationEachChangeOneKeyAndRaiseOneEvent
./cmd/mesh-controller TestOneAcceptForAProviderIsPlacedForEveryConsumer
./cmd/mesh-controller TestReplay164
./cmd/mesh-controller TestAnApprovedLayerOpensTheWalkThatSendsIt
@@ -61,14 +62,3 @@
./cmd/mesh-controller TestAProviderElsewhereServesItsOwnComposedName
./cmd/mesh-controller TestAReusedCheckBuildIsRegisteredByItsDigestAndGatedAsABuiltOne
./internal/inventory TestAReportOfABusyContainerIsKeptAndReadBack
./cmd/mesh-controller TestAReleaseSendsTheRestAtOnceAfterTheFirstPasses
./cmd/mesh-controller TestARestMachineThatFailsIsPutBackAloneAndTheOthersAreJudgedOn
./cmd/mesh-controller TestAWalkTakesOverTheReadingsOfAGateJudgingTheSameBuildThere
./cmd/mesh-controller TestAFirstSendMadeAndNotKeptIsNotSentAgain
./cmd/mesh-controller TestAReleaseSendKeptAndMadeIsNotMadeAgain
./cmd/mesh-controller TestAFirstMachineThatFailsStillStopsTheWalk
./cmd/mesh-controller TestTheControlNodeIsSentLastInAStep
./cmd/mesh-controller TestARestMachineSilentPastItsBoundIsLeftAndSaid
./cmd/mesh-controller TestANoMoveTierWaitsForNoNewReport
./cmd/mesh-controller TestOneProviderDownAndThreeConsumersFailingAreOneCondition
./cmd/mesh-controller TestAConsumerFailingBesideAHealthyProviderIsItsOwn