Compare commits

..
Author SHA1 Message Date
mesh-tools ab210fc5ab Merge pull request 'Design 47's last milestones: watchman, grants gate, assigner wants and placement, builder log, view (issues 774, 1952, 1953, 1954, 1955)' (#113) from feat/774-l3-reland into main 2026-10-30 11:02:07 +00:00
jschoubben 8edc83950c Pin mesh-sdk 2da0de7, the release's sdk with main merged in (issue 774)
mesh/delivery merged: source.main moved to novox/mesh-delivery@ab210fc5ab40
mesh/delivery-group group feat/774-l3-reland merged: every member is merged
mesh/check checking 8edc83950cea on 5588875c3f21 | novox/mesh-delivery#113 in group feat/774-l3-reland
2026-10-30 01:12:57 +01:00
jschoubben 8e470f3af4 Merge origin/main (delivery 1, #112) into the release, keeping the release's mesh-sdk pins (issue 774) 2026-10-30 01:12:57 +01:00
jschoubben 7536d9d14e Pin mesh-sdk 6e4a301, the release's final head, which keeps the old provisioner client's needs grant for one release (issue 774) 2026-10-30 00:18:04 +01:00
jschoubben ce7032c49b Pin mesh-sdk 007b6fd, the release's final head (issue 774) 2026-10-30 00:05:25 +01:00
jschoubben 08ecf10ef3 Grants: write withheld hearings in mesh-sdk's layout, so the self-check reads them by the sdk's names (issue 1955) 2026-10-29 23:56:21 +01:00
jschoubben ba502479e8 Builder: ask and hear the seat mesh-artifact-sweep, the name mesh-sdk 7bd0f6c gives the store's sweep, so its held and artifacts-gone reach their holder (issue 1955, ADR 0571 §2) 2026-10-29 23:56:21 +01:00
jschoubben e564eaf75f Grants: write the hearings it withholds to its state withheld on change, so the self-check's probe D-heard reads them from the bus and not the controller's store (issue 1955) 2026-10-29 23:51:48 +01:00
jschoubben a54d1acf45 View: answer builder_history from the builder's log, each build's end with its duration from its start (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben e82ea89c23 Assigner: read the running commit of a reader of the manifests from the builder's log, and hold the buckets it starts without in one list so the start test skips them (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben 06b58bc83b Indexer: take directories and outside-built registrations from the builder's log, watched from its start, so no build record of the controller's is read (issue 1955, ADR 0571 §1) 2026-10-29 23:48:14 +01:00
jschoubben 17c0734fb2 Builder: keep each build's start and end in its own module log build-log, and answer builder_builds_since from it, so readers and mesh-catalog read the builder's history without a bare stream (issue 1955, ADR 0571 §1, §9) 2026-10-29 23:48:14 +01:00
jschoubben c045057ca0 Assigner: write who holds which seat from placement into seat-holders, only the keys a change moves, and answer it from placement's seats, so readers no longer ask the controller's seats (issue 1955, ADR 0571 §4)
Pins mesh-sdk f9a496d.
2026-10-29 23:14:28 +01:00
jschoubben b44f01f89c Builder: write each build's start and end to its own log on the bus, and ask the seat artifact-store's held and hear its artifacts-gone, the controller's only until the lease write, so build history and the store's sweep leave the controller (issue 1955, ADR 0571 §1, §2)
Pins mesh-sdk f9a496d.
2026-10-29 23:13:32 +01:00
jschoubben 229a3e9855 Assigner: from the lease write rule 7 reads a placed node's own composition digest from its have, an unplaced node gets no want and is said, and the controller's compositions are read only until then (issue 1955, ADR 0571 §6) 2026-10-29 23:10:31 +01:00
jschoubben e5192f55ee View: read no build record and no composition of the controller's: fingerprints against the builder's done keys alone, a move's judgement from the node agent's have (issue 1955, ADR 0571 §1, §6) 2026-10-29 23:08:46 +01:00
jschoubben ee2f1efe78 Watchman: hold a failing network part at the machines it points at, as the controller does, so a machine others cannot reach is said once there and not as each reporter's own (issue 774)
Pins mesh-sdk 6eae076.
2026-10-29 23:06:19 +01:00
jschoubben 8a91871de6 Watchman: raise machine-network, network-rewritten, network-unreachable, machine-units, relogin-needed and directory-used-as-found from each node's own health report, under the controller's keys, so these alerts outlive the controller's lease (issue 774)
Pins mesh-sdk 94d0de5.
2026-10-29 23:01:03 +01:00
jschoubben 64a5074d58 Assigner: lay each held value of the group's own pull requests at their heads over its module's values in the composition check, judged against the head's declaration and named, so a check reads the held values itself (issue 1953 D7, ADR 0565 §5) 2026-10-29 22:43:27 +01:00
jschoubben 7b7796b6fa Assigner: hold a want while a provision its module requires has no key yet, and say it once it stays without one past the delivery bound, so a module never starts without its database unseen (issue 1954 D1)
Pins mesh-sdk 66ff33d.
2026-10-29 22:41:44 +01:00
jschoubben f56eb62d8b Assigner: hold a want until every provision its module consumes on that node is provided, and say a failed one as a finding, so a module never moves ahead of what it needs (issue 1954 D1, ADR 0570 §4)
Pins mesh-sdk 6e39ed1.
2026-10-29 22:39:38 +01:00
jschoubben 78f03b7422 Watchman: say needs-operator from mesh-provisioner's provisions in waits, so the row the haves could not carry has its facts on the bus (issue 1954 D2)
Pins mesh-sdk 4a96793.
2026-10-29 22:37:39 +01:00
jschoubben 3aec7f9b02 Assigner: hold the seat mesh-wants, keep a person's upgrade policy and judge moves by it, say a policy split with the controller as a finding, and answer the bus step, so the policies and the bus step leave the controller (issue 1953 D4-D6) 2026-10-29 22:34:08 +01:00
jschoubben 2488d839ab Bridge, indexer and view: carry the catalogue's predefined list on its merged message, and keep it and each open pull request's config declarations in the indexer's states, so mesh-config reads them in place of the controller's mirror and pull-declares (issue 1953 D1-D3)
The view's tools read both states. Pins mesh-sdk 2d2aa7b.
2026-10-29 22:29:04 +01:00
jschoubben 2531cd42b6 Checker: hold that the change's own failing test ends its level there, as mesh-sdk's build asks now do, so a failing check costs only its own tests (issue 1289)
Pins mesh-sdk 2d2aa7b.
2026-10-29 22:24:50 +01:00
jschoubben 5235bb2e79 Assigner and grants part: record hand acts and warrants through the seat mesh-hand-acts, the controller's log only while that seat has no holder and the controller holds its lease (issue 1953 D8)
Pins mesh-sdk 2d2aa7b in the L5 modules.
2026-10-29 22:24:00 +01:00
jschoubben 3b6f3af38e Grants part: create a module's log with the max age it declares, so a log's entries age out as its manifest says (issue 1952)
Its nats fixture gives the tools bundle's account a secret the tool runtime can reach, as mesh-sdk 2e5cc4f asks. Pins mesh-sdk 2e5cc4f.
2026-10-29 22:19:35 +01:00
jschoubben 15a15cbda5 Watchman: observe S1, S9 and S21 as the controller does and stop the needs-operator row, so the keeper's compare can empty with no condition lost (issue 1952 WM2, WM3, WM5)
S1 is urgent after half an hour for a control node (the node running mesh-controller). S9 also hears the bus module's
slow consumers and closed connections, each closure counted once. S21 is one finding per heard machine from its failed
haves and its provisioner clients' reports (PROV_APPLIED). DEAD_LETTERS is counted again at a letter's age and every
five minutes while letters are held, on the watchman's own pulse. needs-operator waits for L6: its inputs are not on the
bus. Pins mesh-sdk b6ef01d.
2026-10-29 21:59:46 +01:00
jschoubben 7dbf7a19a1 Grants part: compose a principal for every own bus account, a tools bundle's with what it may remove, so the bus tools remove the controller's dead-letter consumer and the engine's acts on their own account (issue 1952 S5)
Pins mesh-sdk 7000bd5.
2026-10-29 21:56:13 +01:00
jschoubben e1fd3eb0f6 View: answer status from the buckets it reads and derive motion with mesh-sdk's MotionOf, so status leaves the controller and the keeper judges a delivery as the view does (issue 1952 V1) 2026-10-29 21:52:42 +01:00
jschoubben 73280f85c3 Assigner: write each condition it says as a finding too, and say it straight only until the take-over, so its holds reach the operator through the condition keeper and never twice (issue 1952 WM6) 2026-10-29 21:49:25 +01:00
jschoubben b8e8981730 Grants part: hear root-not-free from the seat the take-over key names, never both, so the switch to the condition keeper moves it with the router (issue 1952 G1)
Watchman: subscribe exactly what its grant lets it hear (buscompose.WatchmanHears). Both pin mesh-sdk 69fcb6c.
2026-10-29 21:46:29 +01:00
jschoubben da787ae360 Watchman: hold mesh-watchdogs and write its watchdogs as findings, so health can leave the controller (issue 1952 WM1-WM5)
The signals rows S1, S8, S9, S10, S11, S21 and the haves' unhealthy and needs-operator are pulses whose expiry is a finding;
stuck keys and presence are its own findings; it keeps the dead letters from the lease write on its own consumer, lists and
shows them, and refuses deliver and drop until the hand-act log's home (issue 1953). Tools read every state it keeps and
start resync. Pins mesh-sdk 2a04bf0.
2026-10-29 21:44:18 +01:00
jschoubben 013b7420f6 mesh-grants: hold a grant only while the loaded list lacks its permissions, read from the part's own record of the lists it sealed, so a release whose bus tools are down still moves (issue 774) 2026-10-29 21:18:21 +01:00
jschoubben c7fbc530da mesh-grants: compose the user list from the running and the trunk manifest, and write a grant only once the loaded list carries it, so L3's parts never use a subject the bus has not loaded (issue 774) 2026-10-29 21:15:55 +01:00
jschoubben 98ad1aa252 Pin mesh-sdk 22c9ddd in the modules L3 changes, the L3 re-land group's sdk head (issue 774) 2026-10-29 21:07:00 +01:00
jschoubben b01cf63237 Revert "Merge pull request 'Revert L3: its node-engine removed every held module on novox, the bus server among them (issue 774)' (#109) from revert/774-l3 into main"
This reverts commit c1b4533201, reversing
changes made to d8274204fb.
2026-10-29 21:05:14 +01:00
144 changed files with 9939 additions and 2433 deletions
+7
View File
@@ -0,0 +1,7 @@
# The modules of this repository that still read the controller's assignments (mesh-controller.assignments), each with
# why. mesh-grants' test (TestGrants, "assignments readers") refuses any other module.json that reads them, and an entry
# here whose module no longer does: the list only shrinks, and it is empty in the first change after the controller's
# lease is written to a holder that is no controller (novox/hq ADR 0549 §3, §6; issue 774). Each module named here
# uses the assignments only while the controller holds the lease (buscompose.LeaseHandedAway), placement alone after it.
mesh-delivery-assigner unioned with the placement until the lease write (assign.Placed with handedAway); placement alone after
mesh-grants unioned with the placement until the lease write (ReadInputs); placement alone after
@@ -119,11 +119,11 @@ func (p *passer) composeAtHead(ctx context.Context, m assign.Mesh, ch assign.Cha
if h := ch.Members[repository]; h != "" {
commits, member = []string{h}, true
} else {
commits = p.w.runningCommits(ctx, kvGetter(p.kvs[delivery.BuildsBucket]), module)
commits = p.w.runningCommits(module)
}
if len(commits) == 0 {
notJudged = append(notJudged, "not judged whether "+module+" reads the group's manifests: no build of it is assigned with a "+
"commit on record, and the group holds no member of "+repository)
notJudged = append(notJudged, "not judged whether "+module+" reads the group's manifests: the builder's log holds no done "+
"trunk build of it, and the group holds no member of "+repository)
continue
}
for _, commit := range commits {
@@ -106,7 +106,7 @@ func TestAGroupWithAnSDKMemberIsComposedByTheComposerAtItsHead(t *testing.T) {
b.putJSON(t, provider.AssignmentsBucket, "g14.web", []byte(`{"node":"g14","module":"web","build":"build-1"}`))
running := strings.Repeat("3", 40)
b.putJSON(t, provider.AssignmentsBucket, "novox.mesh-controller", []byte(`{"node":"novox","module":"mesh-controller","build":"c-1"}`))
b.putJSON(t, delivery.BuildsBucket, delivery.BuildKey("mesh-controller", "c-1"), []byte(`{"id":"c-1","repository":"novox/mesh-controller","commit":"`+running+`"}`))
b.built(t, "mesh-controller", "novox/mesh-controller", running)
sdkHead := strings.Repeat("4", 40)
heads := []delivery.Head{{Repository: "novox/mesh-sdk", Number: 80, Head: sdkHead}, {Repository: "novox/web", Number: 7, Head: strings.Repeat("2", 40)}}
hash, _ := delivery.HeadsHash(heads)
@@ -21,6 +21,7 @@ import (
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose/judge"
"git.novox.be/novox/mesh-sdk/go/delivery"
@@ -82,6 +83,11 @@ func startBed(t *testing.T, leaveOut string) *bed {
}
b.buckets[name] = kv
}
// The builder's module log, as the runtime raises it (ADR 0297): the running commits are read from it (ADR 0571 clause 1).
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(delivery.BuilderModule, delivery.BuilderLog),
Subjects: []string{buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog) + ".>"}, AllowDirect: true}); err != nil {
t.Fatal(err)
}
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: "EVENTS", Subjects: []string{"mesh.mod.*.event.>"}}); err != nil {
t.Fatal(err)
}
@@ -177,8 +183,8 @@ func saysWhatItKeeps(t *testing.T, b *bed, done <-chan error) {
// 1297/B2b: a bucket it reads that the bus does not hold stops the start, naming the bucket.
func TestTheAssignerDoesNotStartWithoutABucketItReads(t *testing.T) {
for _, missing := range readBuckets {
if missing == delivery.PlacementBucket {
continue // it starts without it, below
if startsWithout(missing) {
continue // it starts without it (the placement and the grants' source: main_test.go; the puts: below)
}
b := startBed(t, missing)
select {
@@ -264,6 +270,19 @@ func (b *bed) putJSON(t *testing.T, bucket, key string, value []byte) {
}
}
// built appends the end of a done trunk build of module at commit to the builder's log, as the builder does.
func (b *bed) built(t *testing.T, module, repository, commit string) {
t.Helper()
raw, err := delivery.EncodeBuildLogEntry(delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: module, FP: delivery.SourcePrefix2 + strings.Repeat("c", 64),
Repository: repository, Directory: delivery.RootPath, Commit: commit, Node: "novox", At: time.Now().UTC(), Outcome: delivery.BuildDone})
if err != nil {
t.Fatal(err)
}
if _, err := b.js.Publish(b.ctx, buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog)+"."+module, raw); err != nil {
t.Fatal(err)
}
}
// compose says the controller composed module for node at fp, with the digest d (a letter repeated) and the judgement of the
// move to it, in mesh-controller_composed.
func (b *bed) compose(t *testing.T, node, module, fp string, d byte, judgement delivery.Judgement) string {
@@ -8,25 +8,32 @@ package main
import (
"context"
"errors"
"fmt"
"io"
"log"
"os"
"os/signal"
"slices"
"sync/atomic"
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/announce"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/process"
"git.novox.be/novox/mesh-sdk/go/provider"
"git.novox.be/novox/mesh-sdk/go/provisions"
"mesh-delivery-assigner/internal/assign"
)
func main() {
@@ -52,8 +59,9 @@ func main() {
var readBuckets = []string{
buscompose.ResolvedBucket,
provider.AssignmentsBucket,
delivery.BuildsBucket,
delivery.ControllerGrantsBucket,
delivery.PartGrantsBucket,
buscompose.LeaseBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
@@ -63,8 +71,19 @@ var readBuckets = []string{
delivery.PresenceBucket,
delivery.HaveBucket,
delivery.StepsBucket(goBuildModule),
takeoverBucket,
provisions.ProvisionsBucket,
keptBucket,
}
// readLogs are the logs of other parts the assigner reads, each named in its manifest's reads: the builder's log, whose
// newest done trunk build of a module names the commit a reader of the manifests runs at (novox/hq ADR 0571 clause 1).
var readLogs = []string{delivery.BuildLogRead}
// takeoverBucket is the condition keeper's take-over key's bucket (novox/hq ADR 0564 §9): whether the assigner's said
// conditions go straight to the router or only as findings (issue 1952 WM6).
var takeoverBucket = health.Bucket(health.KeeperModule, health.TakeoverState)
// goBuildModule is the Go build module, which takes the assigner's compose asks on its seat's check stream and writes
// their steps in its steps state (novox/hq ADR 0482 clause 4).
const goBuildModule = "mesh-build-go"
@@ -73,7 +92,7 @@ const goBuildModule = "mesh-build-go"
// (ADR 0567 §1), named after the assigner as every module state is.
var ownBuckets = []string{delivery.AssignerAsksBucket, delivery.AssignerCompositionsBucket, delivery.AssignerDifferencesBucket,
delivery.AssignerPutsBucket, delivery.AssignerWantsBucket, delivery.PlacementBucket, delivery.PlacementProposalsBucket,
delivery.PlacementCandidatesBucket, delivery.PlacementPulsesBucket}
delivery.PlacementCandidatesBucket, delivery.PlacementPulsesBucket, assign.FindingsBucket, assign.PoliciesBucket, delivery.SeatHoldersBucket}
// seatBuckets are the seat's states, new with it: until the controller has made them from the manifest and the bus grants
// them, the assigner starts without them and binds each once the bus says it was created (issue 1790's kind).
@@ -83,7 +102,8 @@ var seatBuckets = []string{delivery.PlacementBucket, delivery.PlacementProposals
// keptStates is what the assigner keeps and who may read it (ADR 0495 §2-§3): its wants by the watchman and the view
// (and the node agent from release 6), its differences by the view, its compositions by the checker, its asks by the view
// and the teller, its puts by whoever follows one (ADR 0546), and the seat mesh-placement's placement, proposals and
// candidates by the node agents, the grants part, mesh-config and the view (ADR 0567 §9).
// candidates by the node agents, the grants part, mesh-config and the view (ADR 0567 §9), and its placement by
// the builder, the checker and the indexer, through mesh-sdk's toolchain gate (issue 774, ADR 0549 §3).
var keptStates = []announce.State{
{Name: delivery.PlacementState, Readable: true},
{Name: delivery.PlacementProposalsState, Readable: true},
@@ -94,6 +114,9 @@ var keptStates = []announce.State{
{Name: "differences", Readable: true},
{Name: "puts", Readable: true},
{Name: "wants", Readable: true},
{Name: health.FindingsState, Readable: true},
{Name: delivery.AssignerPoliciesState, Readable: true},
{Name: delivery.SeatHoldersState, Readable: true},
}
// buckets is every bucket the assigner opened, by name.
@@ -105,8 +128,33 @@ type buckets map[string]jetstream.KeyValue
// without it and takes no put further until it starts again (ADR 0546).
func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any)) (buckets, error) {
out := buckets{}
// The controller's lease first (ADR 0549 §5, ADR 0561 §3): once it is handed away the seat's placement alone names
// where modules run, and the controller's assignments are no bucket the assigner needs. Not granted or not there, the
// controller holds it, as before.
away, why := false, ""
if kv, err := js.KeyValue(ctx, buscompose.LeaseBucket); err != nil {
logf("the bucket %s cannot be opened (%v): the controller is taken to hold its lease, so the assignments are read beside the placement", buscompose.LeaseBucket, err)
} else {
out[buscompose.LeaseBucket] = kv
e, gerr := kv.Get(ctx, buscompose.ControllerLeaseKey)
var raw []byte
if gerr == nil {
raw = e.Value()
}
if errors.Is(gerr, jetstream.ErrKeyNotFound) {
gerr = nil
}
away, why = buscompose.LeaseHandedAway(raw, raw != nil, gerr)
}
for _, name := range append(append([]string{}, readBuckets...), ownBuckets...) {
if name == buscompose.LeaseBucket {
continue
}
kv, err := js.KeyValue(ctx, name)
if name == provider.AssignmentsBucket && err != nil && away {
logf("the bucket %s cannot be opened (%v), and none is needed: %s, so the seat's placement alone is read", name, err, why)
continue
}
if name == delivery.AssignerPutsBucket && err != nil {
logf("the bucket %s cannot be opened (%v): no put is taken further until the assigner starts again", name, err)
continue
@@ -117,6 +165,41 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == delivery.SeatHoldersBucket && err != nil {
// The seat's state, new with ADR 0571 §4: bound once the bus says it was made; until then nobody is told who
// holds which seat here, and the controller's seats still answer.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == keptBucket && err != nil {
// mesh-config's held values (ADR 0565 §5): without them a composition lays none, said in the journal.
logf("the bucket %s cannot be opened (%v): no held value is laid on a composition until it is bound", name, err)
continue
}
if name == provisions.ProvisionsBucket && err != nil {
// The seat mesh-provisioner's state, with its own delivery (ADR 0570 §4): bound once the bus says it was made;
// until then no want waits for a provision, as before.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if name == assign.PoliciesBucket && err != nil {
// The seat mesh-wants' state, new with it (ADR 0565 §7): bound once the bus says it was made; until then no
// person's policy is read, so the controller's beside each composition holds, as before.
logf("the bucket %s cannot be opened (%v): it is bound once the bus says it was made", name, err)
continue
}
if (name == assign.FindingsBucket || name == takeoverBucket) && err != nil {
// New with health leaving the controller (issue 1952 WM6): without them the conditions are said straight to the
// router and kept as no finding, as before.
logf("the bucket %s cannot be opened (%v): the assigner's conditions are said straight to the router until it starts again", name, err)
continue
}
if sourceOfGrants(name) && err != nil {
// The grants part's grants and the controller's lease choose where grants are read from (issue 774): without
// them the controller's grants are read, as before the lease is handed away.
logf("the bucket %s cannot be opened (%v): the controller's grants are read until the assigner starts again", name, err)
continue
}
if err != nil {
return nil, process.EssentialErr("assigner", name, "write a want from the whole mesh", err)
}
@@ -125,6 +208,23 @@ func open(ctx context.Context, js process.BucketOpener, logf func(string, ...any
return out, nil
}
// startsWithout says the assigner starts without the bucket when it cannot be opened (open says why for each): the seat's
// states and those new with a delivery of their own, bound once made, and the two that choose the grants' source.
func startsWithout(name string) bool {
switch name {
case delivery.SeatHoldersBucket, keptBucket, provisions.ProvisionsBucket, assign.PoliciesBucket, assign.FindingsBucket, takeoverBucket,
delivery.AssignerPutsBucket:
return true
}
return slices.Contains(seatBuckets, name) || sourceOfGrants(name)
}
// sourceOfGrants says the bucket is one of the two that choose the grants' source, the grants part's grants and the
// controller's lease: neither stops the assigner when it is not there (delivery.GrantSource).
func sourceOfGrants(name string) bool {
return name == delivery.PartGrantsBucket || name == buscompose.LeaseBucket
}
// run is the program: it returns when ctx ends or the bus cannot be used.
func run(ctx context.Context, getenv func(string) string, logOut io.Writer) error {
logger := log.New(logOut, "mesh-delivery-assigner: ", log.LstdFlags)
@@ -153,7 +253,71 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
thisNode = account.Node
logger.Printf("started on the account %s, reading %d buckets", account.User, len(kvs))
return follow(ctx, js, conn, kvs, sayVia(ctx, js), emitting{said}, logger)
findings := findingsOf(ctx, kvs, logger.Printf)
buildLog := func(handle func(stdio.LogEntry) error) error {
return said.ReadLog(delivery.BuilderModule, delivery.BuilderLog).Watch("", stdio.LogFromAll, handle)
}
return follow(ctx, js, conn, kvs, findings.Say(sayVia(ctx, js)), findings, emitting{said}, buildLog, logger)
}
// findingsOf is where the assigner's said conditions go as findings (issue 1952 WM6): its findings bucket, and the
// condition keeper's take-over key, watched, so a condition is said straight to the router only until the take-over. A
// bucket it cannot open is said: without its findings the conditions are said straight, as before; without the take-over
// key they are said straight too, the controller being followed.
func findingsOf(ctx context.Context, kvs buckets, logf func(string, ...any)) assign.Findings {
var f assign.Findings
if kv := kvs[assign.FindingsBucket]; kv != nil {
f.Put = func(key string, value []byte) error { _, err := kv.Put(ctx, key, value); return err }
f.Delete = func(key string) error { return kv.Delete(ctx, key) }
f.Keys = func() ([]string, error) {
keys, err := kv.Keys(ctx)
if errors.Is(err, jetstream.ErrNoKeysFound) {
return nil, nil
}
return keys, err
}
}
var keeper atomic.Bool
f.ToKeeper = keeper.Load
takeover := takeoverBucket
kv := kvs[takeover]
if kv == nil {
return f
}
w, err := kv.Watch(ctx, health.TakeoverKey)
if err != nil {
process.NotWatched(logf, takeover, err, "the controller is followed: the conditions are said straight to the router")
return f
}
go func() {
defer w.Stop()
for {
select {
case <-ctx.Done():
return
case e, ok := <-w.Updates():
if !ok {
logf("the watch of %s ended: the last source it named is followed until the assigner starts again", takeover)
return
}
if e == nil {
continue
}
var raw []byte
if e.Operation() == jetstream.KeyValuePut {
raw = e.Value()
}
t, err := health.DecodeTakeover(raw, raw != nil)
if err != nil {
logf("the take-over key cannot be read (%v): the controller is followed", err)
}
if was := keeper.Swap(t.Follow == health.FollowKeeper); was != keeper.Load() {
logf("the assigner's conditions are said by %s from now (the take-over key follows %s)", t.Seat(), t.Follow)
}
}
}
}()
return f
}
// thisNode is the node this process runs on, as its credential names it: said with each of the seat's events.
@@ -170,6 +334,8 @@ var watched = []string{
buscompose.ResolvedBucket,
provider.AssignmentsBucket,
delivery.ControllerGrantsBucket,
delivery.PartGrantsBucket,
buscompose.LeaseBucket,
delivery.ControllerComposedBucket,
delivery.AnswersBucket,
delivery.BuilderBuildsBucket,
@@ -185,11 +351,15 @@ var watched = []string{
delivery.AssignerCompositionsBucket,
delivery.AssignerDifferencesBucket,
delivery.AssignerWantsBucket,
assign.PoliciesBucket,
provisions.ProvisionsBucket,
keptBucket,
delivery.SeatHoldersBucket,
}
// follow watches every bucket a pass reads and makes a pass at start, once every watch handed what stands, and after
// each change; changes that come during a pass make one more. It returns when ctx ends or a watch ends.
func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(asks.Condition) error, bus caller, logger *log.Logger) error {
func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber, kvs buckets, say func(asks.Condition) error, findings assign.Findings, bus caller, buildLog logWatch, logger *log.Logger) error {
w := newWorld()
dirty := make(chan struct{}, 1)
wake := func() {
@@ -198,6 +368,22 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
default:
}
}
// The builder's log (ADR 0571 clause 1): every entry it keeps before the first pass, then each new one as it is appended;
// a module's newer done trunk build makes a pass, as a composition waiting on its commit is judged then. Where it cannot be
// read, a composition names no running reader of the manifests until the assigner starts again, and it is said.
if buildLog != nil {
var first atomic.Bool
if err := buildLog(func(le stdio.LogEntry) error {
if w.putBuilt(le.Value) && first.Load() {
wake()
}
return nil
}); err != nil {
logger.Printf("%s is not read (%v): no composition is judged against a running reader of the manifests until the assigner starts again",
delivery.BuildLogRead, err)
}
first.Store(true)
}
errs := make(chan error, len(watched)+2)
settled := make(chan struct{}, len(watched)+2)
watch := func(name string, opts ...jetstream.WatchOpt) error {
@@ -233,7 +419,13 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
if e.Operation() == jetstream.KeyValuePut {
value = e.Value()
}
before := w.grantsBucket()
w.putEntry(name, e.Key(), value, e.Revision(), e.Created())
if name == buscompose.LeaseBucket && w.grantsBucket() == before {
// The controller renews its lease every few seconds: only a change of where grants are read from
// makes a pass (delivery.GrantSource).
continue
}
wake()
}
}
@@ -269,13 +461,13 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
return
}
if bus != nil {
p.acts = newPutActs(bus, wake, logger.Printf)
p.acts = newPutActs(bus, wake, logger.Printf, w.handedAway)
}
logger.Printf("the bucket %s is open: puts are taken further", delivery.AssignerPutsBucket)
}
started := 0
for _, name := range watched {
if kvs[name] == nil && slices.Contains(seatBuckets, name) {
if kvs[name] == nil && (startsWithout(name) || name == provider.AssignmentsBucket) {
continue // not on the bus at this start (open said so)
}
started++
@@ -297,13 +489,21 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
name string
kv jetstream.KeyValue
}
placementAppeared := make(chan appeared, len(seatBuckets))
placementAppeared := make(chan appeared, len(seatBuckets)+2)
for _, name := range seatBuckets {
if kvs[name] == nil && js != nil {
process.AwaitBucket(ctx, js, nc, name, "the seat mesh-placement waits for it",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} })
}
}
// And the two that choose where grants are read from, with the same delivery as the grants part that writes one.
for _, name := range []string{delivery.PartGrantsBucket, buscompose.LeaseBucket, assign.PoliciesBucket, provisions.ProvisionsBucket, keptBucket, delivery.SeatHoldersBucket} {
if kvs[name] == nil && js != nil {
name := name
process.AwaitBucket(ctx, js, nc, name, "until then the bucket is not read",
logger.Printf).Ready(func(kv jetstream.KeyValue) { placementAppeared <- appeared{name, kv} })
}
}
takePlacement := func(a appeared) error {
if kvs[a.name] != nil {
return nil // bound already, by the other of its two events
@@ -321,13 +521,13 @@ func follow(ctx context.Context, js jetstream.JetStream, nc process.Subscriber,
case <-settled:
}
}
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, said: map[string]bool{}, answeredAt: map[string]uint64{}}
p := &passer{kvs: kvs, w: w, logf: logger.Printf, say: say, findings: findings, said: map[string]bool{}, answeredAt: map[string]uint64{}}
p.asks = newComposeAsks(ctx, buildrun.Asker{Publisher: buildrun.JetStreamAsks{JS: js}, Waiter: buildrun.JetStreamAsks{JS: js}}, wake, logger.Printf)
if bus != nil && kvs[delivery.AssignerPutsBucket] != nil {
p.acts = newPutActs(bus, wake, logger.Printf)
p.acts = newPutActs(bus, wake, logger.Printf, w.handedAway)
}
if bus != nil {
p.seat = newSeatActs(bus, wake, logger.Printf)
p.seat = newSeatActs(bus, wake, logger.Printf, w.handedAway)
}
if js != nil {
p.publish = streamPublisher(ctx, js, thisNode)
@@ -1,6 +1,8 @@
package main
import (
"mesh-delivery-assigner/internal/assign"
"context"
"errors"
"fmt"
@@ -17,8 +19,10 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// 1297/B2a: the buckets the program opens are exactly the manifest's reads and states, as the broker names them
@@ -38,8 +42,16 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
}
var reads []string
for _, r := range m.Reads {
if slices.Contains(readLogs, r) {
continue // a log, read through its own reader (ADR 0571 clause 1), not a bucket
}
reads = append(reads, strings.Replace(r, ".", "_", 1))
}
for _, l := range readLogs {
if !slices.Contains(m.Reads, l) {
t.Errorf("the program reads the log %s, the manifest does not name it", l)
}
}
var states []string
for _, s := range m.State {
states = append(states, m.Module+"_"+s.Name)
@@ -62,20 +74,30 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
}
// ADR 0546: whatever keeps the puts' bucket from opening (not there, not granted, no answer in time), the assigner opens
// the rest and starts, writing wants; any other bucket it cannot open stops the start, named.
// Issue 774 (ADR 0549 §5): the controller's lease not granted, the assigner starts taking it as the controller's; the
// assignments refused while the controller holds it stop the start, and once it is handed away they are not needed.
handed := []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`)
for _, c := range []struct {
missing string
err error
starts bool
lease []byte
said string
}{
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false},
{delivery.AssignerPutsBucket, jetstream.ErrBucketNotFound, true, nil, "no put is taken further"},
{delivery.AssignerPutsBucket, errors.New("nats: permissions violation for subscription to \"$JS.API.STREAM.INFO.KV_mesh-delivery-assigner_puts\""), true, nil, "no put is taken further"},
{delivery.AssignerPutsBucket, context.DeadlineExceeded, true, nil, "no put is taken further"},
{delivery.AssignerWantsBucket, errors.New("nats: permissions violation"), false, nil, ""},
// Issue 774: the grants part's grants and the controller's lease choose the grants' source; neither stops the start.
{delivery.PartGrantsBucket, jetstream.ErrBucketNotFound, true, nil, "the controller's grants are read"},
{buscompose.LeaseBucket, errors.New("nats: permissions violation"), true, nil, "taken to hold its lease"},
{provider.AssignmentsBucket, errors.New("nats: permissions violation"), false, []byte(`{"instance":"novox/1"}`), ""},
{provider.AssignmentsBucket, errors.New("nats: permissions violation"), true, handed, "the seat's placement alone is read"},
} {
var logged string
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err}, func(f string, a ...any) { logged = fmt.Sprintf(f, a...) })
kvs, err := open(context.Background(), refusingJS{missing: c.missing, err: c.err, lease: c.lease}, func(f string, a ...any) { logged += fmt.Sprintf(f, a...) })
switch {
case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, "no put is taken further")):
case c.starts && (err != nil || kvs[c.missing] != nil || len(kvs) != len(readBuckets)+len(ownBuckets)-1 || !strings.Contains(logged, c.said)):
t.Errorf("%s refused with %v: the start answered %v, %d buckets, logged %q", c.missing, c.err, err, len(kvs), logged)
case !c.starts && (err == nil || !strings.Contains(err.Error(), "grant of "+c.missing)): // 1945: named with its grant
t.Errorf("%s refused with %v: the start answered %v", c.missing, c.err, err)
@@ -98,7 +120,7 @@ func TestOpensWhatItsManifestNames(t *testing.T) {
kvs[delivery.ControllerComposedBucket].(*watchKV).entries = []kvEntry{{key: ckey, value: cd}}
kvs[delivery.AssignerPutsBucket].(*watchKV).closed = true
kvs[delivery.AssignerWantsBucket].(*watchKV).wrote = wrote
go follow(ctx, nil, nil, kvs, func(asks.Condition) error { return nil }, nil, log.New(io.Discard, "", 0))
go follow(ctx, nil, nil, kvs, func(asks.Condition) error { return nil }, assign.Findings{}, nil, nil, log.New(io.Discard, "", 0))
select {
case key := <-wrote:
if key != "want.ace.node" {
@@ -114,13 +136,18 @@ type refusingJS struct {
jetstream.JetStream
missing string
err error
lease []byte // the controller's lease, when it holds one
}
func (r refusingJS) KeyValue(_ context.Context, name string) (jetstream.KeyValue, error) {
if name == r.missing {
return nil, r.err
}
return &memKV{values: map[string][]byte{}}, nil
kv := &memKV{values: map[string][]byte{}}
if name == buscompose.LeaseBucket && r.lease != nil {
kv.values[buscompose.ControllerLeaseKey] = r.lease
}
return kv, nil
}
// watchKV is a memKV whose watch hands its entries and then the marker that all stands was handed, or ends at once when
@@ -2,6 +2,7 @@ package main
import (
"context"
"encoding/json"
"errors"
"maps"
"slices"
@@ -11,6 +12,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
"git.novox.be/novox/mesh-sdk/go/sdkversion"
"mesh-delivery-assigner/internal/assign"
@@ -27,6 +29,16 @@ type passer struct {
// pass says a condition once and a restart says it again (the router keeps one per key).
say func(asks.Condition) error
said map[string]bool
// findings is the assigner's findings bucket (its said conditions and policy-split, ADR 0564 §2, ADR 0565 §8); splits the
// modules whose policy-split finding stands, nil until the first pass, which deletes any a previous process left.
findings assign.Findings
splits map[string]bool
// failedProvisions are the failed provisions whose finding this process wrote (ADR 0570 §4).
failedProvisions map[string]bool
// missing are the required provisions with no key this process pulsed, and missingSaid those whose finding it wrote.
missing, missingSaid map[string]bool
// unplaced are the nodes whose unplaced finding this process wrote.
unplaced map[string]bool
// answeredAt is, by the id of a backup ask, how many values the have had when its answer was written into the want.
answeredAt map[string]uint64
asks *composeAsks
@@ -129,6 +141,13 @@ func (p *passer) wants(ctx context.Context) {
}
}
wanted := p.w.wantsOf()
chosen, _ := assign.DecodePolicies(p.w.read(assign.PoliciesBucket))
provisionsOf := p.provisionsOf()
missing := map[string]bool{}
defer func() { p.provisionsMissing(ctx, missing) }()
defer p.unplacedFindings()
defer p.seatHolders(ctx)
defer p.policySplits(manifests, chosen)
frozen := p.sayPlacement(p.withCalls(p.w.placementDiffers()))
placed := p.w.placedNodes()
puts := p.putsByModule()
@@ -149,12 +168,17 @@ func (p *passer) wants(ctx context.Context) {
stale := map[string]neitherOne{}
left := map[string]string{}
defer func() { p.sayNeither(neither, stale, left) }()
// Where grants are read from, once for the pass: the grants part's while the lease is handed away (issue 774).
grants := p.w.grantsBucket()
for name := range names {
fp := trunk[name].FP2
name := name
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name],
m := assign.Module{Name: name, FP: fp, Manifest: manifests[name], Chosen: chosenOf(chosen, name), Provisions: provisionsOf(name), Nodes: nodesOf[name], Now: time.Now(), Puts: puts[name],
Self: func(node string) bool { return placed[node] },
Composed: func(node, fp string) (delivery.ComposedDigest, bool) {
if placed[node] && p.w.handedAway() {
return p.w.ownComposed(node) // from the lease write: the digest its own have names (ADR 0571 §6)
}
if placed[node] {
// The node composes its own (ADR 0547 §2, §4): no composition of the controller's is waited for, the want
// names the build alone, and the node judges the move itself.
@@ -164,7 +188,7 @@ func (p *passer) wants(ctx context.Context) {
},
GrantedAt: func(fp string) bool {
key, err := delivery.ControllerGrantKey(name, fp)
return err == nil && p.w.has(delivery.ControllerGrantsBucket, key)
return err == nil && p.w.has(grants, key)
},
Restored: func(node string) bool {
key, err := delivery.RestoreAskKey(node, name, fp)
@@ -180,8 +204,8 @@ func (p *passer) wants(ctx context.Context) {
hold, _ := delivery.HoldAskKey(name, fp)
m.Released = p.answeredWith(hold, "release")
m.Kept = p.answeredWith(hold, "keep")
m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(delivery.ControllerGrantsBucket, grant)
if raw := p.w.value(delivery.ControllerGrantsBucket, grant); raw != nil {
m.Built, m.Held, m.Granted = p.w.built(name, fp), p.w.has(delivery.AssignerWantsBucket, held) && !m.Released, p.w.has(grants, grant)
if raw := p.w.value(grants, grant); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
m.GrantedOn = func(node string) bool { return assign.GrantNamesNode(g, node, name) }
}
@@ -195,6 +219,11 @@ func (p *passer) wants(ctx context.Context) {
last[node] = p.w.lastHealthy(node, name, fp)
}
out := assign.Wants(m, nodes, wanted[name], last)
for _, node := range m.Nodes {
for _, prov := range m.MissingProvisions(node) {
missing[node+"."+name+"."+prov] = true
}
}
p.yeses(ctx, name, m, wanted[name], &out)
gatherNeither(neither, name, out)
leftNeither(left, m, wanted[name], out)
@@ -930,3 +959,275 @@ func (p *passer) yeses(ctx context.Context, module string, m assign.Module, befo
}
}
}
// chosenOf is a person's choice for module, nil when none.
func chosenOf(chosen map[string]catalogue.PersonPolicy, module string) *catalogue.PersonPolicy {
if c, ok := chosen[module]; ok {
return &c
}
return nil
}
// policySplits writes the finding policy-split for each module whose policy at the assigner differs from one the controller
// writes beside a composition of it, and deletes it once they agree (ADR 0565 §8): written when a split begins, never again
// while it stands. The first pass after a start deletes every one a previous process left that no longer splits.
func (p *passer) policySplits(manifests map[string]catalogue.Manifest, chosen map[string]catalogue.PersonPolicy) {
if p.findings.Put == nil {
return
}
now := time.Now
if p.now != nil {
now = p.now
}
split := map[string]bool{}
theirs := map[string][]string{}
if !p.w.handedAway() { // the controller writes no policy from the lease write (ADR 0571 §6)
theirs = assign.ControllerPolicies(p.w.read(delivery.ControllerComposedBucket))
}
for _, s := range assign.Splits(manifests, chosen, theirs) {
split[s.Module] = true
if p.splits[s.Module] {
continue
}
value, err := assign.PolicySplitFinding(s, now())
if err == nil {
err = p.findings.Put(assign.PolicySplitKey(s.Module), value)
}
if err != nil {
p.logf("the policy split of %s is not written: %v", s.Module, err)
delete(split, s.Module)
}
}
gone := p.splits
if gone == nil && p.findings.Keys != nil {
gone = map[string]bool{}
if keys, err := p.findings.Keys(); err == nil {
for _, k := range keys {
if m, ok := strings.CutSuffix(strings.TrimPrefix(k, "module."), "."+assign.KindPolicySplit); ok && strings.HasPrefix(k, "module.") {
gone[m] = true
}
}
}
}
for m := range gone {
if !split[m] {
if err := p.findings.Delete(assign.PolicySplitKey(m)); err != nil {
p.logf("the policy split of %s is not deleted: %v", m, err)
split[m] = true // tried again at the next pass
}
}
}
p.splits = split
}
// provisionsOf answers, per module, the provisions it consumes on a node, by key (ADR 0570 §4), from the seat
// mesh-provisioner's state; nil while the state is not read (the seat not delivered yet), so no want waits on it. A failed
// provision is also said as the assigner's finding `provider.<key>.provision-failed` (a waiting one is the watchman's
// needs-operator), deleted once it is no longer failed.
func (p *passer) provisionsOf() func(module string) func(node string) map[string]provisions.Provision {
if p.kvs[provisions.ProvisionsBucket] == nil {
return func(string) func(string) map[string]provisions.Provision { return nil }
}
by := map[string]map[string]map[string]provisions.Provision{} // module, node, key
failed := map[string]provisions.Provision{}
for key, raw := range p.w.read(provisions.ProvisionsBucket) {
pr, err := provisions.Decode(raw)
if err != nil {
continue
}
node, module := pr.Consumer.Node, pr.Consumer.Module
if by[module] == nil {
by[module] = map[string]map[string]provisions.Provision{}
}
if by[module][node] == nil {
by[module][node] = map[string]provisions.Provision{}
}
by[module][node][key] = pr
if pr.State == provisions.Failed {
failed[key] = pr
}
}
p.provisionFindings(failed)
return func(module string) func(string) map[string]provisions.Provision {
return func(node string) map[string]provisions.Provision { return by[module][node] }
}
}
// provisionFindings writes the finding of each failed provision and deletes the ones no longer failed.
func (p *passer) provisionFindings(failed map[string]provisions.Provision) {
if p.findings.Put == nil {
return
}
now := time.Now
if p.now != nil {
now = p.now
}
for key, pr := range failed {
if p.failedProvisions[key] {
continue
}
value, err := assign.ProvisionFailedFinding(key, pr, now())
if err == nil {
err = p.findings.Put(assign.ProvisionFailedKey(key), value)
}
if err != nil {
p.logf("the failed provision %s is not said: %v", key, err)
continue
}
if p.failedProvisions == nil {
p.failedProvisions = map[string]bool{}
}
p.failedProvisions[key] = true
}
for key := range p.failedProvisions {
if _, still := failed[key]; still {
continue
}
if err := p.findings.Delete(assign.ProvisionFailedKey(key)); err != nil {
p.logf("the failed provision %s is not cleared: %v", key, err)
continue
}
delete(p.failedProvisions, key)
}
}
// MissingBound is how long a required provision may have no key before it is said: a module's delivery bound, the default
// when its manifest names none (ADR 0564 clause 13 (d)).
var MissingBound = delivery.DefaultDeliveryBound
// missingPulse is the pulse key of a provision with no key, `provision-missing.<node>.<module>.<provision>`.
func missingPulse(key string) string { return "provision-missing." + key }
// provisionsMissing is the loud end of a required provision with no key (the lead, 2026-10-29, ADR 0570 §4): each one is
// pulsed with the bound when it is first seen; once its pulse expired and it still has no key, the finding
// `provider.<node>.<module>.<provision>.provision-missing` is written; one keyed again ends both. The pulse's expiry, heard on
// the watch, is the pass that writes it: no timer.
func (p *passer) provisionsMissing(ctx context.Context, missing map[string]bool) {
pulses := p.kvs[delivery.PlacementPulsesBucket]
if p.missing == nil {
p.missing = map[string]bool{}
}
for key := range missing {
switch {
case !p.missing[key]:
p.missing[key] = true
if pulses != nil {
p.pulse(ctx, missingPulse(key), MissingBound)
}
case pulses == nil || !p.pulsing(missingPulse(key)):
if p.missingSaid[key] || p.findings.Put == nil {
continue
}
value, err := assign.ProvisionMissingFinding(key, MissingBound, p.clock())
if err == nil {
err = p.findings.Put(assign.ProvisionMissingKey(key), value)
}
if err != nil {
p.logf("the missing provision %s is not said: %v", key, err)
continue
}
if p.missingSaid == nil {
p.missingSaid = map[string]bool{}
}
p.missingSaid[key] = true
}
}
for key := range p.missing {
if missing[key] {
continue
}
delete(p.missing, key)
if pulses != nil && p.pulsing(missingPulse(key)) {
if err := pulses.Delete(ctx, missingPulse(key)); err == nil {
p.w.put(delivery.PlacementPulsesBucket, missingPulse(key), nil)
}
}
if p.missingSaid[key] && p.findings.Delete != nil {
if err := p.findings.Delete(assign.ProvisionMissingKey(key)); err != nil {
p.logf("the missing provision %s is not cleared: %v", key, err)
p.missing[key] = true
continue
}
delete(p.missingSaid, key)
}
}
}
// unplacedFindings writes, from the lease write, the finding `machine.<node>.unplaced` for each node that runs but no
// placement names, which gets no want (ADR 0571 §6), and deletes it once the node is placed.
func (p *passer) unplacedFindings() {
if p.findings.Put == nil {
return
}
now := []string(p.w.unplacedNodes())
is := map[string]bool{}
for _, node := range now {
is[node] = true
if p.unplaced[node] {
continue
}
value, err := assign.UnplacedFinding(node, p.clock())
if err == nil {
err = p.findings.Put(assign.UnplacedKey(node), value)
}
if err != nil {
p.logf("the unplaced node %s is not said: %v", node, err)
continue
}
if p.unplaced == nil {
p.unplaced = map[string]bool{}
}
p.unplaced[node] = true
}
for node := range p.unplaced {
if is[node] {
continue
}
if err := p.findings.Delete(assign.UnplacedKey(node)); err == nil {
delete(p.unplaced, node)
}
}
}
// seatHolders writes the seat mesh-placement's state seat-holders from every node's placement (ADR 0571 §4): one key per
// seat held once, `<seat>`, and one per node of a node seat, `<seat>.<node>`, written only where it differs, and a key no
// placement holds any more deleted. A pass follows each placement change and each manifest the indexer takes in, so no
// timer writes it.
func (p *passer) seatHolders(ctx context.Context) {
kv := p.kvs[delivery.SeatHoldersBucket]
if kv == nil {
return
}
placements := map[string]delivery.Placement{}
for key, value := range p.w.read(delivery.PlacementBucket) {
node, err := delivery.ParsePlacementKey(key)
if err != nil {
continue
}
if pl, err := delivery.DecodePlacement(value); err == nil && pl.Node == node {
placements[node] = pl
}
}
if len(placements) == 0 {
return // no placement read yet: nothing is known of who holds what, and nothing is deleted
}
want := delivery.SeatHoldersOf(placements)
have := p.w.read(delivery.SeatHoldersBucket)
for key, sh := range want {
value, err := json.Marshal(sh)
if err != nil || string(have[key]) == string(value) {
continue
}
p.put(ctx, delivery.SeatHoldersBucket, key, func() ([]byte, error) { return value, nil })
}
for key := range have {
if _, ok := want[key]; ok {
continue
}
if err := kv.Delete(ctx, key); err != nil {
p.logf("the seat holders %s are not deleted: %v", key, err)
continue
}
p.w.put(delivery.SeatHoldersBucket, key, nil)
}
}
@@ -13,6 +13,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
@@ -31,6 +32,11 @@ func (m *memKV) Put(_ context.Context, key string, value []byte) (uint64, error)
return uint64(len(m.values)), nil
}
func (m *memKV) Create(_ context.Context, key string, value []byte, _ ...jetstream.KVCreateOpt) (uint64, error) {
m.values[key] = value
return uint64(len(m.values)), nil
}
func (m *memKV) Delete(_ context.Context, key string, _ ...jetstream.KVDeleteOpt) error {
delete(m.values, key)
return nil
@@ -135,6 +141,9 @@ func (m *memKV) Update(_ context.Context, key string, value []byte, revision uin
type memStore map[string]*memKV
func (s memStore) get(state, key string) ([]byte, uint64, bool, error) {
if s[state] == nil {
return nil, 0, false, nil
}
v, ok := s[state].values[key]
return v, uint64(len(v)), ok, nil
}
@@ -183,7 +192,7 @@ func putOfTheBus(t *testing.T) {
p := newPasser(kvs, now)
bus := &fakeBus{calls: make(chan string, 8)}
woke := make(chan struct{}, 8) // each act's goroutine wakes a pass once it is done
p.acts = newPutActs(bus, func() { woke <- struct{}{} }, func(string, ...any) {})
p.acts = newPutActs(bus, func() { woke <- struct{}{} }, func(string, ...any) {}, nil)
p.acts.snapshot = func(context.Context, string, string) (string, error) {
return "novox's restore point of nats (streams) taken 22:01", nil
}
@@ -284,7 +293,7 @@ func putOfTheBus(t *testing.T) {
if got.Step != delivery.PutWanted || err != nil || w.FP != fp || w.Composed != digest || !w.Put || !w.Hold {
t.Fatalf("snapshotted, the put stands at %s (%s); the want %+v, %v", got.Step, got.Words, w, err)
}
if call := <-bus.calls; !strings.Contains(call, "warranted") || !strings.Contains(call, a.Ask.ID) {
if call := <-bus.calls; !strings.HasPrefix(call, "seat:mesh-hand-acts.warranted ") || !strings.Contains(call, a.Ask.ID) { // ADR 0565 §11: into the seat
t.Errorf("the warrant was recorded as %q", call)
}
world(delivery.AssignerWantsBucket, "want.novox.nats", wantsKV.values["want.novox.nats"])
@@ -873,6 +882,24 @@ func aPairOneSourceNamesAloneGetsNoWantAndOneCondition(t *testing.T) {
if m.Input == nil || len(m.Input.Nodes) != 1 || !slices.Equal(m.Input.Nodes[0].Assigned, []string{"a", "b", "c"}) || m.Input.Nodes[0].Place.Key != "k" {
t.Errorf("the check's input from the bus: %+v (%q)", m.Input, m.InputUnread)
}
// Issue 774 (ADR 0549 §5, §6): once the controller's lease is handed away the placement alone names where modules
// run: a pair in the placement only gets its want and no condition, and one in the assignments only is no module of
// the node. A lease that goes back to a controller brings the comparison back.
place("n2", "d")
assigned("g14", "e")
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`))
got, all := p.w.assigned()
if !slices.Equal(got["n2"], []string{"d"}) || !slices.Equal(got["g14"], []string{"a", "b", "c"}) || len(all) != 0 {
t.Errorf("handed away: assigned %v, assignments %v; want the placement alone", got, all)
}
if frozen := p.sayPlacement(p.w.placementDiffers()); len(frozen) != 0 || len(said) != 4 {
t.Errorf("handed away: frozen %v, said %+v; want no condition", frozen, said[4:])
}
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"instance":"novox/1","epoch":58}`))
if frozen := p.sayPlacement(p.w.placementDiffers()); !frozen["n2.d"] || !frozen["g14.e"] {
t.Errorf("a controller's lease again: frozen %v; want both pairs held", frozen)
}
}
// 1126 (ADR 0547 §2 to §4): a node the catalogue places composes its own: its wants name the build alone and its own want
@@ -13,6 +13,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"mesh-delivery-assigner/internal/assign"
)
@@ -46,14 +47,16 @@ type putActs struct {
snapped map[string]string
snapNo map[string]string
warranted map[string]bool
// rec records the put's warrant in the hand-act log (ADR 0565 §11).
rec handacts.Recorder
warrantNo map[string]string
said map[string]string
// snapshot takes the bus's snapshot on node; a variable so a test takes none.
snapshot func(ctx context.Context, node, module string) (string, error)
}
func newPutActs(bus caller, wake func(), logf func(string, ...any)) *putActs {
a := &putActs{bus: bus, wake: wake, logf: logf, snapping: map[string]bool{}, snapped: map[string]string{}, snapNo: map[string]string{}, warranted: map[string]bool{}, warrantNo: map[string]string{},
func newPutActs(bus caller, wake func(), logf func(string, ...any), leaseAway func() bool) *putActs {
a := &putActs{rec: handacts.Recorder{Ask: bus.Call, LeaseAway: leaseAway}, bus: bus, wake: wake, logf: logf, snapping: map[string]bool{}, snapped: map[string]string{}, snapNo: map[string]string{}, warranted: map[string]bool{}, warrantNo: map[string]string{},
said: map[string]string{}}
a.snapshot = a.snapshotTheBus
return a
@@ -183,7 +186,10 @@ func (p *passer) putWorld(node, module string, put delivery.Put, manifest catalo
w.Composed, w.Granted = p.composedFor(node, module, put.FP)
if d := w.Composed; d != nil {
policy := d.Policy
if policy == "" {
chosen, _ := assign.DecodePolicies(p.w.read(assign.PoliciesBucket))
if c, ok := chosen[module]; ok {
policy, _, _, _ = catalogue.Upgrade(manifest, &c) // a person's choice at the assigner (ADR 0565 §7)
} else if policy == "" {
policy, _, _ = catalogue.DerivedUpgrade(manifest)
}
w.Recorded = policy == catalogue.PolicyRecord
@@ -213,8 +219,7 @@ func (p *passer) putWorld(node, module string, put delivery.Put, manifest catalo
// composedFor is the composition of module for node at fp and whether the grant at fp names the node's accounts: the one
// place a put reads them (ADR 0546 clause 4). Today their writer is the controller (mesh-controller_composed and
// mesh-controller_grants); when composing and grants move out of the controller into delivery's own parts, this reads them
// from their new owner and nothing else changes.
// mesh-controller_grants); the grants are the grants part's while the controller's lease is handed away (grantsBucket).
func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest, bool) {
var out *delivery.ComposedDigest
if d, ok := p.w.composed(node, module, fp); ok {
@@ -222,7 +227,7 @@ func (p *passer) composedFor(node, module, fp string) (*delivery.ComposedDigest,
}
granted := false
if gkey, err := delivery.ControllerGrantKey(module, fp); err == nil {
if raw := p.w.value(delivery.ControllerGrantsBucket, gkey); raw != nil {
if raw := p.w.value(p.w.grantsBucket(), gkey); raw != nil {
if g, err := delivery.DecodeControllerGrant(raw); err == nil {
granted = g.NamesNode(node, module)
}
@@ -382,8 +387,9 @@ func (a *putActs) snapshotTheBus(ctx context.Context, node, module string) (stri
}
}
// warrant records once, in the controller's hand-act log, that the bus's want was written on the operator's warrant: the
// router's record of the teller's ask says who chose and how (ADR 0274).
// warrant records once, in the hand-act log (the seat mesh-hand-acts', or the controller's while that seat has no holder and
// the controller holds its lease, ADR 0565 §11), that the bus's want was written on the operator's warrant: the router's
// record of the teller's ask says who chose and how (ADR 0274).
func (a *putActs) warrant(id, ask string) {
a.mu.Lock()
if a.warranted[id] {
@@ -393,10 +399,10 @@ func (a *putActs) warrant(id, ask string) {
a.warranted[id] = true
a.mu.Unlock()
go func() {
if _, err := a.bus.Call("seat:"+catalogue.ControllerSeatName+".warranted", map[string]any{"asker": delivery.TellerModule, "ask": ask}); err != nil {
if _, err := a.rec.RecordWarrant(delivery.TellerModule, ask); err != nil {
a.logf("the put's warrant %s is not recorded in the hand-act log: %v", ask, err)
a.mu.Lock()
a.warrantNo[id] = fmt.Sprintf("the warrant %s is not recorded in the controller's hand-act log: %v", ask, err)
a.warrantNo[id] = fmt.Sprintf("the warrant %s is not recorded in the hand-act log: %v", ask, err)
a.mu.Unlock()
a.wake()
}
@@ -21,6 +21,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/compose/frombus"
"git.novox.be/novox/mesh-sdk/go/compose/judge"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-assigner/internal/assign"
@@ -67,6 +68,8 @@ type seatActs struct {
export map[string]exportCheck
// calls is, by pair (node and module, or node and provision for a pin), what the last call for it came to.
calls map[string]string
// rec records the seat's acts in the hand-act log (ADR 0565 §11).
rec handacts.Recorder
}
// exportCheck is the export read again for one proposal: running, its answer, or why it was not read.
@@ -76,8 +79,9 @@ type exportCheck struct {
err string
}
func newSeatActs(bus caller, wake func(), logf func(string, ...any)) *seatActs {
return &seatActs{bus: bus, wake: wake, logf: logf, export: map[string]exportCheck{}, calls: map[string]string{}}
func newSeatActs(bus caller, wake func(), logf func(string, ...any), leaseAway func() bool) *seatActs {
return &seatActs{bus: bus, wake: wake, logf: logf, export: map[string]exportCheck{}, calls: map[string]string{},
rec: handacts.Recorder{Ask: bus.Call, LeaseAway: leaseAway}}
}
// exportOf is the export read again for a proposal, asked once in the background.
@@ -114,26 +118,31 @@ func (a *seatActs) forget(id string) {
a.mu.Unlock()
}
// record records once, in the controller's hand-act log, that placement was written on the operator's warrant (§5): the
// router's record of the teller's ask says who chose and how (ADR 0274).
// record records once, in the hand-act log, that placement was written on the operator's warrant (§5): the router's
// record of the teller's ask says who chose and how (ADR 0274). The log is the seat mesh-hand-acts' from its delivery, and
// the controller's while that seat has no holder and the controller holds its lease (ADR 0565 §11, handacts.Recorder).
func (a *seatActs) record(warrant string) {
if a == nil || warrant == "" {
return
}
ask, _, _ := strings.Cut(warrant, "@")
go func() {
if _, err := a.bus.Call("seat:"+catalogue.ControllerSeatName+".warranted", map[string]any{"asker": delivery.TellerModule, "ask": ask}); err != nil {
if taken, err := a.rec.RecordWarrant(delivery.TellerModule, ask); err != nil {
a.logf("the placement's warrant %s is not recorded in the hand-act log: %v", ask, err)
} else {
a.logf("the placement's warrant %s is recorded in the log of %s", ask, taken.Log)
}
}()
}
// handAct records a hand-back in the controller's hand-act log (§7).
// handAct records a hand-back in the hand-act log (§7), as record does.
func (a *seatActs) handAct(prop delivery.PlacementProposal) {
go func() {
if _, err := a.bus.Call(handActCall, map[string]any{"what": "mesh-placement hand-back of " + strings.Join(prop.Nodes(), ", "),
"why": prop.Why, "cause": "asked by " + prop.By + ", approved by the operator"}); err != nil {
if taken, err := a.rec.Record(handacts.Request{What: "mesh-placement hand-back of " + strings.Join(prop.Nodes(), ", "),
Why: prop.Why, Cause: "asked by " + prop.By + ", approved by the operator"}); err != nil {
a.logf("the hand-back %s is not recorded in the hand-act log: %v", prop.ID, err)
} else {
a.logf("the hand-back %s is recorded in the log of %s", prop.ID, taken.Log)
}
}()
}
@@ -524,7 +533,7 @@ func (p *passer) layAgain(ctx context.Context, prop delivery.PlacementProposal,
held, recorded := p.lays(ctx)
again, laid, findings := assign.Propose(assign.ProposeIn{ID: prop.ID, By: prop.By, Why: prop.Why, At: prop.At, Changes: prop.Changes,
Current: current, Revisions: revs, Held: held, Recorded: recorded,
Seed: prop.Seed, ExportDigest: prop.ExportDigest})
Seed: prop.Seed, ExportDigest: prop.ExportDigest, Leased: p.w.handedAway()})
if len(findings) > 0 {
p.endProposal(ctx, prop, why+", and laid over the keys now it is refused: "+strings.Join(findings, "; "))
return
@@ -661,7 +670,7 @@ func (p *passer) followWrite(prop delivery.PlacementProposal, answer delivery.An
}
p.seat.record(answer.Warrant)
p.sayChanged(prop, answer, changes)
if p.seat != nil && !assign.Leased && !prop.Seed {
if p.seat != nil && !p.w.handedAway() && !prop.Seed {
p.seat.callAll(assign.ControllerCalls(before, written, prop.ID, answer.Warrant))
}
}
@@ -729,7 +738,7 @@ func (p *passer) proposeSeed(ctx context.Context, was delivery.PlacementProposal
now := p.clock()
seed, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: was.By, Why: was.Why, At: now,
Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{},
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export)})
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export), Leased: p.w.handedAway()})
if len(findings) > 0 {
p.logf("the new export is not proposed as the seed: %s", strings.Join(findings, "; "))
return
@@ -4,11 +4,13 @@ import (
"encoding/json"
"errors"
"fmt"
"slices"
"sort"
"strings"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
@@ -214,7 +216,7 @@ func (s *seatTools) doPropose(args map[string]any, by string) (any, error) {
}
now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: changes, Current: current, Revisions: revs, Held: held, Recorded: recorded})
Changes: changes, Current: current, Revisions: revs, Held: held, Recorded: recorded, Leased: s.leased()})
if len(findings) > 0 {
return nil, fmt.Errorf("the proposal is refused: %s", strings.Join(findings, "; "))
}
@@ -298,7 +300,7 @@ func (s *seatTools) doAdopt(args map[string]any, by string) (any, error) {
now := s.now()
p, laid, findings := assign.Propose(assign.ProposeIn{ID: delivery.NewProposalID(now), By: by, Why: arg(args, "why"), At: now,
Changes: assign.SeedChanges(export), Current: map[string]delivery.Placement{}, Revisions: map[string]uint64{},
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export)})
Held: held, Recorded: recorded, Seed: true, ExportDigest: delivery.ExportDigest(export), Leased: s.leased()})
if len(findings) > 0 {
return nil, fmt.Errorf("the seed is refused: %s", strings.Join(findings, "; "))
}
@@ -308,14 +310,20 @@ func (s *seatTools) doAdopt(args map[string]any, by string) (any, error) {
return proposed{Proposal: p, Follow: followProposal(p.ID)}, nil
}
// handActCall records a hand act in the controller's hand-act log.
const handActCall = "seat:" + catalogue.ControllerSeatName + ".hand-act"
// leased says the controller's lease is handed away (mesh-sdk buscompose.LeaseHandedAway, ADR 0561 §3; issue 774): the lease
// write of ADR 0547 that ends keeping the controller in step (ADR 0567 §8). A lease not read, a controller's, expired or
// unreadable, is not handed away.
func (s *seatTools) leased() bool {
raw, _, found, err := s.st.get(buscompose.ControllerLease, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, err)
return away
}
// doHandBack is the verb hand-back (§7): the operator's alone, so it is asked as a proposal is. It writes a hand-back
// proposal naming every key at its revision; the assigner raises its authorising ask and, on its warrant, deletes the keys,
// records the hand act and says placement-changed; every node returns to the controller's compositions.
func (s *seatTools) doHandBack(args map[string]any, by string) (any, error) {
if assign.Leased {
if s.leased() {
return nil, errors.New("placement is the mesh's alone since the lease write: there is no controller to hand it back to")
}
why := arg(args, "why")
@@ -368,6 +376,9 @@ type historyEntry struct {
// doPlacement is the verb placement (§10, §6): one node's key, one module's nodes, or every key; with history, the
// changes in order from the revisions the state keeps.
func (s *seatTools) doPlacement(args map[string]any) (any, error) {
if arg(args, "seats") == "true" {
return s.seats(arg(args, "node"))
}
all, revs, err := s.placements()
if err != nil {
return nil, err
@@ -531,3 +542,32 @@ func (s *seatTools) tools() []stdio.Tool {
}
return out
}
// seats is the verb placement with seats (ADR 0571 §4): who holds each seat and on which node, as the state seat-holders
// holds it; with node, the keys of seats held on that node.
func (s *seatTools) seats(node string) (any, error) {
keys, err := s.st.keys(delivery.SeatHoldersState)
if err != nil {
return nil, fmt.Errorf("seat-holders cannot be listed: %w", err)
}
sort.Strings(keys)
out := []delivery.SeatHolders{}
for _, key := range keys {
raw, _, found, err := s.st.get(delivery.SeatHoldersState, key)
if err != nil {
return nil, fmt.Errorf("seat-holders %s cannot be read: %w", key, err)
}
if !found {
continue
}
var sh delivery.SeatHolders
if json.Unmarshal(raw, &sh) != nil {
continue
}
if node != "" && sh.Node != node && !slices.ContainsFunc(sh.Holders, func(h delivery.SeatHolder) bool { return h.Node == node }) {
continue
}
out = append(out, sh)
}
return map[string]any{"seats": out, "holds": len(keys)}, nil
}
@@ -1,6 +1,7 @@
package main
import (
"context"
"encoding/json"
"os"
"slices"
@@ -9,8 +10,12 @@ import (
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
"mesh-delivery-assigner/internal/assign"
)
func (s memStore) del(state, key string) error {
@@ -142,9 +147,250 @@ func TestTheSeatsVerbs(t *testing.T) {
if hb := got.(proposed).Proposal; err != nil || !hb.HandBack || hb.Base["anchor"] != uint64(len(raw)) || len(st[delivery.PlacementState].values) != 1 {
t.Errorf("hand-back = %+v, %v", got, err)
}
// Issue 774 (ADR 0567 §8): once the controller's lease is handed away a change its verbs cannot take is proposed, and
// there is no controller to hand placement back to.
st[buscompose.ControllerLease] = &memKV{values: map[string][]byte{buscompose.ControllerLeaseKey: []byte(`{"handed-to":"the operator's hand-over","handed-by":"person.jochen"}`)}}
if _, err := s.doPropose(map[string]any{"why": "hold", "changes": `[{"op":"hold","node":"anchor","module":"web","claim":"node-shell","scope":"node"}]`}, "op"); err != nil {
t.Errorf("a hold after the lease write: %v", err)
}
if _, err := s.doHandBack(map[string]any{"why": "back"}, "op"); err == nil || !strings.Contains(err.Error(), "since the lease write") {
t.Errorf("a hand-back after the lease write: %v", err)
}
if tools := s.tools(); len(tools) != 7 || tools[2].Name != "mesh-placement.propose" || tools[2].Run == nil || tools[2].Input == nil {
t.Errorf("the seat's verbs as served: %+v", tools)
}
t.Run("1953/D4-D6: the seat mesh-wants' upgrade and bus", wantsVerbs)
t.Run("1955/L7 (ADR 0571 §6): rule 7 reads the controller's composition until the lease write, the node's own after; an unplaced node is said", func(t *testing.T) {
w := newWorld()
fp := delivery.SourcePrefix2 + strings.Repeat("2", 64)
ckey, _ := delivery.ComposedKey("ace", "web", fp)
cv, _ := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("c", 64), Judgement: delivery.JudgedNeither,
Policy: catalogue.PolicyRoll, PolicyFrom: catalogue.FromDefault})
w.putEntry(delivery.ControllerComposedBucket, ckey, cv, 1, time.Now())
pl, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "ace", Modules: []delivery.PlacedModule{{Module: "web"}}})
pkey, _ := delivery.PlacementKey("ace")
w.putEntry(delivery.PlacementBucket, pkey, pl, 1, time.Now())
for node, digest := range map[string]string{"ace": "sha256:" + strings.Repeat("a", 64), "g14": ""} {
hn, _ := delivery.EncodeHaveNode(delivery.HaveNode{Composed: digest})
hk, _ := delivery.HaveNodeKey(node)
w.putEntry(delivery.HaveBucket, hk, hn, 1, time.Now())
}
if d, ok := w.composed("ace", "web", fp); !ok || d.Digest != "sha256:"+strings.Repeat("c", 64) || len(w.unplacedNodes()) != 0 {
t.Fatalf("before the lease write: %+v %v, unplaced %v", d, ok, w.unplacedNodes())
}
w.putEntry(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(`{"handed-to":"mesh-vault","handed-by":"operator"}`), 2, time.Now())
if d, ok := w.composed("ace", "web", fp); !ok || d.Digest != "sha256:"+strings.Repeat("a", 64) {
t.Errorf("after the lease write, ace: %+v %v", d, ok)
}
if _, ok := w.composed("g14", "web", fp); ok || !slices.Equal(w.unplacedNodes(), []string{"g14"}) {
t.Errorf("after the lease write, g14 unplaced: composed %v, unplaced %v", ok, w.unplacedNodes())
}
findings := map[string][]byte{}
p := &passer{w: w, logf: func(string, ...any) {}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil }}}
p.unplacedFindings()
if _, ok := findings[assign.UnplacedKey("g14")]; !ok || len(findings) != 1 {
t.Fatalf("findings = %v", findings)
}
pl2, _ := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: "g14", Modules: []delivery.PlacedModule{{Module: "web"}}})
pkey2, _ := delivery.PlacementKey("g14")
w.putEntry(delivery.PlacementBucket, pkey2, pl2, 1, time.Now())
p.unplacedFindings()
if len(findings) != 0 {
t.Errorf("placed again: findings %v", findings)
}
})
t.Run("1955/L7 (ADR 0571 §4): seat-holders is written from placement, only the keys a change moves, and answered by placement seats", func(t *testing.T) {
writes := 0
kv := &countingKV{memKV: memKV{values: map[string][]byte{}}, writes: &writes}
p := &passer{w: newWorld(), logf: t.Logf, kvs: buckets{delivery.SeatHoldersBucket: kv}}
place := func(node string, holds ...delivery.Holding) {
pl, err := delivery.EncodePlacement(delivery.Placement{V: delivery.Version, Node: node, Modules: []delivery.PlacedModule{{Module: "web"}, {Module: "nats"}}, Holds: holds})
if err != nil {
t.Fatal(err)
}
k, _ := delivery.PlacementKey(node)
p.w.putEntry(delivery.PlacementBucket, k, pl, 1, time.Now())
}
place("ace", delivery.Holding{Claim: "mesh-broker", Scope: "mesh", Module: "nats"}, delivery.Holding{Claim: "node-shell", Scope: "node", Module: "web"})
place("g14", delivery.Holding{Claim: "node-shell", Scope: "node", Module: "web"})
ctx := context.Background()
p.seatHolders(ctx)
if writes != 3 || kv.values["mesh-broker"] == nil || kv.values["node-shell.ace"] == nil || kv.values["node-shell.g14"] == nil {
t.Fatalf("%d writes: %v", writes, kv.values)
}
p.seatHolders(ctx)
if writes != 3 {
t.Errorf("an unchanged placement wrote %d keys", writes-3)
}
place("g14") // g14 holds nothing now
p.seatHolders(ctx)
if writes != 3 || kv.values["node-shell.g14"] != nil || kv.values["node-shell.ace"] == nil {
t.Errorf("after g14's change: %d writes, %v", writes, kv.values)
}
st := memStore{delivery.SeatHoldersState: &memKV{values: kv.values}}
out, err := newSeatTools(st, nil).seats("ace")
if err != nil || len(out.(map[string]any)["seats"].([]delivery.SeatHolders)) != 2 {
t.Errorf("seats on ace = %+v, %v", out, err)
}
})
t.Run("1954/D1: a failed provision is the assigner's finding until it is provided", func(t *testing.T) {
findings := map[string][]byte{}
p := &passer{w: newWorld(), logf: func(string, ...any) {}, kvs: buckets{provisions.ProvisionsBucket: nil}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil }}}
failed, _ := provisions.Encode(provisions.Provision{Provision: "db", Consumer: provisions.End{Module: "photos", Node: "ace"},
Provider: provisions.End{Module: "postgres", Node: "novox"}, State: provisions.Failed, Why: "the role exists already"})
p.w.putEntry(provisions.ProvisionsBucket, "ace.photos.db", failed, 1, time.Now())
p.kvs[provisions.ProvisionsBucket] = &memKV{}
of := p.provisionsOf()
if got := of("photos")("ace"); got["ace.photos.db"].State != provisions.Failed {
t.Fatalf("provisions = %+v", got)
}
if _, ok := findings[assign.ProvisionFailedKey("ace.photos.db")]; !ok || len(findings) != 1 {
t.Fatalf("findings = %v", findings)
}
provided, _ := provisions.Encode(provisions.Provision{Provision: "db", Consumer: provisions.End{Module: "photos", Node: "ace"},
Provider: provisions.End{Module: "postgres", Node: "novox"}, State: provisions.Provided})
p.w.putEntry(provisions.ProvisionsBucket, "ace.photos.db", provided, 2, time.Now())
p.provisionsOf()
if len(findings) != 0 {
t.Errorf("findings after provided = %v", findings)
}
// A required provision with no key: pulsed with the bound first, said once the pulse expired, ended once keyed.
ctx := context.Background()
p.kvs[delivery.PlacementPulsesBucket] = &memKV{values: map[string][]byte{}}
gone := map[string]bool{"ace.photos.s3": true}
p.provisionsMissing(ctx, gone)
if !p.pulsing("provision-missing.ace.photos.s3") || len(findings) != 0 {
t.Fatalf("first seen: pulsing %v, findings %v", p.pulsing("provision-missing.ace.photos.s3"), findings)
}
p.provisionsMissing(ctx, gone)
if len(findings) != 0 {
t.Fatal("said before its bound")
}
p.w.put(delivery.PlacementPulsesBucket, "provision-missing.ace.photos.s3", nil) // the server expired it
p.provisionsMissing(ctx, gone)
if _, ok := findings[assign.ProvisionMissingKey("ace.photos.s3")]; !ok {
t.Fatalf("past its bound: findings %v", findings)
}
p.provisionsMissing(ctx, map[string]bool{})
if len(findings) != 0 {
t.Errorf("keyed again: findings %v", findings)
}
})
}
// wantsVerbs (novox/hq ADR 0565 §7, §8, §9): upgrade answers each module's policy and its source; a person's choice is
// written with who and why, recorded as a hand act in the seat's log and said as policy-changed, and default takes it back;
// the bus is refused; compare lists the modules the controller writes another policy for; the process writes policy-split
// for a split and deletes it once they agree; bus names the node behind the indexer's newest.
func wantsVerbs(t *testing.T) {
fpOld, fpNew := delivery.SourcePrefix2+strings.Repeat("1", 64), delivery.SourcePrefix2+strings.Repeat("2", 64)
st := memStore{}
for _, state := range []string{modulesState, manifestsState, composedState, policiesState, haveState, putsState, "findings"} {
st[state] = &memKV{values: map[string][]byte{}}
}
manifest := func(name, extra string) {
raw := `{"module":"` + name + `","version":"1"` + extra + `}`
digest := delivery.ManifestDigest([]byte(raw))
st[manifestsState].values[digest], _ = delivery.EncodeManifest(delivery.Manifest{V: delivery.Version, Module: name, JSON: raw})
st[modulesState].values[name], _ = delivery.EncodeModule(delivery.Module{V: delivery.Version, FP2: fpNew, Manifest: digest, Repository: "novox/x", Commit: "c"})
}
manifest("web", "")
manifest("photos", `,"upgrade":{"policy":"record","why":"a person checks the albums"}`)
manifest("nats", `,"provides":[{"name":"mesh-bus","scope":"mesh"}]`)
for node, fp := range map[string]string{"anchor": fpOld, "laptop": fpNew} {
st[haveState].values[node+".nats"], _ = delivery.EncodeHave(delivery.Have{FP: fp, Phase: delivery.PhaseRunning, Health: delivery.HealthOK})
}
composed := func(module, policy string) {
key, _ := delivery.ComposedKey("anchor", module, fpNew)
st[composedState].values[key], _ = json.Marshal(delivery.ComposedDigest{V: delivery.Version, Digest: "sha256:" + strings.Repeat("c", 64),
Judgement: delivery.JudgedNeither, Policy: policy, PolicyFrom: catalogue.FromPerson, PolicyWhy: "chosen at the controller"})
}
composed("web", catalogue.PolicyRecord)
composed("photos", catalogue.PolicyRecord)
var asked []string
var said []string
w := newWantsTools(st, func(key string, body any) (json.RawMessage, error) {
asked = append(asked, key)
return json.RawMessage(`{"ok":true}`), nil
}, func(event, id string, body any) error { said = append(said, event+" "+id); return nil })
policyOf := func(module string) policyRow {
out, err := w.doUpgrade(map[string]any{"module": module}, "op")
if err != nil {
t.Fatal(err)
}
return out.(map[string]any)["modules"].([]policyRow)[0]
}
if p := policyOf("photos"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromModule {
t.Errorf("photos = %+v", p)
}
if p := policyOf("nats"); p.Policy != catalogue.PolicyRecord || p.From != catalogue.FromBus {
t.Errorf("nats = %+v", p)
}
// compare: web rolls at the assigner and records at the controller.
out, _ := w.doUpgrade(map[string]any{"compare": "true"}, "op")
if splits := out.(map[string]any)["splits"].([]assign.Split); len(splits) != 1 || splits[0].Module != "web" || out.(map[string]any)["findings"] == nil {
t.Errorf("splits = %+v", splits)
}
// The process says the split as its finding, and deletes it once they agree.
findings := map[string][]byte{}
p := &passer{w: newWorld(), logf: func(string, ...any) {}, findings: assign.Findings{
Put: func(k string, v []byte) error { findings[k] = v; return nil },
Delete: func(k string) error { delete(findings, k); return nil },
Keys: func() ([]string, error) { return []string{"module.gone.policy-split"}, nil }}}
for k, v := range st[composedState].values {
p.w.putEntry(delivery.ControllerComposedBucket, k, v, 1, time.Now())
}
man := map[string]catalogue.Manifest{}
for _, name := range []string{"web", "photos"} {
raw, _, _, _ := st.get(modulesState, name)
mod, _ := delivery.DecodeModule(raw)
mraw, _, _, _ := st.get(manifestsState, mod.Manifest)
kept, _ := delivery.DecodeManifest(mraw)
man[name], _ = catalogue.ParseManifest([]byte(kept.JSON))
}
findings["module.gone.policy-split"] = []byte("{}")
p.policySplits(man, nil)
if _, ok := findings[assign.PolicySplitKey("web")]; !ok || len(findings) != 1 {
t.Errorf("findings = %v, want web's split alone (a stale one deleted)", findings)
}
// a person's record for web: written, recorded, said; the split is gone.
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "record", "why": "the operator checks it"}, "person.jochen"); err != nil {
t.Fatal(err)
}
chosen, _ := assign.DecodePolicies(st[policiesState].values)
if c := chosen["web"]; c.Policy != catalogue.PolicyRecord || c.By != "person.jochen" || !slices.Contains(asked, "seat:mesh-hand-acts.hand-act") ||
len(said) != 1 || !strings.HasPrefix(said[0], "policy-changed policy-changed.web.") {
t.Errorf("chosen %+v, asked %v, said %v", c, asked, said)
}
p.policySplits(man, chosen)
if len(findings) != 0 {
t.Errorf("findings after the choice = %v", findings)
}
if _, err := w.doUpgrade(map[string]any{"module": "nats", "policy": "roll", "why": "x"}, "op"); err == nil {
t.Error("the bus was given a policy")
}
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "roll"}, "op"); err == nil {
t.Error("a policy without why was set")
}
if _, err := w.doUpgrade(map[string]any{"module": "web", "policy": "default", "why": "back to the module"}, "op"); err != nil || len(st[policiesState].values) != 0 {
t.Errorf("default left %v (%v)", st[policiesState].values, err)
}
// bus: the node behind the indexer's newest.
b, err := w.doBus(nil)
if err != nil {
t.Fatal(err)
}
nodes := b.(map[string]any)["nodes"].([]busNode)
if b.(map[string]any)["module"] != "nats" || len(nodes) != 2 || !nodes[0].Behind || nodes[1].Behind || !slices.Contains(asked, "seat:mesh-hand-acts.hand-acts") {
t.Errorf("bus = %+v, asked %v", b, asked)
}
if tools := w.tools(); len(tools) != 2 || tools[0].Name != "mesh-wants.upgrade" || tools[1].Run == nil {
t.Errorf("the seat's verbs as served: %+v", tools)
}
}
// readManifest is the module's manifest, read as the controller reads it.
@@ -155,3 +401,14 @@ func readManifest() (catalogue.Manifest, error) {
}
return catalogue.ParseManifest(raw)
}
// countingKV is a memKV that counts its puts.
type countingKV struct {
memKV
writes *int
}
func (c *countingKV) Put(ctx context.Context, key string, value []byte) (uint64, error) {
*c.writes++
return c.memKV.Put(ctx, key, value)
}
@@ -351,7 +351,12 @@ func serveTools() error {
time.Sleep(wait)
}
}()
return stdio.Serve("", append(s.tools(), newSeatTools(runtimeStore{}, stdio.Ask).tools()...))
tools := append(s.tools(), newSeatTools(runtimeStore{}, stdio.Ask).tools()...)
tools = append(tools, newWantsTools(runtimeStore{}, stdio.Ask, func(e, id string, b any) error {
// The seat's event, said on the seat: a claimed seat's event is never the module's own.
return stdio.SeatPublish(wantsEventSubject(e), b, id)
}).tools()...)
return stdio.Serve("", tools)
}
// shortFP is a fingerprint's first twelve hex digits.
@@ -0,0 +1,300 @@
package main
import (
"encoding/json"
"errors"
"fmt"
"sort"
"strconv"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"git.novox.be/novox/mesh-sdk/go/health"
"mesh-delivery-assigner/internal/assign"
)
// The seat mesh-wants' verbs (novox/hq ADR 0565 §7, §8, §9, §15; issue 1953 D4-D6), served through the tool runner beside the
// put tools and mesh-placement's verbs. `upgrade` reads and sets a person's upgrade policy, kept in the state `policies`
// (each set recorded as a hand act and said as policy-changed), and with compare lists every module whose policy differs from
// the one the controller writes beside its compositions; `bus` says what a bus upgrade would do. The assigner's process reads
// `policies` when it judges a move and writes the finding policy-split while a split stands.
// The states the verbs read, as the runtime names them.
const (
policiesState = delivery.AssignerPoliciesState
manifestsState = delivery.IndexerModule + ".manifests"
composedState = "mesh-controller.composed"
// PolicyChangedEvent is the seat's event of a person's policy set or taken back.
PolicyChangedEvent = "policy-changed"
// busStepCause is the cause the bus's step is recorded under in the hand-act log.
busStepCause = "bus-upgrade"
)
// wantsTools serves the seat's verbs over a store, asking seats through ask and saying events through emit.
type wantsTools struct {
st seatStore
ask func(key string, body any) (json.RawMessage, error)
emit func(event, id string, body any) error
now func() time.Time
}
// wantsEventSubject is the subject of one of the seat's events.
func wantsEventSubject(event string) string {
return "mesh.seat." + catalogue.WantsSeat + ".event." + event
}
func newWantsTools(st seatStore, ask func(string, any) (json.RawMessage, error), emit func(event, id string, body any) error) *wantsTools {
return &wantsTools{st: st, ask: ask, emit: emit, now: time.Now}
}
// leased says the controller's lease is handed away.
func (s *wantsTools) leased() bool {
raw, _, found, err := s.st.get(buscompose.ControllerLease, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, err)
return away
}
// values reads every key of a state.
func (s *wantsTools) values(state string) (map[string][]byte, error) {
keys, err := s.st.keys(state)
if err != nil {
return nil, fmt.Errorf("%s cannot be listed: %w", state, err)
}
out := map[string][]byte{}
for _, k := range keys {
raw, _, found, err := s.st.get(state, k)
if err != nil {
return nil, fmt.Errorf("%s %s cannot be read: %w", state, k, err)
}
if found {
out[k] = raw
}
}
return out, nil
}
// manifests is every module the indexer holds, with its trunk manifest and fingerprint.
func (s *wantsTools) manifests() (map[string]catalogue.Manifest, map[string]delivery.Module, error) {
mods, err := s.values(modulesState)
if err != nil {
return nil, nil, err
}
out, held := map[string]catalogue.Manifest{}, map[string]delivery.Module{}
for name, raw := range mods {
m, err := delivery.DecodeModule(raw)
if err != nil {
continue
}
held[name] = m
if m.Manifest == "" {
continue
}
mraw, _, found, err := s.st.get(manifestsState, m.Manifest)
if err != nil || !found {
continue
}
kept, err := delivery.DecodeManifest(mraw)
if err != nil {
continue
}
if man, err := catalogue.ParseManifest([]byte(kept.JSON)); err == nil {
out[name] = man
}
}
return out, held, nil
}
// policyRow is one module's policy as upgrade answers it.
type policyRow struct {
Module string `json:"module"`
Policy string `json:"policy"`
From string `json:"from"`
Why string `json:"why,omitempty"`
By string `json:"by,omitempty"`
}
// doUpgrade is the verb upgrade: read every module's policy or one's; with policy and why set a person's choice, or take it
// back with default; with compare list the splits.
func (s *wantsTools) doUpgrade(args map[string]any, by string) (any, error) {
manifests, _, err := s.manifests()
if err != nil {
return nil, err
}
raw, err := s.values(policiesState)
if err != nil {
return nil, err
}
chosen, unreadable := assign.DecodePolicies(raw)
if arg(args, "compare") == "true" {
composed, err := s.values(composedState)
if err != nil {
return nil, err
}
splits := assign.Splits(manifests, chosen, assign.ControllerPolicies(composed))
if splits == nil {
splits = []assign.Split{}
}
out := map[string]any{"splits": splits, "empty": len(splits) == 0}
// The assigner's findings, its said conditions and its policy splits as the condition keeper reads them (§8, §15).
if found, err := s.values(health.FindingsState); err != nil {
out["findings-unread"] = err.Error()
} else {
keys := []string{}
for k := range found {
keys = append(keys, k)
}
sort.Strings(keys)
out["findings"] = keys
}
return out, nil
}
module, policy := arg(args, "module"), arg(args, "policy")
if policy != "" {
return s.setPolicy(module, policy, arg(args, "why"), by, manifests)
}
var rows []policyRow
for name, m := range manifests {
if module != "" && name != module {
continue
}
p, from, why, who := assign.PolicyOf(m, chosen)
rows = append(rows, policyRow{Module: name, Policy: p, From: from, Why: why, By: who})
}
if module != "" && len(rows) == 0 {
return nil, fmt.Errorf("the indexer holds no manifest of %s", module)
}
sort.Slice(rows, func(i, j int) bool { return rows[i].Module < rows[j].Module })
return map[string]any{"modules": rows, "unreadable": unreadable}, nil
}
// setPolicy writes a person's choice of a module's policy, or deletes it for default, records the act in the hand-act log
// and says policy-changed. The bus is never rolled: a choice for it is refused.
func (s *wantsTools) setPolicy(module, policy, why, by string, manifests map[string]catalogue.Manifest) (any, error) {
if module == "" {
return nil, errors.New("a policy is set for one module: name it")
}
m, ok := manifests[module]
if !ok {
return nil, fmt.Errorf("the indexer holds no manifest of %s", module)
}
if catalogue.ProvidesBus(m) {
return nil, fmt.Errorf("%s is the bus, which is never rolled whoever says so (novox/hq ADR 0236): its step is the put on the operator's warrant", module)
}
if why == "" {
return nil, errors.New("a person's policy says why")
}
if by == "" || by == stdio.Unproven {
return nil, errors.New("a person's policy names who chose it, and the runtime proved no caller")
}
what := fmt.Sprintf("mesh-wants.upgrade %s %s", module, policy)
if policy == "default" {
if err := s.st.del(policiesState, module); err != nil {
return nil, fmt.Errorf("the choice for %s cannot be taken back: %w", module, err)
}
} else {
p := catalogue.PersonPolicy{V: catalogue.PersonPolicyVersion, Module: module, Policy: policy, Why: why, By: by, At: s.now().UTC()}
if err := p.Check(); err != nil {
return nil, err
}
raw, _ := json.Marshal(p)
if err := s.st.put(policiesState, module, raw); err != nil {
return nil, fmt.Errorf("the choice for %s cannot be written: %w", module, err)
}
}
rec := handacts.Recorder{Ask: s.ask, LeaseAway: s.leased}
taken, rerr := rec.Record(handacts.Request{What: what, Why: why, Cause: "upgrade-policy", Verb: "mesh-wants.upgrade"})
recorded := taken.Log
if rerr != nil {
recorded = "not recorded: " + rerr.Error()
}
body := map[string]any{"module": module, "policy": policy, "why": why, "by": by}
said := ""
if s.emit != nil {
id := PolicyChangedEvent + "." + module + "." + strconv.FormatInt(s.now().UnixNano(), 10)
if err := s.emit(PolicyChangedEvent, id, body); err != nil {
said = "not said: " + err.Error()
}
}
return map[string]any{"set": body, "hand-act": recorded, "event": said}, nil
}
// busNode is one node's bus as bus answers it.
type busNode struct {
Node string `json:"node"`
FP string `json:"fp"`
Behind bool `json:"behind"`
}
// doBus is the verb bus: the fingerprint of the bus module each node runs against the indexer's newest, the nodes behind,
// the last bus steps of the hand-act log and the assigner's puts of the bus. Read-only.
func (s *wantsTools) doBus(map[string]any) (any, error) {
manifests, held, err := s.manifests()
if err != nil {
return nil, err
}
bus := ""
for name, m := range manifests {
if catalogue.ProvidesBus(m) {
bus = name
}
}
if bus == "" {
return nil, errors.New("the indexer holds no module that provides the bus")
}
newest := held[bus].FP2
haves, err := s.values(haveState)
if err != nil {
return nil, err
}
var nodes []busNode
for key, raw := range haves {
node, module, isNode, err := delivery.ParseHaveKey(key)
if err != nil || isNode || module != bus {
continue
}
h, err := delivery.DecodeHave(raw)
if err != nil || h.Phase == delivery.PhaseRemoved {
continue
}
nodes = append(nodes, busNode{Node: node, FP: h.FP, Behind: newest != "" && h.FP != newest})
}
sort.Slice(nodes, func(i, j int) bool { return nodes[i].Node < nodes[j].Node })
out := map[string]any{"module": bus, "newest": newest, "nodes": nodes}
if steps, err := s.ask("seat:"+handacts.Seat+".hand-acts", map[string]any{"cause": busStepCause, "days": 90}); err != nil {
out["steps-unread"] = err.Error()
} else {
out["steps"] = steps
}
puts, err := s.values(putsState)
if err != nil {
out["puts-unread"] = err.Error()
} else {
mine := map[string]json.RawMessage{}
for k, v := range puts {
if _, module, err := delivery.ParsePutKey(k); err == nil && module == bus {
mine[k] = v
}
}
out["puts"] = mine
}
return out, nil
}
// tools are the seat's verbs as the runtime serves them, each named <seat>.<verb>, with the seat's own schemas.
func (s *wantsTools) tools() []stdio.Tool {
seat, _ := catalogue.SeatNamed(catalogue.WantsSeat)
run := map[string]func(map[string]any) (any, error){
"upgrade": func(a map[string]any) (any, error) { return s.doUpgrade(a, stdio.CallerOf(a)) },
"bus": s.doBus,
}
var out []stdio.Tool
for _, v := range seat.Serves {
out = append(out, stdio.Tool{Name: catalogue.WantsSeat + "." + v.Name, Description: v.Description, Input: v.Input, Run: run[v.Name]})
}
return out
}
@@ -13,6 +13,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/compose"
@@ -51,6 +52,8 @@ type world struct {
changes map[string]uint64
// meta is, by bucket and key, the revision and the time of the value held now, where the watch said them (putEntry).
meta map[string]entryMeta
// trunkBuilt is, by module, its newest done trunk build in the builder's log (putBuilt).
trunkBuilt map[string]delivery.BuildLogEntry
}
// entryMeta is a value's revision in its bucket and when it was written.
@@ -68,7 +71,7 @@ type parsed struct {
func newWorld() *world {
return &world{values: map[string]map[string][]byte{}, manifests: map[string]parsed{}, healthy: map[string][]string{}, changes: map[string]uint64{},
meta: map[string]entryMeta{}}
meta: map[string]entryMeta{}, trunkBuilt: map[string]delivery.BuildLogEntry{}}
}
// putEntry is put, keeping the value's revision and the time it was written.
@@ -195,21 +198,38 @@ func (w *world) manifest(ctx context.Context, get getter, digest string) parsed
// assigned is each node's modules both the controller's assignments and the seat mesh-placement's placement name, in name order
// (ADR 0549 §5; the assignments alone before any placement was read), with every open assignment by node and module.
// Once the controller's lease is handed away it is the seat's placement alone, with no assignment (ADR 0549 §5, §6).
func (w *world) assigned() (map[string][]string, map[string]assignment) {
byAssignment, all := w.assignedByController()
// A lease handed away before any placement was read keeps the assignments as they last stood, never an empty mesh:
// the lease write's own check refuses that order (ADR 0547 §11).
placed, seeded := w.placed()
both, _ := assign.Placed(byAssignment, placed, seeded)
byAssignment, all := w.assignedByController()
if away := w.handedAway(); away && seeded {
both, _ := assign.Placed(byAssignment, placed, seeded, away)
return both, map[string]assignment{}
}
both, _ := assign.Placed(byAssignment, placed, seeded, false)
return both, all
}
// placementDiffers is every pair the controller's assignments and the seat's placement do not both name.
// placementDiffers is every pair the controller's assignments and the seat's placement do not both name: none once
// the controller's lease is handed away, since the placement is then the one source.
func (w *world) placementDiffers() []assign.Pair {
byAssignment, _ := w.assignedByController()
placed, seeded := w.placed()
_, differ := assign.Placed(byAssignment, placed, seeded)
byAssignment, _ := w.assignedByController()
_, differ := assign.Placed(byAssignment, placed, seeded, w.handedAway())
return differ
}
// handedAway says the controller's lease is handed to a holder that is no controller (mesh-sdk buscompose.LeaseHandedAway,
// ADR 0561 §3): from then the seat mesh-placement's placement alone names where modules run (ADR 0549 §5). A lease not read (its
// bucket not granted or not there), a controller's, expired or unreadable, is not handed away.
func (w *world) handedAway() bool {
raw, found := w.read(buscompose.LeaseBucket)[buscompose.ControllerLeaseKey]
away, _ := buscompose.LeaseHandedAway(raw, found && raw != nil, nil)
return away
}
// placed is each node's modules in the seat's placement (ADR 0567 §1), and whether any placement key was read: a key
// that does not read places nothing on its node and is said by the indexer's own read tool.
func (w *world) placed() (map[string][]string, bool) {
@@ -415,6 +435,11 @@ func (r worldReader) Served(ctx context.Context, node, module string, keys []str
func (w *world) change(ctx context.Context, manifests getter, c delivery.Candidate, trunk map[string]delivery.Module) (assign.Change, string) {
ch := assign.Change{Group: c.Group, Heads: c.HeadsHash, Manifests: map[string]catalogue.Manifest{}, Unreadable: map[string]string{},
Twice: map[string][]string{}, Digests: map[string]string{}, Members: map[string]string{}}
heads := map[string]string{}
for _, p := range c.Pulls {
heads[fmt.Sprintf("%s#%d", p.Repository, p.Number)] = p.Head.Head
}
ch.Held = assign.HeldOf(w.read(keptBucket), heads)
definedIn := map[string]map[string]bool{}
for _, p := range c.Pulls {
if p.Repository == assign.SDKRepository {
@@ -546,6 +571,16 @@ func (w *world) value(bucket, key string) []byte {
return w.values[bucket][key]
}
// grantsBucket is the bucket the grants are read from now (delivery.GrantSource): the grants part's while the controller's
// lease is handed away, as the lease bucket's key says it (buscompose.LeaseHandedAway), the controller's in every other
// case, a lease not held here included.
func (w *world) grantsBucket() string {
raw := w.value(buscompose.LeaseBucket, buscompose.ControllerLeaseKey)
away, _ := buscompose.LeaseHandedAway(raw, raw != nil, nil)
bucket, _ := delivery.GrantSource(away)
return bucket
}
// has says whether a bucket holds a key now.
func (w *world) has(bucket, key string) bool {
w.mu.Lock()
@@ -572,7 +607,17 @@ func (w *world) built(module, fp string) bool {
// composed is the controller's composition of module for node at fp: its digest and the judgement of the move to it
// (mesh-controller_composed `<node>.<module>.<hex>`), and whether it exists and can be read.
//
// From the lease write the controller's copy is no longer read (ADR 0571 §6, a named exception of clause 10 until then): a
// placed node composes its own, and its composition is the digest its own have names; an unplaced node has none, so it gets
// no want (and the pass says so, unplacedFindings).
func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool) {
if w.handedAway() {
if !w.placedNodes()[node] {
return delivery.ComposedDigest{}, false
}
return w.ownComposed(node)
}
key, err := delivery.ComposedKey(node, module, fp)
if err != nil {
return delivery.ComposedDigest{}, false
@@ -590,6 +635,9 @@ func (w *world) composed(node, module, fp string) (delivery.ComposedDigest, bool
// nodeComposed is, by node, the digest of the node's own composition (mesh-controller_composed `<node>.node`).
func (w *world) nodeComposed() map[string]delivery.ComposedDigest {
out := map[string]delivery.ComposedDigest{}
if w.handedAway() {
return out // the controller composes no node from the lease write (ADR 0571 §6)
}
for key, value := range w.read(delivery.ControllerComposedBucket) {
node, _, _, isNode, err := delivery.ParseComposedKey(key)
if err != nil || !isNode {
@@ -621,29 +669,72 @@ func (w *world) changed(bucket, key string) uint64 {
return w.changes[bucket+"|"+key]
}
// runningCommits is, of every open assignment of module, the commit of the build it names, read from the controller's
// build record (`mesh-controller_builds` `<module>.build.<id>`), distinct and sorted: the commits a reader of the
// manifests runs at (novox/hq ADR 0482 clause 5, issue 1623). A build whose record names no full commit is left out.
func (w *world) runningCommits(ctx context.Context, get getter, module string) []string {
_, all := w.assigned()
seen := map[string]bool{}
for _, a := range all {
if a.Module != module || a.Build == "" {
continue
}
value, err := get(ctx, delivery.BuildKey(module, a.Build))
if err != nil {
continue
}
var rec delivery.BuildRecord
if json.Unmarshal(value, &rec) != nil || len(rec.Commit) != 40 {
continue
}
seen[rec.Commit] = true
// logWatch follows the builder's log: handle each entry it keeps, then each new one (process.LogReader.Watch).
type logWatch func(handle func(stdio.LogEntry) error) error
// putBuilt takes one entry of the builder's log in: a done trunk build's end (no ref) naming a full commit, newer than the
// module's last, becomes the module's running commit. It answers whether the commit changed.
func (w *world) putBuilt(raw []byte) bool {
e, err := delivery.DecodeBuildLogEntry(raw)
if err != nil || e.Event != delivery.BuildEndedEvent || e.Outcome != delivery.BuildDone || e.Ref != "" || len(e.Commit) != 40 {
return false
}
out := make([]string, 0, len(seen))
for c := range seen {
out = append(out, c)
w.mu.Lock()
defer w.mu.Unlock()
have, ok := w.trunkBuilt[e.Module]
if ok && !e.At.After(have.At) {
return false
}
w.trunkBuilt[e.Module] = e
return !ok || have.Commit != e.Commit
}
// runningCommits is the commit of module's newest done trunk build in the builder's log (ADR 0571 clause 1): the commit a
// reader of the manifests runs at (novox/hq ADR 0482 clause 5, issue 1623); none when the log holds no such build.
func (w *world) runningCommits(module string) []string {
w.mu.Lock()
defer w.mu.Unlock()
if e, ok := w.trunkBuilt[module]; ok {
return []string{e.Commit}
}
return nil
}
// keptBucket is mesh-config's `kept`: the approved held values of open pull requests (ADR 0565 §5).
var keptBucket = "mesh-config_" + delivery.MeshConfigKeptState
// ownComposed is a placed node's own composition, as the digest its own have names (`<node>.node` of the haves, Composed):
// what rule 7 reads from the lease write. A node whose have names none is not composed yet.
func (w *world) ownComposed(node string) (delivery.ComposedDigest, bool) {
key, err := delivery.HaveNodeKey(node)
if err != nil {
return delivery.ComposedDigest{}, false
}
raw, ok := w.read(delivery.HaveBucket)[key]
if !ok {
return delivery.ComposedDigest{}, false
}
h, err := delivery.DecodeHaveNode(raw)
if err != nil || h.Composed == "" {
return delivery.ComposedDigest{}, false
}
return delivery.ComposedDigest{V: delivery.Version, Digest: h.Composed, Judgement: delivery.JudgedNeither,
Words: "the node composes its own"}, true
}
// unplacedNodes are, from the lease write, the nodes whose engine has a have and that no placement names: they get no
// want, since nothing composes them.
func (w *world) unplacedNodes() []string {
if !w.handedAway() {
return nil
}
placed := w.placedNodes()
var out []string
for key := range w.read(delivery.HaveBucket) {
node, _, isNode, err := delivery.ParseHaveKey(key)
if err == nil && isNode && !placed[node] {
out = append(out, node)
}
}
sort.Strings(out)
return out
@@ -6,7 +6,9 @@ import (
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-assigner/internal/assign"
@@ -125,4 +127,40 @@ func theWorldCountsTheValuesAKeyTakes(t *testing.T) {
if got := w.changed(delivery.HaveBucket, "gone"); got != 0 {
t.Errorf("changed = %d for a key never seen", got)
}
// Issue 774: grants are read from the grants part's bucket only while the controller's lease is handed away: a grant
// only the part holds lets a node's put through then, and with a controller's lease (or none, or an unreadable one) it
// holds it, a grant only the controller holds being the one that counts instead.
fp := delivery.SourcePrefix2 + strings.Repeat("d", 64)
gkey, _ := delivery.ControllerGrantKey("web", fp)
grant, _ := delivery.EncodeControllerGrant(delivery.ControllerGrant{Accounts: []string{"ace.web"}})
for _, row := range []struct {
name, lease string
inPart, inController bool
bucket string
granted bool
}{
{"handed away, the part's grant", `{"handed-to":"the operator's hand-over"}`, true, false, delivery.PartGrantsBucket, true},
{"handed away, only the controller's grant", `{"handed-to":"x"}`, false, true, delivery.PartGrantsBucket, false},
{"a controller holds it, only the part's grant", `{"instance":"novox/1","epoch":3}`, true, false, delivery.ControllerGrantsBucket, false},
{"a controller holds it, its own grant", `{"instance":"novox/1"}`, false, true, delivery.ControllerGrantsBucket, true},
{"no lease read", ``, true, false, delivery.ControllerGrantsBucket, false},
{"an unreadable lease", `[1]`, true, false, delivery.ControllerGrantsBucket, false},
} {
p := newPasser(buckets{}, time.Now())
if row.lease != "" {
p.w.put(buscompose.LeaseBucket, buscompose.ControllerLeaseKey, []byte(row.lease))
}
if row.inPart {
p.w.put(delivery.PartGrantsBucket, gkey, grant)
}
if row.inController {
p.w.put(delivery.ControllerGrantsBucket, gkey, grant)
}
if got := p.w.grantsBucket(); got != row.bucket {
t.Errorf("%s: grants are read from %s, want %s", row.name, got, row.bucket)
}
if _, granted := p.composedFor("ace", "web", fp); granted != row.granted {
t.Errorf("%s: granted = %v, want %v", row.name, granted, row.granted)
}
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-assigner
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -1,12 +1,14 @@
package assign
import (
"slices"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
// 1297/B9 (the assigner's half): the ask about a held fingerprint is a valid ask of the router and of mesh-sdk's layout at
@@ -73,7 +75,13 @@ func TestTheRestoreAskIsValid(t *testing.T) {
// 1906/D3: the condition for a held fingerprint names the module, the node and the check's words, under the assigner's own
// name (ADR 0437 clause 9), and is cleared under the same key.
func TestTheHeldConditionNamesModuleNodeAndWords(t *testing.T) {
func TestTheAssignersConditions(t *testing.T) {
t.Run("the held condition names module, node and words", heldConditionNamesModuleNodeAndWords)
t.Run("a lifted hold clears its condition saying why", liftedHoldClearsItsConditionSayingWhy)
t.Run("1952/WM6: each said condition is a finding, its clear deletes it, said straight only until the take-over", saidConditionsAreFindings)
}
func heldConditionNamesModuleNodeAndWords(t *testing.T) {
c := HeldCondition("mail", fNew, "ace", "connection refused", true, true)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
@@ -87,7 +95,7 @@ func TestTheHeldConditionNamesModuleNodeAndWords(t *testing.T) {
}
// 1906/H2: a hold lifted by itself clears the held condition under its key, saying why.
func TestALiftedHoldClearsItsConditionSayingWhy(t *testing.T) {
func liftedHoldClearsItsConditionSayingWhy(t *testing.T) {
c := LiftedCondition("mail", fNew)
if _, err := asks.EncodeCondition(c); err != nil {
t.Fatalf("the condition is refused: %v", err)
@@ -97,6 +105,48 @@ func TestALiftedHoldClearsItsConditionSayingWhy(t *testing.T) {
}
}
// 1952/WM6 (ADR 0564 §2, issue 1952 note 16): every condition the assigner says is a finding in its own bucket, under
// `<scope>.<id>.<kind>`, written at the raise and deleted at the clear; it is said straight to the router while the take-over
// key follows the controller, and only as a finding from the take-over.
func saidConditionsAreFindings(t *testing.T) {
at := time.Date(2026, 10, 29, 21, 0, 0, 0, time.UTC)
for _, c := range []struct {
raised, cleared asks.Condition
key string
}{
{HeldCondition("mail", fNew, "ace", "refused", true, true), LiftedCondition("mail", fNew), "module.mail." + strings.TrimPrefix(fNew, delivery.SourcePrefix2) + ".held"},
{NodeFailedCondition("ace", "mail", fNew, NodeFailure{Words: "x", Others: []string{"g14"}}, true), NodeFailedCondition("ace", "mail", fNew, NodeFailure{}, false), "module.ace.mail.failed-alone"},
{NeitherCondition("ace", "mail", Holding{Rule: 3, Words: "w"}, true), NeitherCondition("ace", "mail", Holding{}, false), "module.ace.mail.neither"},
{PlacementCondition(Pair{Node: "ace", Module: "mail", Only: OnlyAssignments}, true), PlacementCondition(Pair{Node: "ace", Module: "mail"}, false), "module.ace.mail.placement"},
{SeatWaitsCondition([]string{"placement"}), SeatWaitsCondition(nil), "module.mesh-delivery-assigner.seat-states"},
{BackupCondition("ace", "mail", fNew, "disk full", true), BackupCondition("ace", "mail", fNew, "", false), "module.ace.mail." + strings.TrimPrefix(fNew, delivery.SourcePrefix2) + ".backup"},
} {
puts, deletes, said := map[string]health.Finding{}, []string{}, 0
keeper := false
f := Findings{Put: func(k string, v []byte) error {
got, err := health.DecodeFinding(v)
puts[k] = got
return err
}, Delete: func(k string) error { deletes = append(deletes, k); return nil }, ToKeeper: func() bool { return keeper }, Now: func() time.Time { return at }}
say := f.Say(func(asks.Condition) error { said++; return nil })
if err := say(c.raised); err != nil {
t.Fatalf("%s: %v", c.key, err)
}
got, ok := puts[c.key]
if !ok || got.Severity != health.Severity(c.raised.Severity) || got.Headline != c.raised.Headline || got.Observer != delivery.AssignerModule ||
got.Subject.Machine != c.raised.Subject.Machine || said != 1 {
t.Fatalf("%s: finding %+v (written %v), said %d", c.key, got, puts, said)
}
keeper = true
if err := say(c.cleared); err != nil || !slices.Equal(deletes, []string{c.key}) || said != 1 {
t.Fatalf("%s: deleted %v, said %d after the take-over (%v)", c.key, deletes, said, err)
}
}
if _, _, _, err := FindingOf(asks.Condition{Key: "another.x", State: asks.StateRaised}, at); err == nil {
t.Error("a condition not the assigner's is a finding")
}
}
// 1906/D4: the record ask and the backup ask are valid asks of the router and of mesh-sdk's layout at their keys, answerable in
// the view, and name the move and the node.
func TestTheRecordAndBackupAsksAreValid(t *testing.T) {
@@ -5,6 +5,7 @@
package assign
import (
"encoding/json"
"fmt"
"slices"
"sort"
@@ -59,6 +60,63 @@ type Change struct {
// Members the head of each member by repository: what a composer at the group's mesh-sdk reads (issues 1622, 1623).
Digests map[string]string
Members map[string]string
// Held are the approved held values of the group's pull requests at the heads judged (mesh-config's `kept`, novox/hq ADR
// 0565 §5, ADR 0502 §4): each laid over its module's values on its node in the change's composition, and named.
Held []delivery.KeptValue
}
// HeldOf are the held values kept for the pull requests the candidate names, each at the head it names: one of another
// pull request, or at another head, is never laid (ADR 0502 §4). kept is mesh-config's `kept` by key.
func HeldOf(kept map[string][]byte, heads map[string]string) []delivery.KeptValue {
var out []delivery.KeptValue
for _, raw := range kept {
var k delivery.KeptValue
if json.Unmarshal(raw, &k) != nil || k.Node == "" || k.Module == "" || k.Key == "" {
continue
}
if head, ok := heads[k.Pull]; ok && head == k.Head {
out = append(out, k)
}
}
sort.Slice(out, func(i, j int) bool {
a, b := out[i], out[j]
return a.Pull+a.Node+a.Module+a.Key < b.Pull+b.Node+b.Module+b.Key
})
return out
}
// layHeld lays each held value over its module's values on its node, as a layer of its own after the rest, and answers
// the words that name each; judged first against the head's declaration of its key (catalogue.ValueProblems), a value the
// declaration refuses, or a key it does not declare, is a failure.
func layHeld(in *compose.Input, held []delivery.KeptValue) (said, fails []string) {
for _, k := range held {
var v any
if err := json.Unmarshal(k.Value, &v); err != nil {
said = append(said, fmt.Sprintf("the held value %s of %s on %s for %s cannot be read, so it is not laid: %v", k.Key, k.Module, k.Node, k.Pull, err))
continue
}
d, declared := in.Catalogue[k.Module].Settings[k.Key]
if !declared {
fails = append(fails, fmt.Sprintf("the held value %s of %s on %s for %s: the head declares no such key", k.Key, k.Module, k.Node, k.Pull))
continue
}
if problems := catalogue.ValueProblems(d, v); len(problems) > 0 {
fails = append(fails, fmt.Sprintf("the held value %s of %s on %s for %s does not hold to the head's declaration: %s", k.Key, k.Module,
k.Node, k.Pull, strings.Join(problems, "; ")))
continue
}
at := slices.IndexFunc(in.Nodes, func(n compose.Node) bool { return n.Name == k.Node })
if at < 0 {
said = append(said, fmt.Sprintf("the held value %s of %s for %s names %s, which runs no module of the mesh: not laid", k.Key, k.Module, k.Pull, k.Node))
continue
}
if in.Nodes[at].Settings == nil {
in.Nodes[at].Settings = map[string][]catalogue.Layer{}
}
in.Nodes[at].Settings[k.Module] = append(in.Nodes[at].Settings[k.Module], catalogue.Layer{From: "held for " + k.Pull, Values: map[string]any{k.Key: v}})
said = append(said, fmt.Sprintf("held value laid: %s of %s on %s, held for %s at %.12s, approved by %s (novox/hq ADR 0502 §4)", k.Key, k.Module, k.Node, k.Pull, k.Head, k.Approved))
}
return said, fails
}
// Touched is the machines a change touches (ADR 0446 clause 3): those running a module the change moves, adds or
@@ -171,6 +229,11 @@ func Compose(m Mesh, c Change, sdk string, took time.Duration) delivery.Composit
delete(changed, name)
}
after := input(changed, m.Assigned)
laid, heldFails := layHeld(&after, c.Held)
v.Warnings = append(v.Warnings, laid...)
for _, f := range heldFails {
failAt("", f)
}
for _, node := range machines {
_, before := compose.Resolve(trunk, node)
_, now := compose.Resolve(after, node)
@@ -1,7 +1,9 @@
package assign
import (
"encoding/json"
"slices"
"strconv"
"strings"
"testing"
@@ -38,6 +40,32 @@ func TestAChangeOfOneModuleComposesItsMachine(t *testing.T) {
if _, err := delivery.EncodeCompositionVerdict(v); err != nil {
t.Errorf("the verdict cannot be written: %v", err)
}
// 1953/D7 (ADR 0565 §5, ADR 0502 §4): the group's own pull request's held value at its head is laid and named, judged
// against the head's declaration; one of another pull request or another head is not laid.
web, err := catalogue.ParseManifest([]byte(`{"module":"web","version":"2","config":{"port":{"type":"number","default":80,"description":"the port, a number","runtime":true}}}`))
if err != nil {
t.Fatal(err)
}
kept := map[string][]byte{}
for i, k := range []delivery.KeptValue{
{Pull: "novox/x#1", Head: "h1", Node: "g14", Module: "web", Key: "port", Value: json.RawMessage(`81`), Approved: "op"},
{Pull: "novox/x#2", Head: "h2", Node: "g14", Module: "web", Key: "port", Value: json.RawMessage(`82`)},
{Pull: "novox/x#1", Head: "old", Node: "g14", Module: "web", Key: "other", Value: json.RawMessage(`1`)},
} {
kept[strconv.Itoa(i)], _ = json.Marshal(k)
}
held := HeldOf(kept, map[string]string{"novox/x#1": "h1"})
if len(held) != 1 || string(held[0].Value) != "81" {
t.Fatalf("held = %+v, want #1's at its head alone", held)
}
c = Change{Group: "g", Heads: "0123456789ab", Manifests: map[string]catalogue.Manifest{"web": web}, Held: held}
if v := Compose(meshOf(), c, "x", 0); v.State != delivery.CompositionPass || !hasWarning(v, "held value laid: port of web on g14, held for novox/x#1") {
t.Errorf("composition = %+v", v)
}
c.Held[0].Value = json.RawMessage(`"eighty"`)
if v := Compose(meshOf(), c, "x", 0); v.State != delivery.CompositionFail || !strings.Contains(v.Words, "does not hold to the head's declaration") {
t.Errorf("a held value the declaration refuses: %+v", v)
}
}
// 1297/AC2: a change that removes a module a machine runs fails, naming the machine and the module.
@@ -0,0 +1,105 @@
package assign
import (
"errors"
"fmt"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
// The assigner's said conditions become findings (novox/hq ADR 0564 §2, issue 1952 WM6, note 16): every condition it says
// under its own name (ADR 0437 §9) — a hold and its lift, a node failing alone, a module on neither list, a backup, a pair
// placed by one source, the seat waiting for its states — is also a finding in its own bucket `findings`, written at its
// raise and deleted at its clear, which the condition keeper raises and clears. Until the operator's take-over the router
// hears it straight from the assigner, as before; from the take-over the keeper's condition is the one said, so the
// assigner no longer says it straight (never both).
// FindingsBucket is the assigner's findings.
var FindingsBucket = health.FindingsBucket(delivery.AssignerModule)
// FindingOf is a said condition as a finding: its key `<scope>.<id>.<kind>`, from the condition's own key
// `mesh-delivery-assigner.<kind>.<id…>` (the subject's id when the key names none), its words unchanged. raised is false for
// a clear, whose key is to be deleted; value is nil then.
func FindingOf(c asks.Condition, at time.Time) (key string, value []byte, raised bool, err error) {
rest, ok := strings.CutPrefix(c.Key, delivery.AssignerModule+".")
if !ok || rest == "" {
return "", nil, false, fmt.Errorf("the condition %q is not the assigner's", c.Key)
}
kind, id, _ := strings.Cut(rest, ".")
if id == "" {
id = c.Subject.ID
}
scope := c.Subject.Scope
if scope == "" {
scope = health.ScopeModule
}
key = health.FindingKey(scope, id, kind)
if c.State == asks.StateCleared {
return key, nil, false, nil
}
if c.State != asks.StateRaised {
return "", nil, false, fmt.Errorf("the condition %s is neither raised nor cleared", c.Key)
}
summary := c.Summary
if summary == "" {
summary = c.Headline
}
var actions []health.Action
for _, a := range c.Actions {
actions = append(actions, health.Action{Label: a.Label, Verb: a.Verb, Machine: a.Machine, Arguments: a.Arguments, Level: string(a.Level)})
}
value, err = health.EncodeFinding(health.Finding{Key: key, Kind: kind, Subject: health.Subject{Scope: scope, ID: id, Machine: c.Subject.Machine},
Severity: health.Severity(c.Severity), Summary: summary, Headline: c.Headline, Explanation: c.Explanation, Resolved: c.Resolved,
Needs: c.Needs, Actions: actions, Evidence: []health.Evidence{{At: at.UTC(), Said: summary}}, Source: "assigner." + kind,
Observer: delivery.AssignerModule, At: at.UTC()})
return key, value, true, err
}
// Findings is where the assigner's findings go: the bucket's put and delete, and whether the keeper's conditions are the
// ones said (the take-over key names the keeper).
type Findings struct {
Put func(key string, value []byte) error
Delete func(key string) error
Keys func() ([]string, error)
ToKeeper func() bool
Now func() time.Time
}
// Say is say with each condition also written as a finding (or its finding deleted), and said straight to the router only
// while the take-over key follows the controller. A finding that cannot be written is the error, beside the say's.
func (f Findings) Say(say func(asks.Condition) error) func(asks.Condition) error {
return func(c asks.Condition) error {
var errs []error
if f.Put != nil {
now := time.Now
if f.Now != nil {
now = f.Now
}
key, value, raised, err := FindingOf(c, now())
switch {
case err != nil:
errs = append(errs, err)
case raised:
if err := f.Put(key, value); err != nil {
errs = append(errs, fmt.Errorf("the finding %s cannot be written: %w", key, err))
}
default:
if err := f.Delete(key); err != nil {
errs = append(errs, fmt.Errorf("the finding %s cannot be deleted: %w", key, err))
}
}
}
if f.ToKeeper == nil || !f.ToKeeper() {
if say != nil {
if err := say(c); err != nil {
errs = append(errs, err)
}
}
}
return errors.Join(errs...)
}
}
@@ -2,6 +2,7 @@ package assign
import (
"fmt"
"slices"
"sort"
"strings"
@@ -15,7 +16,8 @@ import (
// removed on one source's word, and it raises a condition naming the pair until both agree. After an approved proposal
// the assigner keeps the controller in step itself, calling its assign, unassign, pin and unpin; a call refused or not
// yet answered leaves the pair differing, and the condition names the call (Pair.Call). Before the first placement key
// is written the controller's assignments are the one source, as before.
// is written the controller's assignments are the one source, as before. From the lease write (the controller's lease
// handed away) the placement is the one source.
// Pair is a module on a node that one source names and the other does not; Only says which: OnlyAssignments or
// OnlyPlacement.
@@ -32,8 +34,17 @@ const (
)
// Placed is, by node, the modules both sources name, in name order, and every pair one names alone, by node then
// module. With no placement read yet (seeded false) it is the assignments, and no pair differs.
func Placed(assigned, placed map[string][]string, seeded bool) (map[string][]string, []Pair) {
// module. With no placement read yet (seeded false) it is the assignments, and no pair differs. Once the controller's
// lease is handed away (handedAway, ADR 0549 §5, §6) it is the placement alone, in name order, and no pair differs: the
// assignments are then a source of nothing.
func Placed(assigned, placed map[string][]string, seeded, handedAway bool) (map[string][]string, []Pair) {
if handedAway && seeded {
out := make(map[string][]string, len(placed))
for node, modules := range placed {
out[node] = slices.Sorted(slices.Values(modules))
}
return out, nil
}
if !seeded {
return assigned, nil
}
@@ -0,0 +1,208 @@
package assign
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/asks"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
// A person's upgrade policy is the assigner's (novox/hq ADR 0565 §7, §8, issue 1953 D4, D5): kept in its state `policies`, one
// key per module, set and taken back through `mesh-wants.upgrade`. Every reader takes one derivation, mesh-sdk's
// catalogue.Upgrade: a person's choice over the module's word, its irreplaceable data and the default, the bus never rolled.
// Until the lease write the controller still writes a policy beside each composition's digest; where it differs from the
// assigner's, the assigner says so as its finding `policy-split`, naming mesh-wants.upgrade as the remedy.
// PoliciesBucket is the assigner's state `policies`.
var PoliciesBucket = delivery.AssignerModule + "_" + delivery.AssignerPoliciesState
// DecodePolicies reads the policies state's values by key (the module); a value that cannot be read is left out, named.
func DecodePolicies(values map[string][]byte) (map[string]catalogue.PersonPolicy, []string) {
out := map[string]catalogue.PersonPolicy{}
var bad []string
for key, raw := range values {
var p catalogue.PersonPolicy
if err := json.Unmarshal(raw, &p); err != nil || p.V != catalogue.PersonPolicyVersion || p.Check() != nil || p.Module != key {
bad = append(bad, key)
continue
}
out[key] = p
}
sort.Strings(bad)
return out, bad
}
// PolicyOf is a module's policy and where it comes from, with the person's choice for it when there is one.
func PolicyOf(m catalogue.Manifest, chosen map[string]catalogue.PersonPolicy) (policy, from, why, by string) {
var c *catalogue.PersonPolicy
if p, ok := chosen[m.Module]; ok {
c = &p
}
return catalogue.Upgrade(m, c)
}
// ControllerPolicies is, by module, every policy the controller writes beside a composition's digest of it.
func ControllerPolicies(composed map[string][]byte) map[string][]string {
out := map[string][]string{}
for key, raw := range composed {
_, module, _, isNode, err := delivery.ParseComposedKey(key)
if err != nil || isNode {
continue
}
d, err := delivery.DecodeComposedDigest(raw)
if err != nil || d.Policy == "" {
continue
}
if !contains(out[module], d.Policy) {
out[module] = append(out[module], d.Policy)
sort.Strings(out[module])
}
}
return out
}
func contains(list []string, s string) bool {
for _, v := range list {
if v == s {
return true
}
}
return false
}
// Split is a module whose policy at the assigner differs from one the controller writes beside a composition of it.
type Split struct {
Module string `json:"module"`
Policy string `json:"policy"`
From string `json:"from"`
Controller []string `json:"controller"`
}
// Splits is every module of manifests the controller writes a policy for that differs from the assigner's, sorted.
func Splits(manifests map[string]catalogue.Manifest, chosen map[string]catalogue.PersonPolicy, controller map[string][]string) []Split {
var out []Split
for module, theirs := range controller {
m, ok := manifests[module]
if !ok {
continue
}
policy, from, _, _ := PolicyOf(m, chosen)
for _, p := range theirs {
if p != policy {
out = append(out, Split{Module: module, Policy: policy, From: from, Controller: theirs})
break
}
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Module < out[j].Module })
return out
}
// KindPolicySplit is the assigner's finding of a split policy (ADR 0565 §8).
const KindPolicySplit = "policy-split"
// PolicySplitKey is a module's policy-split finding's key.
func PolicySplitKey(module string) string {
return health.FindingKey(health.ScopeModule, module, KindPolicySplit)
}
// PolicySplitFinding is the finding of one split, in mesh-sdk's layout.
func PolicySplitFinding(s Split, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s's upgrade policy is %s at the assigner (%s) and %s at the controller", s.Module, s.Policy, s.From,
strings.Join(s.Controller, " and "))
return health.EncodeFinding(health.Finding{Key: PolicySplitKey(s.Module), Kind: KindPolicySplit,
Subject: health.Subject{Scope: health.ScopeModule, ID: s.Module}, Severity: health.Warning, Summary: summary,
Headline: clipHeadline(s.Module + "'s upgrade policy is split"),
Needs: "set it with mesh-wants.upgrade, so the assigner and the controller agree",
Explanation: asks.NeedsYou + "set it with mesh-wants.upgrade, so the assigner and the controller agree." + " The controller still " +
"writes a policy beside each composition until it hands its lease away; a choice made at its upgrade verb after the copy " +
"is not the assigner's.",
Resolved: s.Module + "'s upgrade policy agrees again", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindPolicySplit, Observer: delivery.AssignerModule, At: at.UTC()})
}
func clipHeadline(s string) string {
if len([]rune(s)) <= 60 {
return s
}
return string([]rune(s)[:60])
}
// KindProvisionFailed is the assigner's finding of a provision its module's want waits for that failed (ADR 0570 §4).
const KindProvisionFailed = "provision-failed"
// ProvisionFailedKey is a failed provision's finding's key, `provider.<node>.<consumer>.<provision>[.<local>].provision-failed`.
func ProvisionFailedKey(key string) string {
return health.FindingKey(health.ScopeProvider, key, KindProvisionFailed)
}
// ProvisionFailedFinding is the finding of a failed provision, naming its consumer, its provider and why.
func ProvisionFailedFinding(key string, p provisions.Provision, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s for %s on %s failed at %s on %s: %s, so %s is not moved there until it is provided", p.Provision,
p.Consumer.Module, p.Consumer.Node, p.Provider.Module, p.Provider.Node, p.Why, p.Consumer.Module)
return health.EncodeFinding(health.Finding{Key: ProvisionFailedKey(key), Kind: KindProvisionFailed,
Subject: health.Subject{Scope: health.ScopeProvider, ID: key, Machine: p.Consumer.Node, Also: also(p.Provider.Node, p.Consumer.Node)},
Severity: health.Warning, Summary: summary, Headline: clipHeadline(p.Consumer.Module + " waits for a provision that failed"),
Explanation: asks.NothingToDo + " " + summary + ". It clears once the provider provides it.",
Resolved: p.Provision + " for " + p.Consumer.Module + " is provided", Evidence: []health.Evidence{{At: at.UTC(), Said: p.Why}},
Source: "assigner." + KindProvisionFailed, Observer: delivery.AssignerModule, At: at.UTC()})
}
func also(provider, consumer string) []string {
if provider == "" || provider == consumer {
return nil
}
return []string{provider}
}
// KindProvisionMissing is the assigner's finding of a required provision that has had no key for longer than its bound.
const KindProvisionMissing = "provision-missing"
// ProvisionMissingKey is its key, `provider.<node>.<module>.<provision>.provision-missing`.
func ProvisionMissingKey(key string) string {
return health.FindingKey(health.ScopeProvider, key, KindProvisionMissing)
}
// ProvisionMissingFinding is the finding of a required provision with no key past bound (key `<node>.<module>.<provision>`).
func ProvisionMissingFinding(key string, bound time.Duration, at time.Time) ([]byte, error) {
parts := strings.SplitN(key, ".", 3)
if len(parts) != 3 {
return nil, fmt.Errorf("%q is not <node>.<module>.<provision>", key)
}
node, module, provision := parts[0], parts[1], parts[2]
summary := fmt.Sprintf("%s on %s requires %s and mesh-provisioner has composed no key for it in %s, so %s is not moved there: "+
"no provider of it may be placed", module, node, provision, bound, module)
return health.EncodeFinding(health.Finding{Key: ProvisionMissingKey(key), Kind: KindProvisionMissing,
Subject: health.Subject{Scope: health.ScopeProvider, ID: key, Machine: node},
Severity: health.Warning, Summary: summary, Headline: clipHeadline(module + " on " + node + " lacks " + provision),
Explanation: asks.NothingToDo + " " + summary + ". It clears once the provision has its key.",
Resolved: provision + " for " + module + " on " + node + " has its key", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindProvisionMissing, Observer: delivery.AssignerModule, At: at.UTC()})
}
// KindUnplaced is, from the lease write, a node that runs but no placement names (ADR 0571 §6).
const KindUnplaced = "unplaced"
// UnplacedKey is its finding's key, `machine.<node>.unplaced`.
func UnplacedKey(node string) string {
return health.FindingKey(health.ScopeMachine, node, KindUnplaced)
}
// UnplacedFinding is the finding of a node no placement names since the lease write.
func UnplacedFinding(node string, at time.Time) ([]byte, error) {
summary := fmt.Sprintf("%s runs, but no placement names it: since the controller handed its lease away nothing composes it, "+
"so the assigner writes it no want", node)
return health.EncodeFinding(health.Finding{Key: UnplacedKey(node), Kind: KindUnplaced,
Subject: health.Subject{Scope: health.ScopeMachine, ID: node, Machine: node}, Severity: health.Warning, Summary: summary,
Headline: clipHeadline(node + " has no placement"), Needs: "place it with mesh-placement.propose.",
Explanation: asks.NeedsYou + "place it with mesh-placement.propose. " + summary + ".",
Resolved: node + " is placed", Evidence: []health.Evidence{{At: at.UTC(), Said: summary}},
Source: "assigner." + KindUnplaced, Observer: delivery.AssignerModule, At: at.UTC()})
}
@@ -15,10 +15,6 @@ import (
// `placement`, changed only by a proposal (§2) the nodes it touches answer (§3), judged (§4) and written on the
// operator's Approve at the revision it was laid over (§5). What here decides; the tools and the passes do.
// Leased says whether the lease write of ADR 0547 has happened, after which the controller's assignments are read no
// more (§8). Not before L3: until then the holder keeps the controller in step and refuses what its verbs cannot take.
var Leased = false
// ProposeIn is what a proposal is laid over: its id, who and why, its changes; every node's current placement and the
// revision of its key; which modules the indexer holds and which nodes have a record; and, for the seed, the export's
// digest.
@@ -32,6 +28,10 @@ type ProposeIn struct {
Recorded func(node string) bool
Seed bool
ExportDigest string
// Leased says the lease write of ADR 0547 has happened (the controller's lease handed away, mesh-sdk
// buscompose.LeaseHandedAway; issue 774), after which the controller's assignments are read no more (§8): until then
// the holder keeps the controller in step and refuses what its verbs cannot take.
Leased bool
}
// Propose lays a proposal over the current keys (§2): the proposal (state answering, each changed node's base revision),
@@ -58,7 +58,7 @@ func Propose(in ProposeIn) (delivery.PlacementProposal, map[string]*delivery.Pla
findings = append(findings, fmt.Sprintf("change %d: the seed is adopt's alone, and never mixed with other changes", i+1))
continue
}
if !Leased {
if !in.Leased {
if why := c.BeforeLease(); why != "" {
findings = append(findings, fmt.Sprintf("change %d: %s", i+1, why))
}
@@ -60,18 +60,18 @@ func TestThePlacementSeatDecides(t *testing.T) {
if _, _, f := Propose(in(delivery.PlacementProposalChange{Op: delivery.OpConfig, Node: "anchor", Module: "web", Values: map[string]any{"puid": 1}})); len(f) != 0 {
t.Errorf("config values before the lease write: %v", f)
}
Leased = true
for _, c := range []delivery.PlacementProposalChange{
{Op: delivery.OpPlace, Node: "laptop", Module: "web", Ports: map[string]int{"80": 8080}, Bindings: map[string]delivery.Provider{"db": {Node: "anchor", Module: "store"}}},
{Op: delivery.OpConfig, Node: "anchor", Module: "web", Layer: map[string]any{"places": []any{}}},
{Op: delivery.OpDecide, Node: "laptop", Field: "site", Value: json.RawMessage(`"attic"`)},
{Op: delivery.OpHold, Node: "anchor", Module: "store", Claim: "node-shell", Scope: "node"},
} {
if _, _, f := Propose(in(c)); len(f) != 0 {
leased := in(c)
leased.Leased = true
if _, _, f := Propose(leased); len(f) != 0 {
t.Errorf("after the lease write %s is refused: %v", c.Op, f)
}
}
Leased = false
// A key moved while the warrant wrote: what is left to write is the other nodes' changes, to be laid and asked again.
if rest, ok := Rest(p, map[string]*delivery.Placement{"anchor": laid["anchor"]}); !ok || len(rest.Changes) != 1 || rest.Changes[0].Node != "laptop" ||
@@ -10,6 +10,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
// Module is what the six rules read of one module (ADR 0429 §6, ADR 0528 clause 3): its newest fingerprint F at the
@@ -29,7 +30,12 @@ type Module struct {
// 1945, live on 2026-10-25 at 20:04: ace's mesh-build-go was wanted with no account on the bus). Nil: the grant alone.
GrantedOn func(node string) bool
Manifest catalogue.Manifest
Nodes []string
// Provisions is the seat mesh-provisioner's provisions this module consumes on a node, by key (ADR 0570 §4): the want
// waits until each is provided. Nil: the seat's state is not read (not delivered yet), and nothing waits on it.
Provisions func(node string) map[string]provisions.Provision
// Chosen is a person's choice of the module's upgrade policy (the assigner's `policies`, ADR 0565 §7); nil: none.
Chosen *catalogue.PersonPolicy
Nodes []string
// Composed answers the controller's composition of the module for a node at a fingerprint (mesh-controller_composed): its
// digest and the judgement of the move to it, and whether it exists. Nil: none exists (ADR 0539 §1, §6).
Composed func(node, fp string) (delivery.ComposedDigest, bool)
@@ -124,9 +130,54 @@ func holds(m Module, node string, n Nodes, ordering bool) (bool, int, string) {
if _, ok := m.composition(node, m.FP); !ok {
return false, delivery.RuleComposed, fmt.Sprintf("the controller has not composed %s for %s at %s", m.Name, node, short(m.FP))
}
if words, waiting := m.provisionWaiting(node); waiting {
return false, delivery.RuleProvisions, words
}
return true, 0, ""
}
// provisionWaiting says why the module's want on node waits for a provision (ADR 0570 §4): a provision it requires that has
// no key yet, or the first of its keys that is not provided. Nil Provisions (the seat's state not on the bus) waits for none.
func (m Module) provisionWaiting(node string) (string, bool) {
if m.Provisions == nil {
return "", false
}
if missing := m.MissingProvisions(node); len(missing) > 0 {
return fmt.Sprintf("provision %s has no key yet for %s on %s", missing[0], m.Name, node), true
}
keyed := m.Provisions(node)
for _, key := range slices.Sorted(maps.Keys(keyed)) {
if p := keyed[key]; p.State != provisions.Provided {
words := fmt.Sprintf("the provision %s of %s on %s is %s, not provided", key, m.Name, node, p.State)
if p.Why != "" {
words += ": " + p.Why
}
return words, true
}
}
return "", false
}
// MissingProvisions are the provisions the module requires (not optionally) that have no key on node yet, sorted; none
// while the seat's state is not read.
func (m Module) MissingProvisions(node string) []string {
if m.Provisions == nil {
return nil
}
have := map[string]bool{}
for _, p := range m.Provisions(node) {
have[p.Provision] = true
}
var out []string
for _, r := range m.Manifest.Requires {
if !have[r] && !m.Manifest.OptionalRequires[r] && !slices.Contains(out, r) {
out = append(out, r)
}
}
slices.Sort(out)
return out
}
// providedHealthy is rule 5 for one provision: some module providing it runs healthy on the node, or, assigned
// elsewhere only, at its canary. A provision no module provides is not a module's to judge (the mesh or the node
// offers it) and holds.
@@ -639,9 +690,14 @@ func (m Module) busBar(before delivery.Want, had bool, running string) (prev str
return prev, catalogue.ProvidesBus(m.Manifest) && prev != m.FP
}
// recorded says whether the module is of policy record: as the controller holds it beside the composition's digest, the
// operator's own choice included (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
// recorded says whether the module is of policy record: a person's choice at the assigner over everything but the bus (ADR
// 0565 §7, catalogue.Upgrade); for a module with none, as the controller holds it beside the composition's digest until the
// lease write (ADR 0539 §14); on a composition made by hand, which carries none, as the manifest derives it.
func (m Module) recorded(d delivery.ComposedDigest) bool {
if m.Chosen != nil {
policy, _, _, _ := catalogue.Upgrade(m.Manifest, m.Chosen)
return policy == catalogue.PolicyRecord
}
if d.Policy != "" {
return d.Policy == delivery.PolicyRecord
}
@@ -8,6 +8,7 @@ import (
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provisions"
)
func fpOf(c byte) string { return delivery.SourcePrefix2 + strings.Repeat(string(c), 64) }
@@ -69,6 +70,12 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
delivery.RuleComposed: func(m *Module, _ *Nodes) {
m.Composed = func(string, string) (delivery.ComposedDigest, bool) { return delivery.ComposedDigest{}, false }
},
// 1954/D1 (ADR 0570 §4): a provision of the module on that node still asked holds its want.
delivery.RuleProvisions: func(m *Module, _ *Nodes) {
m.Provisions = func(node string) map[string]provisions.Provision {
return map[string]provisions.Provision{node + ".mail.database": {Provision: "database", State: provisions.Asked}, node + ".mail.oidc": {Provision: "oidc", State: provisions.Provided}}
}
},
} {
m, n := ready()
breakIt(&m, &n)
@@ -82,6 +89,29 @@ func TestEachRuleHoldsTheWantBack(t *testing.T) {
if w := out.Wants["g14"]; w.FP != "" && (w.FP != fOld || w.Newest != fNew || w.Rule != rule) {
t.Errorf("rule %d broken: g14's want is %+v", rule, w)
}
if rule == delivery.RuleProvisions && (!strings.Contains(out.Rules["g14"].Words, "g14.mail.database") || !strings.Contains(out.Rules["g14"].Words, "asked")) {
t.Errorf("the provision's hold does not name its key and state: %s", out.Rules["g14"].Words)
}
}
// 1954/D1: every provision provided, the want moves; with the seat's state not read, nothing waits on it.
for _, all := range []func(string) map[string]provisions.Provision{
func(node string) map[string]provisions.Provision {
return map[string]provisions.Provision{node + ".mail.database": {Provision: "database", State: provisions.Provided}}
}, nil} {
m, n := ready()
m.Provisions = all
if out := Wants(m, n, wantedOld(), nil); out.Wants["g14"].FP != fNew {
t.Errorf("provided provisions held g14's want: %+v (%s)", out.Wants["g14"], out.Rules["g14"].Words)
}
}
// 1954/D1 (the lead, 2026-10-29): a required provision with no key yet holds the want, named.
{
m, n := ready()
m.Provisions = func(string) map[string]provisions.Provision { return map[string]provisions.Provision{} }
if out := Wants(m, n, wantedOld(), nil); out.Rules["g14"].Rule != delivery.RuleProvisions ||
!strings.Contains(out.Rules["g14"].Words, "provision database has no key yet") || len(m.MissingProvisions("g14")) != 1 {
t.Errorf("a provision with no key: %+v", out.Rules["g14"])
}
}
// 1960: rule 5 holds no want, and no module is left without a first want for a provider down or a node offline; the
// provider down is said (Unreached). novox on 2026-10-29: route-proxy down held de-spiegel and link2pay, minio not
@@ -802,6 +832,12 @@ func TestTheRecordPolicyIsTheOneTheControllerHolds(t *testing.T) {
if out := Wants(m, n, wanted, nil); len(out.Record) == 0 {
t.Errorf("a hand-made composition fell through the manifest's record policy: %+v", out)
}
// 1953/D4 (ADR 0565 §7): a person's choice at the assigner is over the controller's beside the composition.
with(delivery.PolicyRecord, "person")
m.Chosen = &catalogue.PersonPolicy{V: catalogue.PersonPolicyVersion, Module: m.Name, Policy: catalogue.PolicyRoll, Why: "w", By: "op"}
if out := Wants(m, n, wanted, nil); len(out.Record) != 0 || out.Wants["ace"].FP != fNew {
t.Errorf("a person's roll at the assigner was not honoured over the controller's record: %+v", out)
}
}
// Review round 1, item 2 (ADR 0539 §15): a node that runs the bus with no readable want compares the new fingerprint with the one
@@ -33,9 +33,9 @@ func TestManifest(t *testing.T) {
}
reads := slices.Clone(m.Reads)
slices.Sort(reads)
want := []string{"mesh-build-go.steps", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-controller.grants",
"mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "node-engine.have"}
want := []string{"mesh-build-go.steps", "mesh-conditions.takeover", "mesh-config.kept", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.composed", "mesh-controller.grants", "mesh-controller.lease",
"mesh-delivery-builder.build-log", "mesh-delivery-builder.builds", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-grants.grants", "mesh-provisioner.provisions", "node-engine.have"}
if !reflect.DeepEqual(reads, want) {
t.Errorf("reads = %v, want exactly %v", reads, want)
}
@@ -44,7 +44,7 @@ func TestManifest(t *testing.T) {
states[s.Name] = s.History
}
if want := map[string]int{"asks": 5, "compositions": 5, "differences": 64, "puts": 20, "wants": 20,
"placement": 64, "proposals": 5, "candidates": 5, "pulses": 0}; !reflect.DeepEqual(states, want) {
"placement": 64, "proposals": 5, "candidates": 5, "pulses": 0, "findings": 5, "policies": 20, "seat-holders": 5}; !reflect.DeepEqual(states, want) {
t.Errorf("state = %v, want %v", states, want)
}
if _, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 {
@@ -63,9 +63,14 @@ func TestManifest(t *testing.T) {
// ADR 0567 §1, §6-§10: it holds the seat mesh-placement, serving its seven verbs; keeps the seat's events on its stream
// placement for 30 days; calls the controller's export, its four verbs that keep its assignments in step and its
// hand-act log; and reads mesh-config's resolved values to compose every node a proposal touches.
if len(m.Claims) != 1 || m.Claims[0].Name != catalogue.PlacementSeat || m.Claims[0].At() != catalogue.ScopeMesh ||
!slices.Equal(m.Claims[0].Serves, []string{"placement", "proposals", "propose", "withdraw", "again", "adopt", "hand-back"}) {
t.Errorf("the assigner claims %+v, want mesh-placement with its seven verbs", m.Claims)
// ADR 0565 §7, §9 (issue 1953 D4): and the seat mesh-wants, serving upgrade and bus.
if len(m.Claims) != 2 || m.Claims[0].Name != catalogue.PlacementSeat || m.Claims[0].At() != catalogue.ScopeMesh ||
!slices.Equal(m.Claims[0].Serves, []string{"placement", "proposals", "propose", "withdraw", "again", "adopt", "hand-back"}) ||
m.Claims[1].Name != catalogue.WantsSeat || m.Claims[1].At() != catalogue.ScopeMesh || !slices.Equal(m.Claims[1].Serves, []string{"upgrade", "bus"}) {
t.Errorf("the assigner claims %+v, want mesh-placement with its seven verbs and mesh-wants with its two", m.Claims)
}
if seat, _ := catalogue.SeatNamed(catalogue.WantsSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold mesh-wants: %v", catalogue.CanHold(m, seat))
}
if seat, _ := catalogue.SeatNamed(catalogue.PlacementSeat); catalogue.CanHold(m, seat) != nil {
t.Errorf("the assigner cannot hold the seat: %v", catalogue.CanHold(m, seat))
@@ -79,7 +84,8 @@ func TestManifest(t *testing.T) {
}
if want := []string{"seat:mesh-controller.assign", "seat:mesh-controller.hand-act", "seat:mesh-controller.pin",
"seat:mesh-controller.placement-export", "seat:mesh-controller.unassign", "seat:mesh-controller.unpin",
"seat:mesh-controller.warranted", "seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) {
"seat:mesh-controller.warranted", "seat:mesh-hand-acts.hand-act", "seat:mesh-hand-acts.hand-acts", "seat:mesh-hand-acts.warranted", // ADR 0565 §9, §11: the seat, the controller's only while it has no holder
"seat:node-backup.backed-up", "seat:node-backup.now"}; !slices.Equal(slices.Sorted(slices.Values(m.Invokes)), want) {
t.Errorf("invokes = %v, want %v", m.Invokes, want)
}
if !slices.Equal(m.Emits, []string{"put"}) {
+21 -2
View File
@@ -15,21 +15,34 @@
"adopt",
"hand-back"
]
},
{
"name": "mesh-wants",
"scope": "mesh",
"serves": [
"upgrade",
"bus"
]
}
],
"reads": [
"mesh-build-go.steps",
"mesh-conditions.takeover",
"mesh-config.kept",
"mesh-config.resolved",
"mesh-controller.assignments",
"mesh-controller.builds",
"mesh-controller.composed",
"mesh-controller.grants",
"mesh-controller.lease",
"mesh-delivery-builder.build-log",
"mesh-delivery-builder.builds",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
"mesh-delivery-indexer.modules",
"mesh-delivery-teller.answers",
"mesh-delivery-watchman.presence",
"mesh-grants.grants",
"mesh-provisioner.provisions",
"node-engine.have"
],
"uses": [
@@ -44,6 +57,9 @@
"seat:mesh-controller.unassign",
"seat:mesh-controller.unpin",
"seat:mesh-controller.warranted",
"seat:mesh-hand-acts.hand-act",
"seat:mesh-hand-acts.hand-acts",
"seat:mesh-hand-acts.warranted",
"seat:node-backup.backed-up",
"seat:node-backup.now"
],
@@ -64,7 +80,10 @@
{ "name": "compositions", "history": 5 },
{ "name": "differences", "history": 64 },
{ "name": "puts", "history": 20 },
{ "name": "wants", "history": 20 }
{ "name": "wants", "history": 20 },
{ "name": "findings", "history": 5 },
{ "name": "policies", "history": 20 },
{ "name": "seat-holders", "history": 5 }
],
"streams": [
{
@@ -26,7 +26,7 @@ type queueAsker func(key string, value []byte) (json.RawMessage, error)
// Tools are the builder's tools: stuck reads, askAgain writes the reasks value, queue writes a queues value and waits for
// the builder's read, buildAgain writes a reasks value of mode build and waits for the builder's answer.
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue, buildAgain queueAsker, now func() time.Time) []stdio.Tool {
func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, queue, buildAgain queueAsker, since build.LogSince, now func() time.Time) []stdio.Tool {
str := func(args map[string]any, k string) string { s, _ := args[k].(string); return strings.TrimSpace(s) }
return []stdio.Tool{
{Name: build.StuckTool,
@@ -94,6 +94,20 @@ func Tools(r build.StuckReader, askAgain func(key string, value []byte) error, q
return map[string]any{"key": key, "revision": r.Revision, "asks": r.Asks, "answer": r.Answer,
"next": build.StuckTool + " shows the key and its asks while they wait"}, nil
}},
{Name: delivery.BuildsSinceTool,
Description: "Every build that ended since a time, oldest first, from the builder's log build-log (kept 30 days): per build its module, " +
"fingerprint, source revision, repository, directory, commit and ref, the build node, when it ended, its outcome (done, failed or " +
"queued), the manifest built, the bases it was built against and each artifact made with its digest and reference. more says the " +
"log held more than one answer carries: ask again from the last build's time. What mesh-catalog registers a build from. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"since"}, "properties": map[string]any{
"since": map[string]any{"type": "string", "description": "RFC 3339 time; the builds that ended at or after it"}}},
Run: func(args map[string]any) (any, error) {
at, err := time.Parse(time.RFC3339Nano, str(args, "since"))
if err != nil {
return nil, fmt.Errorf("since %q is not an RFC 3339 time: %w", str(args, "since"), err)
}
return build.BuildsSince(since, at)
}},
{Name: build.AnnounceAgainTool,
Description: "Announce a done build again (take in again, compose again): the builder writes the done key of module and fingerprint " +
"again, unchanged, as a new revision of the same key (its content, artifacts and manifest stay identical), so the controller's " +
@@ -250,7 +264,7 @@ func main() {
a, err := build.DecodeReAsk(v)
return err == nil && !a.Answered.IsZero()
})
if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), buildAgain, time.Now)); err != nil {
if err := stdio.Serve("", Tools(r, put, askQueue(stdio.State(build.QueuesState), 25*time.Second), buildAgain, stdio.Log(delivery.BuilderLog).Since, time.Now)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-builder-tools] %v\n", err)
os.Exit(1)
}
@@ -23,6 +23,7 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
@@ -268,6 +269,12 @@ func startBedWith(t *testing.T, closed bool) *bed {
mk(delivery.ManifestsBucket, delivery.ManifestsHistory)
b.builds = mk(delivery.BuilderBuildsBucket, delivery.BuilderBuildsHistory)
b.instances = mk(instancesBucket, 1)
// The builder's module log, as the runtime raises it (ADR 0297): its stream on the log's subjects.
logStream := buscompose.LogStreamName(delivery.BuilderModule, delivery.BuilderLog)
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: logStream, Subjects: []string{buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog) + ".>"},
AllowDirect: true}); err != nil {
t.Fatal(err)
}
b.fake = &fakeBuildModules{js: js, ctx: ctx, calls: map[string]int{}}
b.fake.closed.Store(closed)
b.fake.serve(t, "mesh-build-go", "n1", "n2")
@@ -280,6 +287,29 @@ func startBedWith(t *testing.T, closed bool) *bed {
return b
}
// logged is every entry of module's key in the builder's log, read through process.Conn.ReadLog on the controller's user.
func (b *bed) logged(t *testing.T, module string) []delivery.BuildLogEntry {
t.Helper()
conn, err := process.Connect(process.Account{URL: b.srvURL, Module: "mesh-delivery-view", User: "ctl", Password: "pw"})
if err != nil {
t.Fatal(err)
}
defer conn.Close()
entries, err := conn.ReadLog(delivery.BuilderModule, delivery.BuilderLog).Since(time.Unix(0, 0), module)
if err != nil {
t.Fatal(err)
}
var out []delivery.BuildLogEntry
for _, le := range entries {
e, err := delivery.DecodeBuildLogEntry(le.Value)
if err != nil {
t.Fatal(err)
}
out = append(out, e)
}
return out
}
// report writes a node's toolchains key of a build module and says so on the module's stream, as the module does.
func (b *bed) report(t *testing.T, module, node, language, version, recipe string) {
t.Helper()
@@ -406,6 +436,13 @@ func TestBuilderMakesADoneKeyWithTheDigests(t *testing.T) { // 1607/AC7
if v.Toolchains["go"].Version != "go1.27.1" || v.Toolchains["typescript"].Version != "tsc5" {
t.Errorf("toolchains = %+v", v.Toolchains)
}
// 1955/L7 (ADR 0571 §1): the build's start and end are in the module log build-log under the module's key, read as a
// reader with the log's read reads them, the end naming the manifest and what was made.
ended := b.logged(t, "netcheck")
if len(ended) != 2 || ended[0].Event != delivery.BuildStartedEvent || ended[1].Event != delivery.BuildEndedEvent ||
ended[1].Manifest != digest || len(ended[1].Made) != 2 || ended[1].Node != "n1" {
t.Errorf("logged = %+v, want the start and the end of netcheck's build", ended)
}
for tool, want := range map[string]int{"mesh-build-go.build_go_fetch": 1, "mesh-build-go.build_go_build": 1, "mesh-build-node.build_node_fetch": 1, "mesh-build-node.build_node_build": 1} {
if got := b.fake.count(tool); got != want {
t.Errorf("%s called %d times, want %d (on one node, ADR 0478 clause 4)", tool, got, want)
@@ -120,6 +120,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
Manifests: manifestReader{manifests},
Logf: logger.Printf,
Stored: storedVia(calls.Call),
Node: cred.Node,
}
// A candidate's preview is merged onto the trunk head the indexer previewed it onto: source.main at the candidate's base
// revision (the gate's source state). The source's revisions are watched with their history (a direct get of one revision
@@ -129,6 +130,13 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
trunks := &trunkSource{ctx: ctx, js: js, heads: &sourceHeads{}, moved: b.TrunkMoved, errs: errs, logf: logger.Printf}
trunks.start()
b.Trunk = trunks.head
// The builder's log (ADR 0571 §1): each build's start and end in its module log `build-log`, read by the view, the
// indexer and the assigner through their reads, and by mesh-catalog through the tool builder_builds_since.
b.Log = func(e delivery.BuildLogEntry) {
if err := appendLog(context.WithoutCancel(ctx), js, cred, e); err != nil {
logger.Printf("%v", err)
}
}
b.Said = func(e build.AskedAgain) {
if err := emitAskedAgain(context.WithoutCancel(ctx), js, cred, e); err != nil {
logger.Printf("%v", err)
@@ -153,11 +161,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
}
// The toolchains first, so a build that starts from the first modules value finds the nodes.
watch := newToolchainWatch(ctx, js, tools, b.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
// Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// placement is settled first, so the toolchains settle through the gate already knowing it.
gate := buildrun.NewToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, b.NodesChanged, logger)
placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, b.NodesChanged, logger.Printf)
if err != nil {
return err
}
@@ -254,7 +262,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil
case err := <-errs:
return err
case err := <-assignErrs:
case err := <-placementErrs:
return err
case err := <-watch.errs:
return err
@@ -8,6 +8,8 @@ import (
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
@@ -138,3 +140,26 @@ func announcedAgain(ctx context.Context, b askAgainer, kv jetstream.KeyValue, e
logf("%s.%s: the answer could not be written: %v", build.ReAsksState, e.Key(), err)
}
}
// appendLog appends one entry to the builder's module log `build-log` (ADR 0297, ADR 0571 §1) under the module's key
// (delivery.BuildLogKey), on the log's own subject: a module's program appends to the logs it keeps on its own credential.
// The id makes a repeated append of one entry one entry.
func appendLog(ctx context.Context, js jetstream.JetStream, cred process.Account, e delivery.BuildLogEntry) error {
data, err := delivery.EncodeBuildLogEntry(e)
if err != nil {
return err
}
key, err := delivery.BuildLogKey(e.Module)
if err != nil {
return err
}
msg := nats.NewMsg(buscompose.LogSubject(cred.Module, delivery.BuilderLog) + "." + key)
msg.Data = data
id := e.Event + "." + e.Module + "." + e.FP + "." + e.At.Format(time.RFC3339Nano)
pub, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
if _, err := js.PublishMsg(pub, msg, jetstream.WithMsgID(id)); err != nil {
return fmt.Errorf("logging %s of %s at %s: %w", e.Event, e.Module, e.FP, err)
}
return nil
}
@@ -0,0 +1,73 @@
package main
import (
"bytes"
"context"
"errors"
"log"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// 1945 (a part exits at start for a bucket it works without): a bus that refuses every toolchains bucket leaves the
// watch unstarted and the part running, said; a refused bucket it cannot work without is an error naming the bucket and
// its grant. The gate over the seat's placement, and its refusal, are mesh-sdk's (buildrun.ToolchainGate,
// issue 774).
func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
set := &buildrun.ToolchainSet{}
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-build-node_toolchains")}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "builder", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
// The gate's source (the previews' trunk heads): refused at start, the builder goes on, and a candidate asking a trunk
// head is told why and opens it again then.
src := &trunkSource{ctx: context.Background(), js: refusingBus{refuseOpen: true, err: refused.err}, heads: &sourceHeads{}, errs: make(chan error, 1), logf: logger.Printf}
src.start()
if _, err := src.head(context.Background(), 7, "novox/mesh-catalog"); err == nil || !strings.Contains(err.Error(), delivery.SourceBucket) {
t.Errorf("a trunk head with the source refused: %v, want why", err)
}
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -1,211 +0,0 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -1,228 +0,0 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
// 1945 (a part exits at start for a bucket it works without): a bus that refuses the assignments' watch, or every
// toolchains bucket, leaves the watches unstarted and the part running, said; a refused bucket it cannot work without
// is an error naming the bucket and its grant.
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments")}
if _, err := watchAssignments(context.Background(), refused, newToolchainGate(nil, set.Put, t.Logf), func(context.Context) {}, logger); err != nil {
t.Errorf("a refused watch of the assignments stopped the start: %v", err)
}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "builder", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
// The gate's source (the previews' trunk heads): refused at start, the builder goes on, and a candidate asking a trunk
// head is told why and opens it again then.
src := &trunkSource{ctx: context.Background(), js: refusingBus{refuseOpen: true, err: refused.err}, heads: &sourceHeads{}, errs: make(chan error, 1), logf: logger.Printf}
src.start()
if _, err := src.head(context.Background(), 7, "novox/mesh-catalog"); err == nil || !strings.Contains(err.Error(), delivery.SourceBucket) {
t.Errorf("a trunk head with the source refused: %v, want why", err)
}
if src.opened {
t.Error("a refused source counts as opened, so it is never opened again")
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
@@ -9,7 +9,9 @@ import (
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/handacts"
"mesh-delivery-builder/internal/build"
)
@@ -18,17 +20,27 @@ import (
// consumer of artifacts-gone (mesh-controller internal/broker EventsStream).
const eventsStream = "EVENTS"
// SeatHeld is the seat mesh-artifact-sweep's verb held (ADR 0571 §2).
const SeatHeld = "seat:" + catalogue.ArtifactSweepSeat + ".held"
// toolCaller calls a tool or a seat verb as the builder's own account (mesh-sdk process.Conn.Call).
type toolCaller func(key string, body any) (json.RawMessage, error)
// storedVia is build.Builder's Stored: it asks the controller's verb held (mesh-sdk delivery.HeldVerb; novox/hq issue
// 1937) and answers the references the store no longer holds. One it could not ask about, with none gone, is an error:
// the key is then left done, and the controller's take-in still refuses a gone blob.
//
// It asks the seat mesh-artifact-sweep's `held` (novox/hq ADR 0571 §2), and the controller's while that seat has no holder that
// answers (a named exception of clause 10, until the lease write).
func storedVia(call toolCaller) func(ctx context.Context, references []string) ([]string, error) {
return func(_ context.Context, references []string) ([]string, error) {
raw, err := call(delivery.HeldVerb, map[string]any{"references": references})
body := map[string]any{"references": references}
raw, err := call(SeatHeld, body)
if handacts.NoResponder(err) {
raw, err = call(delivery.HeldVerb, body)
}
if err != nil {
return nil, fmt.Errorf("asking %s: %w", delivery.HeldVerb, err)
return nil, fmt.Errorf("asking held: %w", err)
}
a, err := delivery.DecodeHeldAnswer(raw, references)
if err != nil {
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-builder
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -7,6 +7,7 @@ import (
"fmt"
"maps"
"sort"
"strconv"
"strings"
"sync"
"time"
@@ -68,6 +69,11 @@ type Builder struct {
Askings int
// Said is told each ask asked again (the builder's event build.asked-again). Nil tells nobody.
Said func(AskedAgain)
// Log is told each build's start and end, the entries of the builder's log (novox/hq ADR 0571 §1). Nil tells nobody.
Log func(delivery.BuildLogEntry)
// Now is the clock of the log; nil is time.Now. Node is the node the builder runs on, which asked the build.
Now func() time.Time
Node string
once sync.Once
sem chan struct{}
@@ -429,6 +435,7 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
return
}
b.handClaimed(key, cl.Rev)
b.logBuild(LogStarted(module, m, b.refOf(module, m.FP2), b.Node, b.now()))
if cl.Unasked != nil {
b.logf("%v", cl.Unasked)
}
@@ -478,6 +485,7 @@ func (b *Builder) try(ctx context.Context, module string, m delivery.Module) {
b.logf("%s: the build could not be written: %v", cl.Key, err)
return
}
b.logBuild(LogEnded(module, m, b.refOf(module, m.FP2), b.Node, b.now(), result))
if result.State == delivery.BuildFailed {
b.logf("%s: failed: %s", cl.Key, result.Error)
}
@@ -872,3 +880,60 @@ func needsFetch(tasks []Task) bool {
}
return false
}
func (b *Builder) now() time.Time {
if b.Now != nil {
return b.Now()
}
return time.Now()
}
// logBuild tells the log an entry, said in the journal when it cannot be one.
func (b *Builder) logBuild(e delivery.BuildLogEntry) {
if b.Log == nil {
return
}
if _, err := delivery.EncodeBuildLogEntry(e); err != nil {
b.logf("%s at %s: no log entry: %v", e.Module, e.FP, err)
return
}
b.Log(e)
}
// refOf is the ref a candidate's build of module's fingerprint is built for, "" for the trunk's.
func (b *Builder) refOf(module, fp string) string {
key, err := delivery.BuilderBuildKey(module, fp)
if err != nil {
return ""
}
b.mu.Lock()
defer b.mu.Unlock()
return b.via[key].Ref
}
// LogStarted is the log's entry of a build of module's fingerprint starting (ADR 0571 §1): the module, the fingerprint, the
// source revision, where it is built from (repository, directory, commit, the candidate's ref), the node of the builder
// that asked it and the time.
func LogStarted(module string, m delivery.Module, ref, by string, at time.Time) delivery.BuildLogEntry {
dir := m.Path
if dir == "" {
dir = delivery.RootPath
}
return delivery.BuildLogEntry{Event: delivery.BuildStartedEvent, Module: module, FP: m.FP2, Source: strconv.FormatUint(m.Src, 10),
Repository: m.Repository, Directory: dir, Commit: m.Commit, Ref: ref, Node: by, At: at.UTC()}
}
// LogEnded is the log's entry of that build ending: the outcome (done, failed or queued), the manifest built, the bases it
// was built against and each artifact made, as the done key has them: what mesh-catalog registers a build from.
func LogEnded(module string, m delivery.Module, ref, by string, at time.Time, result delivery.BuilderBuild) delivery.BuildLogEntry {
e := LogStarted(module, m, ref, by, at)
e.Event, e.Outcome, e.Manifest = delivery.BuildEndedEvent, result.State, m.Manifest
if result.Manifest != "" {
e.Manifest = result.Manifest
}
if m.Inputs2 != nil {
e.Against = append([]string(nil), m.Inputs2.Bases...)
}
e.Made = append([]delivery.BuiltArtifact(nil), result.Artifacts...)
return e
}
@@ -5,7 +5,6 @@ import (
"encoding/json"
"errors"
"fmt"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"strconv"
"strings"
"sync"
@@ -13,9 +12,11 @@ import (
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/buildstep"
"git.novox.be/novox/mesh-sdk/go/catalogue"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// buildModules simulates the two build modules on the nodes: a long tool answers a step and key at once and writes the
@@ -221,7 +222,42 @@ var (
func TestABundleAndAnArchiveAreBuiltOnOneNodeAndDone(t *testing.T) { // 1607/AC7, ADR 0478 clauses 1, 4
r := newRig(t, goBundle, archiveA, tsBundle)
var logged []delivery.BuildLogEntry
var lmu sync.Mutex
r.b.Node, r.b.Log = "novox", func(e delivery.BuildLogEntry) { lmu.Lock(); logged = append(logged, e); lmu.Unlock() }
v := r.consider(t, "modules/m")
// 1955/L7 (ADR 0571 §1): the build is one start and one end in the builder's log, every field named, its duration read
// from the two.
if len(logged) != 2 || logged[0].Event != delivery.BuildStartedEvent || logged[1].Event != delivery.BuildEndedEvent ||
logged[1].Outcome != delivery.BuildDone || len(logged[1].Made) != 3 || logged[1].Made[0].Reference == "" ||
logged[1].Manifest != manifestD || logged[0].Manifest != "" || logged[0].Repository != "novox/mesh-catalog" ||
logged[0].Directory != "modules/m" || logged[0].Commit != "7a74cee83f88e6585a5511c3cc53bdf0f5303d20" || logged[0].Node != "novox" ||
logged[0].FP != fp1 || logged[0].Module != "m" {
t.Fatalf("logged = %+v", logged)
}
for _, e := range logged {
if _, err := delivery.EncodeBuildLogEntry(e); err != nil {
t.Errorf("an entry the log refuses: %v", err)
}
}
if _, ok := delivery.BuildDuration(logged[0], logged[1]); !ok {
t.Error("no duration from the two entries")
}
// builder_builds_since (ADR 0571 clause 9) answers the end alone from the log's page, and says when the log holds more.
var page stdio.LogPage
for i, e := range logged {
raw, _ := delivery.EncodeBuildLogEntry(e)
page.Entries = append(page.Entries, stdio.LogEntry{Sequence: uint64(i + 1), Key: e.Module, Value: raw, At: e.At})
}
page.Next = 3
since, err := BuildsSince(func(time.Time, string, int) (*stdio.LogPage, error) { return &page, nil }, logged[0].At)
if err != nil || len(since.Builds) != 1 || since.Builds[0].Event != delivery.BuildEndedEvent || len(since.Builds[0].Made) != 3 || !since.More {
t.Errorf("builds since = %+v, %v, want the one end and more", since, err)
}
page.Entries[0].Value = []byte(`{"v":0}`)
if _, err := BuildsSince(func(time.Time, string, int) (*stdio.LogPage, error) { return &page, nil }, logged[0].At); err == nil {
t.Error("an entry that does not read was skipped, want it refused")
}
if v.State != delivery.BuildDone || v.By != "novox.builder" || v.Manifest != manifestD || len(v.Artifacts) != 3 {
t.Fatalf("build = %+v, want done by novox.builder with three artifacts", v)
}
@@ -0,0 +1,43 @@
package build
import (
"fmt"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// BuildsSinceLimit is how many entries one answer of BuildsSinceTool reads of the log; More says the log held more.
const BuildsSinceLimit = 500
// LogSince is one page of the builder's log from a time on: stdio.Log(BuilderLog).Since.
type LogSince func(at time.Time, keys string, limit int) (*stdio.LogPage, error)
// BuildsSince answers delivery.BuildsSinceTool (ADR 0571 clause 9): every ended build in the builder's log from since on,
// oldest first, read through the runtime's log verbs; More when the page was not the log's last, so the caller asks again
// from the last build's time. An entry the log holds that does not read is refused, naming its sequence: a reader that
// skipped it would register too little.
func BuildsSince(read LogSince, since time.Time) (delivery.BuildsSince, error) {
out := delivery.BuildsSince{Since: since.UTC(), Builds: []delivery.BuildLogEntry{}}
page, err := read(since, "", BuildsSinceLimit)
if err != nil {
return out, fmt.Errorf("the builder's log %s from %s: %w", delivery.BuilderLog, since.Format(time.RFC3339), err)
}
if page == nil {
return out, nil
}
var entries []delivery.BuildLogEntry
for _, le := range page.Entries {
e, err := delivery.DecodeBuildLogEntry(le.Value)
if err != nil {
return out, fmt.Errorf("the builder's log %s, entry %d: %w", delivery.BuilderLog, le.Sequence, err)
}
entries = append(entries, e)
}
if ended := delivery.EndedSince(entries, since); ended != nil {
out.Builds = ended
}
out.More = page.Next != 0
return out, nil
}
+22 -10
View File
@@ -10,6 +10,8 @@ import (
"slices"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// The controller's manifest reader is mesh-controller internal/catalogue (Manifest, manifest.go: Consumes :521,
@@ -40,10 +42,14 @@ type manifest struct {
Emits []string `json:"emits"`
Consumes []string `json:"consumes"`
State []state `json:"state"`
Reads []string `json:"reads"`
Uses []string `json:"uses"`
Invokes []string `json:"invokes"`
Config map[string]struct {
Logs []struct {
Name string `json:"name"`
MaxAgeDays int `json:"max-age-days"`
} `json:"logs"`
Reads []string `json:"reads"`
Uses []string `json:"uses"`
Invokes []string `json:"invokes"`
Config map[string]struct {
Type string `json:"type"`
Description string `json:"description"`
Pattern string `json:"pattern"`
@@ -92,8 +98,8 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
}
// 1887/A4c: the builder chooses no node, so it reads neither presence nor have, invokes no build module's tool and
// holds no capacity; it puts asks on the three build seats and reads their steps (ADR 0525 §1, §4, §5).
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments",
"mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules"}; !reflect.DeepEqual(sorted(m.Reads), want) {
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-image.steps", "mesh-build-node.steps", "mesh-build-node.toolchains",
"mesh-delivery-assigner.placement", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want)
}
if want := []string{"node-build-go", "node-build-image", "node-build-node"}; !reflect.DeepEqual(sorted(m.Uses), want) {
@@ -101,12 +107,14 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
}
// 1937/AC3: the builder asks the controller whether the artifact store still holds a done key's blobs (held), and no
// other tool; it holds no capacity.
if want := []string{"seat:mesh-controller.held"}; !reflect.DeepEqual(m.Invokes, want) || len(m.Config) != 0 {
// 1955/L7 (ADR 0571 §2): the seat mesh-artifact-sweep's held, the controller's only while the seat has no holder (clause 10).
if want := []string{"seat:mesh-artifact-sweep.held", "seat:mesh-controller.held"}; !reflect.DeepEqual(sorted(m.Invokes), want) || len(m.Config) != 0 {
t.Errorf("invokes = %v, config = %v, want %v and no config", m.Invokes, m.Config, want)
}
// The builder is woken by a build module's report of a toolchain, as the indexer is: the bucket may not exist when it
// starts; and by the controller's artifacts-gone, a take-in refused for a gone blob (1937/AC4).
if want := []string{"mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-controller.artifacts-gone"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
// 1955/L7: and the seat mesh-artifact-sweep's, the controller's until the lease write (its take-in sweeps until then).
if want := []string{"mesh-artifact-sweep.artifacts-gone", "mesh-build-go.toolchain.reported", "mesh-build-node.toolchain.reported", "mesh-controller.artifacts-gone"}; !reflect.DeepEqual(sorted(m.Consumes), want) {
t.Errorf("consumes = %v, want %v", m.Consumes, want)
}
if b, ok := m.OwnSecrets["broker"]; !ok || len(m.OwnSecrets) != 1 || b != "${dir:state}/broker" {
@@ -116,13 +124,17 @@ func TestManifestDeclarations(t *testing.T) { // 1607/AC1
t.Errorf("secrets-owner = %q", m.SecretsOwner)
}
// 1945: the builder's tools (builder_seat_queue: note 62), served by its tools bundle, and the event it says when it asks an ask again.
if want := []string{"builder_announce_again", "builder_ask_again", "builder_build_again", "builder_seat_queue", "builder_stuck"}; !reflect.DeepEqual(sorted(m.Tools), want) {
if want := []string{"builder_announce_again", "builder_ask_again", "builder_build_again", "builder_builds_since", "builder_seat_queue", "builder_stuck"}; !reflect.DeepEqual(sorted(m.Tools), want) {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
if want := []string{"build.asked-again"}; !reflect.DeepEqual(m.Emits, want) {
t.Errorf("emits = %v, want %v", m.Emits, want)
}
for _, k := range []string{"streams", "logs", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
// 1955/L7 (ADR 0571 §1): its module log, each build's start and end, kept 30 days; no bare stream.
if len(m.Logs) != 1 || m.Logs[0].Name != delivery.BuilderLog || m.Logs[0].MaxAgeDays != delivery.BuilderLogDays {
t.Errorf("logs = %+v, want %s kept %d days", m.Logs, delivery.BuilderLog, delivery.BuilderLogDays)
}
for _, k := range []string{"streams", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the builder has none", k)
}
+7 -2
View File
@@ -6,6 +6,7 @@
"builder_announce_again",
"builder_ask_again",
"builder_build_again",
"builder_builds_since",
"builder_seat_queue",
"builder_stuck"
],
@@ -13,11 +14,13 @@
"build.asked-again"
],
"consumes": [
"mesh-artifact-sweep.artifacts-gone",
"mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported",
"mesh-controller.artifacts-gone"
],
"invokes": [
"seat:mesh-artifact-sweep.held",
"seat:mesh-controller.held"
],
"state": [
@@ -27,13 +30,16 @@
{ "name": "queues", "history": 1 },
{ "name": "reasks", "history": 1 }
],
"logs": [
{ "name": "build-log", "max-age-days": 30 }
],
"reads": [
"mesh-build-go.steps",
"mesh-build-go.toolchains",
"mesh-build-image.steps",
"mesh-build-node.steps",
"mesh-build-node.toolchains",
"mesh-controller.assignments",
"mesh-delivery-assigner.placement",
"mesh-delivery-gate.source",
"mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests",
@@ -44,7 +50,6 @@
"node-build-image",
"node-build-node"
],
"own-secrets": {
"broker": "${dir:state}/broker"
},
@@ -170,6 +170,21 @@ func TryTools(tr Trying) []stdio.Tool {
}
}
// Seat is the seat the checker holds (novox/hq ADR 0547 §10, issue 774); its verbs are served as tools named <seat>.<verb>.
const Seat = "mesh-delivery-checker"
// SeatTryTools are the try's tools as the seat's verbs: try, try-read and try-end, run by the same code as checker_try,
// checker_try_read and checker_try_end, so the answer is the same. The module tools stay beside them.
func SeatTryTools(tr Trying) []stdio.Tool {
verbs := map[string]string{check.TryTool: "try", check.TryReadTool: "try-read", check.TryEndTool: "try-end"}
var out []stdio.Tool
for _, t := range TryTools(tr) {
t.Name = Seat + "." + verbs[t.Name]
out = append(out, t)
}
return out
}
// Tools are the read tools over the checker's states.
func Tools(r check.Reader) []stdio.Tool {
return []stdio.Tool{
@@ -193,24 +208,6 @@ func Tools(r check.Reader) []stdio.Tool {
n, _ := args["number"].(float64)
return r.Pull(repository, int(n))
}},
{Name: check.FailureReadTool,
Description: "The full output of a pull request's failed test, which the verdict and its mesh/check status cut to their last lines " +
"(\"... (N earlier lines)\"): the test, the words the verdict keeps, the build step that holds the rest (its id, build module, " +
"machine, the check it ran for, and the call that reads it) and the first page of its build log. Read on with from = the " +
"output's next. Answers from the pull request's last failed verdict also once no candidate holds it. Read-only.",
Input: map[string]any{"type": "object", "required": []string{"repository", "number"}, "properties": map[string]any{
"repository": map[string]any{"type": "string", "description": "<owner>/<name>"},
"number": map[string]any{"type": "integer", "description": "the pull request's number"},
"from": map[string]any{"type": "integer", "description": "the log's stream sequence to read on from: the output's next; 0 is the start"}}},
Run: func(args map[string]any) (any, error) {
repository, _ := args["repository"].(string)
n, _ := args["number"].(float64)
from, _ := args["from"].(float64)
if from < 0 {
return nil, fmt.Errorf("from is a stream sequence, 0 or more")
}
return r.Failure(repository, int(n), uint64(from))
}},
{Name: check.KeptReadTool,
Description: "The kept verdicts of passing packages, each as stored under kept.<level>.<digest>: the package, its tests, the time they took, the toolchain " +
"and when it was kept. With a level, that level's; with a level and a digest, that one. Read-only.",
@@ -287,8 +284,7 @@ func main() {
HaveState: read(stdio.State("node-engine.have")), StepBound: stepBound(),
TurnState: read(stdio.State("mesh-delivery-gate.turns")), HearingState: read(stdio.State(buildrun.HearingState)),
AskState: read(stdio.State(check.AsksState))}
r.Logs = func(tool string, args map[string]any) (json.RawMessage, error) { return stdio.Ask(tool, args) } // the module's invokes
r.Runs = runsOf{read(stdio.State("mesh-delivery-indexer.modules"))} // the node-engine at the commit the mesh runs, as the checker reads it
r.Runs = runsOf{read(stdio.State("mesh-delivery-indexer.modules"))} // the node-engine at the commit the mesh runs, as the checker reads it
tries := stdio.State(check.TriesState)
source := stdio.State("mesh-delivery-gate.source")
trunks := func() (map[string]string, error) {
@@ -327,7 +323,7 @@ func main() {
_, err := verdicts.Update(key, json.RawMessage(value), revision)
return err
}}
tools := append(append(Tools(r), TryTools(trying)...), RecoveryTools(r, written)...)
tools := append(append(append(Tools(r), TryTools(trying)...), SeatTryTools(trying)...), RecoveryTools(r, written)...)
if err := stdio.Serve("", logged(tools)); err != nil {
fmt.Fprintf(os.Stderr, "[mesh-delivery-checker-tools] %v\n", err)
os.Exit(1)
@@ -11,7 +11,7 @@ import (
"mesh-delivery-checker/internal/check"
)
func TestTheToolsServedAreTheSixReadTools(t *testing.T) {
func TestTheToolsServedAreTheFiveReadTools(t *testing.T) {
var names []string
for _, tl := range Tools(check.Reader{}) {
names = append(names, tl.Name)
@@ -19,7 +19,7 @@ func TestTheToolsServedAreTheSixReadTools(t *testing.T) {
t.Errorf("%s has no description or no run", tl.Name)
}
}
want := []string{check.VerdictsReadTool, check.PullReadTool, check.FailureReadTool, check.KeptReadTool, check.PreviewsReadTool, check.StatusReadTool}
want := []string{check.VerdictsReadTool, check.PullReadTool, check.KeptReadTool, check.PreviewsReadTool, check.StatusReadTool}
if !slices.Equal(names, want) {
t.Errorf("tools = %v, want %v", names, want)
}
@@ -88,6 +88,24 @@ func TestTheTryToolsAskAndReadATry(t *testing.T) { // ADR 0501 §5, the agreed i
if err != nil || !strings.Contains(string(raw.(json.RawMessage)), `"state":"queued"`) {
t.Errorf("read = %s, %v", raw, err)
}
// The seat's verbs answer as the tools do (ADR 0547 §10, issue 774): the same ask is the same try, and the read is alike.
verbs := SeatTryTools(Trying{Tries: store, Ender: store, Now: func() time.Time { return time.Unix(1, 5) },
Trunks: func() (map[string]string, error) {
return map[string]string{"novox/mesh-delivery": strings.Repeat("a", 40)}, nil
}})
if len(verbs) != 3 || verbs[0].Name != Seat+".try" || verbs[1].Name != Seat+".try-read" || verbs[2].Name != Seat+".try-end" {
t.Fatalf("verbs = %+v", verbs)
}
again, err := verbs[0].Run(map[string]any{"repository": "novox/mesh-delivery", "head": strings.Repeat("1", 40), "trunk": "main"})
if err != nil || again.(map[string]any)["id"] != id {
t.Errorf("verb try = %v, %v, want the waiting try %s", again, err, id)
}
if viaVerb, err := verbs[1].Run(map[string]any{"id": id}); err != nil || string(viaVerb.(json.RawMessage)) != string(raw.(json.RawMessage)) {
t.Errorf("verb try-read = %s, %v; tool = %s", viaVerb, err, raw)
}
if _, err := verbs[0].Run(map[string]any{"repository": "novox/hq", "head": strings.Repeat("1", 40)}); err == nil {
t.Error("the verb tried a repository nothing checks")
}
// Issue 734: a try whose ask was lost is ended by hand with why, and its head is asked anew.
ended, err := tools[2].Run(map[string]any{"id": id, "why": "its ask was lost"})
if err != nil || ended.(map[string]any)["state"] != "error" {
@@ -208,11 +208,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// The toolchains first, so a check that starts from the first candidate finds the nodes.
watch := newToolchainWatch(ctx, js, tools, ch.NodesChanged, logger)
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
// Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// placement is settled first, so the toolchains settle through the gate already knowing it.
gate := buildrun.NewToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, tools.Put, logger.Printf)
watch.put = gate.Toolchain
assignErrs, err := watchAssignments(ctx, js, gate, ch.NodesChanged, logger)
placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, ch.NodesChanged, logger.Printf)
if err != nil {
return err
}
@@ -362,7 +362,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return nil
case err := <-errs:
return err
case err := <-assignErrs:
case err := <-placementErrs:
return err
case err := <-watch.errs:
return err
@@ -0,0 +1,62 @@
package main
import (
"bytes"
"context"
"errors"
"log"
"strings"
"testing"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
// 1945 (a part exits at start for a bucket it works without): a bus that refuses every toolchains bucket leaves the
// watch unstarted and the part running, said; a refused bucket it cannot work without is an error naming the bucket and
// its grant. The gate over the seat's placement, and its refusal, are mesh-sdk's (buildrun.ToolchainGate,
// issue 774).
func TestARefusedBucketIsSaidAndThePartGoesOn(t *testing.T) {
set := &buildrun.ToolchainSet{}
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-build-node_toolchains")}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "checker", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
@@ -1,211 +0,0 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -1,218 +0,0 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"bytes"
"context"
"errors"
"fmt"
"log"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/process"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
// 1945 (a part exits at start for a bucket it works without): a bus that refuses the assignments' watch, or every
// toolchains bucket, leaves the watches unstarted and the part running, said; a refused bucket it cannot work without
// is an error naming the bucket and its grant.
var logs bytes.Buffer
logger := log.New(&logs, "", 0)
refused := refusingBus{err: errors.New("nats: permissions violation for publish to $JS.API.CONSUMER.CREATE.KV_mesh-controller_assignments")}
if _, err := watchAssignments(context.Background(), refused, newToolchainGate(nil, set.Put, t.Logf), func(context.Context) {}, logger); err != nil {
t.Errorf("a refused watch of the assignments stopped the start: %v", err)
}
if err := newToolchainWatch(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, set, func(context.Context) {}, logger).start(); err != nil {
t.Errorf("a refused toolchains bucket stopped the start: %v", err)
}
if !strings.Contains(logs.String(), "cannot be watched") || !strings.Contains(logs.String(), "cannot be read") {
t.Errorf("the refusals were not said: %q", logs.String())
}
if _, err := process.EssentialBucket(context.Background(), refusingBus{refuseOpen: true, err: refused.err}, "checker", "mesh-delivery-indexer_modules", "work"); err == nil ||
!strings.Contains(err.Error(), "grant of mesh-delivery-indexer_modules") {
t.Errorf("a refused bucket the part needs: %v, want an error naming its grant", err)
}
}
// refusingBus is a bus that refuses: every open (refuseOpen), or every watch of a bucket it opens, with err.
type refusingBus struct {
refuseOpen bool
err error
}
func (r refusingBus) KeyValue(context.Context, string) (jetstream.KeyValue, error) {
if r.refuseOpen {
return nil, r.err
}
return refusedWatch{err: r.err}, nil
}
type refusedWatch struct {
jetstream.KeyValue
err error
}
func (r refusedWatch) WatchAll(context.Context, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
func (r refusedWatch) Watch(context.Context, string, ...jetstream.WatchOpt) (jetstream.KeyWatcher, error) {
return nil, r.err
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-checker
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030100935-c2ec8b6b2fde
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -6
View File
@@ -1,9 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030095816-530d8d3007dc h1:Wdjl2gomooHswJ5qpkwZPq/ureZgXVw6FMCsgzPJhPo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030095816-530d8d3007dc/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030100935-c2ec8b6b2fde h1:eKXSllHCS31GgMazfvZ7BeiLoXIUibnUmzxWVkj+Vcw=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030100935-c2ec8b6b2fde/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -425,15 +425,13 @@ func TestANodePackageIsNeverKeptAndRunsEveryCheck(t *testing.T) { // 1288/AC4, 1
}
}
// 1289/AC1, AC2 as issue 1921 changed it: the change's own failing test runs first, the level's other packages still run
// after it (so one check names every failure), and no later level runs.
func TestTheChangesOwnFailingTestRunsFirstThenTheRestAndNoLaterLevel(t *testing.T) {
// 1289/AC1, AC2 (R4c, mesh-sdk buildrun at 0b12b62): the change's own tests run first, and when one fails the level ends
// there: no other package of the level and no later level runs, so the rest of the suite is not paid for.
func TestTheChangesOwnFailingTestEndsTheLevelThere(t *testing.T) {
b := newBench(t)
b.fetch.Changed = []string{"modules/m/internal/b/b_test.go"}
b.digest[delivery.LevelUnit] = Digested{Level: "unit", Packages: map[string]string{"m/internal/a": digA, "m/internal/b": digB}}
failing := captured[Tested](t, "go-test-failing")
// The step said the failure's whole output on its log (the operator, 2026-10-30): the verdict names where.
failing.Failed[0].Step, failing.Failed[0].Module, failing.Failed[0].Machine, failing.Failed[0].Run = "s1", "mesh-build-go", "ace", "r1"
b.test = func(c stepCall) (any, error) {
if got := packagesOf(c); len(got) == 1 && got[0] == "m/internal/b" {
return failing, nil
@@ -441,13 +439,12 @@ func TestTheChangesOwnFailingTestRunsFirstThenTheRestAndNoLaterLevel(t *testing.
return captured[Tested](t, "go-test-slow"), nil
}
out := b.judge(t, caseOf(t, pullOf(repoA, 7, head1)))
if f := out.Verdict.Failed; out.Verdict.State != delivery.VerdictFail || !strings.Contains(f.Test, "TestThreeIsFour") ||
f.Step != "s1" || f.Module != "mesh-build-go" || f.Machine != "ace" || f.Run != "r1" || f.Pull != repoA+"#7" || out.Verdict.Failures[0].Step != "s1" {
if out.Verdict.State != delivery.VerdictFail || !strings.Contains(out.Verdict.Failed.Test, "TestThreeIsFour") {
t.Fatalf("verdict = %+v", out.Verdict)
}
tests := b.steps.tool("test")
if len(tests) != 2 || strings.Join(packagesOf(tests[0]), ",") != "m/internal/b" || strings.Join(packagesOf(tests[1]), ",") != "m/internal/a" {
t.Fatalf("test calls = %+v, want m/internal/b, then m/internal/a, each once", tests)
if len(tests) != 1 || strings.Join(packagesOf(tests[0]), ",") != "m/internal/b" {
t.Fatalf("test calls = %+v, want m/internal/b alone: its failure ends the level", tests)
}
if n := len(b.steps.tool("digest")); n != 1 {
t.Errorf("%d digest calls, want the unit level's alone (the check stopped)", n)
@@ -1,132 +0,0 @@
package check
import (
"encoding/json"
"fmt"
"strings"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// A pull request's failed test, its whole output read from the build step that said it (the operator, 2026-10-30: the
// verdict cuts a failure to its last lines, and its cause was guessed). The verdict names the step, its build module and
// machine and the check it ran for (delivery.FailedFirst); the build module's log tool reads the step's log from its own
// stream (mesh-sdk buildstep.LogTool). The verdict is found through the pull request's candidate, or, once no candidate
// holds it, as the newest verdict that names the pull request's failure.
// FailureStep is where a failure's whole output is: the step id, its build module and machine, the run it ran for, and the
// call that reads it.
type FailureStep struct {
ID string `json:"id"`
Module string `json:"module"`
Machine string `json:"machine,omitempty"`
Run string `json:"run"`
Read string `json:"read"`
}
// FailureRead is a pull request's failed test: the verdict that names it, the test and the words the verdict keeps, the
// step that holds the rest, and the first page of its log (Output, the log tool's answer), or why there is none.
type FailureRead struct {
Pull string `json:"pull"`
Verdict string `json:"verdict"`
Test string `json:"test,omitempty"`
Words string `json:"words,omitempty"`
Step *FailureStep `json:"step,omitempty"`
Output json.RawMessage `json:"output,omitempty"`
Note string `json:"note,omitempty"`
}
// logToolOf is a build module's log tool: mesh-build-go's build_go_log.
func logToolOf(module string) string {
lang := strings.TrimPrefix(module, "mesh-build-")
return module + ".build_" + lang + "_log"
}
// failureOf is the failure a verdict names of the pull request pull: the first that names it, or, in a verdict that names
// no pull request (written before), the first failure.
func failureOf(v delivery.Verdict, pull string) (delivery.FailedFirst, bool) {
for _, f := range append([]delivery.FailedFirst{v.Failed}, v.Failures...) {
if f.Pull == pull && f.Test != "" {
return f, true
}
}
if v.Failed.Pull == "" && v.Failed.Test != "" {
return v.Failed, true
}
return delivery.FailedFirst{}, false
}
// Failure answers a pull request's failed test, with the page of its log from the stream sequence from on (0 the start).
func (r Reader) Failure(repository string, number int, from uint64) (FailureRead, error) {
pull := fmt.Sprintf("%s#%d", repository, number)
out := FailureRead{Pull: pull}
var v delivery.Verdict
found := false
if read, err := r.Pull(repository, number); err == nil && len(read.Verdict) > 0 {
if v, err = delivery.DecodeVerdict(read.Verdict); err != nil {
return out, fmt.Errorf("%s holds what is not a verdict: %w", read.Key, err)
}
out.Verdict, found = read.Key, true
} else if err != nil && !strings.HasPrefix(err.Error(), "no candidate") {
return out, err
}
if _, ok := failureOf(v, pull); !found || v.State != delivery.VerdictFail || !ok {
// No candidate holds it, or its candidate's verdict did not fail of it: the newest verdict that names its failure.
key, last, err := r.lastFailure(pull)
if err != nil {
return out, err
}
if key == "" {
if found {
out.Note = "its verdict names no failed test of it, and no other verdict does"
return out, nil
}
return out, fmt.Errorf("no candidate holds %s, and no verdict names a failed test of it", pull)
}
out.Verdict, v = key, last
}
f, _ := failureOf(v, pull)
out.Test, out.Words = f.Test, f.Words
if f.Step == "" {
out.Note = "the verdict names no build step that kept the whole output (a build module or a checker from before it)"
return out, nil
}
out.Step = &FailureStep{ID: f.Step, Module: f.Module, Machine: f.Machine, Run: f.Run,
Read: fmt.Sprintf("%s {run: %s, step: %s}", logToolOf(f.Module), f.Run, f.Step)}
if r.Logs == nil {
out.Note = "the build module's log cannot be asked here"
return out, nil
}
page, err := r.Logs(logToolOf(f.Module), map[string]any{"run": f.Run, "step": f.Step, "from": from})
if err != nil {
out.Note = "its log could not be read: " + err.Error()
return out, nil
}
out.Output = page
return out, nil
}
// lastFailure is the newest verdict that names a failed test of pull, by its step id (an id begins with its time), and
// its key; "" when none does among the first MaxReads.
func (r Reader) lastFailure(pull string) (string, delivery.Verdict, error) {
rows, _, err := r.Verdicts("")
if err != nil {
return "", delivery.Verdict{}, err
}
var key, newest string
var last delivery.Verdict
for _, row := range rows {
v, err := delivery.DecodeVerdict(row.Verdict)
if err != nil || v.State != delivery.VerdictFail {
continue
}
f, ok := failureOf(v, pull)
if !ok || f.Pull != pull {
continue
}
if key == "" || len(f.Step) > len(newest) || (len(f.Step) == len(newest) && f.Step > newest) {
key, newest, last = row.Key, f.Step, v
}
}
return key, last, nil
}
@@ -318,15 +318,6 @@ func (st *state) outcome(v delivery.Verdict) Outcome {
return out
}
// runOf is the run a case's asks name (delivery.StepAsk.Run), under which their steps say their logs: the check's heads
// hash, or delivery.NoRun when it is no token.
func runOf(c *Case) string {
if c == nil || c.HeadsHash == "" || strings.ContainsAny(c.HeadsHash, ".*> ") {
return delivery.NoRun
}
return c.HeadsHash
}
// fail is a verdict of failure at a machine.
func fail(machine, words string) *delivery.Verdict {
v := failure(machine, words)
@@ -439,7 +430,7 @@ func (st *state) askNeeding(ctx context.Context, t *target, merged *[]Fetched, l
kind = delivery.AskRecheck // a re-judge after a trunk move goes after every new head's check (ADR 0536 §4)
}
a := delivery.StepAsk{ID: buildstep.NewID(), Kind: kind, Language: askLanguage[t.module], Asker: delivery.CheckerModule,
At: st.j.now().UTC(), Check: job, Needs: needs, Run: runOf(st.c)}
At: st.j.now().UTC(), Check: job, Needs: needs}
var step delivery.Step
var firstDecline time.Time
for {
@@ -840,12 +831,7 @@ func levelFailure(t *target, level string, results ...Tested) *delivery.Verdict
if t.module == buildrun.NodeBuildModule && f.Package == "" {
name = res.Module
}
ff := delivery.FailedFirst{Test: level + ": " + name, Words: failureWordsAfter(f, hidWords(t.module, res.By, f)),
Pull: fmt.Sprintf("%s#%d", t.pull.Repository, t.pull.Number)}
if f.Step != "" { // its build step said its whole output on the module's log: where (checker_failure reads it)
ff.Step, ff.Module, ff.Machine, ff.Run = f.Step, f.Module, f.Machine, f.Run
}
all = append(all, ff)
all = append(all, delivery.FailedFirst{Test: level + ": " + name, Words: failureWordsAfter(f, hidWords(t.module, res.By, f))})
}
}
if len(all) == 0 { // no result: a level said failed with nothing to name
@@ -862,8 +848,7 @@ func failureLines(all []delivery.FailedFirst) []delivery.FailedFirst {
n := min(len(all), delivery.MaxFailures)
out := make([]delivery.FailedFirst, 0, n+1)
for _, f := range all[:n] {
f.Words = clipLine(firstLine(f.Words), MaxFailureLine)
out = append(out, f)
out = append(out, delivery.FailedFirst{Test: f.Test, Machine: f.Machine, Words: clipLine(firstLine(f.Words), MaxFailureLine)})
}
if more := len(all) - n; more > 0 {
out = append(out, delivery.FailedFirst{Test: fmt.Sprintf("and %d more", more)})
@@ -312,7 +312,7 @@ func parts(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []s
// Report is the full line of a pull request's status, with no limit: what the read tools and the full report say.
func Report(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []string, judged map[string]string) string {
return joinSet(parts(v, cs, p, builds, slowest, judged).join(0), fullOutputWords(v, p))
return parts(v, cs, p, builds, slowest, judged).join(0)
}
// Line is the words of a status for one pull request of a case (ADR 0447 clause 3, ADR 0484 clause 7): the verdict first (a
@@ -324,52 +324,26 @@ func Report(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []
// The program that judged is never dropped: a line naming it gives the counts in their compact form, then drops them and
// what follows, and is cut in its first part rather than lose it.
func Line(v delivery.Verdict, cs *Case, p Pull, builds []BuildNote, slowest []string, judged map[string]string) string {
full := fullOutputWords(v, p)
if full == "" {
return lineWithin(parts(v, cs, p, builds, slowest, judged), MaxLine)
}
return lineWithin(parts(v, cs, p, builds, slowest, judged), MaxLine-utf8.RuneCountInString(full)-3) + " | " + full
}
// FullOutputTool is the read tool that answers a failed test's whole output (the checker's tools).
const FullOutputTool = "checker_failure"
// fullOutputWords end the line of a pull request whose failed test's whole output a build step said on its log: where to
// read it, "full output: checker_failure <repo>#<n>"; "" when none was said (no test failed, or an older build module).
func fullOutputWords(v delivery.Verdict, p Pull) string {
if v.State != delivery.VerdictFail {
return ""
}
pull := fmt.Sprintf("%s#%d", p.Repository, p.Number)
for _, f := range append([]delivery.FailedFirst{v.Failed}, v.Failures...) {
if f.Step != "" && (f.Pull == "" || f.Pull == pull) {
return "full output: " + FullOutputTool + " " + pull
}
}
return ""
}
// lineWithin is the line of the pieces in at most most characters (Line).
func lineWithin(pc pieces, most int) string {
pc := parts(v, cs, p, builds, slowest, judged)
for drop := 0; drop <= 4; drop++ {
if s := pc.join(drop); utf8.RuneCountInString(s) <= most {
if s := pc.join(drop); utf8.RuneCountInString(s) <= MaxLine {
return s
}
}
if pc.judged != "" {
for _, counts := range []string{pc.shortCounts, ""} {
if s := joinSet(pc.head, pc.commits, counts, pc.judged); utf8.RuneCountInString(s) <= most {
if s := joinSet(pc.head, pc.commits, counts, pc.judged); utf8.RuneCountInString(s) <= MaxLine {
return s
}
}
rest := " | " + joinSet(pc.commits, pc.judged)
room := most - utf8.RuneCountInString(rest) - 1
room := MaxLine - utf8.RuneCountInString(rest) - 1
if head := []rune(pc.head); room > 0 && len(head) > room {
return string(head[:room]) + "…" + rest
}
}
runes := []rune(pc.join(4))
return string(runes[:most-1]) + "…"
return string(runes[:MaxLine-1]) + "…"
}
// joinSet joins the parts that are not empty.
@@ -50,18 +50,11 @@ func TestAFailureNamesTheTestTheConflictOrTheRefusedStepFirst(t *testing.T) { //
"test": {delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Test: "unit: m/a.TestX", Words: "want 4"}}, "failed: unit: m/a.TestX: want 4"},
"conflict": {delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Machine: "shanks", Words: "novox/mesh-delivery#7 does not merge into the trunk: files in conflict: go.mod"}}, "failed: shanks: novox/mesh-delivery#7 does not merge into the trunk: files in conflict: go.mod"},
"refused": {delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Machine: "mesh-build-go", Words: "no go build node qualifies: builders disagree"}}, "failed: mesh-build-go: no go build node qualifies"},
// The operator, 2026-10-30: a test whose step said its whole output ends the line saying where to read it.
"whole output": {delivery.Verdict{State: delivery.VerdictFail, Failed: delivery.FailedFirst{Test: "unit: m/a.TestX", Words: "want 4\n... " + strings.Repeat("a long line ", 40),
Step: "s1", Module: "mesh-build-go", Run: "r1", Pull: "novox/mesh-delivery#7"}}, "failed: unit: m/a.TestX: want 4"},
} {
line := Line(c.v, cs, p, nil, nil, nil)
if !strings.HasPrefix(line, c.want) || !strings.Contains(line, "111111111111 on aaaaaaaaaaaa") {
t.Errorf("%s: line = %q, want it to open with %q and name the commits", name, line, c.want)
}
full := strings.HasSuffix(line, " | full output: checker_failure novox/mesh-delivery#7")
if full != (c.v.Failed.Step != "") || utf8.RuneCountInString(line) > MaxLine {
t.Errorf("%s: line = %q: the full output named %v, want %v, within %d characters", name, line, full, !full, MaxLine)
}
}
}
@@ -18,9 +18,7 @@ import (
const (
VerdictsReadTool = "checker_verdicts"
PullReadTool = "checker_pull"
// FailureReadTool is FullOutputTool: a pull request's failed test with its whole output (Failure).
FailureReadTool = FullOutputTool
KeptReadTool = "checker_kept"
KeptReadTool = "checker_kept"
// PreviewsReadTool reads the previews state (issue 1680); StatusReadTool reads what the checker runs (issue 1685: nothing holds it since R4g).
PreviewsReadTool = "checker_previews"
StatusReadTool = "checker_status"
@@ -51,10 +49,7 @@ type Reader struct {
HearingState StateRead
// AskState is the checker's state asks: the asks each running check put (AsksState), which checker_ask_again reads.
AskState StateRead
// Logs asks a build module's log tool (`<module>.build_<lang>_log`) as the checker (stdio.Ask): a failed test's whole
// output (Failure); nil asks none.
Logs func(tool string, args map[string]any) (json.RawMessage, error)
Now func() time.Time
Now func() time.Time
}
// VerdictRead is a verdict as stored with the full report of the status of each pull request it judged (Report), when the
@@ -67,41 +67,6 @@ func TestTheVerdictToolsAnswerTheStoredVerdictWithTheFullReportOfEachPullRequest
if only, _, _ := rd.Verdicts("feat/other"); len(only) != 0 {
t.Errorf("another group's verdicts = %+v", only)
}
// checker_failure (the operator, 2026-10-30): a passing pull request has no failure to read; one no candidate holds any
// more is answered from its newest failed verdict, with the step that holds its whole output and that log's first page.
if got, err := rd.Failure(repoA, 7, 0); err != nil || got.Step != nil || !strings.Contains(got.Note, "names no failed test") {
t.Errorf("a passing pull request's failure: %+v, %v", got, err)
}
if _, err := rd.Failure("novox/none", 1, 0); err == nil || !strings.Contains(err.Error(), "no verdict names a failed test") {
t.Errorf("a pull request no candidate and no verdict holds: %v", err)
}
verdicts := fakeState{}
keys, _ := rd.VerdictState.Keys()
for _, k := range keys {
verdicts[k], _ = rd.VerdictState.Get(k)
}
for i, step := range []string{"mh0aaaaa000001", "mh0aaaab000002"} {
v := delivery.Verdict{V: 1, Group: "fix/y", Heads: fmt.Sprintf("%012d", i), State: delivery.VerdictFail,
Failed: delivery.FailedFirst{Test: "unit: m/a.TestX", Words: "want 4", Step: step, Module: "mesh-build-go", Machine: "ace", Run: "r1", Pull: "novox/y#9"}}
raw, err := delivery.EncodeVerdict(v)
if err != nil {
t.Fatal(err)
}
key, _ := delivery.CheckerVerdictKey(v.Group, v.Heads)
verdicts[key] = raw
}
rd.VerdictState = verdicts.read()
var asked string
rd.Logs = func(tool string, args map[string]any) (json.RawMessage, error) {
asked = fmt.Sprintf("%s %v %v %v", tool, args["run"], args["step"], args["from"])
return json.RawMessage(`{"lines":["=== m/a TestX","want 4"],"next":0}`), nil
}
got, err := rd.Failure("novox/y", 9, 0)
if err != nil || got.Step == nil || got.Step.ID != "mh0aaaab000002" || got.Step.Read != "mesh-build-go.build_go_log {run: r1, step: mh0aaaab000002}" ||
asked != "mesh-build-go.build_go_log r1 mh0aaaab000002 0" || !strings.Contains(string(got.Output), "want 4") {
t.Errorf("the failure of novox/y#9: %+v (%s), asked %q, %v", got, got.Output, asked, err)
}
}
// 1887/A10: an hq pull request is answered as any other, its report its verdict's (ADR 0527 §2), never a note that a script
+18 -10
View File
@@ -38,6 +38,11 @@ type stream struct {
}
type manifest struct {
Claims []struct {
Name string `json:"name"`
Scope string `json:"scope"`
Serves []string `json:"serves"`
} `json:"claims"`
Streams []stream `json:"streams"`
Tools []string `json:"tools"`
Module string `json:"module"`
@@ -96,17 +101,15 @@ func TestManifestDeclarations(t *testing.T) { // 1678/AC1
if want := []state{{Name: "asks", History: 1}, {Name: "hearing", History: 1}, {Name: "kept", History: 1}, {Name: "owed", History: 5}, {Name: "previews", History: 5}, {Name: "tries", History: 1}, {Name: "verdicts", History: 5}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want)
}
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-node.steps", "mesh-build-node.toolchains", "mesh-controller.assignments",
"mesh-delivery-assigner.compositions", "mesh-delivery-builder.builds", "mesh-delivery-gate.source", "mesh-delivery-gate.turns", "mesh-delivery-indexer.candidates",
if want := []string{"mesh-build-go.steps", "mesh-build-go.toolchains", "mesh-build-node.steps", "mesh-build-node.toolchains",
"mesh-delivery-assigner.compositions", "mesh-delivery-assigner.placement", "mesh-delivery-builder.builds", "mesh-delivery-gate.source", "mesh-delivery-gate.turns", "mesh-delivery-indexer.candidates",
"mesh-delivery-indexer.manifests", "mesh-delivery-indexer.modules", "mesh-delivery-watchman.presence", "node-engine.have"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want)
}
// 1887/A5b: the checker reads nothing of the controller's plans-building (ADR 0483 superseded by ADR 0525 §7), runs no
// build module's step tool and holds no capacity: it puts check asks on the Go and Node build seats (ADR 0525 §1, §4).
// Its checker_failure reads a failed step's whole output with the build modules' log tools (the operator, 2026-10-30).
if want := []string{"gitea.gitea_commits", "gitea.gitea_compare", "gitea.gitea_get_file", "gitea.gitea_list_branches",
"mesh-build-go.build_go_log", "mesh-build-node.build_node_log"}; !reflect.DeepEqual(sorted(m.Invokes), want) {
t.Errorf("invokes = %v, want the forge's reads for a mesh-sdk pin (ADR 0484 clause 9) and the build logs' reads", m.Invokes)
// 1887/A5b: the checker reads nothing of the controller's plans-building (ADR 0483 superseded by ADR 0525 §7), calls no
// build module's tool and holds no capacity: it puts check asks on the Go and Node build seats (ADR 0525 §1, §4).
if want := []string{"gitea.gitea_commits", "gitea.gitea_compare", "gitea.gitea_get_file", "gitea.gitea_list_branches"}; !reflect.DeepEqual(sorted(m.Invokes), want) {
t.Errorf("invokes = %v, want only the forge's reads for a mesh-sdk pin (ADR 0484 clause 9)", m.Invokes)
}
if want := []string{"node-build-go", "node-build-node"}; !reflect.DeepEqual(sorted(m.Uses), want) {
t.Errorf("uses = %v, want the Go and Node build seats", m.Uses)
@@ -128,10 +131,15 @@ func TestManifestDeclarations(t *testing.T) { // 1678/AC1
// The read tools (issue 1686) are served by the tools bundle, not by the checker's process.
// The try's tools (ADR 0501 §5; the end by hand, issue 734) beside them: mesh-controller's try calls them. A lost check
// is asked again by hand (checker_ask_again).
if want := []string{"checker_ask_again", "checker_failure", "checker_kept", "checker_previews", "checker_pull", "checker_status", "checker_try", "checker_try_end", "checker_try_read", "checker_verdicts"}; !reflect.DeepEqual(sorted(m.Tools), want) {
if want := []string{"checker_ask_again", "checker_kept", "checker_previews", "checker_pull", "checker_status", "checker_try", "checker_try_end", "checker_try_read", "checker_verdicts"}; !reflect.DeepEqual(sorted(m.Tools), want) {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
for _, k := range []string{"logs", "emits", "claims", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
// The try is the checker's own seat's verbs (ADR 0547 §10, issue 774), served beside the tools by the same code.
if len(m.Claims) != 1 || m.Claims[0].Name != "mesh-delivery-checker" || m.Claims[0].Scope != "mesh" ||
!reflect.DeepEqual(m.Claims[0].Serves, []string{"try", "try-read", "try-end"}) {
t.Errorf("claims = %+v, want the mesh seat mesh-delivery-checker serving try, try-read and try-end", m.Claims)
}
for _, k := range []string{"logs", "emits", "provides", "requires", "defines-seats", "seat", "seats", "bus", "config"} {
if _, ok := top[k]; ok {
t.Errorf("the manifest declares %q: the checker has none", k)
}
+13 -5
View File
@@ -2,6 +2,17 @@
"module": "mesh-delivery-checker",
"version": "1",
"slug": "checker",
"claims": [
{
"name": "mesh-delivery-checker",
"scope": "mesh",
"serves": [
"try",
"try-read",
"try-end"
]
}
],
"consumes": [
"mesh-build-go.toolchain.reported",
"mesh-build-node.toolchain.reported"
@@ -23,8 +34,8 @@
"mesh-build-go.toolchains",
"mesh-build-node.steps",
"mesh-build-node.toolchains",
"mesh-controller.assignments",
"mesh-delivery-assigner.compositions",
"mesh-delivery-assigner.placement",
"mesh-delivery-builder.builds",
"mesh-delivery-gate.source",
"mesh-delivery-gate.turns",
@@ -40,7 +51,6 @@
],
"tools": [
"checker_ask_again",
"checker_failure",
"checker_kept",
"checker_previews",
"checker_pull",
@@ -54,9 +64,7 @@
"gitea.gitea_commits",
"gitea.gitea_compare",
"gitea.gitea_get_file",
"gitea.gitea_list_branches",
"mesh-build-go.build_go_log",
"mesh-build-node.build_node_log"
"gitea.gitea_list_branches"
],
"own-secrets": {
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-forge-bridge
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 h1:ycu/sNieTdLpD3bzhZOCXdzcoqXUOxPRmqWV019REpo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -46,6 +46,13 @@ var ErrNotFound = errors.New("the forge has no such file")
// says is not there is ErrNotFound, any other failure any other error (issue 1448).
type Files func(owner, repo, path, ref string) ([]byte, error)
// CatalogRepository is the catalogue, whose merged message carries its predefined list, and PredefinedFile that list's
// file at its root (novox/hq ADR 0565 §2, ADR 0348 §7).
const (
CatalogRepository = "novox/mesh-catalog"
PredefinedFile = "predefined.json"
)
// LockFiles are the lock files read beside a module's manifest, by name relative to its directory (novox/hq ADR 0465
// clause 1: go.sum for a Go module, package-lock.json for a Node one).
var LockFiles = []string{"go.sum", "package-lock.json"}
@@ -315,6 +322,18 @@ func merged(subject string, p pull, trees Trees, files Files, kept Kept) (Messag
}
f := delivery.Forge{Kind: delivery.ForgeMerged, Repository: repository, Branch: p.Head, Number: p.Number,
Head: p.HeadSHA, Merge: p.MergeSHA, Root: root, Trees: dirs, Manifests: manifests, Locks: locks}
if repository == CatalogRepository {
// The catalogue's predefined list at the merge (novox/hq ADR 0565 §2): the indexer keeps it for mesh-config.
raw, err := files(p.Owner, p.Repo, PredefinedFile, p.MergeSHA)
switch {
case errors.Is(err, ErrNotFound):
f.PredefinedRead = true
case err != nil:
return Message{}, fmt.Errorf("%w: %s: %s of %s at the merge commit %s: %v", ErrRetry, subject, PredefinedFile, repository, p.MergeSHA, err)
default:
f.Predefined = json.RawMessage(raw)
}
}
body, err := delivery.EncodeForge(f)
if err != nil {
return Message{}, unreadable(subject, "%v", err)
@@ -97,6 +97,27 @@ func TestMergedCarriesManifestsAndLocks(t *testing.T) { // 1448/AC1
t.Errorf("the forge was asked for %s", c.path)
}
}
// 1953/D1 (ADR 0565 §2): the catalogue's merged message carries its predefined.json at the merge, or says it holds
// none; another repository's asks the forge for none.
if !f.PredefinedRead || len(f.Predefined) != 0 {
t.Errorf("a merge of the catalogue with no predefined.json does not say so: %s", msg.Data)
}
forge.have[PredefinedFile] = []byte(`{"font-size":{"type":"integer","value":13}}`)
msg, _ = Convert(mergedSubject, mergedBody(map[string]any{"repo": "mesh-catalog"}), treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get)
if f, err = delivery.DecodeForge(msg.Data); err != nil || string(f.Predefined) != string(forge.have[PredefinedFile]) || f.PredefinedRead {
t.Errorf("predefined = %s (%v)", f.Predefined, err)
}
forge.calls = nil
msg, _ = Convert(mergedSubject, mergedBody(nil), treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get)
for _, c := range forge.calls {
if c.path == PredefinedFile {
t.Errorf("another repository's merge asked for %s", c.path)
}
}
forge.errs = map[string]error{PredefinedFile: errors.New("forge down")}
if _, err := Convert(mergedSubject, mergedBody(map[string]any{"repo": "mesh-catalog"}), treesOf(&calls, "r0", map[string]string{"modules/a": "t1"}, nil), forge.get); !errors.Is(err, ErrRetry) {
t.Errorf("a forge that cannot say is %v, want a retry", err)
}
// Issue 1126: a merge reads the forge only for the module directories whose tree changed.
t.Run("a merge reads only the directories whose tree changed", mergeReadsOnlyChangedDirs)
}
@@ -19,6 +19,7 @@ import (
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/buscompose"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
@@ -44,21 +45,10 @@ accounts {
// manifest's state (mesh-delivery-indexer_candidates, history 10) and mesh-controller_builds holding build records, and
// no mesh-delivery-gate_source bucket.
func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10
js, ctx, candidates, _ := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) {
// Build records in the controller's layout (mesh-sdk go/delivery/testdata/build-record-build-agent.json, captured from
// the controller's encoder), under delivery.BuildKey, with the newest pointer beside them.
for _, b := range []struct{ repo, module, path, id string }{
{"novox/mesh-delivery", "a", "modules/a", "build-1"}, {"novox/mesh-catalog", "c", "modules/c", "build-2"},
} {
rec, _ := json.Marshal(map[string]any{"id": b.id, "repository": b.repo, "ref": "main", "module": b.module,
"commit": "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", "path": b.path, "at": "2026-10-19T09:38:00Z"})
if _, err := builds.Put(ctx, delivery.BuildKey(b.module, b.id), rec); err != nil {
t.Fatal(err)
}
if _, err := builds.Put(ctx, delivery.NewestKey(b.module), []byte(b.id)); err != nil {
t.Fatal(err)
}
}
js, ctx, candidates, _ := startIndexer(t, func(ctx context.Context, builds buildLog) {
// The builder's log entries of two builds' ends (ADR 0571 clause 1), as the builder appends them.
builds.end(t, "novox/mesh-delivery", "a", "modules/a")
builds.end(t, "novox/mesh-catalog", "c", "modules/c")
})
h1, h2 := "0123456789abcdef0123456789abcdef01234567", "89abcdef0123456789abcdef0123456789abcdef"
for _, f := range []delivery.Forge{
@@ -99,7 +89,7 @@ func TestIndexerWhole(t *testing.T) { // 1238/AC2, AC10
// startIndexer starts the bus, the stream FORGE with the indexer's durable consumer, the buckets, lets put fill
// mesh-controller_builds, and runs the program. mesh-delivery-gate_source is never created.
func startIndexer(t *testing.T, put func(context.Context, jetstream.KeyValue), before ...func(context.Context, jetstream.JetStream)) (jetstream.JetStream, context.Context, jetstream.KeyValue, jetstream.KeyValue) {
func startIndexer(t *testing.T, put func(context.Context, buildLog), before ...func(context.Context, jetstream.JetStream)) (jetstream.JetStream, context.Context, jetstream.KeyValue, jetstream.KeyValue) {
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "i")
if err != nil {
t.Fatal(err)
@@ -162,11 +152,12 @@ func startIndexer(t *testing.T, put func(context.Context, jetstream.KeyValue), b
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ManifestsBucket, History: delivery.ManifestsHistory}); err != nil {
t.Fatal(err)
}
builds, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.BuildsBucket, History: 5})
if err != nil {
// The builder's module log, as the runtime raises it (ADR 0297), filled by put before the indexer starts.
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: buscompose.LogStreamName(delivery.BuilderModule, delivery.BuilderLog),
Subjects: []string{buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog) + ".>"}, AllowDirect: true}); err != nil {
t.Fatal(err)
}
put(ctx, builds)
put(ctx, buildLog{ctx: ctx, js: js})
for _, b := range before {
b(ctx, js)
}
@@ -185,15 +176,32 @@ func startIndexer(t *testing.T, put func(context.Context, jetstream.KeyValue), b
return js, ctx, candidates, modules
}
// buildLog appends entries to the builder's log as the builder does.
type buildLog struct {
ctx context.Context
js jetstream.JetStream
}
// end appends the end of a done trunk build of module from repo's directory path.
func (l buildLog) end(t *testing.T, repo, module, path string) {
t.Helper()
raw, err := delivery.EncodeBuildLogEntry(delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: module, FP: "src2:" + module,
Repository: repo, Directory: path, Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", Node: "n1",
At: time.Date(2026, 10, 19, 9, 38, 0, 0, time.UTC), Outcome: delivery.BuildDone})
if err != nil {
t.Fatal(err)
}
if _, err := l.js.Publish(l.ctx, buscompose.LogSubject(delivery.BuilderModule, delivery.BuilderLog)+"."+module, raw); err != nil {
t.Fatal(err)
}
}
// TestIndexerWritesModulesAtAMerge: a module's record and its newest pointer are in mesh-controller_builds; a merged
// message on FORGE makes modules.<m> within 5 s, the fingerprint of the merge's tree and the recorded inputs.
func TestIndexerWritesModulesAtAMerge(t *testing.T) { // 1239/AC9; I1, I3: the modules are the merged message's, no build record names them (issue 1851)
// The bucket holds only build-agent's record, which no fingerprint reads since R4g (ADR 0530 §1).
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) {
rec := `{"id":"build-1","repository":"novox/mesh-controller","ref":"main","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","at":"2026-10-19T09:38:00Z"}`
if _, err := builds.Put(ctx, delivery.BuildKey("build-agent", "build-1"), []byte(rec)); err != nil {
t.Fatal(err)
}
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds buildLog) {
builds.end(t, "novox/mesh-controller", "build-agent", "modules/build-agent")
})
const merge = "89abcdef0123456789abcdef0123456789abcdef"
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-delivery", Branch: "feat/x", Number: 12,
@@ -229,15 +237,8 @@ func TestIndexerKeepsManifestsAtAMerge(t *testing.T) { // 1444/AC4, 1449/AC4
t.Fatal(err)
}
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
const inputs = `{"Module":"avahi","Tree":"oldtree","Bases":null,"Contexts":null,"Toolchains":{"code":"go sha256:ccc 0123456789abcdef"},"Unpinned":""}`
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) {
rec := `{"id":"build-1","repository":"novox/mesh-catalog","ref":"main","module":"avahi","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/avahi","at":"2026-10-19T09:38:00Z","source-inputs":` + inputs + `}`
if _, err := builds.Put(ctx, delivery.BuildKey("avahi", "build-1"), []byte(rec)); err != nil {
t.Fatal(err)
}
if _, err := builds.Put(ctx, delivery.NewestKey("avahi"), []byte("build-1")); err != nil {
t.Fatal(err)
}
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds buildLog) {
builds.end(t, "novox/mesh-catalog", "avahi", "modules/avahi")
})
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 301,
Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0",
@@ -287,15 +288,8 @@ func TestIndexerWritesAGoModuleAgainWhenTheBuildModuleReportsItsToolchain(t *tes
t.Fatal(err)
}
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
const inputs = `{"Module":"avahi","Tree":"oldtree","Bases":null,"Contexts":null,"Toolchains":{"code":"go sha256:ccc 0123456789abcdef"},"Unpinned":""}`
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) {
rec := `{"id":"build-1","repository":"novox/mesh-catalog","ref":"main","module":"avahi","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/avahi","at":"2026-10-19T09:38:00Z","source-inputs":` + inputs + `}`
if _, err := builds.Put(ctx, delivery.BuildKey("avahi", "build-1"), []byte(rec)); err != nil {
t.Fatal(err)
}
if _, err := builds.Put(ctx, delivery.NewestKey("avahi"), []byte("build-1")); err != nil {
t.Fatal(err)
}
js, ctx, _, modules := startIndexer(t, func(ctx context.Context, builds buildLog) {
builds.end(t, "novox/mesh-catalog", "avahi", "modules/avahi")
})
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 301,
Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0",
@@ -362,16 +356,9 @@ func TestIndexerGivesACandidateItsFingerprintWhenTheCheckerCreatesThePreviewsBuc
t.Fatal(err)
}
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
const inputs = `{"Module":"avahi","Tree":"oldtree","Bases":null,"Contexts":null,"Toolchains":{"code":"go sha256:ccc 0123456789abcdef"},"Unpinned":""}`
const trunk, head = "5555555555555555555555555555555555555555", "0123456789abcdef0123456789abcdef01234567"
js, ctx, candidates, _ := startIndexer(t, func(ctx context.Context, builds jetstream.KeyValue) {
rec := `{"id":"build-1","repository":"novox/mesh-catalog","ref":"main","module":"avahi","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/avahi","at":"2026-10-19T09:38:00Z","source-inputs":` + inputs + `}`
if _, err := builds.Put(ctx, delivery.BuildKey("avahi", "build-1"), []byte(rec)); err != nil {
t.Fatal(err)
}
if _, err := builds.Put(ctx, delivery.NewestKey("avahi"), []byte("build-1")); err != nil {
t.Fatal(err)
}
js, ctx, candidates, _ := startIndexer(t, func(ctx context.Context, builds buildLog) {
builds.end(t, "novox/mesh-catalog", "avahi", "modules/avahi")
}, func(ctx context.Context, js jetstream.JetStream) {
if _, err := js.CreateStream(ctx, jetstream.StreamConfig{Name: PreviewsStream, Subjects: []string{previewWrittenSubject}}); err != nil {
t.Fatal(err)
@@ -451,7 +438,7 @@ func TestAMergeOfMeshSDKChangingOnlyGoLeavesItsNamedTreeEqual(t *testing.T) {
}
objects := map[string]string{"src": "1111111111111111111111111111111111111111", "test": "2222222222222222222222222222222222222222",
"package.json": "3333333333333333333333333333333333333333", "package-lock.json": "4444444444444444444444444444444444444444", "tsconfig.json": "5555555555555555555555555555555555555555"}
js, ctx, _, modules := startIndexer(t, func(context.Context, jetstream.KeyValue) {})
js, ctx, _, modules := startIndexer(t, func(context.Context, buildLog) {})
publish := func(merge, root string, objects map[string]string) {
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-sdk", Branch: "feat/x", Number: 70,
Head: "0123456789abcdef0123456789abcdef01234567", Merge: merge, Root: root, Trees: map[string]string{"": root},
@@ -518,7 +505,7 @@ func TestIndexerJudgesAnUnreadableVerdictAgainAtStart(t *testing.T) { // 1923/AC
}
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
digest := delivery.ManifestDigest(manifest)
_, ctx, _, modules := startIndexer(t, func(context.Context, jetstream.KeyValue) {}, func(ctx context.Context, js jetstream.JetStream) {
_, ctx, _, modules := startIndexer(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
mkey, err := delivery.ManifestsKey(digest)
if err != nil {
t.Fatal(err)
@@ -578,23 +565,39 @@ func TestIndexerJudgesAnUnreadableVerdictAgainAtStart(t *testing.T) { // 1923/AC
}
}
// TestIndexerCountsOnlyTheAssignedBuildNodesToolchains (issue 1940): the toolchains bucket holds the report of n2, where
// mesh-build-go was unassigned (its assignment ended) at another version than n1's, as ace's v26.12.0 stayed beside the
// v26.12.1 of the nodes mesh-build-node runs on. The indexer gives the merged module a fingerprint from n1's toolchain
// alone; when the controller assigns mesh-build-go to n2 again, its report counts and the builders disagree (the guard
// stays); when the assignment ends again, the fingerprint comes back. No restart, no timer: the assignments' watch.
func TestIndexerCountsOnlyTheAssignedBuildNodesToolchains(t *testing.T) {
// TestIndexerCountsOnlyThePlacedBuildNodesToolchains (issue 1940, issue 774): the toolchains bucket holds the report of
// n2, where the catalogue no longer places mesh-build-go, at another version than n1's, as ace's v26.12.0 stayed beside
// the v26.12.1 of the nodes mesh-build-node runs on. The indexer gives the merged module a fingerprint from n1's
// toolchain alone; when the placement puts mesh-build-go on n2 again, its report written since counts and the builders
// disagree (the guard stays); when it is taken off again, the fingerprint comes back. No restart, no timer: the
// placement's watch (ADR 0549 §3), not the controller's assignments.
func TestIndexerCountsOnlyThePlacedBuildNodesToolchains(t *testing.T) {
manifest, err := os.ReadFile("../../internal/index/testdata/catalogue-a67f0196-avahi.module.json")
if err != nil {
t.Fatal(err)
}
const goSum = "3b5028f0c1e9bde04dd766abd63eb5931bc516bae695cf46f00abcd8510b0381"
var assignments jetstream.KeyValue
js, ctx, _, modules := startIndexer(t, func(context.Context, jetstream.KeyValue) {}, func(ctx context.Context, js jetstream.JetStream) {
var placement, toolchains jetstream.KeyValue
place := func(ctx context.Context, t *testing.T, node string, modules ...string) {
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
key, _ := delivery.PlacementKey(node)
value, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
if _, err := placement.Put(ctx, key, value); err != nil {
t.Fatal(err)
}
}
js, ctx, _, modules := startIndexer(t, func(context.Context, buildLog) {}, func(ctx context.Context, js jetstream.JetStream) {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.ToolchainsBucket("mesh-build-go"), History: 1})
if err != nil {
t.Fatal(err)
}
toolchains = kv
for node, version := range map[string]string{"n1": "go1.27.3", "n2": "go1.27.2"} {
key, _ := delivery.ToolchainKey(node, "go")
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "go", Version: version, Recipe: "1", At: time.Now()})
@@ -605,18 +608,12 @@ func TestIndexerCountsOnlyTheAssignedBuildNodesToolchains(t *testing.T) {
t.Fatal(err)
}
}
// The controller's assignments mirror, in its layout (mesh-controller internal/chain Assignment).
if assignments, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: "mesh-controller_assignments", History: 1}); err != nil {
// The seat mesh-placement's placement, as the assigner writes it (ADR 0567): n1 places mesh-build-go, n2 nothing.
if placement, err = js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: delivery.PlacementBucket, History: delivery.PlacementHistory}); err != nil {
t.Fatal(err)
}
for key, value := range map[string]string{
"n1.mesh-build-go": `{"node":"n1","module":"mesh-build-go","assigned":"2026-10-20T08:00:00Z"}`,
"n2.mesh-build-go": `{"node":"n2","module":"mesh-build-go","assigned":"2026-10-20T08:00:00Z","ended":"2026-10-23T08:00:00Z"}`,
} {
if _, err := assignments.Put(ctx, key, []byte(value)); err != nil {
t.Fatal(err)
}
}
place(ctx, t, "n1", "mesh-build-go")
place(ctx, t, "n2")
})
data, err := delivery.EncodeForge(delivery.Forge{Kind: delivery.ForgeMerged, Repository: "novox/mesh-catalog", Branch: "feat/x", Number: 302,
Head: "0123456789abcdef0123456789abcdef01234567", Merge: "89abcdef0123456789abcdef0123456789abcdef", Root: "r0",
@@ -647,16 +644,22 @@ func TestIndexerCountsOnlyTheAssignedBuildNodesToolchains(t *testing.T) {
if !agreed(held) {
t.Fatalf("within 5 s avahi has fp2 %q, none2 %q, toolchains %v: want a fingerprint of n1's go1.27.3 alone", held.FP2, held.None2, held.Toolchains)
}
if _, err := assignments.Put(ctx, "n2.mesh-build-go", []byte(`{"node":"n2","module":"mesh-build-go","assigned":"2026-10-24T08:00:00Z"}`)); err != nil {
place(ctx, t, "n2", "mesh-build-go")
key, _ := delivery.ToolchainKey("n2", "go")
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "go", Version: "go1.27.2", Recipe: "1", At: time.Now().Add(time.Second)})
if err != nil {
t.Fatal(err)
}
// n2's report, written since its assignment, counts: the builders disagree. A fingerprint already computed is kept
if _, err := toolchains.Put(ctx, key, value); err != nil {
t.Fatal(err)
}
// n2's report, written since its placement, counts: the builders disagree. A fingerprint already computed is kept
// (issue 1940), and only a new one needs the agreement: a merge moving avahi gets none until they agree.
time.Sleep(500 * time.Millisecond)
if e, err := modules.Get(ctx, "avahi"); err != nil {
t.Fatal(err)
} else if m, err := delivery.DecodeModule(e.Value()); err != nil || m.FP2 != held.FP2 {
t.Fatalf("after n2's assignment avahi has fp2 %q, none2 %q: want its fingerprint %q kept", m.FP2, m.None2, held.FP2)
t.Fatalf("after n2's placement avahi has fp2 %q, none2 %q: want its fingerprint %q kept", m.FP2, m.None2, held.FP2)
}
moved := strings.Replace(string(data), `"newtree"`, `"movedtree"`, 1)
if moved == string(data) {
@@ -669,12 +672,10 @@ func TestIndexerCountsOnlyTheAssignedBuildNodesToolchains(t *testing.T) {
return m.FP2 == "" && strings.HasPrefix(m.None2, "builders disagree: go1.27.2 recipe 1 on n2")
}
if m := read(disagree); !disagree(m) {
t.Fatalf("within 5 s of a merge after n2's assignment avahi has fp2 %q, none2 %q: want the builders to disagree", m.FP2, m.None2)
}
if _, err := assignments.Put(ctx, "n2.mesh-build-go", []byte(`{"node":"n2","module":"mesh-build-go","ended":"2026-10-24T09:00:00Z"}`)); err != nil {
t.Fatal(err)
t.Fatalf("within 5 s of a merge after n2's placement avahi has fp2 %q, none2 %q: want the builders to disagree", m.FP2, m.None2)
}
place(ctx, t, "n2")
if m := read(agreed); !agreed(m) {
t.Fatalf("within 5 s of n2's unassignment avahi has fp2 %q, none2 %q", m.FP2, m.None2)
t.Fatalf("within 5 s of n2's placement taken away avahi has fp2 %q, none2 %q", m.FP2, m.None2)
}
}
@@ -10,20 +10,22 @@ package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"log"
"os"
"os/signal"
"sync/atomic"
"syscall"
"time"
"github.com/nats-io/nats.go"
"github.com/nats-io/nats.go/jetstream"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
"git.novox.be/novox/mesh-sdk/go/process"
"mesh-delivery-indexer/internal/index"
@@ -273,28 +275,15 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
// Its taken state (ADR 0556 clause 1) is bound when it appears, as every state a delivery adds (issue 1790): until then
// a merge's taken key is said and not written, and the rest of the indexer runs.
taken := process.AwaitBucket(ctx, js, conn, delivery.TakenBucket, "no repository's taken key is written", logger.Printf)
// The controller's build records only name the directory of a module whose key names none (fillPaths): without them
// such a module waits for its path, every other is indexed (issue 1945).
builds := process.OptionalBucket(ctx, js, delivery.BuildsBucket, "a module whose key names no directory is not given one from the controller's build records until the indexer starts again", logger.Printf)
// The control records (ADR 0565 §2), new with this delivery and bound when they appear: until then nothing is written to them.
predefined := process.AwaitBucket(ctx, js, conn, index.PredefinedBucket, "no predefined list is written", logger.Printf)
configCandidates := process.AwaitBucket(ctx, js, conn, index.ConfigCandidatesBucket, "no pull request's config declarations are written", logger.Printf)
// The builder's log (ADR 0571 clause 1), read on the indexer's own credential (the calls connection) through its declared read: every entry it
// keeps at the start, then each new one as the builder appends it (no timer). It names the directory of a module whose key
// names none (fillPaths) and the places a registered module was built from; no build moves a fingerprint (ADR 0530 §1).
// Where the log cannot be read such a module waits for its path, every other is indexed (issue 1945), and it is said.
known := index.NewBuilds()
// The controller's build records name the directory of a module whose key names none (fillPaths); no build of any
// module moves a fingerprint since no fingerprint takes the builder (ADR 0530 §1).
each := func(e jetstream.KeyValueEntry) {
if opOf(e) {
known.Put(e.Key(), e.Value())
} else {
known.Delete(e.Key())
}
}
errs := make(chan error, 1)
var buildsWatch jetstream.KeyWatcher
if builds != nil {
if w, err := settle(ctx, builds, each); err != nil {
process.NotWatched(logger.Printf, delivery.BuildsBucket, err, "a module whose key names no directory is not given one until the indexer starts again")
} else {
buildsWatch = w
}
}
// The indexer asks the forge for the objects of a manifest's named paths as its own account (ADR 0499 clause 3).
calls, err := process.Connect(cred, nats.Name(cred.Node+"."+cred.Module+".calls"), nats.RetryOnFailedConnect(true))
@@ -302,9 +291,29 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err
}
defer calls.Close()
ix := &index.Indexer{Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Taken: awaitedStore{taken}, Builds: known, Logf: logger.Printf,
ix := &index.Indexer{Store: kvStore{candidates}, Modules: kvStore{modules}, Manifests: kvStore{manifests}, Taken: awaitedStore{taken}, Builds: known,
Predefined: awaitedStore{predefined}, ConfigCandidates: awaitedStore{configCandidates}, Logf: logger.Printf,
Named: namedVia(calls.Call), Trunk: trunkVia(calls.Call), Contains: containsVia(calls.Call), Read: readVia(calls.Call),
ReadOpen: openVia(calls.Call)}
// The log's entries at the start are read before any fingerprint is computed or registered module indexed (below); each
// new one indexes the module it names when that module is registered and the bucket holds none (registered.go).
var started atomic.Bool
if err := calls.ReadLog(delivery.BuilderModule, delivery.BuilderLog).Watch("", stdio.LogFromAll, func(le stdio.LogEntry) error {
if err := known.Put(le.Value); err != nil {
logger.Printf("%s entry %d is not read: %v", delivery.BuildLogRead, le.Sequence, err)
return nil
}
if started.Load() {
if err := ix.IndexRegisteredModule(ctx, le.Key); err != nil {
logger.Printf("the registered module %s is not indexed: %v", le.Key, err)
}
}
return nil
}); err != nil {
logger.Printf("%s is not read (%v): a module whose key names no directory is not given one, and no registered module is "+
"indexed from it, until the indexer starts again", delivery.BuildLogRead, err)
}
started.Store(true)
// The gate's source bucket (mesh-delivery-gate_source) comes with the gate (release 3e, novox/hq ADR 0450 §1): its key
// main is the mesh's source revision, which stamps the modules (issue 1344). Where it does not exist yet the indexer does
// not wait for it (that needs a timer, ADR 0437 §8): it says so, and is started again with the gate.
@@ -321,14 +330,14 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
go func() { sourceErrs <- follow(ctx, sw, each) }()
}
}
// Only the toolchain reports of nodes the build module is assigned to count (issue 1940): the controller's
// assignments are settled first, so the toolchains settle through the gate already knowing them.
gate := newToolchainGate(sortedModules(), ix.PutToolchain, logger.Printf)
assignErrs, err := watchAssignments(ctx, js, gate, func(ctx context.Context) {
// Only the toolchain reports of nodes the catalogue places the build module on count (issue 1940, ADR 0549 §3): the
// placement is settled first, so the toolchains settle through the gate already knowing it.
gate := buildrun.NewToolchainGate(sortedModules(), ix.PutToolchain, logger.Printf)
placementErrs, err := buildrun.FollowPlacement(ctx, js, gate, func(ctx context.Context) {
if err := ix.Recompute(ctx); err != nil {
logger.Printf("writing the modules again after an assignment of a build module changed: %v", err)
logger.Printf("writing the modules again after the placement of a build module changed: %v", err)
}
}, logger)
}, logger.Printf)
if err != nil {
return err
}
@@ -355,28 +364,11 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
if err := ix.RecomputeAtStart(ctx); err != nil {
logger.Printf("computing the new-form fingerprints at start: %v", err)
}
// A module registered in the catalogue from its own repository gets its manifest indexed when the controller's records
// name it and the bucket holds none (registered.go); the records that follow are heard below.
// A module registered in the catalogue from its own repository gets its manifest indexed when the builder's log names
// it and the bucket holds none (registered.go); the entries that follow are heard above.
if err := ix.IndexRegisteredAll(ctx); err != nil {
logger.Printf("indexing the registered modules at start: %v", err)
}
if buildsWatch != nil {
go func() {
errs <- follow(ctx, buildsWatch, func(e jetstream.KeyValueEntry) {
each(e)
if !opOf(e) {
return
}
var r delivery.BuildRecord
if json.Unmarshal(e.Value(), &r) != nil || r.Module == "" {
return
}
if err := ix.IndexRegisteredModule(ctx, r.Module); err != nil {
logger.Printf("the registered module %s is not indexed: %v", r.Module, err)
}
})
}()
}
// A person takes a registered module in now with indexer_take_in, which writes the takeins state (issue 1945): watched,
// as every other input. A module the forge could not read is also read again by itself on the next source revision and
// merged message (index.RetryUnread).
@@ -442,7 +434,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err
case err := <-sourceErrs:
return err
case err := <-assignErrs:
case err := <-placementErrs:
return err
case err := <-tools.errs:
return err
@@ -1,211 +0,0 @@
package main
// This file is the same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker (issue 1940): it moves
// to mesh-sdk go/delivery/buildrun once mesh-sdk can be delivered again, since a change of mesh-sdk waits on the very
// toolchain agreement this file repairs.
import (
"context"
"encoding/json"
"fmt"
"log"
"maps"
"slices"
"strings"
"sync"
"time"
"github.com/nats-io/nats.go/jetstream"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/provider"
)
// A build module writes <node>.<language> of its toolchains state on every node it runs on, and nothing deletes the key
// when the module is unassigned from a node: the controller writes no module's state, and the module is gone. A report of
// a node the build module no longer runs on is therefore not a builder (issue 1940: mesh-build-node unassigned from ace
// on 2026-10-23 left ace.typescript at v26.12.0, and every Node fingerprint said "builders disagree").
//
// toolchainGate stands between the toolchains buckets and what takes their entries in (the indexer's PutToolchain, a
// buildrun.ToolchainSet): an entry is passed on only while the controller's assignments (mesh-controller_assignments,
// key <node>.<module>) hold an open assignment of its build module on its node, and only once it was written since that
// assignment; it is passed on as deleted otherwise, said once, and passed on when its node writes it again. A report
// written before the assignment is the one an earlier run of the module left (issue 1940, 2026-10-25 20:56: mesh-build-node
// assigned to ace again brought ace.typescript v26.12.0 of 2026-10-23 back into the agreement, before the module ran
// there, and the controller's take-in refused every module "builders disagree"). Until the assignments are settled (or where they cannot
// be read) every entry is passed on, as before. Builders that run and report different versions still disagree: that
// refusal is the guard of novox/hq ADR 0465, and it stays.
type toolchainGate struct {
mu sync.Mutex
modules map[string]bool // the build modules whose toolchains pass the gate
forward func(module, key string, value []byte, deleted bool) bool // answers whether it changed what it holds
logf func(format string, a ...any)
known bool // the assignments are settled
standing map[string]time.Time // <node>.<module> with an open assignment -> since when it stands
raw map[string]map[string][]byte // module -> toolchains key -> value, as the bucket holds it
left map[string]string // <module>/<key> -> why it was said left out, so each reason is said once
}
func newToolchainGate(modules []string, forward func(module, key string, value []byte, deleted bool) bool, logf func(string, ...any)) *toolchainGate {
g := &toolchainGate{modules: map[string]bool{}, forward: forward, logf: logf, standing: map[string]time.Time{},
raw: map[string]map[string][]byte{}, left: map[string]string{}}
for _, m := range modules {
g.modules[m] = true
}
return g
}
// Toolchain takes one change of a build module's toolchains bucket in and answers whether what it passed on changed.
func (g *toolchainGate) Toolchain(module, key string, value []byte, deleted bool) bool {
g.mu.Lock()
defer g.mu.Unlock()
if g.raw[module] == nil {
g.raw[module] = map[string][]byte{}
}
if deleted {
delete(g.raw[module], key)
} else {
g.raw[module][key] = value
}
return g.pass(module, key)
}
// Assignment takes one change of the controller's assignments in (at is when the bus wrote that revision) and answers
// whether what it passed on changed.
func (g *toolchainGate) Assignment(key string, value []byte, deleted bool, at time.Time) bool {
node, module, ok := strings.Cut(key, ".")
if !ok || !g.modules[module] {
return false
}
assigned, open := openAssignment(value)
open = open && !deleted
g.mu.Lock()
defer g.mu.Unlock()
was, stood := g.standing[key]
since := assigned
if since.IsZero() {
// A value with no time of its own (an assignment a module pulled in) stands since the revision that opened it.
since = at
if stood {
since = was
}
}
if open == stood && (!open || since.Equal(was)) {
return false
}
if open {
g.standing[key] = since
} else {
delete(g.standing, key)
}
if !g.known {
return false
}
changed := false
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
if n, _, err := delivery.ParseToolchainKey(k); err == nil && n == node {
changed = g.pass(module, k) || changed
}
}
return changed
}
// Settled says the assignments are read: from now on only the reports of assigned nodes pass. It answers whether what it
// passed on changed.
func (g *toolchainGate) Settled() bool {
g.mu.Lock()
defer g.mu.Unlock()
g.known = true
changed := false
for _, module := range slices.Sorted(maps.Keys(g.raw)) {
for _, k := range slices.Sorted(maps.Keys(g.raw[module])) {
changed = g.pass(module, k) || changed
}
}
return changed
}
// pass hands one key on as it stands: its value when its node is assigned and wrote it since (or the assignments are not
// known), deleted otherwise. The caller holds mu.
func (g *toolchainGate) pass(module, key string) bool {
value, held := g.raw[module][key]
if !held {
return g.forward(module, key, nil, true)
}
node, _, err := delivery.ParseToolchainKey(key)
why := ""
if err == nil && g.known {
why = g.leftOut(module, node, value)
}
if why == "" {
delete(g.left, module+"/"+key)
return g.forward(module, key, value, false)
}
if g.left[module+"/"+key] != why {
g.left[module+"/"+key] = why
g.logf("%s.%s is left out of the agreed toolchain: %s (issue 1940)", delivery.ToolchainsBucket(module), key, why)
}
return g.forward(module, key, nil, true)
}
// leftOut is why a node's report is no builder's, or empty when it counts: its node has no open assignment of the module,
// or the report was written before that assignment. A value that is no toolchain is passed on for its reader to refuse.
// The caller holds mu.
func (g *toolchainGate) leftOut(module, node string, value []byte) string {
since, open := g.standing[node+"."+module]
if !open {
return fmt.Sprintf("%s has no assignment on %s, so it is no builder", module, node)
}
tc, err := delivery.DecodeToolchain(value)
if err == nil && tc.At.Before(since) {
return fmt.Sprintf("it was written at %s, before %s's assignment on %s at %s: it counts once %s writes it again",
tc.At.UTC().Format(time.RFC3339), module, node, since.UTC().Format(time.RFC3339), node)
}
return ""
}
// openAssignment is whether a value of the controller's assignments holds an assignment that stands (one that has not
// ended and has no state: a pending one is not on the machine yet; mesh-controller internal/chain Assignment), and when
// it was assigned, zero where the value does not say.
func openAssignment(value []byte) (time.Time, bool) {
var a struct {
Assigned time.Time `json:"assigned"`
Ended time.Time `json:"ended"`
State string `json:"state"`
}
if json.Unmarshal(value, &a) != nil {
return time.Time{}, false
}
return a.Assigned, a.Ended.IsZero() && a.State == ""
}
// watchAssignments settles the controller's assignments into the gate, says them settled, and follows them; changed is
// told whenever what the gate passes on changed. Where the bucket cannot be read (not there, or this module may not read
// it) it says so and the gate passes every report on, as before; an error of the watch after that ends on the channel.
func watchAssignments(ctx context.Context, js bucketOpener, gate *toolchainGate, changed func(context.Context), logger *log.Logger) (<-chan error, error) {
errs := make(chan error, 1)
kv, err := js.KeyValue(ctx, provider.AssignmentsBucket)
if err != nil {
// Not fatal: an indexer, builder or checker that stops for it stops every delivery (a grant comes with a delivery).
logger.Printf("the bucket %s cannot be read, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
each := func(e jetstream.KeyValueEntry) { gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) }
w, err := settle(ctx, kv, each)
if err != nil {
// Not fatal either (issue 1945): a watch the bus refuses is the same missing grant as a refused read.
logger.Printf("the bucket %s cannot be watched, so every node's toolchain report counts, an unassigned build node's too (issue 1940): %v", provider.AssignmentsBucket, err)
return errs, nil
}
if gate.Settled() {
changed(ctx)
}
go func() {
errs <- follow(ctx, w, func(e jetstream.KeyValueEntry) {
if gate.Assignment(e.Key(), e.Value(), !opOf(e), e.Created()) {
changed(ctx)
}
})
}()
return errs, nil
}
@@ -1,165 +0,0 @@
package main
// The same in mesh-delivery-indexer, mesh-delivery-builder and mesh-delivery-checker, as standing.go is (issue 1940).
import (
"fmt"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/delivery/buildrun"
)
func gateReport(t *testing.T, node, version string) (string, []byte) {
t.Helper()
key, err := delivery.ToolchainKey(node, "typescript")
if err != nil {
t.Fatal(err)
}
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: time.Unix(0, 0).UTC()})
if err != nil {
t.Fatal(err)
}
return key, value
}
func agreedOf(set *buildrun.ToolchainSet) (string, string) {
tc, none := delivery.AgreedToolchain(set.Entries("typescript"))
return tc.Version, none
}
// The live case of issue 1940: mesh-build-node was unassigned from ace, whose report stayed at v26.12.0 while the
// three nodes it runs on report v26.12.1. Without the assignments every report counts (and they disagree); with them
// ace's is left out and the three agree; assigned to ace again, its report counts again and the guard refuses again.
func TestAnUnassignedBuildNodesReportIsNoBuilder(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.GoBuildModule, buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, f) })
for node, version := range map[string]string{"ace": "v26.12.0", "g14": "v26.12.1", "novox": "v26.12.1", "shanks": "v26.12.1"} {
key, value := gateReport(t, node, version)
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree") {
t.Fatalf("before the assignments are read every report counts, so they disagree; got %q", none)
}
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+"."+buildrun.NodeBuildModule, []byte(`{"node":"`+node+`","module":"mesh-build-node"}`), false, time.Time{})
}
gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","ended":"2026-10-23T08:00:00Z"}`), false, time.Time{})
gate.Assignment("ace.mesh-build-go", []byte(`{"node":"ace","module":"mesh-build-go"}`), false, time.Time{}) // another module's: no matter
if !gate.Settled() {
t.Fatal("settling the assignments left ace's report out, so what was passed on changed")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("the assigned nodes agree on v26.12.1; got %q, %q", v, none)
}
if len(said) != 1 {
t.Fatalf("ace's report is said left out once; said %v", said)
}
// ace reports again while unassigned (an old value seen again): still left out.
key, value := gateReport(t, "ace", "v26.12.0")
if gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("an unassigned node's report changed what was passed on")
}
// Assigned to ace again: its report counts, and the builders really disagree: the guard stays.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node"}`), false, time.Time{}) {
t.Fatal("assigning ace again passed nothing on")
}
if _, none := agreedOf(set); !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("builders that run different versions still disagree; got %q", none)
}
// The assignment's key deleted, or a pending one written: no assignment stands, and ace's report goes again.
if !gate.Assignment("ace.mesh-build-node", []byte(`{"node":"ace","module":"mesh-build-node","state":"pending"}`), false, time.Time{}) {
t.Fatal("a pending assignment is no assignment on the machine")
}
if v, none := agreedOf(set); v != "v26.12.1" || none != "" {
t.Fatalf("got %q, %q", v, none)
}
gate.Assignment("ace.mesh-build-node", nil, true, time.Time{})
if v, _ := agreedOf(set); v != "v26.12.1" {
t.Fatalf("got %q", v)
}
// A report deleted from the bucket goes whether its node is assigned or not.
key, _ = gateReport(t, "g14", "v26.12.1")
if !gate.Toolchain(buildrun.NodeBuildModule, key, nil, true) {
t.Fatal("deleting g14's report passed nothing on")
}
if n := len(set.Entries("typescript")); n != 2 {
t.Fatalf("novox and shanks are left; got %d entries", n)
}
}
// A build node's report counts from the first one it writes since its assignment: a newly assigned node that has not
// reported changes nothing, and the report an earlier run left on the bus (issue 1940, 2026-10-25 20:56: ace.typescript
// v26.12.0 of 2026-10-23 06:29, mesh-build-node assigned to ace again at 20:56:45) is left out, said once, until the node
// writes it again. An assignment whose value names no time stands since the revision that opened it.
func TestANewlyAssignedNodeCountsFromItsReport(t *testing.T) {
left := time.Date(2026, 10, 23, 6, 29, 18, 0, time.UTC) // ace's report from its earlier run
assigned := time.Date(2026, 10, 25, 19, 56, 45, 0, time.UTC) // mesh-build-node assigned to ace again
report := func(t *testing.T, node, version string, at time.Time) (string, []byte) {
key, _ := gateReport(t, node, version)
value, err := delivery.EncodeToolchain(delivery.Toolchain{Language: "typescript", Version: version, Recipe: "f59571cf4be75adf", At: at})
if err != nil {
t.Fatal(err)
}
return key, value
}
for _, c := range []struct {
name string
assignment string // ace's assignment value
revision time.Time // when the bus wrote it
at time.Time // when ace wrote its report
counts bool
}{
{"no report yet, then one", `{}`, time.Time{}, time.Now(), true},
{"a report left before the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, left, false},
{"a report left before a value with no time", `{"node":"ace","module":"mesh-build-node"}`, assigned, left, false},
{"a report written since the assignment", `{"node":"ace","module":"mesh-build-node","assigned":"2026-10-25T19:56:45Z"}`, assigned, assigned.Add(30 * time.Second), true},
} {
t.Run(c.name, func(t *testing.T) {
set := &buildrun.ToolchainSet{}
var said []string
gate := newToolchainGate([]string{buildrun.NodeBuildModule}, set.Put, func(f string, a ...any) { said = append(said, fmt.Sprintf(f, a...)) })
for _, node := range []string{"g14", "novox", "shanks"} {
gate.Assignment(node+".mesh-build-node", []byte(`{}`), false, time.Time{})
key, value := gateReport(t, node, "v26.12.1")
gate.Toolchain(buildrun.NodeBuildModule, key, value, false)
}
gate.Settled()
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision) {
t.Fatal("an assignment with no report passed something on")
}
key, value := report(t, "ace", "v26.12.0", c.at)
if got := gate.Toolchain(buildrun.NodeBuildModule, key, value, false); got != c.counts {
t.Fatalf("ace's report passed on: %v, want %v", got, c.counts)
}
// A rewrite of the assignment that does not move it changes nothing, and nothing is said again.
if gate.Assignment("ace.mesh-build-node", []byte(c.assignment), false, c.revision.Add(time.Minute)) {
t.Fatal("the same assignment written again changed what was passed on")
}
v, none := agreedOf(set)
if c.counts {
if !strings.HasPrefix(none, "builders disagree: v26.12.0 recipe f59571cf4be75adf on ace") {
t.Fatalf("ace's own report counts and disagrees; got %q, %q", v, none)
}
return
}
if v != "v26.12.1" || none != "" {
t.Fatalf("ace's left report is no builder's, so g14, novox and shanks agree; got %q, %q", v, none)
}
if len(said) != 1 || !strings.Contains(said[0], "before mesh-build-node's assignment on ace at 2026-10-25T19:56:45Z") {
t.Fatalf("said %q, want once why ace's report is left out", said)
}
// ace writes its report again since it was assigned: it counts.
key, value = report(t, "ace", "v26.12.1", assigned.Add(time.Minute))
if !gate.Toolchain(buildrun.NodeBuildModule, key, value, false) {
t.Fatal("ace's new report was not passed on")
}
if n := len(set.Entries("typescript")); n != 4 {
t.Fatalf("four builders report; got %d", n)
}
})
}
}
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-indexer
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771 h1:Tt16HJk8JZRce1i3JzzDSNdCWOrv7dw8FQ/bckNQ5Ig=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261029233535-78c9ec881771/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -1,72 +1,63 @@
package index
import (
"encoding/json"
"fmt"
"maps"
"slices"
"strings"
"sync"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
// Builds holds the controller's build records (mesh-controller_builds) for the two things the indexer still reads from
// them until release R4g, when build-agent goes (the lead's decision of R4e, 2026-10-22):
// - the Builder input of the builder's own module, which build-agent builds (TrunkOf; novox/hq ADR 0453 clause 5,
// issue 1727);
// - the directory of a module whose modules key has none (PathOf; issue 1614), filled once and changing no fingerprint.
// Builds holds the builder's log (mesh-delivery-builder.build-log, novox/hq ADR 0571 clause 1) for what the indexer reads
// of past builds: the directory of a module whose modules key has none (PathOf; issue 1614), filled once and changing no
// fingerprint, and the places a module registered from its own repository was built from (RegistrationsOf). It is filled
// from the log's entries at the indexer's start (every one the log keeps, 30 days) and then each new one as the builder
// appends it; no entry is ever taken out, since only age takes one from the log.
//
// It lists no module and maps no directory to a module: the indexer's modules are those its own messages and bucket name
// (ADR 0501 clause 6, issue 1851). Safe for use from a watch and from the message handler at once.
type Builds struct {
mu sync.Mutex
records map[string]delivery.BuildRecord // by bucket key
records map[string]delivery.BuildRecord // by the entry's event, module, fingerprint and time
}
// NewBuilds is an empty Builds.
func NewBuilds() *Builds { return &Builds{records: map[string]delivery.BuildRecord{}} }
// Put takes a key of the builds bucket and its value. Only a build record of a module (key `<module>.build.<id>`,
// delivery.BuildKey) counts; the newest-pointer keys and requests that knew no module are not records of one.
func (b *Builds) Put(key string, value []byte) {
if !strings.Contains(key, ".build.") {
return
// Put takes one entry of the builder's log. A build's end is an outcome: done registers, failed or queued does not; its
// start is a request, which registers nothing either. An entry that does not read, or names no module or repository, is
// refused and said.
func (b *Builds) Put(value []byte) error {
e, err := delivery.DecodeBuildLogEntry(value)
if err != nil {
return err
}
var r delivery.BuildRecord
if err := json.Unmarshal(value, &r); err != nil || r.Module == "" || r.Repository == "" {
return
if e.Module == "" || e.Repository == "" {
return fmt.Errorf("a build log entry naming no module or repository: %s", value)
}
r := delivery.BuildRecord{ID: e.At.UTC().Format(time.RFC3339Nano) + "." + e.FP, Repository: e.Repository, Ref: e.Ref, Module: e.Module,
Path: e.Directory, Commit: e.Commit, At: e.At}
switch {
case e.Event != delivery.BuildEndedEvent:
r.State = e.Event
case e.Outcome == delivery.BuildFailed:
r.Failed = "the build failed"
case e.Outcome != delivery.BuildDone:
r.State = e.Outcome
}
b.mu.Lock()
defer b.mu.Unlock()
b.records[key] = r
}
// Delete takes a key out.
func (b *Builds) Delete(key string) {
b.mu.Lock()
defer b.mu.Unlock()
delete(b.records, key)
b.records[e.Event+"."+e.Module+"."+r.ID] = r
return nil
}
func newer(a, b delivery.BuildRecord) bool {
return a.At.After(b.At) || (a.At.Equal(b.At) && a.ID > b.ID)
}
// TrunkOf is the newest successful trunk build record of a module, in whatever repository it was built from; nil when
// there is none (issue 1727).
func (b *Builds) TrunkOf(module string) *delivery.BuildRecord {
var best *delivery.BuildRecord
b.mu.Lock()
defer b.mu.Unlock()
for _, r := range b.records {
if r.Module == module && r.IsTrunkBuild() && (best == nil || newer(r, *best)) {
c := r
best = &c
}
}
return best
}
// PathOf is the directory of a module of the repository as its newest build record of that repository names it, its
// newest successful trunk build's first ("." for the repository's root), and false when no record of the repository
// names the module (issue 1614).
@@ -79,7 +70,7 @@ func (b *Builds) PathOf(module, repository string) (string, bool) {
if r.Module != module || !delivery.SameRepository(r.Repository, repository) {
continue
}
t := r.IsTrunkBuild()
t := registered(r) && trunkLike[r.Ref]
if best == nil || (t && !trunk) || (t == trunk && newer(r, *best)) {
c := r
best, trunk = &c, t
@@ -91,15 +82,14 @@ func (b *Builds) PathOf(module, repository string) (string, bool) {
return pathOf(strings.Trim(best.Path, "/")), true
}
// Registration is a module registered in the catalogue from its own repository, as the controller's newest outcome record
// Registration is a module registered in the catalogue from its own repository, as the builder log.s newest end of a build
// of it names it: the repository, the directory of its manifest there ("" or "." for the root) and the commit it was
// built at.
type Registration struct {
Module, Repository, Path, Commit, Ref string
}
// registered is whether the record is an outcome that did not fail: what registers a module. Unlike IsTrunkBuild it
// asks for no recorded source inputs, which no build records since ADR 0530 clause 1, and for no particular branch: a
// registered is whether the record is an outcome that did not fail: what registers a module. It asks for no particular branch: a
// repository's trunk is named main, master, or what its owner chose (the registered photos, de-spiegel and link2pay were
// built from nox-mesh, and every record of them names that ref).
func registered(r delivery.BuildRecord) bool {
@@ -48,6 +48,8 @@ type Indexer struct {
Modules Store
// Manifests is the manifests bucket; nil writes no manifests (issue 1449).
Manifests Store
// Predefined and ConfigCandidates are the control records' states (control_records.go, ADR 0565 §2); nil writes none.
Predefined, ConfigCandidates Store
// Taken is the taken bucket: per repository the last merge taken in (taken.go, ADR 0556); nil writes none.
Taken Store
// Now is the clock of a taken value; nil is time.Now.
@@ -121,6 +123,14 @@ func (ix *Indexer) ApplyMessage(ctx context.Context, f delivery.Forge) (Applied,
if err := ix.writeTakenKnown(ctx, f, w.asked); err != nil {
return applied, err
}
// The control records (ADR 0565 §2): the catalogue's predefined list, and the open pull requests' config
// declarations judged again against the trunk this merge moved.
if err := ix.writePredefined(ctx, f); err != nil {
return applied, err
}
if err := ix.configCandidatesAfterMerge(ctx, f, writtenModules(w.written)); err != nil {
return applied, err
}
}
ix.kickRetry(ctx) // a registered module the forge could not read is read again on a merge, in the background (issue 1945)
}
@@ -134,6 +144,12 @@ func (ix *Indexer) ApplyMessage(ctx context.Context, f delivery.Forge) (Applied,
if held, err = ix.heldModules(ctx); err != nil {
return applied, err
}
if err := ix.configCandidateOfPull(ctx, f, held); err != nil {
if !errors.Is(err, ErrUnreadable) {
return applied, err
}
ix.logf("%s#%d: its config declarations are not written: %v", f.Repository, f.Number, err)
}
}
if trunkLike[f.Branch] {
ix.logf("%s#%d: its branch %q is no delivery group, so no candidate", f.Repository, f.Number, f.Branch)
@@ -0,0 +1,292 @@
package index
import (
"context"
"encoding/json"
"errors"
"fmt"
"reflect"
"strconv"
"strings"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// The indexer's two control records (novox/hq ADR 0565 §2, issue 1953 D2, D3): the catalogue's predefined list, read from the
// forge bridge's merged message, and each open pull request's config declarations, read from its pull message's manifests
// against the trunk's manifests the indexer holds. mesh-config reads both in place of the controller's mirror and its
// `pull-declares`; a key's write is its event, nothing polls.
// The two states' buckets.
var (
PredefinedBucket = delivery.IndexerModule + "_" + delivery.IndexerPredefinedState
ConfigCandidatesBucket = delivery.IndexerModule + "_" + delivery.IndexerConfigCandidatesState
)
// writePredefined keeps the predefined list a merged message carries, unless the key names a later merge: a merge older
// than the one it names never replaces it (ADR 0348 §7). The same list at the same merge writes nothing (ADR 0289 §1).
func (ix *Indexer) writePredefined(ctx context.Context, f delivery.Forge) error {
if ix.Predefined == nil || len(f.Predefined) == 0 {
return nil
}
return ix.notYet(PredefinedBucket, write(ctx, ix.Predefined, delivery.PredefinedListKey, func(value []byte, found bool) ([]byte, bool, error) {
if found {
var cur delivery.PredefinedKept
if err := json.Unmarshal(value, &cur); err == nil {
if cur.Commit == f.Merge && string(cur.List) == string(f.Predefined) {
return nil, false, nil
}
older, err := ix.older(ctx, f.Repository, f.Merge, cur.Commit)
if err != nil {
return nil, false, err
}
if older {
ix.logf("%s: the predefined list of %.12s is older than the one kept (%.12s): left", f.Repository, f.Merge, cur.Commit)
return nil, false, nil
}
}
}
next, err := json.Marshal(delivery.PredefinedKept{V: delivery.Version, Commit: f.Merge, At: ix.now(), List: f.Predefined})
return next, false, err
}))
}
// notYet says a state not on the bus yet (it comes with this delivery, bound when it appears) and lets the message go:
// what it would have written is written at the next message that carries it.
func (ix *Indexer) notYet(bucket string, err error) error {
if errors.Is(err, process.ErrNotYet) {
ix.logf("the state %s is not on the bus yet: nothing is written to it for this message", bucket)
return nil
}
return err
}
func (ix *Indexer) now() time.Time {
if ix.Now != nil {
return ix.Now()
}
return time.Now()
}
// PullName is a pull request as mesh-config names it, `owner/repository#number`.
func PullName(repository string, number int) string { return repository + "#" + strconv.Itoa(number) }
// DeclarationsOf is a manifest's config declarations by key, in the layout of the controller's declared-config mirror: the
// config form's `config`, else the settings form's `settings`, each default and enum encoded the one way.
func DeclarationsOf(raw string) (map[string]delivery.ConfigDeclaration, error) {
var m struct {
Config map[string]delivery.ConfigDeclaration `json:"config"`
Settings map[string]delivery.ConfigDeclaration `json:"settings"`
}
if err := json.Unmarshal([]byte(raw), &m); err != nil {
return nil, err
}
declared := m.Config
if declared == nil {
declared = m.Settings
}
out := map[string]delivery.ConfigDeclaration{}
for key, d := range declared {
for _, field := range []*json.RawMessage{&d.Default, &d.Enum} {
if len(*field) == 0 {
continue
}
var v any
if err := json.Unmarshal(*field, &v); err != nil {
return nil, fmt.Errorf("%s: %w", key, err)
}
canonical, err := json.Marshal(v)
if err != nil {
return nil, err
}
*field = canonical
}
out[key] = d
}
return out, nil
}
// changedDeclarations are the declarations head adds or changes against trunk.
func changedDeclarations(head, trunk map[string]delivery.ConfigDeclaration) map[string]delivery.ConfigDeclaration {
out := map[string]delivery.ConfigDeclaration{}
for key, d := range head {
if before, there := trunk[key]; !there || !reflect.DeepEqual(before, d) {
out[key] = d
}
}
return out
}
// trunkManifest is the manifest the indexer holds for a module on the trunk, "" when it holds none.
func (ix *Indexer) trunkManifest(ctx context.Context, held map[string]delivery.Module, module string) (string, error) {
m, ok := held[module]
if !ok || m.Manifest == "" || ix.Manifests == nil {
return "", nil
}
raw, _, found, err := ix.Manifests.Get(ctx, m.Manifest)
if err != nil || !found {
return "", err
}
kept, err := delivery.DecodeManifest(raw)
if err != nil {
return "", err
}
return kept.JSON, nil
}
// candidateDeclares are, per module of a pull request's head, the declarations its manifest adds or changes against the
// trunk's (an empty set for a module that adds none): heads are the head's manifests by module.
func (ix *Indexer) candidateDeclares(ctx context.Context, held map[string]delivery.Module, heads map[string]string) (map[string]map[string]delivery.ConfigDeclaration, error) {
out := map[string]map[string]delivery.ConfigDeclaration{}
for module, raw := range heads {
head, err := DeclarationsOf(raw)
if err != nil {
return nil, fmt.Errorf("%w: the manifest of %s at the head: %v", ErrUnreadable, module, err)
}
trunkRaw, err := ix.trunkManifest(ctx, held, module)
if err != nil {
return nil, err
}
trunk := map[string]delivery.ConfigDeclaration{}
if trunkRaw != "" {
if trunk, err = DeclarationsOf(trunkRaw); err != nil {
return nil, fmt.Errorf("the trunk's manifest of %s: %w", module, err)
}
}
if changed := changedDeclarations(head, trunk); len(changed) > 0 {
out[module] = changed
}
}
return out, nil
}
// writeConfigCandidate writes a pull request's key of config-candidates from its head's manifests by module: written when
// what it says differs (its time apart), never for the same head and declarations twice.
func (ix *Indexer) writeConfigCandidate(ctx context.Context, repository string, number int, head string, held map[string]delivery.Module, heads map[string]string) error {
if ix.ConfigCandidates == nil {
return nil
}
pull := PullName(repository, number)
key, err := delivery.ConfigCandidateKey(pull)
if err != nil {
return fmt.Errorf("%w: %v", ErrUnreadable, err)
}
declares, err := ix.candidateDeclares(ctx, held, heads)
if err != nil {
return err
}
return ix.notYet(ConfigCandidatesBucket, write(ctx, ix.ConfigCandidates, key, func(value []byte, found bool) ([]byte, bool, error) {
if found {
var cur delivery.ConfigCandidate
if json.Unmarshal(value, &cur) == nil && cur.Head == head && reflect.DeepEqual(cur.Declares, declares) {
return nil, false, nil
}
}
next, err := json.Marshal(delivery.ConfigCandidate{V: delivery.Version, Pull: pull, Head: head, Declares: declares, At: ix.now()})
return next, false, err
}))
}
// dropConfigCandidate deletes a pull request's key: it merged or closed.
func (ix *Indexer) dropConfigCandidate(ctx context.Context, repository string, number int) error {
if ix.ConfigCandidates == nil {
return nil
}
key, err := delivery.ConfigCandidateKey(PullName(repository, number))
if err != nil {
return nil
}
return ix.notYet(ConfigCandidatesBucket, write(ctx, ix.ConfigCandidates, key, func(_ []byte, found bool) ([]byte, bool, error) { return nil, found, nil }))
}
// configCandidateOfPull is a pull message's part in config-candidates: a head whose manifests the message carries is written,
// a closed one deleted. A head held again without its directories leaves the key as it is.
func (ix *Indexer) configCandidateOfPull(ctx context.Context, f delivery.Forge, held map[string]delivery.Module) error {
if f.Action == delivery.PullClosed {
return ix.dropConfigCandidate(ctx, f.Repository, f.Number)
}
if !f.DirsKnown {
return nil
}
heads := map[string]string{}
for dir, raw := range f.Manifests {
heads[moduleOfDir(held, f, dir)] = raw
}
return ix.writeConfigCandidate(ctx, f.Repository, f.Number, f.Head, held, heads)
}
// configCandidatesAfterMerge writes again every open pull request's key that names a module the merge changed, since what
// it adds is judged against the trunk (ADR 0565 §2), from the head's manifests its candidate keeps; and deletes the merged
// pull request's.
func (ix *Indexer) configCandidatesAfterMerge(ctx context.Context, f delivery.Forge, changed []string) error {
if ix.ConfigCandidates == nil {
return nil
}
if err := ix.dropConfigCandidate(ctx, f.Repository, f.Number); err != nil {
return err
}
if len(changed) == 0 || ix.Store == nil || ix.Manifests == nil {
return nil
}
held, err := ix.heldModules(ctx)
if err != nil {
return err
}
if _, err := ix.ConfigCandidates.Keys(ctx); errors.Is(err, process.ErrNotYet) {
return nil
}
groups, err := ix.Store.Keys(ctx)
if err != nil {
return err
}
var errs []error
for _, g := range groups {
raw, _, found, err := ix.Store.Get(ctx, g)
if err != nil || !found {
continue
}
c, err := delivery.DecodeCandidate(raw)
if err != nil {
continue
}
for _, p := range c.Pulls {
if !touchesAny(p, changed) {
continue
}
heads := map[string]string{}
for module, fp := range p.Fps {
mraw, _, found, err := ix.Manifests.Get(ctx, fp.Manifest)
if err != nil || !found {
continue
}
if kept, err := delivery.DecodeManifest(mraw); err == nil {
heads[module] = kept.JSON
}
}
if err := ix.writeConfigCandidate(ctx, p.Repository, p.Number, p.Head.Head, held, heads); err != nil {
errs = append(errs, err)
}
}
}
return errors.Join(errs...)
}
func touchesAny(p delivery.CandidatePull, modules []string) bool {
for _, m := range modules {
if _, ok := p.Fps[m]; ok {
return true
}
}
return false
}
// writtenModules are the modules a merged message wrote, from its written keys (`modules.<module>`).
func writtenModules(written []string) []string {
var out []string
for _, k := range written {
out = append(out, strings.TrimPrefix(k, "modules."))
}
return out
}
@@ -0,0 +1,88 @@
package index
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
)
// 1953/D2, D3 (novox/hq ADR 0565 §2): the indexer keeps the catalogue's predefined list from the merged message, a merge older
// than the one kept never replacing it; and each open pull request's config declarations its head adds or changes against the
// trunk's manifest, an empty set when none, written only when they change and deleted when it merges or closes.
func TestTheControlRecords(t *testing.T) {
ctx := context.Background()
at := time.Date(2026, 10, 29, 22, 0, 0, 0, time.UTC)
t.Run("the predefined list, never replaced by an older merge", func(t *testing.T) {
ix, _, _ := manifestsIndexer(t)
pre := newMem()
ix.Predefined, ix.Now = pre, func() time.Time { return at }
f := mergedMsg(catalogueRepo, map[string]string{"modules/a": "t1"})
f.Predefined = json.RawMessage(`{"font-size":{"type":"integer","value":13}}`)
if err := ix.Apply(ctx, f); err != nil {
t.Fatal(err)
}
var kept delivery.PredefinedKept
if err := json.Unmarshal(pre.vals[delivery.PredefinedListKey], &kept); err != nil || kept.Commit != merge || string(kept.List) != string(f.Predefined) {
t.Fatalf("kept = %+v (%v)", kept, err)
}
writes := pre.writes
if err := ix.Apply(ctx, f); err != nil || pre.writes != writes {
t.Errorf("the same list at the same merge wrote again (%v)", err)
}
older := f
older.Merge, older.Predefined = strings.Repeat("e", 40), json.RawMessage(`{}`)
ix.Contains = func(context.Context, string, string, string) (bool, error) { return true, nil }
if err := ix.writePredefined(ctx, older); err != nil || pre.writes != writes {
t.Errorf("an older merge replaced the list (%v)", err)
}
ix.Predefined = &memStore{unbound: process.ErrNotYet}
if err := ix.writePredefined(ctx, f); err != nil {
t.Errorf("a state not on the bus yet stopped the message: %v", err)
}
})
t.Run("a pull request's config declarations against the trunk", func(t *testing.T) {
ix, _, _ := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs))
cands := newMem()
ix.ConfigCandidates, ix.Now = cands, func() time.Time { return at }
trunk := `{"module":"avahi","version":"1","config":{"port":{"type":"integer","default":5353,"why":"mDNS"}}}`
if err := ix.Apply(ctx, withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "t1"}), map[string]string{"modules/avahi": trunk}, nil)); err != nil {
t.Fatal(err)
}
key := "novox.mesh-catalog.7"
head := strings.Replace(trunk, `"why":"mDNS"}`, `"why":"mDNS"},"name":{"type":"string","default":"x","why":"shown"}`, 1)
p := withManifests(pull(catalogueRepo, 7, "feat/y", h1, "modules/avahi"), map[string]string{"modules/avahi": head}, nil)
if err := ix.Apply(ctx, p); err != nil {
t.Fatal(err)
}
var c delivery.ConfigCandidate
if err := json.Unmarshal(cands.vals[key], &c); err != nil || c.Pull != "novox/mesh-catalog#7" || c.Head != h1 || len(c.Declares["avahi"]) != 1 ||
string(c.Declares["avahi"]["name"].Default) != `"x"` {
t.Fatalf("candidate = %+v (%v)", c, err)
}
writes := cands.writes
if err := ix.Apply(ctx, p); err != nil || cands.writes != writes {
t.Errorf("the same head wrote again (%v)", err)
}
same := withManifests(pull(catalogueRepo, 7, "feat/y", strings.Repeat("d", 40), "modules/avahi"), map[string]string{"modules/avahi": trunk}, nil)
if err := ix.Apply(ctx, same); err != nil {
t.Fatal(err)
}
c = delivery.ConfigCandidate{}
if err := json.Unmarshal(cands.vals[key], &c); err != nil || len(c.Declares) != 0 || c.Declares == nil {
t.Errorf("a head declaring nothing new = %+v, want an empty set", c)
}
closed := pull(catalogueRepo, 7, "feat/y", strings.Repeat("d", 40))
closed.Action = delivery.PullClosed
if err := ix.Apply(ctx, closed); err != nil {
t.Fatal(err)
}
if _, ok := cands.vals[key]; ok {
t.Error("a closed pull request's declarations stayed")
}
})
}
@@ -2,7 +2,6 @@ package index
import (
"context"
"encoding/json"
"strings"
"testing"
@@ -85,14 +84,7 @@ func (m *memStore) Delete(_ context.Context, key string, rev uint64) error {
// the controller's encoder; copied here, the vendored SDK does not carry testdata) has it, with the repository, module and path changed.
func buildRecord(t *testing.T, repo, module, path, at string) []byte {
t.Helper()
raw := []byte(`{"id":"build-1792395447088936520","repository":"novox/mesh-controller","ref":"main","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","at":"2026-10-19T09:38:00Z"}`)
var m map[string]any
if err := json.Unmarshal(raw, &m); err != nil {
t.Fatal(err)
}
m["repository"], m["module"], m["path"], m["at"] = repo, module, path, at
b, _ := json.Marshal(m)
return b
return logged(t, "build-"+module, module, repo, path, "", "", at)
}
func pull(repo string, number int, branch, head string, dirs ...string) delivery.Forge {
@@ -104,9 +96,9 @@ func newIndexer(t *testing.T) (*Indexer, *memStore, *strings.Builder) {
t.Helper()
st, logs := newMem(), &strings.Builder{}
b := NewBuilds()
b.Put(delivery.BuildKey("a", "build-1"), buildRecord(t, "novox/mesh-delivery", "a", "modules/a", "2026-10-19T09:00:00Z"))
b.Put(delivery.BuildKey("c", "build-2"), buildRecord(t, "novox/mesh-catalog", "c", "modules/c", "2026-10-19T09:00:00Z"))
b.Put(delivery.BuildKey("odd", "build-3"), buildRecord(t, "novox/mesh-delivery", "odd-name", "modules/odd", "2026-10-19T09:00:00Z"))
_ = b.Put(buildRecord(t, "novox/mesh-delivery", "a", "modules/a", "2026-10-19T09:00:00Z"))
_ = b.Put(buildRecord(t, "novox/mesh-catalog", "c", "modules/c", "2026-10-19T09:00:00Z"))
_ = b.Put(buildRecord(t, "novox/mesh-delivery", "odd-name", "modules/odd", "2026-10-19T09:00:00Z"))
return &Indexer{Store: st, Builds: b, Logf: func(f string, a ...any) {
logs.WriteString(strings.TrimSpace(strings.ReplaceAll(f, "\n", " ")) + "\n")
_ = a
@@ -2,6 +2,7 @@ package index
import (
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -23,13 +24,10 @@ const (
)
func liveRecord(id, module, repository, ref, commit, path, at, source string) [2]any {
raw := `{"id":"` + id + `","repository":"` + repository + `",`
if ref != "" {
raw += `"ref":"` + ref + `",`
}
raw += `"module":"` + module + `","commit":"` + commit + `","path":"` + path + `","at":"` + at + `","source-fingerprint":"` + source +
`","source-inputs":{"Module":"` + module + `",` + liveInputsTail + `}`
return [2]any{delivery.BuildKey(module, id), []byte(raw)}
when, _ := time.Parse(time.RFC3339, at)
raw, _ := delivery.EncodeBuildLogEntry(delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: module, FP: source, Repository: repository,
Directory: path, Commit: commit, Ref: ref, Node: "n1", At: when, Outcome: delivery.BuildDone})
return [2]any{module, raw}
}
func liveBuilds() [][2]any {
@@ -42,7 +42,12 @@ func withManifests(f delivery.Forge, manifests map[string]string, locks map[stri
return f
}
func TestAMergedManifestIsKeptByItsDigestAndNamedByTheModule(t *testing.T) { // 1449/AC1
func TestAMergedManifestIsKeptOnceByItsDigestAndNamedByTheModule(t *testing.T) {
t.Run("kept by its digest and named by the module", mergedManifestIsKeptByItsDigestAndNamedByTheModule)
t.Run("the same manifest applied again is written once", sameManifestAppliedAgainIsWrittenOnce)
}
func mergedManifestIsKeptByItsDigestAndNamedByTheModule(t *testing.T) { // 1449/AC1
avahi := catalogueManifest(t, "avahi")
ix, mods, manifests := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs))
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, nil)
@@ -63,7 +68,7 @@ func TestAMergedManifestIsKeptByItsDigestAndNamedByTheModule(t *testing.T) { //
}
}
func TestTheSameManifestAppliedAgainIsWrittenOnce(t *testing.T) { // 1449/AC2
func sameManifestAppliedAgainIsWrittenOnce(t *testing.T) { // 1449/AC2
avahi := catalogueManifest(t, "avahi")
ix, _, manifests := manifestsIndexer(t, rec(t, "build-1", "avahi", catalogueRepo, "modules/avahi", "main", "", "2026-10-19T09:00:00Z", aInputs))
f := withManifests(mergedMsg(catalogueRepo, map[string]string{"modules/avahi": "newtree"}), map[string]string{"modules/avahi": avahi}, nil)
@@ -2,12 +2,12 @@ package index
import (
"context"
"encoding/json"
"errors"
"fmt"
"path"
"strings"
"testing"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/process"
@@ -15,29 +15,38 @@ import (
const merge = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"
// record is a build record as the controller writes it (mesh-sdk go/delivery/testdata/build-record-build-agent.json is
// the captured shape): SourceInputs under its Go field names, no json keys. inputs is that object's JSON.
func record(t *testing.T, id, module, repo, path, ref, state, at, inputs string) (string, []byte) {
// logged is one entry of the builder's log (mesh-sdk go/delivery BuildLogEntry, as the builder appends it): a build of
// module from repo's directory path at ref ("" for the trunk), its outcome ("" done, "failed", or "asked" for a build only
// started), at the time at; id makes its fingerprint.
func logged(t *testing.T, id, module, repo, path, ref, outcome, at string) []byte {
t.Helper()
// an outcome has no state key (mesh-controller internal/chain/records.go:57-59); state asked is a request
raw := `{"id":"` + id + `",`
switch state {
case "":
case "failed": // a failure is an outcome with failed set and no state (mesh-controller internal/chain/records.go:109)
raw += `"failed":"the build failed",`
default:
raw += `"state":"` + state + `",`
}
raw += `"repository":"` + repo + `","ref":"` + ref + `","module":"` + module + `","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"` + path + `","at":"` + at + `"`
if inputs != "" {
raw += `,"source-fingerprint":"src1:ignored","source-inputs":` + inputs
}
raw += `}`
var check delivery.BuildRecord
if err := json.Unmarshal([]byte(raw), &check); err != nil {
when, err := time.Parse(time.RFC3339, at)
if err != nil {
t.Fatal(err)
}
return delivery.BuildKey(module, id), []byte(raw)
if path == "" {
path = delivery.RootPath
}
e := delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: module, FP: "src2:" + id, Repository: repo, Directory: path,
Commit: "7a74cee83f88e6585a5511c3cc53bdf0f5303d20", Ref: ref, Node: "n1", At: when, Outcome: delivery.BuildDone}
switch outcome {
case "":
case "failed":
e.Outcome = delivery.BuildFailed
default:
e.Event, e.Outcome = delivery.BuildStartedEvent, ""
}
raw, err := delivery.EncodeBuildLogEntry(e)
if err != nil {
t.Fatal(err)
}
return raw
}
// record is logged under the module's log key, as the tests' tables carry it.
func record(t *testing.T, id, module, repo, path, ref, state, at, _ string) (string, []byte) {
t.Helper()
return module, logged(t, id, module, repo, path, ref, state, at)
}
func modulesIndexer(t *testing.T, recs ...[2]any) (*Indexer, *memStore) {
@@ -45,7 +54,9 @@ func modulesIndexer(t *testing.T, recs ...[2]any) (*Indexer, *memStore) {
mods := newMem()
ix := &Indexer{Store: newMem(), Modules: mods, Builds: NewBuilds()}
for _, r := range recs {
ix.Builds.Put(r[0].(string), r[1].([]byte))
if err := ix.Builds.Put(r[1].([]byte)); err != nil {
t.Fatal(err)
}
}
return ix, mods
}
@@ -302,14 +302,14 @@ func TestARecordOfAPullRequestDoesNotReplaceTheRegisteredRef(t *testing.T) { //
rec(t, "b2", "photos", repo, "", "nox-mesh", "", "2026-10-02T09:00:00Z", ""),
rec(t, "b3", "photos", repo, "", "feat/x", "", "2026-10-09T09:00:00Z", ""),
} {
b.Put(r[0].(string), r[1].([]byte))
_ = b.Put(r[1].([]byte))
}
got := b.RegistrationsOf("photos")
if len(got) != 1 || got[0].Ref != "nox-mesh" {
t.Fatalf("registrations = %+v, want the ref most records name, not the newer feat/x", got)
}
k, v := record(t, "b4", "photos", repo, "", "main", "", "2026-10-03T09:00:00Z", "")
b.Put(k, v) // a trunk's ref beats a branch of its own
_, _ = k, b.Put(v) // a trunk's ref beats a branch of its own
if got := b.RegistrationsOf("photos"); len(got) != 1 || got[0].Ref != "main" {
t.Fatalf("registrations = %+v", got)
}
@@ -46,13 +46,10 @@ const (
)
func agentBuild(id, src, at string) [2]any {
commit := "cada7e3c"
if id == "build-1792587792086548572" {
commit = "f10fbda1"
}
raw := `{"id":"` + id + `","repository":"` + agentCloneURL + `","ref":"main","module":"build-agent","commit":"` + commit + `",` +
`"path":"modules/build-agent","at":"` + at + `","source-fingerprint":"` + src + `","source-inputs":` + aInputs + `}`
return [2]any{delivery.BuildKey("build-agent", id), []byte(raw)}
when, _ := time.Parse(time.RFC3339, at)
raw, _ := delivery.EncodeBuildLogEntry(delivery.BuildLogEntry{Event: delivery.BuildEndedEvent, Module: "build-agent", FP: src, Repository: agentCloneURL,
Directory: "modules/build-agent", Commit: "cada7e3c", Node: "n1", At: when, Outcome: delivery.BuildDone})
return [2]any{"build-agent", raw}
}
type mod struct {
@@ -184,7 +181,7 @@ func TestBasesAreComputedFirstAndNoFingerprintTakesTheBuilder(t *testing.T) { //
t.Errorf("other: fp2 = %q none2 = %q, want none2 naming the base ts-base", m.FP2, m.None2)
}
a := agentBuild("build-1792587792086548572", agentSrcY, "2026-10-21T15:03:00Z")
ix.Builds.Put(a[0].(string), a[1].([]byte))
_ = ix.Builds.Put(a[1].([]byte))
if err := ix.Recompute(context.Background()); err != nil {
t.Fatal(err)
}
@@ -86,10 +86,10 @@ func TestManifestDeclarations(t *testing.T) { // 1238/AC1, 1444 (reads the build
t.Errorf("emits = %v, want none", m.Emits)
}
// 1126 note 76: heard keeps the last 64 FORGE messages the indexer handled, for indexer_heard.
if want := []state{{Name: "candidates", History: 10}, {Name: "heard", History: 64}, {Name: "manifests", History: 1}, {Name: "modules", History: 20}, {Name: "taken", History: 1}, {Name: "takeins", History: 1}}; !reflect.DeepEqual(m.State, want) {
if want := []state{{Name: "candidates", History: 10}, {Name: "heard", History: 64}, {Name: "manifests", History: 1}, {Name: "modules", History: 20}, {Name: "predefined", History: 5}, {Name: "config-candidates", History: 5}, {Name: "taken", History: 1}, {Name: "takeins", History: 1}}; !reflect.DeepEqual(m.State, want) {
t.Errorf("state = %+v, want %+v", m.State, want)
}
if want := []string{"mesh-build-go.toolchains", "mesh-build-node.toolchains", "mesh-controller.assignments", "mesh-controller.builds", "mesh-delivery-checker.previews", "mesh-delivery-gate.source"}; !reflect.DeepEqual(sorted(m.Reads), want) {
if want := []string{"mesh-build-go.toolchains", "mesh-build-node.toolchains", "mesh-delivery-assigner.placement", "mesh-delivery-builder.build-log", "mesh-delivery-checker.previews", "mesh-delivery-gate.source"}; !reflect.DeepEqual(sorted(m.Reads), want) {
t.Errorf("reads = %v, want %v", m.Reads, want)
}
// "Reads FORGE" is a consumes of the indexer's stream: the broker grants only buckets by reads (mesh-control
+4 -2
View File
@@ -16,14 +16,16 @@
{ "name": "heard", "history": 64 },
{ "name": "manifests", "history": 1 },
{ "name": "modules", "history": 20 },
{ "name": "predefined", "history": 5 },
{ "name": "config-candidates", "history": 5 },
{ "name": "taken", "history": 1 },
{ "name": "takeins", "history": 1 }
],
"reads": [
"mesh-build-go.toolchains",
"mesh-build-node.toolchains",
"mesh-controller.assignments",
"mesh-controller.builds",
"mesh-delivery-assigner.placement",
"mesh-delivery-builder.build-log",
"mesh-delivery-checker.previews",
"mesh-delivery-gate.source"
],
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-teller
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 h1:ycu/sNieTdLpD3bzhZOCXdzcoqXUOxPRmqWV019REpo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -19,7 +19,11 @@ import (
func serve(t *testing.T) string {
t.Helper()
m := view.New()
m.Apply(view.Entry{Bucket: view.AssignmentsBucket, Key: "a.x", Value: []byte(`{"node":"a","module":"x","build":"b1"}`)})
placed, err := delivery.EncodePlacement(delivery.Placement{Node: "a", Modules: []delivery.PlacedModule{{Module: "x"}}})
if err != nil {
t.Fatal(err)
}
m.Apply(view.Entry{Bucket: delivery.PlacementBucket, Key: "placement.a", Value: placed})
dir, err := os.MkdirTemp(os.Getenv("TMPDIR"), "c")
if err != nil {
t.Fatal(err)
@@ -8,6 +8,7 @@ import (
"fmt"
"os"
"strings"
"time"
"mesh-delivery-view/internal/view"
@@ -19,6 +20,14 @@ import (
// Tools are the read tools over a modules state, and build_usage over the build modules' states.
func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
return []stdio.Tool{
{Name: view.StatusTool,
Description: "The mesh's status, derived by the view from the buckets it reads (novox/hq ADR 0564 §10), nothing written as a " +
"summary: every open condition of the condition keeper (mesh-conditions), urgent first then oldest; what is quiet (each " +
"node the watchman says is offline, each key it says stopped moving); each module whose want and have differ on a node " +
"(moving, or stuck with the have's or the watchman's words), as mesh-sdk's delivery motion derives it; and what " +
"mesh-healers did in the last seven days, by outcome. A part that cannot be read says so, never as none; well is true " +
"only when nothing is open, quiet or stuck and every part was read. Read-only.",
Run: func(map[string]any) (any, error) { return r.Status(time.Now()), nil }},
{Name: view.ReachReadTool,
Description: "Whether a merge is on the machines: per repository, the reached head (the newest merge every node runs a build " +
"containing, as the view derives it from the indexer's modules, the wants and the haves; a merge at or before it is on the " +
@@ -95,6 +104,23 @@ func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
group, _ := args["group"].(string)
return r.Candidate(group)
}},
{Name: view.PredefinedReadTool,
Description: "The catalogue's predefined list as the indexer keeps it (novox/hq ADR 0565 §2): its key list, the list whole as " +
"predefined.json holds it with the merge it was read at; a merge older than that never replaces it. mesh-config " +
"publishes the predefined keys from it after its take-over. Read-only.",
Run: func(map[string]any) (any, error) {
return view.ControlRecord(r.PredefinedState, delivery.IndexerPredefinedState, delivery.PredefinedListKey)
}},
{Name: view.ConfigCandidatesReadTool,
Description: "Each open pull request's config declarations as the indexer keeps them (novox/hq ADR 0565 §2): per pull " +
"request (owner.repository.number), its head and per module the config keys its manifests add or change against the " +
"trunk's, an empty set when none; deleted when it merges or closes. Give key for one. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"key": map[string]any{"type": "string", "description": "one pull request's key, owner.repository.number"}}},
Run: func(args map[string]any) (any, error) {
key, _ := args["key"].(string)
return view.ControlRecord(r.ConfigCandidatesState, delivery.IndexerConfigCandidatesState, key)
}},
{Name: view.CandidatesReadTool,
Description: "Every candidate the indexer holds, by group, each as stored under candidates.<group>; with module, only those with a " +
"pull request touching that module. Keys that cannot be read are listed as unreadable. Read-only.",
@@ -119,6 +145,27 @@ func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
limit, _ := args["limit"].(float64)
return r.BuilderBuilds(module, state, int(limit))
}},
{Name: view.BuildHistoryTool,
Description: "Every build that ended since a time (default the last 24 hours), oldest first, from the builder's log build-log (kept 30 " +
"days): its module, fingerprint, repository, directory, commit and ref, the build node, when it ended, its outcome (done, failed or " +
"queued), the manifest, the bases it was built against, each artifact made, and seconds, how long it took from its start (absent " +
"when the start is older than since). With module, that module's builds only; more says the log held more: ask again from the " +
"last build's time. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{
"module": map[string]any{"type": "string", "description": "only this module's builds"},
"since": map[string]any{"type": "string", "description": "RFC 3339 time; default 24 hours ago"}}},
Run: func(args map[string]any) (any, error) {
module, _ := args["module"].(string)
since := time.Now().Add(-view.DefaultHistorySince)
if raw, _ := args["since"].(string); strings.TrimSpace(raw) != "" {
at, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(raw))
if err != nil {
return nil, fmt.Errorf("since %q is not an RFC 3339 time: %w", raw, err)
}
since = at
}
return r.BuildHistory(module, since)
}},
{Name: view.BuilderBuildReadTool,
Description: "One build of the builder as stored. Give the module and its fingerprint (fp2, src2:...), or the module alone for the build " +
"at the fingerprint the indexer holds for it now, joined through modules.<module>; with commit, only if the indexer holds the module at that commit (a module's value names the commit its fingerprint first appeared at, and stays there through merges that leave the module alone). Read-only.",
@@ -166,6 +213,15 @@ func Tools(r view.Reader, u view.UsageReader) []stdio.Tool {
node, module := nodeModuleOf(args)
return r.Haves(node, module)
}},
{Name: view.TrustsReadTool,
Description: "What every node trusts a mesh signature from, in one call (novox/hq ADR 0550 §2): each node's record's trusts " +
"(from node-engine_have <node>.node), whether it names the vault's signing key as mesh-vault publishes it in its " +
"state signers and whether it still names the controller's, the nodes that do not name the vault's yet (not_yet), " +
"and every: true once all do, the gate before the controller's lease is handed away. Read-only.",
Input: map[string]any{"type": "object", "properties": map[string]any{}},
Run: func(map[string]any) (any, error) {
return r.Trusts()
}},
}
}
@@ -223,8 +279,14 @@ func read(state stdio.KeptState) view.StateRead {
func main() {
r := view.Reader{State: read(stdio.State("mesh-delivery-indexer.modules")), Builds: read(stdio.State("mesh-delivery-builder.builds")), CandidatesState: read(stdio.State("mesh-delivery-indexer.candidates")),
WantsState: read(stdio.State("mesh-delivery-assigner.wants")), DifferencesState: read(stdio.State("mesh-delivery-assigner.differences")), HavesState: read(stdio.State("node-engine.have")),
ReachState: read(stdio.State("reach")),
HeardState: read(stdio.State("mesh-delivery-indexer.heard"))}
ReachState: read(stdio.State("reach")),
HeardState: read(stdio.State("mesh-delivery-indexer.heard")),
SignersState: read(stdio.State("mesh-vault.signers")),
ConditionsState: read(stdio.State(view.StatusConditionsState)), AttemptsState: read(stdio.State(view.StatusAttemptsState)),
PresenceState: read(stdio.State(view.StatusPresenceState)), StuckState: read(stdio.State(view.StatusStuckState)),
PredefinedState: read(stdio.State(delivery.IndexerModule + "." + delivery.IndexerPredefinedState)),
ConfigCandidatesState: read(stdio.State(delivery.IndexerModule + "." + delivery.IndexerConfigCandidatesState)),
BuildLog: stdio.Log(delivery.BuildLogRead).Since}
u := view.UsageReader{Haves: r.HavesState, Resolved: read(stdio.State("mesh-config.resolved")), Tries: read(stdio.State("mesh-delivery-checker.tries")),
Steps: map[string]view.StateRead{}, Pauses: map[string]view.StateRead{}, Backlog: backlogOf(socket())}
for _, lang := range delivery.AskLanguages {
@@ -1,13 +1,18 @@
package main
import (
"crypto/ed25519"
"encoding/json"
"fmt"
"reflect"
"strings"
"sync/atomic"
"testing"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
"mesh-delivery-view/internal/view"
)
@@ -173,6 +178,28 @@ func TestBuilderBuildsListsKeysAsStored(t *testing.T) {
t.Errorf("%s filter = %+v, want %s", args, got, want)
}
}
// 1955/L7 (ADR 0571 clause 1): builder_history answers each end from the builder's log with its duration from its start,
// the start of a later build of the same fingerprint not counted, and narrows by module through the log's key.
t0 := time.Date(2026, 10, 29, 22, 0, 0, 0, time.UTC)
entry := func(event string, at time.Duration) []byte {
e := delivery.BuildLogEntry{Event: event, Module: "netcheck", FP: fp2A, Repository: "novox/mesh-catalog", Directory: "modules/netcheck",
Commit: strings.Repeat("7", 40), Node: "ace", At: t0.Add(at)}
if event == delivery.BuildEndedEvent {
e.Outcome = delivery.BuildDone
}
raw, _ := delivery.EncodeBuildLogEntry(e)
return raw
}
var asked string
r.BuildLog = func(at time.Time, keys string, limit int) (*stdio.LogPage, error) {
asked = keys
return &stdio.LogPage{Entries: []stdio.LogEntry{{Sequence: 1, Value: entry(delivery.BuildStartedEvent, 0)},
{Sequence: 2, Value: entry(delivery.BuildEndedEvent, 90*time.Second)}, {Sequence: 3, Value: entry(delivery.BuildStartedEvent, time.Hour)}}}, nil
}
out, err = run(t, r, view.BuildHistoryTool, map[string]any{"module": "netcheck", "since": t0.Format(time.RFC3339)})
if h, _ := out.(view.HistoryAnswer); err != nil || len(h.Builds) != 1 || h.Builds[0].Seconds != 90 || h.Builds[0].Outcome != delivery.BuildDone || asked != "netcheck" || h.More {
t.Errorf("history = %+v, %v (keys %q), want the one end at 90 s", out, err, asked)
}
}
// 1723/AC2: builder_build answers one key by module and fp2, and by module alone through the indexer's fp2 (joined at its commit).
@@ -438,6 +465,41 @@ func TestAssignerDifferencesAndNodeHavesToolsAnswerAsStored(t *testing.T) {
t.Errorf("haves of ace (the node's own key too) = %v", got)
}
// L3 s6b (ADR 0550 §2): node_trusts answers for every node in one call whether its record names the vault's key the
// vault publishes, and every is true only once all do.
vaultPub, _, _ := ed25519.GenerateKey(nil)
vaultKey, _ := json.Marshal(delivery.NewSignerKey(vaultPub, time.Now()))
vaultFP := delivery.SignerFingerprint(vaultPub)
record := func(node string, trusts ...string) string {
raw, _ := delivery.EncodeHaveNode(delivery.HaveNode{V: delivery.Version, Record: &delivery.NodeRecord{Node: node, Trusts: trusts}})
return string(raw)
}
for _, row := range []struct {
name string
haves fake
every bool
notYet []string
}{
{"one node trusts the controller's alone", fake{"ace.node": record("ace", "controller key sha256:aa", "vault key "+vaultFP),
"g14.node": record("g14", "controller key sha256:aa"), "ace.zsh": `{"fp":"x"}`}, false, []string{"g14"}},
{"every node trusts the vault's key", fake{"ace.node": record("ace", "vault key "+vaultFP),
"g14.node": record("g14", "controller key sha256:aa", "vault key "+vaultFP)}, true, []string{}},
} {
tr := view.Reader{HavesState: row.haves.reader().State, SignersState: fake{"key": string(vaultKey)}.reader().State}
out, err := run(t, tr, view.TrustsReadTool, map[string]any{})
if err != nil {
t.Fatalf("%s: %v", row.name, err)
}
a := out.(view.TrustsAnswer)
if a.Every != row.every || !reflect.DeepEqual(a.NotYet, row.notYet) || a.VaultKey != vaultFP || len(a.Nodes) != 2 {
t.Errorf("%s: %+v", row.name, a)
}
}
if out, err := run(t, view.Reader{HavesState: fake{"ace.node": record("ace", "vault key "+vaultFP)}.reader().State,
SignersState: fake{}.reader().State}, view.TrustsReadTool, map[string]any{}); err != nil || out.(view.TrustsAnswer).Every {
t.Errorf("with no key published: %+v, %v", out, err)
}
// More haves than one call reads (4 nodes, 300 each, 1 ms a Get): a node and a module read that one key and list
// nothing; the whole answer reads MaxStateReads values in parallel and says it was cut.
slow := newSlow(300, time.Millisecond, func(n string, i int) string { return fmt.Sprintf("%s.m%03d", n, i) })
@@ -48,7 +48,7 @@ func TestProcessWholeSeesTheBuckets(t *testing.T) { // 1134/AC8
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
kvs := map[string]jetstream.KeyValue{}
for _, b := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket} {
for _, b := range []string{delivery.HaveBucket, delivery.PlacementBucket, delivery.PresenceBucket, delivery.StuckBucket} {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5})
if err != nil {
t.Fatal(err)
@@ -68,11 +68,20 @@ func TestProcessWholeSeesTheBuckets(t *testing.T) { // 1134/AC8
}
return b
}
put(view.AssignmentsBucket, "a.x", []byte(`{"node":"a","module":"x","build":"b1"}`))
put(view.AssignmentsBucket, "a.y", []byte(`{"node":"a","module":"y","build":"b2"}`))
put(view.AssignmentsBucket, "b.x", []byte(`{"node":"b","module":"x","build":"b1","ended":"2026-10-18T11:30:00Z"}`))
put(view.BuildsBucket, "x.build.b1", []byte(`{"id":"b1","module":"x"}`))
put(view.BuildsBucket, "x.newest", []byte("b1"))
// The seat's placement (issue 774, ADR 0549 §3): x and y on a, nothing on b.
placeValue := func(node string, modules ...string) []byte {
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
v, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
return v
}
put(delivery.PlacementBucket, "placement.a", placeValue("a", "x", "y"))
put(delivery.PlacementBucket, "placement.b", placeValue("b"))
put(delivery.HaveBucket, "a.x", haveValue("b1", delivery.PhaseChecking, delivery.HealthWaits))
put(delivery.HaveBucket, "b.x", haveValue("b1", delivery.PhaseRunning, delivery.HealthOK))
@@ -150,7 +159,7 @@ func TestProcessWholeShowsWhatTheWatchmanSays(t *testing.T) { // 1168/AC5
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
kvs := map[string]jetstream.KeyValue{}
for _, b := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket} {
for _, b := range []string{delivery.HaveBucket, delivery.PlacementBucket, delivery.PresenceBucket, delivery.StuckBucket} {
kv, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5})
if err != nil {
t.Fatal(err)
@@ -236,7 +245,7 @@ func TestProcessWholeShowsTheIndexersBucketsOnceTheyExist(t *testing.T) { // 124
}
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
defer cancel()
for _, b := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket} {
for _, b := range []string{delivery.HaveBucket, delivery.PlacementBucket, delivery.PresenceBucket, delivery.StuckBucket} {
if _, err := js.CreateKeyValue(ctx, jetstream.KeyValueConfig{Bucket: b, History: 5}); err != nil {
t.Fatal(err)
}
@@ -1,4 +1,4 @@
// Command mesh-delivery-view is the operator's view of delivery (novox/hq ADR 0429 §3): it watches the have, assignments
// Command mesh-delivery-view is the operator's view of delivery (novox/hq ADR 0429 §3): it watches the have, the seat's placement
// and builds buckets on its own bus credential, derives one row per module and node in memory, and serves them on a
// local socket (for mesh-delivery-cli) and on a local web page. It starts, stops or retries nothing; the one thing it writes is the operator's answer to an ask that needs no proof of who
// answered, under `<ask key>.view` in its bucket answers (ADR 0528 clause 6).
@@ -106,7 +106,7 @@ func run(ctx context.Context, getenv func(string) string, logOut io.Writer) erro
return err
})
model.SetBacklog(view.BacklogOver(js, 5*time.Second))
for _, bucket := range []string{delivery.HaveBucket, view.AssignmentsBucket, view.BuildsBucket, delivery.PresenceBucket, delivery.StuckBucket, delivery.CandidatesBucket, delivery.ModulesBucket, delivery.AsksBucket, delivery.AnswersBucket, delivery.BuilderBuildsBucket, delivery.AssignerWantsBucket, delivery.SourceBucket, delivery.ControllerComposedBucket, delivery.AssignerAsksBucket, delivery.ViewAnswersBucket, delivery.TakenBucket} {
for _, bucket := range []string{delivery.HaveBucket, delivery.PlacementBucket, delivery.PresenceBucket, delivery.StuckBucket, delivery.CandidatesBucket, delivery.ModulesBucket, delivery.AsksBucket, delivery.AnswersBucket, delivery.BuilderBuildsBucket, delivery.AssignerWantsBucket, delivery.SourceBucket, delivery.AssignerAsksBucket, delivery.ViewAnswersBucket, delivery.TakenBucket} {
go follow(ctx, js, bucket, false, model, logger)
}
// The modules' and the source's history give each fingerprint's first commit and each head's source revision (ADR 0556).
+1 -1
View File
@@ -3,7 +3,7 @@ module mesh-delivery-view
go 1.26.0
require (
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0
github.com/nats-io/nats-server/v2 v2.11.17
github.com/nats-io/nats.go v1.54.0
)
+2 -2
View File
@@ -1,5 +1,5 @@
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388 h1:ycu/sNieTdLpD3bzhZOCXdzcoqXUOxPRmqWV019REpo=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261028174713-38dbe1d3a388/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0 h1:lKU3HjNXGiVbLW2hUgFjU/TMTUYMlNnnmZxG5T7biQA=
git.novox.be/novox/mesh-sdk/go v0.1.14-0.20261030001116-2da0de75c3f0/go.mod h1:ouERNpZizzwZ0HLEtAypsY9FR20FyUgUYRni6J7kVCY=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op h1:Z/MZK75wC/NSrkgqeNIa7jexam9uWzhLmFTSCPI/kn0=
github.com/antithesishq/antithesis-sdk-go v0.7.0-default-no-op/go.mod h1:FQyySiasQQM8735Ddel3MRojmy4dA1IqCeyJ5jmPMbI=
github.com/google/go-tpm v0.9.8 h1:slArAR9Ft+1ybZu0lBwpSmpwhRXaa85hWtMinMyRAWo=
@@ -50,30 +50,6 @@ func modEntry(t *testing.T, name string, m delivery.Module) Entry {
return Entry{Bucket: delivery.ModulesBucket, Key: name, Value: b, At: t0}
}
// trunkBuild is a build record as the controller writes it (mesh-sdk go/delivery/testdata/build-record-build-agent.json is
// the captured shape: id, state, repository, ref, module, commit, path, source-fingerprint, source-inputs, at).
func trunkBuild(module, id, ref, state, fp, at string) []Entry {
// an outcome has no state key; a failed one has "failed" (mesh-controller internal/chain/records.go:57-59, :109)
rec := `{"id":"` + id + `",`
switch state {
case "":
case "failed":
rec += `"failed":"the build failed",`
default:
rec += `"state":"` + state + `",`
}
rec += `"repository":"novox/mesh-delivery","ref":"` + ref + `","module":"` + module + `","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/` + module + `","at":"` + at + `"`
if fp != "" {
// the controller writes the inputs whenever it writes the fingerprint (mesh-sdk testdata/build-record-build-agent.json)
rec += `,"source-fingerprint":"` + fp + `","source-inputs":{"Module":"` + module + `","Tree":"t","Bases":null,"Contexts":null,"Toolchains":null,"Unpinned":""}`
}
rec += `}`
return []Entry{
{Bucket: BuildsBucket, Key: module + ".build." + id, Value: []byte(rec), At: t0},
{Bucket: BuildsBucket, Key: module + ".newest", Value: []byte(id), At: t0},
}
}
func fpRow(t *testing.T, s Snapshot, module string) FingerprintRow {
t.Helper()
for _, f := range s.Fingerprints {
@@ -111,62 +87,27 @@ func TestSnapshotListsTheCandidateGroups(t *testing.T) { // 1241/AC1
}
}
func TestFingerprintAgainstTheNewestTrunkBuild(t *testing.T) { // 1241/AC2
// 1955/L7 (ADR 0571 §1): the view reads no build record of the controller's: a module with no fingerprint of the new
// form is said so, never compared with the controller's builds.
func TestAModuleOfTheOldFormIsComparedWithNoBuild(t *testing.T) {
m := New()
fill(m, trunkBuild("a", "build-1", "main", "", fF, "2026-10-19T10:00:00Z")...)
fill(m, trunkBuild("d", "build-2", "main", "", fF, "2026-10-19T10:00:00Z")...)
fill(m,
modEntry(t, "a", delivery.Module{FP: fF, Repository: "novox/mesh-delivery", Commit: hA}),
modEntry(t, "d", delivery.Module{FP: fG, Repository: "novox/mesh-delivery", Commit: hA}),
modEntry(t, "b", delivery.Module{None: "an artifact compiles a Go program from a build source that git trees do not pin: no fingerprint before release 4", Repository: "novox/mesh-delivery", Commit: hA}),
modEntry(t, "e", delivery.Module{FP: fG, Repository: "novox/mesh-delivery", Commit: hA}))
modEntry(t, "b", delivery.Module{None: "an artifact compiles a Go program from a build source that git trees do not pin: no fingerprint before release 4", Repository: "novox/mesh-delivery", Commit: hA}))
s := m.Snapshot()
if r := fpRow(t, s, "a"); r.State != FPSame || r.Indexer != fF || r.Build != fF {
t.Errorf("a = %+v, want same", r)
}
if r := fpRow(t, s, "d"); r.State != FPDiffers || r.Indexer != fG || r.Build != fF {
t.Errorf("d = %+v, want differs with both", r)
}
if r := fpRow(t, s, "b"); r.State != FPNone || !strings.Contains(r.Reason, "release 4") || r.Indexer != "" {
t.Errorf("b = %+v, want none with the reason", r)
}
if r := fpRow(t, s, "e"); r.State != FPNoBuild || r.Indexer != fG || r.Build != "" {
t.Errorf("e = %+v, want no build with the indexer's alone", r)
}
}
func TestOnlyTheNewestSuccessfulTrunkBuildCounts(t *testing.T) { // 1241/AC2 (the newest key is only a hint)
m := New()
fill(m, trunkBuild("a", "build-1", "main", "", fG, "2026-10-19T09:00:00Z")...)
fill(m, trunkBuild("a", "build-2", "main", "", fF, "2026-10-19T10:00:00Z")...)
fill(m, trunkBuild("a", "build-3", "main", "failed", fG, "2026-10-19T11:00:00Z")...) // newer, not built
fill(m, trunkBuild("a", "build-4", "feat/x", "", fG, "2026-10-19T12:00:00Z")...) // newer, not trunk; also the newest key
fill(m, modEntry(t, "a", delivery.Module{FP: fF, Repository: "novox/mesh-delivery", Commit: hA}))
if r := fpRow(t, m.Snapshot(), "a"); r.State != FPSame || r.Build != fF {
t.Errorf("a = %+v, want same with the 10:00 trunk build's fingerprint", r)
}
}
func TestBuildFingerprintIsReadWithTheSDKRecord(t *testing.T) { // 1241/AC3
// The record as the controller's encoder wrote it (mesh-sdk go/delivery/testdata/build-record-build-agent.json), whole,
// with its source-inputs under their Go field names: the view reads source-fingerprint from it.
const rec = `{"id":"build-1792395447088936520","repository":"novox/mesh-controller","ref":"main","module":"build-agent","commit":"7a74cee83f88e6585a5511c3cc53bdf0f5303d20","path":"modules/build-agent","manifest-digest":"sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","source-fingerprint":"src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7","source-inputs":{"Module":"build-agent","Tree":"11c0b4f90cae0d6e4ef9cc48cbfdf9f6dbe52bca","Bases":["sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"],"Contexts":{"server":"narrow:d51ca744fd11af59d616c858f76c27b3e58e16e11fbdc0c720eb9d7d19d55fac"},"Toolchains":null,"Unpinned":""},"made":[{"name":"server","kind":"image","reference":"registry.example/build-agent@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc"}],"at":"2026-10-19T09:38:00Z"}`
const fp = "src1:e846efe73716b78e80860baaac96153de2cb76551325753838ca245d2160a3d7"
m := New()
fill(m, Entry{Bucket: BuildsBucket, Key: delivery.BuildKey("build-agent", "build-1792395447088936520"), Value: []byte(rec), At: t0},
Entry{Bucket: BuildsBucket, Key: delivery.NewestKey("build-agent"), Value: []byte("build-1792395447088936520"), At: t0},
modEntry(t, "build-agent", delivery.Module{FP: fp, Repository: "novox/mesh-controller", Commit: hA}))
if r := fpRow(t, m.Snapshot(), "build-agent"); r.State != FPSame || r.Build != fp {
t.Errorf("build-agent = %+v, want same with %s", r, fp)
for _, name := range []string{"a", "b"} {
if r := fpRow(t, s, name); r.State != FPNone || !(strings.Contains(r.Reason, "new form") || strings.Contains(r.Reason, "release 4")) || r.Build != "" {
t.Errorf("%s = %+v, want none: no build of the builder's is compared", name, r)
}
}
}
func shown(t *testing.T) *Model {
m := New()
fill(m, trunkBuild("a", "build-1", "main", "", fF, "2026-10-19T10:00:00Z")...)
fill(m,
candidate(t, "feat/x", pullOf("novox/mesh-delivery", 12, hA, "a"), pullOf("novox/mesh-catalog", 300, hB, "c")),
modEntry(t, "a", delivery.Module{FP: fG, Repository: "novox/mesh-delivery", Commit: hA}),
modEntry(t, "a", delivery.Module{FP2: fp2A, Repository: "novox/mesh-delivery", Commit: hA}),
builderKey(t, "a", fp2A, delivery.BuilderBuild{State: delivery.BuildFailed, Error: "the compile failed"}),
modEntry(t, "b", delivery.Module{None: "no tree", Repository: "novox/mesh-delivery", Commit: hA}))
return m
}
@@ -175,14 +116,14 @@ func TestTableAndPagePrintTheGroupsAndTheComparison(t *testing.T) { // 1241/AC6
m := shown(t)
var out bytes.Buffer
Table(&out, m.Snapshot())
for _, want := range []string{"CANDIDATES", "feat/x", "novox/mesh-delivery#12", hA[:12], "novox/mesh-catalog#300", "FINGERPRINT", "differs", "indexer " + fG, "build " + fF, "no tree"} {
for _, want := range []string{"CANDIDATES", "feat/x", "novox/mesh-delivery#12", hA[:12], "novox/mesh-catalog#300", "FINGERPRINT", "the compile failed", fp2A, "no tree"} {
if !strings.Contains(out.String(), want) {
t.Errorf("the table lacks %q:\n%s", want, out.String())
}
}
rec := httptest.NewRecorder()
Web(m).ServeHTTP(rec, httptest.NewRequest(http.MethodGet, "/", nil))
for _, want := range []string{"CANDIDATES", "feat/x", "novox/mesh-delivery#12", hA[:12], "FINGERPRINTS", "differs", fG, fF, "no tree"} {
for _, want := range []string{"CANDIDATES", "feat/x", "novox/mesh-delivery#12", hA[:12], "FINGERPRINTS", "the compile failed", fp2A, "no tree"} {
if !strings.Contains(rec.Body.String(), want) {
t.Errorf("the page lacks %q:\n%s", want, rec.Body.String())
}
@@ -198,23 +139,3 @@ func TestUnreadableIndexerValuesAreNamed(t *testing.T) { // never dropped silent
t.Errorf("snapshot = %+v", s)
}
}
func TestAPlanAskedBuildWithNoRefIsATrunkBuild(t *testing.T) { // 1281/AC5
// The build record of module mesh-delivery from mesh-catalog, build-1792435539606203626, as `mesh-controller.command
// builds` showed it on 2026-10-19 at about 21:20 local: built on ace at 20:45 local, the repository the clone URL, the
// source fingerprint below, and no ref (a delivery plan asked it after a merge; the listing shows no "at main").
// No reading command shows the whole record: its path is the module's directory in the catalogue, and its
// source-inputs are mailu's live ones (mesh-sdk go/delivery/fingerprint_test.go) with the module's name put in.
const live = "src1:f79e2eedc2689cf6b3591ebfdb81ec65899131122fb2fe3dd9e384091d8a2a6d"
const rec = `{"id":"build-1792435539606203626","repository":"http://novox.internal:20000/novox/mesh-catalog.git",` +
`"module":"mesh-delivery","commit":"9efdd87a","path":"modules/mesh-delivery","at":"2026-10-19T18:45:00Z","source-fingerprint":"` + live + `",` +
`"source-inputs":{"Module":"mesh-delivery","Tree":"cfc6fa8ff5fc1e220132e51ade653b9d3f387b3e","Bases":["sha256:25f3cfeaceca14921366af4d1240b56457ef46273bdb508c7b0e8f469f6fd228"],"Contexts":null,"Toolchains":null,"Unpinned":""}}`
m := New()
fill(m,
Entry{Bucket: BuildsBucket, Key: "mesh-delivery.build.build-1792435539606203626", Value: []byte(rec), At: t0},
Entry{Bucket: BuildsBucket, Key: "mesh-delivery.newest", Value: []byte("build-1792435539606203626"), At: t0},
modEntry(t, "mesh-delivery", delivery.Module{FP: live, Repository: "novox/mesh-catalog", Commit: hA}))
if r := fpRow(t, m.Snapshot(), "mesh-delivery"); r.State != FPSame || r.Build != live {
t.Errorf("mesh-delivery = %+v, want same as the no-ref build's fingerprint", r)
}
}
@@ -12,7 +12,9 @@ import (
"strings"
"sync"
"sync/atomic"
"time"
stdio "git.novox.be/novox/mesh-sdk/go"
"git.novox.be/novox/mesh-sdk/go/delivery"
)
@@ -44,6 +46,57 @@ type Reader struct {
ReachState StateRead // the view's own reach (ADR 0556)
// HeardState is the FORGE messages the indexer handled (issue 1126 note 76); unset reads none.
HeardState StateRead
// SignersState is the vault's public signing halves (ADR 0550 §2); unset reads none.
SignersState StateRead
// ConditionsState, AttemptsState, PresenceState and StuckState are what status reads beside the wants and the haves
// (ADR 0564 §10): the condition keeper's open conditions, mesh-healers' attempts, the watchman's presence and stuck keys.
ConditionsState, AttemptsState, PresenceState, StuckState StateRead
// PredefinedState and ConfigCandidatesState are the indexer's control records (ADR 0565 §2, §15); unset reads none.
PredefinedState, ConfigCandidatesState StateRead
// BuildLog reads the builder's log from a time (stdio.Log(delivery.BuildLogRead).Since); unset reads none.
BuildLog LogSince
}
// LogSince is one page of a log from a time on, at most limit entries of keys (stdio.KeptLog.Since).
type LogSince func(at time.Time, keys string, limit int) (*stdio.LogPage, error)
// The read tools of the indexer's control records (ADR 0565 §15): the indexer serves no tool of its own (ADR 0437 §4).
const (
PredefinedReadTool = "indexer_predefined"
ConfigCandidatesReadTool = "indexer_config_candidates"
)
// ControlRecord is every key of one of the indexer's control records as stored, or one key, with how many it holds.
func ControlRecord(s StateRead, name, key string) (any, error) {
if s.Keys == nil || s.Get == nil {
return nil, fmt.Errorf("the indexer's %s cannot be read here", name)
}
if key = strings.TrimSpace(key); key != "" {
raw, err := s.Get(key)
if err != nil {
return nil, err
}
if raw == nil {
return nil, fmt.Errorf("the indexer's %s holds no key %s", name, key)
}
return raw, nil
}
keys, err := s.Keys()
if err != nil {
return nil, err
}
sort.Strings(keys)
values, err := s.values(keys)
if err != nil {
return nil, err
}
return map[string]any{"state": name, "holds": len(keys), "keys": values}, nil
}
// Status is the view's derivation of status at now (status.go).
func (r Reader) Status(now time.Time) Status {
return DeriveStatus(StatusRead{Conditions: r.ConditionsState, Attempts: r.AttemptsState, Presence: r.PresenceState,
Stuck: r.StuckState, Wants: r.WantsState, Haves: r.HavesState}, now)
}
// ReachReadTool reads the view's reach: per repository, how far each module reached and the reached head (ADR 0556).
@@ -649,3 +702,168 @@ func (r Reader) Differences(node, module string) (KeptAnswer, error) {
func (r Reader) Haves(node, module string) (KeptAnswer, error) {
return r.kept(r.HavesState, "the haves", node, module)
}
// TrustsReadTool reads what every node trusts a mesh signature from, in one call (novox/hq ADR 0550 §2; design 47's L3
// s6b): the gate before the lease write is that every node's record names the vault's key.
const TrustsReadTool = "node_trusts"
// NodeTrusts is what one node's record says it trusts: its lines (delivery.Trusts) and whether they name the vault's key
// the vault publishes and the controller's key.
type NodeTrusts struct {
Trusts []string `json:"trusts"`
TrustsVault bool `json:"trusts_vault"`
TrustsController bool `json:"trusts_controller"`
}
// TrustsAnswer is node_trusts' answer: the vault's key as it publishes it, each node's trusts, the nodes whose record does
// not name the vault's key yet, and whether every node does.
type TrustsAnswer struct {
VaultKey string `json:"vault_key,omitempty"`
Nodes map[string]NodeTrusts `json:"nodes"`
NotYet []string `json:"not_yet"`
Every bool `json:"every"`
Unreadable []string `json:"unreadable,omitempty"`
Note string `json:"note,omitempty"`
}
// Trusts reads every node's record (`<node>.node` of the haves) and the vault's `signers` key, and answers which nodes
// trust the vault's key. Read-only.
func (r Reader) Trusts() (TrustsAnswer, error) {
if r.HavesState.Get == nil || r.HavesState.Keys == nil {
return TrustsAnswer{}, fmt.Errorf("the haves cannot be read here")
}
out := TrustsAnswer{Nodes: map[string]NodeTrusts{}, NotYet: []string{}}
if r.SignersState.Get != nil {
raw, err := r.SignersState.Get(delivery.SignersKeyKey)
if err != nil {
return TrustsAnswer{}, err
}
if len(raw) > 0 {
k, err := delivery.DecodeSignerKey(raw)
if err != nil {
return TrustsAnswer{}, err
}
out.VaultKey = k.Fingerprint
}
}
keys, err := r.HavesState.Keys()
if err != nil {
return TrustsAnswer{}, err
}
var records []string
for _, k := range keys {
if _, _, isNode, err := delivery.ParseHaveKey(k); err == nil && isNode {
records = append(records, k)
}
}
values, _, err := r.HavesState.read(records)
if err != nil {
return TrustsAnswer{}, err
}
for key, raw := range values {
node := strings.TrimSuffix(key, "."+delivery.HaveNodeWord)
h, err := delivery.DecodeHaveNode(raw)
if err != nil {
out.Unreadable = append(out.Unreadable, key)
continue
}
var t NodeTrusts
if h.Record != nil {
t.Trusts = h.Record.Trusts
}
for _, line := range t.Trusts {
t.TrustsController = t.TrustsController || strings.HasPrefix(line, delivery.SignerController+" key ")
}
t.TrustsVault = out.VaultKey != "" && delivery.TrustsKey(t.Trusts, delivery.SignerVault, out.VaultKey)
out.Nodes[node] = t
if !t.TrustsVault {
out.NotYet = append(out.NotYet, node)
}
}
sort.Strings(out.NotYet)
sort.Strings(out.Unreadable)
out.Every = out.VaultKey != "" && len(out.Nodes) > 0 && len(out.NotYet) == 0 && len(out.Unreadable) == 0
if out.VaultKey == "" {
out.Note = "the vault publishes no signing key yet (" + delivery.SignersRead + "), so no node can trust it"
}
return out, nil
}
// BuildHistoryTool is the view's tool of the builds that ended, read from the builder's log (novox/hq ADR 0571 clause 1).
const BuildHistoryTool = "builder_history"
// The bounds of builder_history: how far back it reads when no since is given, and the entries it reads of the log.
const (
DefaultHistorySince = 24 * time.Hour
MaxHistoryEntries = 1000
)
// HistoryRow is one ended build of the builder's log, with how long it took from its start (none when its start is not in
// what was read).
type HistoryRow struct {
delivery.BuildLogEntry
Seconds float64 `json:"seconds,omitempty"`
}
// HistoryAnswer is builder_history's answer: every build that ended since, oldest first, and More when the log held more
// entries than one read takes (ask again from the last build's time).
type HistoryAnswer struct {
Since time.Time `json:"since"`
Builds []HistoryRow `json:"builds"`
More bool `json:"more,omitempty"`
}
// BuildHistory is every build of module (every module when empty) that ended at or after since, oldest first, each with its
// duration from the start the log holds of the same module and fingerprint before it (delivery.BuildDuration). An entry that
// does not read is refused, naming its sequence.
func (r Reader) BuildHistory(module string, since time.Time) (HistoryAnswer, error) {
out := HistoryAnswer{Since: since.UTC(), Builds: []HistoryRow{}}
if r.BuildLog == nil {
return out, fmt.Errorf("the builder's log cannot be read here")
}
keys := ""
if module = strings.TrimSpace(module); module != "" {
k, err := delivery.BuildLogKey(module)
if err != nil {
return out, err
}
keys = k
}
page, err := r.BuildLog(since, keys, MaxHistoryEntries)
if err != nil {
return out, fmt.Errorf("the builder's log %s from %s: %w", delivery.BuildLogRead, since.Format(time.RFC3339), err)
}
if page == nil {
return out, nil
}
var entries []delivery.BuildLogEntry
for _, le := range page.Entries {
e, err := delivery.DecodeBuildLogEntry(le.Value)
if err != nil {
return out, fmt.Errorf("the builder's log %s, entry %d: %w", delivery.BuildLogRead, le.Sequence, err)
}
entries = append(entries, e)
}
starts := map[string][]delivery.BuildLogEntry{}
for _, e := range entries {
if e.Event == delivery.BuildStartedEvent {
starts[e.Module+" "+e.FP] = append(starts[e.Module+" "+e.FP], e)
}
}
for _, e := range delivery.EndedSince(entries, since) {
row := HistoryRow{BuildLogEntry: e}
// the newest start of the same build at or before its end: a fingerprint built again later starts again after it
var start delivery.BuildLogEntry
for _, s := range starts[e.Module+" "+e.FP] {
if !s.At.After(e.At) && s.At.After(start.At) {
start = s
}
}
if d, ok := delivery.BuildDuration(start, e); ok {
row.Seconds = d.Seconds()
}
out.Builds = append(out.Builds, row)
}
out.More = page.Next != 0
return out, nil
}
@@ -0,0 +1,218 @@
package view
import (
"encoding/json"
"fmt"
"sort"
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
// `status` is the view's derivation (novox/hq ADR 0564 §10, Limits 7, issue 1952 V1): what is wrong, quiet and on its way,
// read from the buckets it watches, never from a summary someone writes (research 072 §2), so the controller's status, its
// summary and the probe D9 go with its lease. Each part comes from one bucket:
// - Conditions: the condition keeper's open conditions (mesh-conditions.conditions), urgent first, then oldest;
// - Quiet: the watchman's presence of each node that is offline, and its stuck keys not moved;
// - Moving: each module whose want and have differ, as mesh-sdk's delivery.MotionOf derives it (moving or stuck);
// - Heals: what mesh-healers did in the last seven days (mesh-healers.attempts), by outcome.
// A bucket that cannot be read says so in its part's Unread, never as none.
// StatusTool is the view's tool that answers status.
const StatusTool = "view_status"
// The states status reads, as the view's manifest names them.
const (
StatusConditionsState = health.KeeperModule + "." + health.ConditionsState
StatusAttemptsState = health.HealersModule + "." + health.AttemptsState
StatusPresenceState = delivery.WatchmanModule + ".presence"
StatusStuckState = delivery.WatchmanModule + ".stuck"
)
// HealsWithin is how far back status counts the healers' acts.
const HealsWithin = 7 * 24 * time.Hour
// StatusRead is the five buckets status reads, as a tool reads them.
type StatusRead struct {
Conditions, Attempts, Presence, Stuck, Wants, Haves StateRead
}
// Status is the view's answer to status.
type Status struct {
Conditions []StatusCondition `json:"conditions"`
ConditionsUnread string `json:"conditions-unread,omitempty"`
Quiet []StatusQuiet `json:"quiet"`
QuietUnread string `json:"quiet-unread,omitempty"`
Moving []StatusMoving `json:"moving"`
MovingUnread string `json:"moving-unread,omitempty"`
Heals map[string]int `json:"heals-this-week"`
HealsUnread string `json:"heals-unread,omitempty"`
// Well says nothing is open, quiet or stuck; a part unread is not well.
Well bool `json:"well"`
}
// StatusCondition is one open condition in brief.
type StatusCondition struct {
Key string `json:"key"`
Severity string `json:"severity"`
Machine string `json:"machine,omitempty"`
Summary string `json:"summary"`
Raised time.Time `json:"raised"`
Silenced bool `json:"silenced,omitempty"`
}
// StatusQuiet is a node offline, or a key the watchman says stopped moving.
type StatusQuiet struct {
Key string `json:"key"`
What string `json:"what"`
Since time.Time `json:"since,omitzero"`
}
// StatusMoving is a module whose want and have differ on a node.
type StatusMoving struct {
Node string `json:"node"`
Module string `json:"module"`
State string `json:"state"`
Outcome string `json:"outcome,omitempty"`
Words string `json:"words,omitempty"`
FromFP string `json:"from_fp,omitempty"`
ToFP string `json:"to_fp"`
}
func values(s StateRead) (map[string]json.RawMessage, error) {
if s.Keys == nil || s.Get == nil {
return nil, fmt.Errorf("not read here")
}
keys, err := s.Keys()
if err != nil {
return nil, err
}
out := map[string]json.RawMessage{}
for _, k := range keys {
v, err := s.Get(k)
if err != nil {
return nil, fmt.Errorf("%s: %w", k, err)
}
if v != nil {
out[k] = v
}
}
return out, nil
}
// DeriveStatus is status from the buckets r reads, at now.
func DeriveStatus(r StatusRead, now time.Time) Status {
st := Status{Conditions: []StatusCondition{}, Quiet: []StatusQuiet{}, Moving: []StatusMoving{}, Heals: map[string]int{}}
if raw, err := values(r.Conditions); err != nil {
st.ConditionsUnread = "the condition keeper's conditions cannot be read: " + err.Error()
} else {
var open []health.Condition
for _, v := range raw {
var c health.Condition
if json.Unmarshal(v, &c) == nil && c.Key != "" {
open = append(open, c)
}
}
health.Order(open)
for _, c := range open {
st.Conditions = append(st.Conditions, StatusCondition{Key: c.Key, Severity: string(c.Severity), Machine: c.Subject.Machine,
Summary: c.Summary, Raised: c.Raised, Silenced: c.SilencedAt(now)})
}
}
presence, perr := values(r.Presence)
stuck, serr := values(r.Stuck)
switch {
case perr != nil:
st.QuietUnread = "the watchman's presence cannot be read: " + perr.Error()
case serr != nil:
st.QuietUnread = "the watchman's stuck keys cannot be read: " + serr.Error()
default:
for k, v := range presence {
if p, err := delivery.DecodePresence(v); err == nil && !p.Online {
st.Quiet = append(st.Quiet, StatusQuiet{Key: k, What: "offline"})
}
}
for k, v := range stuck {
if s, err := delivery.DecodeStuck(v); err == nil && !s.Moved {
st.Quiet = append(st.Quiet, StatusQuiet{Key: k, What: "stopped moving: it waited for " + s.Waited, Since: s.Since})
}
}
sort.Slice(st.Quiet, func(i, j int) bool { return st.Quiet[i].Key < st.Quiet[j].Key })
}
wants, werr := values(r.Wants)
haves, herr := values(r.Haves)
switch {
case werr != nil:
st.MovingUnread = "the assigner's wants cannot be read: " + werr.Error()
case herr != nil:
st.MovingUnread = "the nodes' haves cannot be read: " + herr.Error()
default:
for wk, v := range wants {
node, module, err := delivery.ParseWantKey(wk)
if err != nil {
continue
}
w, err := delivery.DecodeWant(v)
if err != nil {
continue
}
hk, _ := delivery.HaveKey(node, module)
var h delivery.Have
hv, has := haves[hk]
if has {
if h, err = delivery.DecodeHave(hv); err != nil {
has = false
}
}
words, stopped := "", false
if sk, err := delivery.StuckKey(delivery.AssignerWantsBucket, wk); err == nil && serr == nil {
if s, err := delivery.DecodeStuck(stuck[sk]); err == nil && !s.Moved {
words, stopped = s.Waited, true
}
}
m := delivery.MotionOf(w, true, h, has, words, stopped)
if m.State == "" || m.State == delivery.MotionSettled {
continue
}
row := StatusMoving{Node: node, Module: module, State: m.State, Outcome: m.Outcome, Words: m.Words, ToFP: w.FP}
if has {
row.FromFP = h.FP
}
st.Moving = append(st.Moving, row)
}
sort.Slice(st.Moving, func(i, j int) bool {
if st.Moving[i].Node != st.Moving[j].Node {
return st.Moving[i].Node < st.Moving[j].Node
}
return st.Moving[i].Module < st.Moving[j].Module
})
}
if raw, err := values(r.Attempts); err != nil {
st.HealsUnread = "mesh-healers' attempts cannot be read: " + err.Error()
} else {
for _, v := range raw {
var a health.Attempt
if json.Unmarshal(v, &a) != nil || now.Sub(a.Begun) > HealsWithin {
continue
}
outcome := a.Outcome
if outcome == "" {
outcome = health.OutcomeActing
}
st.Heals[outcome]++
}
}
stuckMoving := false
for _, m := range st.Moving {
stuckMoving = stuckMoving || m.State == delivery.MotionStuck
}
st.Well = len(st.Conditions) == 0 && len(st.Quiet) == 0 && !stuckMoving &&
st.ConditionsUnread == "" && st.QuietUnread == "" && st.MovingUnread == ""
return st
}
+53 -145
View File
@@ -4,6 +4,7 @@ package view
import (
"encoding/json"
"fmt"
"maps"
"slices"
"sort"
"strings"
@@ -102,8 +103,8 @@ type WantRow struct {
Newest string `json:"newest,omitempty"`
Rule int `json:"rule,omitempty"`
Words string `json:"words,omitempty"`
// Judgement is the controller's judgement of the move to the composition the want names (recreates, moves-data, neither or
// cannot-judge), and Judged its words; empty when the controller wrote none (ADR 0539 §14).
// Judgement is the node agent's judgement of the move that waits for a person's yes (its have in waiting-yes, ADR 0547
// §3), and Judged its words; empty otherwise. The controller's composed copy is no longer read (ADR 0571 §6).
Judgement string `json:"judgement,omitempty"`
Judged string `json:"judged,omitempty"`
}
@@ -215,7 +216,7 @@ type UnreadableRow struct {
// Model holds the rows' inputs in memory and nothing else: a restart fills it again from the buckets' last values.
type Model struct {
mu sync.Mutex
assigns map[string]assignment
placed *delivery.Placed // the seat's placement (the assigner's state, ADR 0567, ADR 0549 §3)
builds map[string]buildEntry // by <module>.build.<id>
groups map[string]groupEntry // the indexer's candidates, by group
modules map[string]moduleEntry // the indexer's modules, by module
@@ -263,22 +264,20 @@ type stuckEntry struct {
err string
}
// assignment holds only the fields of the controller's Assignment the view reads.
type assignment struct {
Node string `json:"node"`
Module string `json:"module"`
Build string `json:"build"`
Commit string `json:"commit"`
Since time.Time `json:"since"`
Ended time.Time `json:"ended"`
State string `json:"state"`
PulledIn bool `json:"pulled-in"`
RequiredBy string `json:"required-by"`
}
// pair is a module the seat's placement puts on a node.
type pair struct{ Node, Module string }
// open says whether the value is an assignment that stands for the module on its node, as the controller's
// Assignment.Stands does. A module pulled in by another stands too.
func (a assignment) open() bool { return a.Ended.IsZero() && a.State == "" }
// placedPairs is every module the seat's placement puts on a node, by node then module. The caller holds mu.
func (m *Model) placedPairs() []pair {
byNode := m.placed.ByNode()
var out []pair
for _, node := range slices.Sorted(maps.Keys(byNode)) {
for _, module := range byNode[node] {
out = append(out, pair{Node: node, Module: module})
}
}
return out
}
// build holds only the fields of the controller's Build record the view reads. A record with no state and no failure
// is a done build.
@@ -356,7 +355,7 @@ type nodeEntry struct {
// New is an empty model.
func New() *Model {
return &Model{assigns: map[string]assignment{}, builds: map[string]buildEntry{}, groups: map[string]groupEntry{}, modules: map[string]moduleEntry{}, keys: map[string]keyEntry{}, asks: map[string]askRecord{}, wants: map[string]wantEntry{}, told: map[string]bool{}, initial: map[string]bool{}, composed: map[string]composedEntry{}, aasks: map[string]assignerAskRecord{}, chosen: map[string]delivery.ViewAnswer{}, answers: map[string]string{}, answerOf: map[string]delivery.Answer{}, holds: map[string]heldEntry{}, newest: map[string]string{},
return &Model{placed: delivery.NewPlaced(), builds: map[string]buildEntry{}, groups: map[string]groupEntry{}, modules: map[string]moduleEntry{}, keys: map[string]keyEntry{}, asks: map[string]askRecord{}, wants: map[string]wantEntry{}, told: map[string]bool{}, initial: map[string]bool{}, composed: map[string]composedEntry{}, aasks: map[string]assignerAskRecord{}, chosen: map[string]delivery.ViewAnswer{}, answers: map[string]string{}, answerOf: map[string]delivery.Answer{}, holds: map[string]heldEntry{}, newest: map[string]string{},
haves: map[string]haveEntry{}, nodes: map[string]nodeEntry{},
presence: map[string]presenceEntry{}, stuck: map[string]stuckEntry{},
modHist: map[string][]delivery.Module{}, srcHist: map[uint64]map[string]string{}, taken: map[string]delivery.Taken{},
@@ -529,16 +528,9 @@ func (m *Model) apply(e Entry) {
}
st, err := delivery.DecodeStuck(e.Value)
m.stuck[e.Key] = stuckEntry{value: st, err: errText(err)}
case AssignmentsBucket:
if e.Deleted {
delete(m.assigns, e.Key)
return
}
var a assignment
if json.Unmarshal(e.Value, &a) != nil || a.Node == "" || a.Module == "" {
return
}
m.assigns[a.Node+"."+a.Module] = a
case delivery.PlacementBucket:
// A key that does not read places nothing on its node; the seat's verb placement says why.
_, _, _ = m.placed.Put(e.Key, e.Value, e.Deleted, e.At)
case delivery.CandidatesBucket:
if e.Deleted {
delete(m.groups, e.Key)
@@ -597,13 +589,6 @@ func (m *Model) apply(e Entry) {
}
w, err := delivery.DecodeWant(e.Value)
m.wants[key] = wantEntry{value: w, err: errText(err), at: e.At}
case delivery.ControllerComposedBucket:
if e.Deleted {
delete(m.composed, e.Key)
return
}
d, err := delivery.DecodeComposedDigest(e.Value)
m.composed[e.Key] = composedEntry{value: d, err: errText(err)}
case delivery.TakenBucket:
if e.Deleted {
delete(m.taken, e.Key)
@@ -651,34 +636,6 @@ func (m *Model) apply(e Entry) {
return
}
m.keys[e.Key] = keyEntry{value: b}
case BuildsBucket:
module, rest, ok := strings.Cut(e.Key, ".")
if !ok {
return
}
switch {
case rest == "newest":
if e.Deleted {
delete(m.newest, module)
return
}
m.newest[module] = strings.TrimSpace(string(e.Value))
case strings.HasPrefix(rest, "build."):
if e.Deleted {
delete(m.builds, e.Key)
return
}
var b build
if json.Unmarshal(e.Value, &b) != nil {
return
}
// The source fingerprint is read with the SDK's record, not with the view's own struct.
var rec delivery.BuildRecord
if json.Unmarshal(e.Value, &rec) != nil {
return
}
m.builds[e.Key] = buildEntry{build: b, rec: rec}
}
}
}
@@ -714,18 +671,12 @@ func (m *Model) snapshot() Snapshot {
m.mu.Lock()
defer m.mu.Unlock()
rows := []Row{}
for key, a := range m.assigns {
if !a.open() {
continue
}
r := Row{Node: a.Node, Module: a.Module, Status: StatusAssigned, Assigned: a.Build}
if a.PulledIn {
r.RequiredBy = a.RequiredBy
}
for _, a := range m.placedPairs() {
r := Row{Node: a.Node, Module: a.Module, Status: StatusAssigned}
var waits []string
hk, herr := delivery.HaveKey(a.Node, a.Module)
if herr != nil {
hk = key
continue
}
h, has := m.haves[hk]
// The controller no longer writes the build of an assignment (R6): the row is the want and the have of the node.
@@ -745,17 +696,8 @@ func (m *Model) snapshot() Snapshot {
r.Have = h.have.FP
}
r.Since, r.Detail = h.at, h.have.Detail
if wanted {
if h.have.FP != w.value.FP {
waits = append(waits, "the node has not applied "+w.value.FP)
}
} else if a.Build != "" && h.have.Build != a.Build {
waits = append(waits, "the node has not applied "+a.Build)
}
}
if id := m.newest[a.Module]; id != "" && a.Build != "" && id != a.Build {
if b, ok := m.builds[a.Module+".build."+id]; ok && b.done() {
waits = append(waits, a.Module+" has a newer build "+id+" not yet assigned")
if wanted && h.have.FP != w.value.FP {
waits = append(waits, "the node has not applied "+w.value.FP)
}
}
r.Waits = strings.Join(waits, "; ")
@@ -766,7 +708,7 @@ func (m *Model) snapshot() Snapshot {
if perr != nil {
continue
}
if a, ok := m.assigns[node+"."+module]; ok && a.open() {
if _, ok := m.placed.Since(node, module); ok {
continue
}
r := Row{Node: node, Module: module, Status: StatusLeftBehind, Since: h.at, Error: h.err}
@@ -892,38 +834,22 @@ func (m *Model) groupRows() []GroupRow {
// successful trunk build (delivery.BuildRecord.IsTrunkBuild: an outcome, not failed, on main or with no ref) that recorded one. The newest key is any state and ref, so it is not
// used: the records are.
func (m *Model) fingerprintRows() []FingerprintRow {
newest := map[string]buildEntry{}
for _, b := range m.builds {
if b.rec.Module == "" || !b.rec.IsTrunkBuild() || b.rec.SourceFingerprint == "" {
continue
}
if old, ok := newest[b.rec.Module]; !ok || b.rec.At.After(old.rec.At) || (b.rec.At.Equal(old.rec.At) && b.rec.ID > old.rec.ID) {
newest[b.rec.Module] = b
}
}
out := []FingerprintRow{}
for name, mod := range m.modules {
if mod.err != "" {
continue
}
row := FingerprintRow{Module: name, Repository: mod.value.Repository, Commit: mod.value.Commit}
if mod.value.FP2 != "" || mod.value.None2 != "" {
// The new form (R4e): the indexer's fp2 against the builder's key of it.
out = append(out, m.newFormRow(name, row, mod.value))
if mod.value.FP2 == "" && mod.value.None2 == "" {
// The builder builds the new form alone, and its done keys are the build history the view reads (ADR 0571 §1).
row.State, row.Reason = FPNone, mod.value.None
if row.Reason == "" {
row.Reason = "the indexer holds no fingerprint of the new form for it, so no build of the builder's is compared"
}
out = append(out, row)
continue
}
b, hasBuild := newest[name]
switch {
case mod.value.None != "":
row.State, row.Reason = FPNone, mod.value.None
case !hasBuild:
row.State, row.Indexer = FPNoBuild, mod.value.FP
case b.rec.SourceFingerprint == mod.value.FP:
row.State, row.Indexer, row.Build = FPSame, mod.value.FP, b.rec.SourceFingerprint
default:
row.State, row.Indexer, row.Build = FPDiffers, mod.value.FP, b.rec.SourceFingerprint
}
out = append(out, row)
out = append(out, m.newFormRow(name, row, mod.value))
}
sort.Slice(out, func(i, j int) bool { return out[i].Module < out[j].Module })
return out
@@ -982,12 +908,6 @@ func (s Snapshot) Counted() int {
return n
}
// The buckets the view reads besides the have bucket (which mesh-sdk's delivery package names).
const (
AssignmentsBucket = "mesh-controller_assignments"
BuildsBucket = "mesh-controller_builds"
)
// The statuses of a row.
const (
StatusAssigned = "assigned"
@@ -1113,10 +1033,8 @@ func (m *Model) wantRows() []WantRow {
if has && h.err == "" {
row.Have, row.Phase, row.Health = h.have.FP, string(h.have.Phase), string(h.have.Health)
}
if ckey, err := delivery.ComposedKey(node, module, v.FP); err == nil && !v.Removed {
if c, ok := m.composed[ckey]; ok && c.err == "" {
row.Judgement, row.Judged = string(c.value.Judgement), c.value.Words
}
if has && h.err == "" && h.have.Phase == delivery.PhaseWaitingYes && !v.Removed {
row.Judgement, row.Judged = string(h.have.Judgement), h.have.Detail
}
switch {
case v.Removed:
@@ -1176,13 +1094,10 @@ func (m *Model) sourceRow() *SourceRow {
waiting = append(waiting, fmt.Sprintf("%s on %s runs %s, not yet running and healthy", module, node, short(w.value.FP)))
}
}
for key, a := range m.assigns {
if !a.open() {
continue
}
for _, a := range m.placedPairs() {
wk, werr := delivery.HaveKey(a.Node, a.Module)
if werr != nil {
wk = key
continue
}
if _, ok := m.wants[wk]; ok {
continue
@@ -1212,9 +1127,9 @@ func splitKey(key string) (node, module string) {
// module's want on a node differs from its have (moving), that the have reached the want running and healthy (settled), or that it
// is held or failed or stopped moving past the watchman's bound (stuck). Each is said once for a node, a module and a composition.
const (
MotionMoving = "moving"
MotionSettled = "settled"
MotionStuck = "stuck"
MotionMoving = delivery.MotionMoving
MotionSettled = delivery.MotionSettled
MotionStuck = delivery.MotionStuck
// MotionBoundSeconds is how long a node may take to apply a want before the watchman says it stuck (watch.StepBound).
MotionBoundSeconds = 1800
)
@@ -1343,7 +1258,6 @@ func (m *Model) motionOf(hk string) []MotionEvent {
id := func(kind string) string {
return kind + "." + node + "." + module + "." + strings.TrimPrefix(w.value.Composed, "sha256:")
}
applied := has && h.have.FP == w.value.FP && (h.have.Composed == "" || h.have.Composed == w.value.Composed)
var out []MotionEvent
say := func(kind string, b MotionBody) {
if m.told[id(kind)] {
@@ -1352,31 +1266,25 @@ func (m *Model) motionOf(hk string) []MotionEvent {
m.told[id(kind)] = true
out = append(out, MotionEvent{Type: kind, ID: id(kind), Body: b})
}
switch {
case applied && h.have.Phase == delivery.PhaseRunning && h.have.Health == delivery.HealthOK:
// The derivation is mesh-sdk's (delivery.MotionOf, issue 1952 S7), shared with the condition keeper's hold.
words, stuck := m.stuckWant(w.value, node, module)
mo := delivery.MotionOf(w.value, true, h.have, has, words, stuck)
if mo.State == delivery.MotionSettled {
if m.told[id(MotionMoving)] {
b := body
b.Outcome = "running"
b.Outcome = mo.Outcome
say(MotionSettled, b)
}
return out
case !m.told[id(MotionMoving)]:
}
if !m.told[id(MotionMoving)] {
say(MotionMoving, body)
}
stop := body
switch {
case has && h.have.Phase == delivery.PhaseHeld:
stop.Outcome, stop.Words = "stopped", h.have.Detail
case has && (h.have.Health == delivery.HealthFailed || h.have.Phase == delivery.PhaseFailed) && h.have.FP == w.value.FP:
stop.Outcome, stop.Words = "failed", h.have.Detail
default:
if s, ok := m.stuckWant(w.value, node, module); ok {
stop.Outcome, stop.Words = "stopped", s
} else {
return out
}
if mo.State == delivery.MotionStuck {
stop := body
stop.Outcome, stop.Words = mo.Outcome, mo.Words
say(MotionStuck, stop)
}
say(MotionStuck, stop)
return out
}
@@ -1,6 +1,7 @@
package view
import (
"encoding/json"
"errors"
"reflect"
"strings"
@@ -8,6 +9,7 @@ import (
"time"
"git.novox.be/novox/mesh-sdk/go/delivery"
"git.novox.be/novox/mesh-sdk/go/health"
)
var t0 = time.Date(2026, 10, 18, 12, 0, 0, 0, time.UTC)
@@ -25,17 +27,22 @@ func haveE(t *testing.T, node, module string, v []byte, at time.Time) Entry {
return Entry{Bucket: delivery.HaveBucket, Key: node + "." + module, Value: v, At: at}
}
func assign(node, module, build, extra string) Entry {
v := `{"node":"` + node + `","module":"` + module + `","build":"` + build + `","commit":"c","since":"2026-10-18T11:00:00Z"` + extra + `}`
return Entry{Bucket: AssignmentsBucket, Key: node + "." + module, Value: []byte(v), At: t0}
}
func buildRec(module, id, extra string) []Entry {
rec := `{"id":"` + id + `","module":"` + module + `","commit":"c"` + extra + `}`
return []Entry{
{Bucket: BuildsBucket, Key: module + ".build." + id, Value: []byte(rec), At: t0},
{Bucket: BuildsBucket, Key: module + ".newest", Value: []byte(id), At: t0},
// place is the seat's placement of node, as the assigner keeps it (ADR 0567): the modules placed there.
func place(t *testing.T, node string, modules ...string) Entry {
t.Helper()
p := delivery.Placement{Node: node}
for _, m := range modules {
p.Modules = append(p.Modules, delivery.PlacedModule{Module: m})
}
key, err := delivery.PlacementKey(node)
if err != nil {
t.Fatal(err)
}
v, err := delivery.EncodePlacement(p)
if err != nil {
t.Fatal(err)
}
return Entry{Bucket: delivery.PlacementBucket, Key: key, Value: v, At: t0}
}
func find(t *testing.T, s Snapshot, node, module string) Row {
@@ -58,14 +65,11 @@ func fill(m *Model, es ...Entry) {
// base: two nodes (a, b), three modules (x, y, z).
func base(t *testing.T) []Entry {
es := []Entry{
assign("a", "x", "b1", ""), assign("a", "y", "b2", ""), assign("b", "x", "b1", ""), assign("b", "z", "b3", ""),
place(t, "a", "x", "y"), place(t, "b", "x", "z"),
haveE(t, "a", "x", have(t, "b1", delivery.PhaseRunning, delivery.HealthOK, ""), t0.Add(1*time.Minute)),
haveE(t, "b", "x", have(t, "b1", delivery.PhaseChecking, delivery.HealthWaits, "starting"), t0.Add(2*time.Minute)),
haveE(t, "b", "z", have(t, "b3", delivery.PhaseRunning, delivery.HealthOK, ""), t0.Add(3*time.Minute)),
}
es = append(es, buildRec("x", "b1", "")...)
es = append(es, buildRec("y", "b2", "")...)
es = append(es, buildRec("z", "b3", "")...)
return es
}
@@ -77,68 +81,35 @@ func TestRowsPerAssignedModuleAndNode(t *testing.T) { // 1134/AC1
t.Fatalf("want 4 rows, got %d counted %d: %+v", len(s.Rows), s.Counted(), s.Rows)
}
r := find(t, s, "b", "x")
want := Row{Node: "b", Module: "x", Status: StatusAssigned, Assigned: "b1", Have: "b1", Phase: "checking", Health: "waits",
want := Row{Node: "b", Module: "x", Status: StatusAssigned, Have: "b1", Phase: "checking", Health: "waits",
Since: t0.Add(2 * time.Minute), Detail: "starting"}
if !reflect.DeepEqual(r, want) {
t.Fatalf("row = %+v, want %+v", r, want)
}
y := find(t, s, "a", "y")
if y.Waits != "no have yet" || y.Assigned != "b2" || y.Have != "" {
if y.Waits != "no have yet" || y.Have != "" {
t.Fatalf("a.y = %+v, want 'no have yet'", y)
}
}
func TestWaitsForTheNodeAndForANewerBuild(t *testing.T) { // 1134/AC2
func TestLeftBehindIsWhatThePlacementDoesNotPut(t *testing.T) { // 1134/AC3, issue 774 (the seat's placement)
m := New()
fill(m, base(t)...)
fill(m, haveE(t, "a", "x", have(t, "b0", delivery.PhaseRunning, delivery.HealthOK, ""), t0))
if w := find(t, m.Snapshot(), "a", "x").Waits; w != "the node has not applied b1" {
t.Fatalf("waits = %q", w)
}
fill(m, buildRec("z", "b4", "")...)
if w := find(t, m.Snapshot(), "b", "z").Waits; w != "z has a newer build b4 not yet assigned" {
t.Fatalf("waits = %q", w)
}
// a newest build that is not done is not a newer build to assign
fill(m, buildRec("z", "b5", `,"failed":"boom"`)...)
if w := find(t, m.Snapshot(), "b", "z").Waits; w != "" {
t.Fatalf("a failed newest build must not be waited on, waits = %q", w)
}
fill(m, buildRec("z", "b6", `,"state":"asked"`)...)
if w := find(t, m.Snapshot(), "b", "z").Waits; w != "" {
t.Fatalf("an asked newest build must not be waited on, waits = %q", w)
}
}
func TestLeftBehindIsNotRunningAndPulledInIs(t *testing.T) { // 1134/AC3 (both halves)
for name, extra := range map[string]string{"ended": `,"ended":"2026-10-18T11:30:00Z"`, "state": `,"state":"pending"`} {
m := New()
fill(m, base(t)...)
fill(m, assign("b", "z", "b3", extra))
s := m.Snapshot()
r := find(t, s, "b", "z")
if r.Status != StatusLeftBehind {
t.Errorf("%s: status = %q, want %q", name, r.Status, StatusLeftBehind)
}
if s.Counted() != 3 {
t.Errorf("%s: counted = %d, want 3", name, s.Counted())
}
}
m := New()
fill(m, base(t)...)
fill(m, haveE(t, "a", "w", have(t, "b9", delivery.PhaseRunning, delivery.HealthOK, ""), t0)) // no assignment at all
fill(m, place(t, "b", "x")) // z taken off b
s := m.Snapshot()
if find(t, s, "a", "w").Status != StatusLeftBehind || s.Counted() != 4 {
t.Fatalf("a have with no assignment: %+v counted %d", find(t, s, "a", "w"), s.Counted())
if r := find(t, s, "b", "z"); r.Status != StatusLeftBehind || s.Counted() != 3 {
t.Errorf("status = %q counted %d, want %q and 3", r.Status, s.Counted(), StatusLeftBehind)
}
// A module pulled in by another stands: shown and counted, with what required it.
m = New()
fill(m, base(t)...)
fill(m, assign("b", "z", "b3", `,"pulled-in":true,"required-by":"x"`))
fill(m, haveE(t, "a", "w", have(t, "b9", delivery.PhaseRunning, delivery.HealthOK, ""), t0)) // placed nowhere
s = m.Snapshot()
r := find(t, s, "b", "z")
if r.Status != StatusAssigned || r.RequiredBy != "x" || r.Have != "b3" || s.Counted() != 4 {
t.Fatalf("pulled-in row = %+v counted %d, want shown and counted with required-by x", r, s.Counted())
if find(t, s, "a", "w").Status != StatusLeftBehind || s.Counted() != 4 {
t.Fatalf("a have with no placement: %+v counted %d", find(t, s, "a", "w"), s.Counted())
}
fill(m, Entry{Bucket: delivery.PlacementBucket, Key: "placement.b", Deleted: true})
if s := m.Snapshot(); s.Counted() != 2 || find(t, s, "b", "x").Status != StatusLeftBehind {
t.Fatalf("b's placement deleted: counted %d, %+v", s.Counted(), s.Rows)
}
}
@@ -181,18 +152,13 @@ func TestOnlyTheAffectedRowChanges(t *testing.T) { // 1134/AC5
if r := find(t, after, "b", "x"); r.Phase != "running" || r.Health != "ok" || !r.Since.Equal(t0.Add(9*time.Minute)) {
t.Fatalf("changed row = %+v", r)
}
// an assignment change and a build change touch only their rows
fill(m, assign("a", "y", "b7", ""))
if r := find(t, m.Snapshot(), "a", "y"); r.Assigned != "b7" {
t.Fatalf("assignment change not seen: %+v", r)
// a want's change touches only its row
fill(m, wantE(t, "a", "y", delivery.Want{FP: fpNew}))
if r := find(t, m.Snapshot(), "a", "y"); r.Assigned != fpNew {
t.Fatalf("want change not seen: %+v", r)
}
if !reflect.DeepEqual(find(t, m.Snapshot(), "b", "z"), find(t, after, "b", "z")) {
t.Fatal("an assignment of another row changed b.z")
}
fill(m, buildRec("x", "b8", "")...)
s := m.Snapshot()
if find(t, s, "a", "x").Waits == "" || find(t, s, "b", "x").Waits == "" || find(t, s, "b", "z").Waits != "" {
t.Fatalf("a build of x must touch x's rows only: %+v", s.Rows)
t.Fatal("a want of another row changed b.z")
}
// a deleted key (a bucket's expiry) removes its value
m.Apply(Entry{Bucket: delivery.HaveBucket, Key: "a.x", Deleted: true})
@@ -237,7 +203,13 @@ func stuckE(t *testing.T, bucket, key, waited string, since time.Time, moved boo
return Entry{Bucket: delivery.StuckBucket, Key: bucket + "." + key, Value: b, At: since}
}
func TestOfflineAndStuckAreListed(t *testing.T) { // 1168/AC2
func TestTheWatchmansKeysAndStatus(t *testing.T) {
t.Run("offline and stuck are listed", offlineAndStuckAreListed)
t.Run("an unknown watchman version is unreadable, not dropped", unknownWatchmanVersionIsUnreadableNotDropped)
t.Run("1952/V1: status derives each part from its bucket, and says a part it cannot read", statusDerivesEachPartFromItsBucket)
}
func offlineAndStuckAreListed(t *testing.T) { // 1168/AC2
m := New()
fill(m, base(t)...)
fill(m,
@@ -265,7 +237,7 @@ func TestOfflineAndStuckAreListed(t *testing.T) { // 1168/AC2
}
}
func TestUnknownWatchmanVersionIsUnreadableNotDropped(t *testing.T) { // 1168/AC3
func unknownWatchmanVersionIsUnreadableNotDropped(t *testing.T) { // 1168/AC3
m := New()
fill(m,
Entry{Bucket: delivery.PresenceBucket, Key: "c", Value: []byte(`{"v":9,"online":false}`), At: t0},
@@ -292,3 +264,70 @@ func TestUnknownWatchmanVersionIsUnreadableNotDropped(t *testing.T) { // 1168/AC
t.Fatal("a readable value must clear the unreadable mark")
}
}
// 1952/V1 (ADR 0564 §10): status reads the keeper's open conditions, the watchman's presence and stuck keys, the wants against
// the haves and the healers' attempts of the week; nothing writes a summary, and a part it cannot read is said, never well.
func statusDerivesEachPartFromItsBucket(t *testing.T) {
now := t0.Add(time.Hour)
composedC := "sha256:" + strings.Repeat("c", 64)
fp1, fp2, fp3 := delivery.SourcePrefix2+strings.Repeat("1", 64), delivery.SourcePrefix2+strings.Repeat("2", 64), delivery.SourcePrefix2+strings.Repeat("3", 64)
enc := func(v any) string { b, _ := json.Marshal(v); return string(b) }
want := func(fp string) string {
b, _ := delivery.EncodeWant(delivery.Want{FP: fp, Composed: composedC})
return string(b)
}
have := func(fp string, phase delivery.Phase, health delivery.Health, detail string) string {
b, err := delivery.EncodeHave(delivery.Have{FP: fp, Composed: composedC, Phase: phase, Health: health, Detail: detail})
if err != nil {
t.Fatal(err)
}
return string(b)
}
presence := func(on bool) string { b, _ := delivery.EncodePresence(delivery.Presence{Online: on}); return string(b) }
stuckOf := func(bucket, key, waited string, moved bool) string {
b, _ := delivery.EncodeStuck(delivery.Stuck{Bucket: bucket, Key: key, Waited: waited, Since: t0, Moved: moved})
return string(b)
}
stuckWant, _ := delivery.StuckKey(delivery.AssignerWantsBucket, "want.b.web")
fin := t0.Add(time.Minute)
r := StatusRead{
Conditions: states(map[string]string{
"machine.b.silent": enc(health.Condition{Key: "machine.b.silent", Severity: health.Warning, Summary: "b silent", Raised: t0}),
"core.doctor.self-check": enc(health.Condition{Key: "core.doctor.self-check", Severity: health.Urgent, Summary: "no run", Raised: t0.Add(time.Minute)}),
}),
Presence: states(map[string]string{"a": presence(true), "b": presence(false)}),
Stuck: states(map[string]string{stuckWant: stuckOf(delivery.AssignerWantsBucket, "want.b.web", "b to run web", false),
"x.moved": stuckOf("x", "moved", "y", true)}),
Wants: states(map[string]string{"want.a.web": want(fp2), "want.a.mail": want(fp1), "want.b.web": want(fp2), "want.a.db": want(fp3)}),
Haves: states(map[string]string{"a.web": have(fp1, delivery.PhaseRunning, delivery.HealthOK, ""), "a.mail": have(fp1, delivery.PhaseRunning, delivery.HealthOK, ""),
"b.web": have(fp1, delivery.PhaseRunning, delivery.HealthOK, ""), "a.db": have(fp3, delivery.PhaseRunning, delivery.HealthFailed, "the unit failed")}),
Attempts: states(map[string]string{"1": enc(health.Attempt{Outcome: health.OutcomeActed, Begun: t0, Finished: &fin}),
"2": enc(health.Attempt{Outcome: health.OutcomeEscalated, Begun: t0}), "old": enc(health.Attempt{Outcome: health.OutcomeActed, Begun: now.Add(-8 * 24 * time.Hour)})}),
}
st := DeriveStatus(r, now)
if len(st.Conditions) != 2 || st.Conditions[0].Key != "core.doctor.self-check" || st.Conditions[1].Key != "machine.b.silent" {
t.Errorf("conditions = %+v, want the urgent one first", st.Conditions)
}
if len(st.Quiet) != 2 || st.Quiet[0].Key != "b" || st.Quiet[1].Key != stuckWant {
t.Errorf("quiet = %+v", st.Quiet)
}
wantMoving := []StatusMoving{
{Node: "a", Module: "db", State: delivery.MotionStuck, Outcome: delivery.OutcomeFailed, Words: "the unit failed", FromFP: fp3, ToFP: fp3},
{Node: "a", Module: "web", State: delivery.MotionMoving, FromFP: fp1, ToFP: fp2},
{Node: "b", Module: "web", State: delivery.MotionStuck, Outcome: delivery.OutcomeStopped, Words: "b to run web", FromFP: fp1, ToFP: fp2},
}
if !reflect.DeepEqual(st.Moving, wantMoving) {
t.Errorf("moving = %+v\nwant %+v", st.Moving, wantMoving)
}
if !reflect.DeepEqual(st.Heals, map[string]int{health.OutcomeActed: 1, health.OutcomeEscalated: 1}) || st.Well {
t.Errorf("heals = %v, well %v", st.Heals, st.Well)
}
// A part it cannot read is said, and the mesh is not well; with nothing wrong and every part read it is.
if st := DeriveStatus(StatusRead{}, now); st.ConditionsUnread == "" || st.QuietUnread == "" || st.MovingUnread == "" || st.HealsUnread == "" || st.Well {
t.Errorf("nothing read = %+v", st)
}
empty := states(map[string]string{})
if st := DeriveStatus(StatusRead{Conditions: empty, Presence: empty, Stuck: empty, Wants: empty, Haves: empty, Attempts: empty}, now); !st.Well {
t.Errorf("an empty mesh = %+v, want well", st)
}
}
@@ -101,7 +101,7 @@ func TestEveryWantIsARowAgainstItsHave(t *testing.T) {
func TestASourceRevisionIsDerivedFromWantsAndHaves(t *testing.T) {
m := New()
fill(m, sourceE(t, 14355), moduleE(t, "web", fpNew), moduleE(t, "mail", fpNew),
assign("ace", "web", "b-1", ""), assign("g14", "web", "b-1", ""),
place(t, "ace", "web"), place(t, "g14", "web"),
wantE(t, "ace", "web", delivery.Want{FP: fpNew}), haveFP(t, "ace", "web", fpNew, delivery.HealthOK),
wantE(t, "g14", "web", delivery.Want{FP: fpNew}), haveFP(t, "g14", "web", fpNew, delivery.HealthOK),
wantE(t, "ace", "mail", delivery.Want{FP: fpNew}), haveFP(t, "ace", "mail", fpNew, delivery.HealthOK))
@@ -119,7 +119,7 @@ func TestASourceRevisionIsDerivedFromWantsAndHaves(t *testing.T) {
if s.State != SourceNotDelivered || !strings.Contains(strings.Join(s.Waiting, ";"), "mail") {
t.Errorf("a want that is not the module's fingerprint: %+v", s)
}
fill(m, wantE(t, "ace", "mail", delivery.Want{FP: fpNew}), assign("novox", "web", "b-1", ""))
fill(m, wantE(t, "ace", "mail", delivery.Want{FP: fpNew}), place(t, "novox", "web"))
s = m.Snapshot().Source
if s.State != SourceNotDelivered || !strings.Contains(strings.Join(s.Waiting, ";"), "novox") {
t.Errorf("a node assigned the module with no want: %+v", s)
@@ -143,16 +143,15 @@ func TestTheWantsPageSaysWantsAgainstHaves(t *testing.T) {
// 1906/D7 (ADR 0539 §14): the controller's judgement of the move a want names is shown beside it, with its words.
func TestTheJudgementOfTheMoveIsShownBesideTheWant(t *testing.T) {
// 1955/L7 (ADR 0571 §6): the judgement is the node agent's, in its have waiting for a yes; the controller's composed copy
// is not read.
m := New()
key, err := delivery.ComposedKey("ace", "web", fpNew)
h, err := delivery.EncodeHave(delivery.Have{FP: fpOld, Phase: delivery.PhaseWaitingYes, Health: delivery.HealthOK,
Judgement: delivery.JudgedCannot, Waits: "sha256:" + strings.Repeat("c", 64), Detail: "the composition it runs is gone"})
if err != nil {
t.Fatal(err)
}
v, err := delivery.EncodeComposedDigest(delivery.ComposedDigest{Digest: "sha256:" + strings.Repeat("c", 64), Judgement: delivery.JudgedCannot, Words: "the composition it runs is gone"})
if err != nil {
t.Fatal(err)
}
fill(m, wantE(t, "ace", "web", delivery.Want{FP: fpNew}), Entry{Bucket: delivery.ControllerComposedBucket, Key: key, Value: v, At: t0})
fill(m, wantE(t, "ace", "web", delivery.Want{FP: fpNew}), Entry{Bucket: delivery.HaveBucket, Key: "ace.web", Value: h, At: t0})
rows := m.Snapshot().Wants
if len(rows) != 1 || rows[0].Judgement != string(delivery.JudgedCannot) || !strings.Contains(rows[0].Judged, "gone") {
t.Fatalf("rows = %+v", rows)
@@ -177,7 +176,7 @@ func TestTheViewSaysOnceThatASourceRevisionIsDelivered(t *testing.T) {
said = append(said, e)
return nil
})
fill(m, sourceE(t, 14355), moduleE(t, "web", fpNew), assign("ace", "web", "b-1", ""),
fill(m, sourceE(t, 14355), moduleE(t, "web", fpNew), place(t, "ace", "web"),
wantE(t, "ace", "web", delivery.Want{FP: fpNew}))
if len(said) != 0 {
t.Fatalf("said %v before any have", said)
@@ -213,7 +212,7 @@ func TestTheViewSaysOnceThatASourceRevisionIsDelivered(t *testing.T) {
// have: the row shows the fingerprint wanted and waits while the node runs another.
func TestARowReadsTheWantWhenTheAssignmentNamesNoBuild(t *testing.T) {
m := New()
fill(m, assign("ace", "web", "", ""), wantE(t, "ace", "web", delivery.Want{FP: fpNew}), haveFP(t, "ace", "web", fpOld, delivery.HealthOK))
fill(m, place(t, "ace", "web"), wantE(t, "ace", "web", delivery.Want{FP: fpNew}), haveFP(t, "ace", "web", fpOld, delivery.HealthOK))
r := find(t, m.Snapshot(), "ace", "web")
if r.Assigned != fpNew || r.Have != fpOld || !strings.Contains(r.Waits, "has not applied") {
t.Fatalf("row = %+v, want it to wait for the want's fingerprint", r)
@@ -228,10 +227,12 @@ func TestARowReadsTheWantWhenTheAssignmentNamesNoBuild(t *testing.T) {
// and a have nothing is assigned to keeps the module's name.
func TestADottedModuleNameIsFoundAndKept(t *testing.T) {
m := New()
fill(m, assign("ace", "novox.be", "", ""), wantE(t, "ace", "novox.be", delivery.Want{FP: fpNew}), haveFP(t, "ace", "novox.be", fpNew, delivery.HealthOK),
// The seat's placement refuses a dotted module name (mesh-sdk delivery.Placement), so such a module is never placed:
// its have is a row left behind under its own name.
fill(m, place(t, "ace"), wantE(t, "ace", "novox.be", delivery.Want{FP: fpNew}), haveFP(t, "ace", "novox.be", fpNew, delivery.HealthOK),
haveFP(t, "ace", "old.name", fpNew, delivery.HealthOK))
s := m.Snapshot()
if r := find(t, s, "ace", "novox.be"); r.Have != fpNew || r.Waits != "" {
if r := find(t, s, "ace", "novox.be"); r.Have != fpNew || r.Status != StatusLeftBehind {
t.Errorf("the dotted module's row = %+v", r)
}
if r := find(t, s, "ace", "old.name"); r.Status != StatusLeftBehind {
+2 -2
View File
@@ -87,7 +87,7 @@ func TestManifestIsTakenAndReadsOnly(t *testing.T) { // 1134/AC7
if m.Module != "mesh-delivery-view" {
t.Errorf("module = %q", m.Module)
}
want := []string{"mesh-build-go.pauses", "mesh-build-go.steps", "mesh-build-image.pauses", "mesh-build-image.steps", "mesh-build-node.pauses", "mesh-build-node.steps", "mesh-config.resolved", "mesh-controller.assignments", "mesh-controller.builds", "mesh-controller.composed", "mesh-delivery-assigner.asks", "mesh-delivery-assigner.differences", "mesh-delivery-assigner.wants", "mesh-delivery-builder.builds", "mesh-delivery-checker.tries", "mesh-delivery-gate.asks", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.heard", "mesh-delivery-indexer.modules", "mesh-delivery-indexer.taken", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-delivery-watchman.stuck", "node-engine.have"} // build_usage reads the build modules' steps and pauses, mesh-config's capacities and the checker's tries; 1168/AC1, 1241/AC4, 1345/AC3, 1297/B8, B9
want := []string{"mesh-build-go.pauses", "mesh-build-go.steps", "mesh-build-image.pauses", "mesh-build-image.steps", "mesh-build-node.pauses", "mesh-build-node.steps", "mesh-conditions.conditions", "mesh-config.resolved", "mesh-delivery-assigner.asks", "mesh-delivery-assigner.differences", "mesh-delivery-assigner.placement", "mesh-delivery-assigner.wants", "mesh-delivery-builder.build-log", "mesh-delivery-builder.builds", "mesh-delivery-checker.tries", "mesh-delivery-gate.asks", "mesh-delivery-gate.source", "mesh-delivery-indexer.candidates", "mesh-delivery-indexer.config-candidates", "mesh-delivery-indexer.heard", "mesh-delivery-indexer.modules", "mesh-delivery-indexer.predefined", "mesh-delivery-indexer.taken", "mesh-delivery-teller.answers", "mesh-delivery-watchman.presence", "mesh-delivery-watchman.stuck", "mesh-healers.attempts", "mesh-vault.signers", "node-engine.have"} // view_status reads the keeper's conditions and the healers' attempts (1952/V1); build_usage reads the build modules' steps and pauses, mesh-config's capacities and the checker's tries; 1168/AC1, 1241/AC4, 1345/AC3, 1297/B8, B9
got := slices.Clone(m.Reads)
slices.Sort(got)
if !reflect.DeepEqual(got, want) {
@@ -119,7 +119,7 @@ func TestManifestIsTakenAndReadsOnly(t *testing.T) { // 1134/AC7
}
// The two read tools of the indexer's modules (issue 1562), carried by a bundle the runtime launches; the indexer
// itself serves no tool (ADR 0437 §4). Carrying them gives the tool runner read access to every bucket the view reads.
if want := []string{"indexer_module", "indexer_modules", "indexer_candidate", "indexer_candidates", "builder_build", "builder_builds", "assigner_wants", "assigner_want", "assigner_differences", "node_haves", "build_usage", "view_reach", "indexer_module_history", "indexer_heard"}; !reflect.DeepEqual(m.Tools, want) {
if want := []string{"view_status", "indexer_module", "indexer_modules", "indexer_candidate", "indexer_candidates", "indexer_predefined", "indexer_config_candidates", "builder_build", "builder_builds", "builder_history", "assigner_wants", "assigner_want", "assigner_differences", "node_haves", "node_trusts", "build_usage", "view_reach", "indexer_module_history", "indexer_heard"}; !reflect.DeepEqual(m.Tools, want) {
t.Errorf("tools = %v, want %v", m.Tools, want)
}
for _, a := range m.Build.Artifacts {

Some files were not shown because too many files have changed in this diff Show More